1109
flat / full
normal / watch
| Forums » Selected ISP Support » Speakeasy » Packet Loss/Latency Reports |
| Uniqs: 1109 | Share Topic: | RSS topic: | toggle: flat / full normal / watch | ||||
| whats up in seattle ... is this normal.... » « Traffic Management | |
page: 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · 9  | |
| Author | All Replies | |||
 KatOak VIP join:2001-09-10 Seattle, WA | reply to KatOak Re: Packet Loss/Latency Reports Status: Although we've been aggressively pursuing customers that have been reported as infected with the MSBlaster worm or its variants, the packet loss that is occurring as a result of this has decreased only marginally. The Blaster worm & its variants basically scan everything within its class B subnet relentlessly, and we've a couple of these shared with other organizations with whom we've also been in contact with in order to cut down the traffic we're seeing as a result of this worm. Packet loss is as a result of a maxing of the packets per second that can be processed by our routers. We will be upgrading all of our equipment so in the future we are able to filter/block these kinds of attacks based on more individualized rule sets, instead of the global filters we'd need to put in place right now. We did attempt a filter on port 135 about 3 weeks ago, but had to pull it due to complaints from Exchange and Windows Filesharing users. All recorded complaints of packet loss and latency in the SFO area have been customers on one of our routers - we will be transitioning 40% of the customers off of this router and circuit and onto a new router and circuit to alleviate the issue in the short term. In Seattle, we will begin our POP upgrade next week and will have all new equipment in place within the next few weeks. We have just a handful of issues in a couple of our other POPs, and will continue to monitor those and work with customers infected with this worm in order to alleviate any load that we're seeing. -- Kat Oak Speakeasy kat@speakeasy.net | |||
 GeminiCub4U Premium join:2003-04-07 Livermore, CA |  Kat can you please put me on the new router? Im a new customer and am experiencing latency and packet loss.-- Reality TV has taken over me. | |||
 GamingGirl @speakeasy.n | reply to sh0V3L "my feelings are this se knows its got problems ok but if they can buy a lil time by having you do the ping ploter thing they will play that game with you as they did me till i called their bluff. just tell them you cannot reproduce the symptoms and document them because its udp connectionless service when i told them thats when things started going my way so give that a shot" Hiya Catskinner... well, that didn't work for me. I called and spoke to the first person, told him what was going on and he proceeded to tell me that there is nothing they could do for UDP traffic...something about it not being able to be corrected en route or something. He told me that it was my game that was causing it (huh?!). He wanted me to believe that it was the netcode or something... two years of never experiencing packet loss and then all of a sudden the game socks me in the back? I don't think so =(. Of course I began to ask him why there is no way it could have to do with the network if the packets still use the network to reach me. He decided to transfer me to advanced support (finally!). The advanced support guy said that he sent 100 packets to me and saw no packet loss. When I explained about my game and the UDP traffic he said "we can't even count what happens in your game"... Ok, my big problem with this is, I pay for the so called Gamer Package. Like duh, I use my connection for gaming. When my connection becomes useless to me for that purpose, of course I want things fixed. For them to tell me that the problems I am experiencing in game (and to a lesser extent outside of it) are not relevant and deserve no time for discussion is ludicrous. Why sell me a package geared for gaming (knowing that UDP traffic is what consists of gaming) if they refuse to support me when I am having a problem with the very marketing tool they sold me? That's like me buying shampoo, finding out there's dirt inside and being told "I'm sorry, it may be called shampoo but we didn't say it worked like it!" =( I'm not normally someone who gets upset easily but I feel ignored and I'm just very frustrated. I'm sure that if all I did was surf the net I wouldn't have much to complain about, but then again I wouldn't be paying 100 bucks a month to do it... sigh =(. He did say that he is going to have Covad reprovision my line to see if that helps my latency (a whole other issue) so who knows, maybe that will work. | |||
 koitsu Premium join:2002-07-16 Mountain View, CA | *sigh* Shady technical support reps. Thumbs down. First off, what the rep. was referring to was the fact that UDP traffic by nature is lossy -- there is absolutely no guarantee that the packet will reach it's destination. The protocol is basically a "send the packet and pray" medium. TCP, on the other hand, guarantees that your packet makes it to it's destination due to handshaking (TCP SYN/ACK). Most games use UDP for their actual transmission of movement, shots fired, blah blah blah, because of two reasons: 1) UDP is by far less intensive on a server (as far as processing goes), and 2) if some bloke running along happens to miss a delivery packet, big whoop, he'll send another one half a second later anyways. Now you know why gamers who have slow or saturated net connections "jump around" a lot, and why latency plays a big role. The issue I have with what the rep. told you is that the problem we're all experience has _NOTHING_ to do with the kind-of traffic that's being propagated -- TCP, UDP, ICMP. It doesn't matter. The results are the same no matter which protocol you use: the POPs are a mess. The rep. shouldn't be giving you some schmooze about how they "can't do anything about UDP traffic." I get stalls and lag via ssh to my co-lo, which is _entirely_ TCP-based, and I still don't see them bending over backwards about it (because the problem has NOTHING to do with protocols!  ).The fact that SE calls their "gamer package" a "package for gamers" is misleading. All of the packages are the same, just with different speeds or different account options (i.e. shell accounts, 10 Email boxes, and so on). There's absolutely no difference, DSL or network wise, between the SysAdmin package and the Gamer package. It's just marketing schmooze. Your latency problem will probably get solved when they deploy what Kat said above in her most recent post. I can attest to seeing a _very_ marginal increase in stability since 9/9 (re: MSBlast terminating itself), but the problem (not to sound like a braggart) is definitely looking like oversaturation, which is what I claimed in the first place. *shakes head* Ah well. We'll see how it goes. I'm glad the Seattle folks are getting the upgrade first since their POP is absolutely horrid. I just hope SFO and LAX are next. -- Making life hard for others since 1977. | |||
 TrueAudio 192khz Premium join:2002-02-24 Richmond, CA | reply to KatOak Nobody wants to upgrade there systems till its absolutely necessary. Those routers and servers are super expensive. Its just gets to a point where customers know what there talking about and experiencing, and start calling the company on it, and then they are forced to take some kind of action. Since they do a great deal of advertising from DSL reports I would think it would be in there best interest to keep the customers that post here happy, and with service they are expecting to receive. I ordered this service because of the ads on this website. I saw allot of people praising the service and that really turned me on. Im 3 or 4 days into this service, and im already experiencing complete loss of connection for about 15-20 seconds, then it pops back, and there seems to be allot of route changes going on. I haven't been keep much track of the packet loss, but I do notice it. Well... I hope they are upgrading around here(SFO). Its seems like they bay area has allot of packet loss on some of the major ISP's. Tons of data flowing out of Cali & Seattle.. that's for sure. I guess we wait and see. Good luck to those who are having problems. -- »www.mtndewbuzz.com | |||
 paulhaskew Unoffical Dominos Spokesman join:2002-01-10 Vancouver, WA clubs:  | reply to KatOak agreed, i am out of the PDX pop, so far no issues as your guys are talking about... only the two major outages that lasted long than 4 hours, once was the who dang pop went down, then AT&T screwed up everything... Did that to the company i work for Charter, numerous times...  I hate seeing posts like these anywhere especially for people who use the same service as me, or the company I work for... Oh, did I mention that tracerts no longer work correctly??? Have the time every other hop times out now... hmm... My fingers are crossed for resolution of the issue you are all having... As for my just look at my thread, Transfer Nightmare (still is by the way, being ignored still )-- I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries. [text was edited by author 2003-09-13 00:53:11] | |||
GamingGirl join:2003-09-13 Folsom, CA | reply to KatOak Ok, this is what I found in my service ticket area when I logged into MySpeakeasy... 2003-09-12 17:00:30 Backhaul installed Statused TT. ISP has reported Dropping Packets (Slow throughput, latency, packet loss). Training Starts: 0 ( 69 ) Reprovisioned loop per ISP request. Training Starts: 0 ( 70 ). This circuit has had 1 previous trouble tickets for RMA. TT will be placed into CPPV> TT will auto close w/in 24 hrs. Not able to reproduce. Trace routes die at our redback, and could be condition of latency between redback and customer. Please depro/repro the port to see if this helps the situation. Thank you. DSLAM Trunk Status: OK Technology: DMT8-2 Card Status: OK Port Status: Up Actual Port Rates: 1536 kbps Downstream / 768 kbps Upstream Margin: 26.0 dB Downstream / 9.5 dB Upstream 2003-09-12 17:01:06 Backhaul installed CLOSED-Pending Partner Verify Unfortunately the reprovision didn't do anything =(. I see lots of latency at my second hop and dropped packets from that point on. What does it mean when they say "Trace routes die at our redback, and could be condition of latency between redback and customer"? I think I'm just out of luck =( Unfortunately, I still have six months on my contract so it doesn't look like I have many options at this point... too sad. =( | |||
GamingGirl join:2003-09-13 Folsom, CA | reply to TrueAudiosaid by TrueAudioI've noticed a lot of routing changes as well. I asked the rep. that I spoke with earlier why I was seeing all these different routes to one of the websites I typically trace (out of curiosity of course =) ) and he told me that "sometimes when a router goes down the route will get changed but Speakeasy hasn't made any changes to the way things get routed". However, I am seeing a lot of sprintlink and att routes that I never saw before. I used to see mainly ALTER for my trips across the US. However, as I said in a previous post, I'm not technically inclined in the least so it could all still be the same but with a different name... what do I know =) | |||
paulhaskew Unoffical Dominos Spokesman join:2002-01-10 Vancouver, WA clubs: | reply to KatOak my stuff is getting routed sadly through level 3 and keeps dropping tons of packets with tracerts... it sucks... AT&T is the carried for the PDX area... and they have problems big time... not sure why speakeasy chose to this provider as the backbone transport... Verio/Sprintlink are nice... good ping times etc... anywho... i dunno... i am sooo tired... example... traceroutes to both yahoo & here... Tracing route to broadbandreports.com [209.123.109.175] over a maximum of 30 hops: 1 19 ms 19 ms 19 ms gate.tailie.net [66.93.174.1] 2 39 ms 44 ms * ge0-0-0.brd-1-sea.speakeasy.net [206.191.168.200 ] 3 17 ms 17 ms 18 ms border26s.g3-2.speakeasy-14.sea.pnap.net [206.19 1.168.220] 4 17 ms 17 ms 16 ms core1.ge2-0-bbnet1.sea.pnap.net [206.253.192.129 ] 5 17 ms 17 ms 16 ms sl-gw11-sea-1-0.sprintlink.net [144.228.95.45] 6 16 ms 15 ms 15 ms sl-bb21-sea-9-3.sprintlink.net [144.232.6.117] 7 59 ms 59 ms 59 ms sl-bb25-chi-2-0.sprintlink.net [144.232.20.157] 8 59 ms 59 ms 59 ms sl-bb21-chi-13-0.sprintlink.net [144.232.26.89] 9 61 ms 61 ms 60 ms sl-st20-chi-15-1.sprintlink.net [144.232.20.80] 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. Tracing route to yahoo.com [66.218.71.198] over a maximum of 30 hops: 1 20 ms 19 ms 19 ms gate.tailie.net [66.93.174.1] 2 * * * Request timed out. 3 19 ms 19 ms 19 ms border26s.g3-2.speakeasy-14.sea.pnap.net [206.19 1.168.220] 4 17 ms 18 ms 18 ms core2.ge2-0-bbnet1.sea.pnap.net [206.253.192.130 ] 5 17 ms 17 ms 17 ms 12.124.173.37 6 17 ms 16 ms 16 ms gbr2-p60.st6wa.ip.att.net [12.123.44.118] 7 18 ms 18 ms 18 ms tbr1-p012601.st6wa.ip.att.net [12.122.12.161] 8 15 ms 16 ms 15 ms ggr1-p340.st6wa.ip.att.net [12.123.44.129] 9 33 ms 33 ms 32 ms att-gw.sea.level3.net [192.205.32.22] 10 34 ms 34 ms 34 ms ge-6-0-1.mp1.Seattle1.level3.net [209.247.9.45] 11 34 ms 34 ms 33 ms so-3-0-0.mp2.SanJose1.Level3.net [64.159.1.130] 12 33 ms 34 ms 34 ms gige10-2.ipcolo3.SanJose1.Level3.net [64.159.2.1 69] 13 36 ms 35 ms 34 ms unknown.Level3.net [64.152.69.30] 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. They never complete anymore... can ping the sites though-- I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries. | |||
koitsu Premium join:2002-07-16 Mountain View, CA | Sadly, Verio here in SFO is absolute garbage (I can vouch for this, as I worked for them for 3 years here in the Bay); packet loss madness. I'm presently seeing a consistent 5-6% loss from Verio. It does not fluxuate. It's good to know Verio's PDX POP is still standing (the head west coast network engineer lives there  ).The two best peering points through SFO InterNAP look to be Level3 and Alter. At least for where I'm commonly going those look to be the best. I'll cover the "random route" thing that GamingGirl asked in the next post. -- Making life hard for others since 1977. | |||
koitsu Premium join:2002-07-16 Mountain View, CA | reply to GamingGirl InterNAP is who Speakeasy peers with. InterNAP peers with all sorts-of different providers; it's synonymous with the PAIX in Palo Alto (i.e. a central point for many backbone providers to connect to one another, hence making the Internet work). InterNAP is a double-edged sword, and I'm sure Kat can talk a bit about why this is as well. It's fantastic -- when everything is working how it should be. Otherwise it's an absolute nightmare to debunk, since they seem to change routes almost at the drop of a coin. InterNAP has some seriously magic BGP voodoo going on -- they drop/swap routes practically on an hourly basis. Even when the interface you're going through is absolutely 100% clean (e.g. prior to all of this packet loss crap), they will _still_ drop the route and switch you from, for sake of example, GBLX to ALTER for absolutely no reason. That's why I say it's "magic BGP voodoo." This is NOT how BGP is normally supposed to be used, and it looks to me like InterNAP simply uses some excessively anal settings, or simply plays spin-a-roo with their routes (I've heard of peering providers doing this as well, but it's not very common). The rep. you spoke to was correct in his description of what's supposed to happen -- when there's a network outage or major anomaly (i.e. packet loss, CRC errors, line errors, or anything else that generally affects POS (Packet Over Sonet) links), BGP will automatically say "Link is screwed, dropping route automatically, moving to another peer." In about 90% of the cases out there, this works great. InterNAP simply deploys this along side a bunch of other stuff -- it's the other stuff which is causing your traceroutes to show different routes every hour, or sometimes as often as every 15 minutes. Oh, BGP stands for Border Gateway Protocol, by the way -- it's just a monitoring protocol that allow routers to make automatic configuration changes based upon rulesets to determine if a network is functioning "up to par" or not. The best way to understand all of this is to get the utility called Ping Plotter and let it run for a few hours again a website you commonly visit (use www.news.com if you need a default). If you know how to use traceroute, then PP will make perfect sense to you -- except that it's a graphical representation of a continual traceroute. One of the coolest features of PP is that it detects and logs route changes, so you can see exactly where, when, and why a route was changed. The other thing it's useful for is showing actual graphs of packet loss and latency. Try it out, you might like it -- for *IX I prefer mtr (matt's traceroute), while for Windows I prefer Ping Plotter. Hope this helps. -- Making life hard for others since 1977. | |||
paulhaskew Unoffical Dominos Spokesman join:2002-01-10 Vancouver, WA clubs: | reply to koitsu any idea why my tracerts no longer complete? | |||
Bondman join:2001-08-24 Livonia, MI | Paul: I get timeouts on tracerts to WWW.BroadbandReports.com as well unfortunately but I am able to ping them. I think Speakeasy needs to push for better service from their backbone providers. tracert www.broadbandreports.com Tracing route to broadbandreports.com [209.123.109.175] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms ml350.server-tech.local [10.1.1.1] 2 17 ms 16 ms 16 ms dsl093-002-001.det1.dsl.speakeasy.net [66.93.2.1 ] 3 30 ms 19 ms 32 ms border5.ge3-2.speakeasy-28.chg.pnap.net [64.94.3 5.212] 4 19 ms 19 ms 19 ms core2.fe0-1-bbnet2.chg.pnap.net [64.94.32.66] 5 19 ms 18 ms 20 ms POS3-1.GW1.CHI13.ALTER.NET [65.207.239.21] 6 20 ms 18 ms 18 ms 0.so-1-0-0.XL1.CHI13.ALTER.NET [152.63.69.178] 7 21 ms 20 ms 20 ms 0.so-2-2-0.XL1.CHI2.ALTER.NET [152.63.70.102] 8 19 ms 20 ms 19 ms POS6-0.BR2.CHI2.ALTER.NET [152.63.67.241] 9 * * * Request timed out. 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * ping www.broadbandreports.com Pinging broadbandreports.com [209.123.109.175] with 32 bytes of data: Reply from 209.123.109.175: bytes=32 time=53ms TTL=53 Reply from 209.123.109.175: bytes=32 time=52ms TTL=53 Reply from 209.123.109.175: bytes=32 time=53ms TTL=53 Reply from 209.123.109.175: bytes=32 time=63ms TTL=53 Ping statistics for 209.123.109.175: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 52ms, Maximum = 63ms, Average = 55ms [text was edited by author 2003-09-13 17:28:49] | |||
paulhaskew Unoffical Dominos Spokesman join:2002-01-10 Vancouver, WA clubs: | reply to KatOak well they used to complete | |||
wattjg Premium join:2003-05-24 San Jose, CA | reply to Bondman They may be blocking UDP but not ICMP echo requests. The only thing that's odd is that the Windows "tracert" uses ICMP. Unix traceroutes commonly use UDP by default. Both Windows and Unix "ping" use ICMP. That's not consistent with your data, but I was able to "tracert" (Windows) to www.dslreports.com, but not "traceroute" (Unix/FreeBSD) to it. I can "ping" it from both machines. Jim | |||
 sporkme drop the crantini and move it, sister Premium,MVM join:2000-07-01 Morristown, NJ  ·Optimum Online | reply to koitsusaid by koitsuIt's not really synonymous. InterNAP buys *transit* from mutliple providers, likely at some fixed price point if they guarantee "X" number of megabits/sec average. At a peering point like PAIX, providers go there to exhange *peering*, which is *not* the same as paying for transit. They will agree on a pipe of some size, who will pay what portion of the interconnect, and agree that this is of mutual benefit to them as ISP X's customer's can now directly reach ISP Y's customers without either side paying each other for transit. These days it's a bit more complicated than that, but that's the theory at least. InterNAP is simply a paying customer of a number of backbones. They have no network interconnecting their PoPs, each one is an "island". The original plan was to sell "premium connectivity", but ever since they've had financial difficulties, it's become a bit different. You now see more and more traffic going out the cheapest providers. One day we'll see Cogent in the mix. said by koitsuThis is certainly on purpose, and it's so that they meet their contractual agreements with each provider without going over/under what they've agreed to buy from each provider. I would not be at all surprised to find a RouteScience box at each PoP twiddling all these paths for them. RouteScience, »www.routescience.com/, is a really nifty thing, but it's just a tool. It's generally used to *improve* performance, but it also has many options that let you say "we don't want to average more than 10Mb/s this month, we don't want to average more than 40Mb/x this month to Level3, but we'll go up to 100Mb/s to Verio, and Cogent is unlimited". Hopefully one day SE will get more neteng people on board and think about simply multihoming themselves to 5 or 6 providers at each PoP and managing this themselves. It requires more time and talent, but in today's fluctuating bandwidth market, it makes sense to add and drop providers on a regular basis. If they are in good facilities with low x-connect charges, it's a win-win. -- just a minute | |||
koitsu Premium join:2002-07-16 Mountain View, CA | reply to paulhaskew The examples you show above make it all the way to a Sprint peering point in Chicago (for BBR) and a peering point with Level 3 in (probably) their Santa Clara POP (for Yahoo). I can't provide you traceroutes for comparison to either www.yahoo.com or www.broadbandreports.com because the route packets go from my co-lo to either of those sites are completely different. (Heck, my co-lo is 6 hops from www.yahoo.com, go figure). Try tracerouting to a different destination. For example, 4.2.2.1 (which is a public nameserver hosted by GTEI): Tracing route to vnsc-pri.sys.gtei.net [4.2.2.1] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 192.168.0.1 2 11 ms 11 ms 11 ms dsl081-051-001.sfo1.dsl.speakeasy.net [64.81.51.1] 3 11 ms 12 ms 12 ms border5.g3-4.speakeasy-29.sfo.pnap.net [63.251.53.219] 4 12 ms 11 ms 11 ms core1.ge0-0-bbnet1.sfo.pnap.net [63.251.63.1] 5 12 ms 12 ms 12 ms sl-gw13-sj-0-2-155M.sprintlink.net [160.81.100.1] 6 13 ms 12 ms 11 ms sl-bb21-sj-6-0.sprintlink.net [144.232.3.173] 7 13 ms 13 ms 12 ms sl-st20-sj-13-0.sprintlink.net [144.232.9.58] 8 15 ms 14 ms 13 ms interconnect-eng.SanJose1.Level3.net [209.245.146.245] 9 13 ms 14 ms 14 ms so-5-0-0.gar1.SanJose1.level3.net [209.244.3.137] 10 15 ms 15 ms 13 ms so-7-0-0.mp1.SanJose1.Level3.net [64.159.1.73] 11 78 ms 79 ms 79 ms so-0-2-0.bbr1.Atlanta1.level3.net [209.247.9.101] 12 78 ms 78 ms 79 ms pos8-0.hsa1.Atlanta1.Level3.net [209.247.9.166] 13 79 ms 81 ms 78 ms vlan521.public-msf1.Atlanta2.Level3.net [67.72.92.18] 14 78 ms 81 ms 79 ms vnsc-pri.sys.gtei.net [4.2.2.1] One thing to note: it looks to me that Ping Plotter and Windows traceroute may be using different protocols (possibly ICMP vs. UDP) for their tests. I cannot traceroute to my co-lo using Windows tracert (dies at hop #5), while using Ping Plotter from the same box works fine. Hop #5, in this case, happens to be so-3-0-0.ar4.SFO1.gblx.net. -- Making life hard for others since 1977. | |||
koitsu Premium join:2002-07-16 Mountain View, CA | reply to sporkme So to sum everything up, literally InterNAP is just a customer who happens to have service with multiple backbone providers (Global Crossing, Verio, Level 3, Alter, and a few other who's names are slipping my mind since I just woke up), then proceed to resell portions of their own POP, with their mysterious BGP setup deployed, to customers (like SE) for network connectivity. Likewise, the reason for their entertaining merry-go-round routing procedures is a purely financial one due to the costs of aggregate bandwidth total per peering point (hence balancing out the traffic between all providers rather than shoving it all through one which happens to have the lowest hop count and more reliable link (at that time)). What a bummer. -- Making life hard for others since 1977. | |||
paulhaskew Unoffical Dominos Spokesman join:2002-01-10 Vancouver, WA clubs: | reply to koitsu yeah, i noticed that as well... i downloaded the free pingplotter yesterday and the tracert completed... i will test some other stuff once i get home tonight... here is something i did from work, ping test almost all day, from 1:30pm to 8:53pm... Ping statistics for 66.93.174.127: Packets: Sent = 26297, Received = 26290, Lost = 7 (0% loss), Approximate round trip times in milli-seconds: Minimum = 15ms, Maximum = 547ms, Average = 18ms -- I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries. [text was edited by author 2003-09-13 23:53:26] | |||
koitsu Premium join:2002-07-16 Mountain View, CA | reply to KatOak
I'm somewhat bothered by the fact that today -- Saturday -- shows no ramping usage spike from 09:00 until midnight (or, well, technically 21:33). Just posting this for Kat and anyone else who's interested in how SFO is doing. That huge packet loss problem around 23:30 to midnight last night, I'm hoping, was caused either by Verio or possibly changes with InterNAP or SE. That's all for now! -- Making life hard for others since 1977. | |||
