| [northeast] Speed/Latency issues every night at 2amHello all,
So I've been a happy fios customer for over a year now. Everything's been running 100% until last week sometime. I thought it was a fluke, so I ignored it. But I no longer think that, and I'm afraid it looks like there's some sort of issue I may have to get in touch with Verizon about.
Every night, sometime after 2am, I am experiencing horrible latency issues. It makes the internet more or less unusable. No issues at all during the day as daytime pings are C:\>ping dslreports.com
Pinging dslreports.com [209.123.109.175] with 32 bytes of data:
Reply from 209.123.109.175: bytes=32 time=356ms TTL=48
Reply from 209.123.109.175: bytes=32 time=168ms TTL=48
Request timed out.
Reply from 209.123.109.175: bytes=32 time=3353ms TTL=48
Ping statistics for 209.123.109.175:
Packets: Sent = 4, Received = 3, Lost = 1 (25% loss),
Approximate round trip times in milli-seconds:
Minimum = 168ms, Maximum = 3353ms, Average = 1292ms
C:\>ping google.com
Pinging google.com [72.14.204.99] with 32 bytes of data:
Reply from 72.14.204.99: bytes=32 time=2181ms TTL=51
Reply from 72.14.204.99: bytes=32 time=1611ms TTL=51
Reply from 72.14.204.99: bytes=32 time=1406ms TTL=51
Reply from 72.14.204.99: bytes=32 time=2753ms TTL=51
Ping statistics for 72.14.204.99:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1406ms, Maximum = 2753ms, Average = 1987ms
With ping times in the order of seconds, I think it's clear there's an issue. This happens with all computers on the network.
Tracert to the same google server:
C:\>tracert gooogle.com
Tracing route to gooogle.com [72.14.204.99]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms WRT54G-TM.WRT54G-TM [192.168.1.1]
2 1819 ms 1172 ms 452 ms L100.PHLAPA-VFTTP-65.verizon-gni.net [173.49.125
.1]
3 2980 ms 1392 ms 1690 ms G5-0-7-865.PHLAPA-LCR-08.verizon-gni.net [130.81
.104.154]
4 167 ms 2255 ms 1287 ms so-3-1-0-0.PHIL-BB-RTR2.verizon-gni.net [130.81.
29.6]
5 1823 ms 931 ms 2374 ms so-7-1-0-0.RES-BB-RTR2.verizon-gni.net [130.81.1
9.106]
6 1648 ms 1359 ms 2193 ms 152.63.32.161
7 1809 ms 2941 ms 501 ms xe-11-2-0.edge1.Washington4.level3.net [4.68.63.
165]
8 3012 ms 1804 ms 1226 ms ae-4-99.edge1.Washington1.Level3.net [4.68.17.20
8]
9 1651 ms 1434 ms 1833 ms GOOGLE-INC.edge1.Washington1.Level3.net [4.79.23
1.6]
10 3005 ms 2889 ms 2009 ms 209.85.240.136
11 1299 ms 3506 ms 2115 ms 66.249.94.46
12 1430 ms 3166 ms 2823 ms iad04s01-in-f99.1e100.net [72.14.204.99]
Trace complete.
I see a lot of latency in the verizon network, before it even goes out to the general internet. 1800ms to the first hop?
I'm a bit of a night owl, so it's rather annoying that this happens at night, which is kind of primetime for me. Again, any info/tips/etc appreciated. THANK YOU in advance for taking the time to share your knowledge. |
|
|
|
 birdfeedrPremium,MVM
join:2001-08-11
Warwick, RI
kudos:8 | When does it come back to normal? Is it predictable?
My suggestion is to call FSC while the problem exists. They can read the ONT logs in case it shows up there.
Perhaps it's CO maintenance in the wee hours. |
|
 SparkChaserPremium
join:2000-06-06
Downingtown, PA
kudos:3
1 edit | reply to yyonline
I'm also from Dtown. I don't see anything in the DSLR monitoring. Of course, it's them pinging me.
Edit: See link below. |
|
birdfeedrPremium,MVM
join:2001-08-11
Warwick, RI
kudos:8 | This link works better. »/testhistory?v···t=m&p=13
In future views, SparkChaser results may be further down the screen as new monitors are added. |
|
| reply to yyonline
I'm in the Pittsburgh, PA area, and I've also seen connectivity problems intermittently between about 2am and 3am with both phone and internet ever since I got FIOS about a year ago. Sometimes the phone will go out (no dial tone, line appears to be off hook), but the internet still works. Other times, the phone seems fine (I get dial tone, but I don't try calling anyone that late!), but the internet is slow or completely non-responsive. Sometimes they both go out. I don't know if the TV is effected, as I just got that a few days ago.
I've never called support about it, 'cause I just assume that they are doing maintenance, at an hour that inconveniences only night owls and vampires.  |
|
SparkChaserPremium
join:2000-06-06
Downingtown, PA
kudos:3 Reviews:
 ·Verizon FiOS
| reply to yyonline
Last night in Downingtown
|
|
 FTTC
join:2008-10-01
Mount Rainier, MD | reply to yyonline
i ping your 3rd hope in about the same time you do. im only a few hundred miles away  . also isn't philly closer to newyork? where dslreports.com server is. i ping there in 14ms. im beating you there by 30ms & your closer! i would say you have bad routing. not terrible but not good. |
|
FTTC
join:2008-10-01
Mount Rainier, MD
1 edit | reply to yyonline
pinged your 4th hop. i beat you there as well my max, is 10ms yours is 120ms. thats ridiculous. i call verizon and complain. |
|
| reply to yyonline
Thanks for the feedback guys. The problem continues...
When pings are good, they're really good. Under 20ms to most of the NE. But at times, they're really bad, usually at night. I have noticed them getting into the 200ms range even during the day, however.
I'll have to call up Verizon sometime when it's happening. While I'm usually awake, I don't normally have time at 2am for a troubleshooting phone call. I'll have to add that to the list of things to do this weekend.
I have a Linksys router as primary. Will Verizon troubleshoot with that hooked up, or would it be a good idea to reconnect the Actiontec before calling support? Hmm, might be a good idea anyway just to rule out a malfunctioning router. But yeah, if anybody has any experience on whether or not it's necessary to dust off the Actiontec, that'd be great. |
|
birdfeedrPremium,MVM
join:2001-08-11
Warwick, RI
kudos:8 | said by yyonline:I have a Linksys router as primary. Will Verizon troubleshoot with that hooked up, or would it be a good idea to reconnect the Actiontec before calling support? I've gotten "call us back when the Actiontec is primary" once when I had a connection problem with one specific site. I rebuild my home network in one fashion or another every few months anyway. I can't say whether it keeps things running smoothly or not, but I do think that since your problem appears to be local (in the CO or just upstream), taking steps to acquire a new WAN IP address may just clear it up. Swapping to the Actiontec primary will probably do that.
»Verizon Online FiOS FAQ »How do I release my DHCP lease |
|
SparkChaserPremium
join:2000-06-06
Downingtown, PA
kudos:3 Reviews:
·Verizon FiOS
| reply to yyonline
another check from Downingtown, last night.
|
|
| reply to yyonline
Just got FIOS 15/5 last month and I today I reported ping times spiking from 100ms to over 800ms! I called Verizon FIOS support and was on the phone for over 30min. The tech rep didn't even know what ping was.
Turns out the best solution she and her supervisor had to offer was nothing at all. Quote: "Can't help you.".
I noticed more people are complaining. Generally happens once in a blue but they should advertised their service more clearly: "Fast gaming sometimes". The sole reason why I switched from cable to FIOS was for low latency. I mean light speed isn't good enough now? |
|
 Smith6612Premium,MVM
join:2008-02-01
North Tonawanda, NY
kudos:22
 ·Verizon Online DSL
·Frontier Communi..
| said by Shadowseph :
I noticed more people are complaining. Generally happens once in a blue but they should advertised their service more clearly: "Fast gaming sometimes". The sole reason why I switched from cable to FIOS was for low latency. I mean light speed isn't good enough now?
What we need, is ludicrous speed, not light speed as it's too slow! :P
--
This is a signature, eaten by spring. |
|
| reply to yyonline
The saga continues... I managed to dust off and reconnect the Actiontec. No difference in performance. I still have really bad latency at night. Happens throughout the day as well, just not as bad as at night.
I've called Verizon twice about this now, both times the technician has said that they take their network analysis tools down for maintenance from 12am-6am. Really frustrating when trying to troubleshoot an issue that seems to occur most often during that time. |
|
SparkChaserPremium
join:2000-06-06
Downingtown, PA
kudos:3 Reviews:
·Verizon FiOS
| said by yyonline:The saga continues... I managed to dust off and reconnect the Actiontec. No difference in performance. I still have really bad latency at night. Happens throughout the day as well, just not as bad as at night. I'm over here in West Bradford section of Downingtown and pinging all the Verizon nodes in your list I get numbers mostly in the single digits.
Question for the gurus, wouldn't this indicate something between the OP and first node. e.g. ONT or the node itself?
--
--
--
"Facts do not cease to exist because they are ignored." - Aldous Huxley
"Children will not remember you for the material things you provided, but for the feeling that you cherished them." - Richard Evans |
|
hubrisnxs
join:2009-12-30
Fountain Valley, CA
kudos:1 | reply to Shadowseph
said by Shadowseph :
Just got FIOS 15/5 last month and I today I reported ping times spiking from 100ms to over 800ms! I called Verizon FIOS support and was on the phone for over 30min. The tech rep didn't even know what ping was.
Turns out the best solution she and her supervisor had to offer was nothing at all. Quote: "Can't help you.".
I noticed more people are complaining. Generally happens once in a blue but they should advertised their service more clearly: "Fast gaming sometimes". The sole reason why I switched from cable to FIOS was for low latency. I mean light speed isn't good enough now?
it's called a bottleneck. you're only as good as your weakest link, and if you have a bad link either at your house or a hop out on the net, it doesn't matter what you have for a connection. Fios, T1, DS3, OC3 or not.
Their Network technicians or NT's can report bad latency to level 3 (which is their backbone provider) And so if you see heavy latency on a Verizon specific hop, you can call in. The problem here is that the front line agents aren't instructed on how to troubleshoot pings and latency. they basically say "if you can download fast, we don't care about latency"
Which is understandable as they don't have anything in their TOS talking about latency and have no garuntee's about it. BUT if you call in, what you are hoping for is that they (first line supprt) reach out to a NT or network technician and that its made CLEAR that the problem resides on the first hop. AT that point they may either just take the IP for the first hop and then report it to level 3 or they will ask for screen shots like you are sharing here. But your goal is to talk to them over at the regular tech support line, emphasize the 400 -1800 ms on the first hop. and get them to talk to their NT's and have them in turn emphasize to the nt's that it's a problem at the first hop and it's 400-1800 ms right there, and that is in fact a verizon owned hop.
without them notifying level 3 the problem will be fixed on their own time, which could be as little as hours, or as long as a few weeks to a month. it all depends on their management software and if it's setting off red alarms at their NOC |
|
birdfeedrPremium,MVM
join:2001-08-11
Warwick, RI
kudos:8 | said by hubrisnxs:it's called a bottleneck. you're only as good as your weakest link, and if you have a bad link either at your house or a hop out on the net, it doesn't matter what you have for a connection. Fios, T1, DS3, OC3 or not. If the bottleneck is before it gets out of the CO, consider trying to get a new IP address. You may need to release the WAN IP at the router and shut it off overnight (or daytime), basically longer than a good 2 hours.
However, you may pull a new IP sooner than that.
Did I ask you that already? |
|
1 edit | reply to yyonline
It is my experience that they (Verizon network engineers) complete maintenance mostly early morning. However, the left and right hands don't talk to each other.
When it comes to network issues "deep in the network" I find it more productive to talk to the VZ NOC. As they are more than just bodies, and they can actually help you troubleshoot a network issue, they also have core and edge router access.
If you need there contact details, you can find it by doing a whois lookup on your external ip address. The number you'll want to call is the OrgTechHandle.
EDIT: But please remember that ICMP (ping) is of lowest priority on backbone networks, so if the links are carrying a lot of traffic, the icmp traffic gets discarded. The congestion occurs usually when they bring another backbone circuit down for maintenance, as they remaining paths have to make up for the downed link. |
|
1 edit | reply to yyonline
Well, another update...
So, I unplugged everything for 2 hours and got a 96.x.x.x IP instead of a 173.x.x.x IP.
Even with the new IP, the routing remains the same. First hop is still always 173.49.125.1, regardless of the site I trace.
Even now at 6pm, ping to my first hop is 60-100 ms. I understand ping is deprioritized, but when latency gets very bad at night, I see time outs in web pages, etc. Transfer speeds haven't been an issue, just latency, which contributes to slow page loads as the browser will have a delay as it opens a separate connection for each page, image, etc.
I decided to ask my next door neighbor if I could use their computer today. They also have fios and a 96.x.x.x IP address. Neighbors routing to a few sites I tested is 100% identical to mine. Their first hop is the same address as my first hop. Neighbors can ping the first hop at 5-6 ms. During the same time, my ping to the same first hop is 60-100 ms.
So, given the neighbors can access the same first hop with no latency problems at the same time I have latency issues, and that the latency continues with multiple routers and multiple IP addresses, my uneducated guess is that the router/switch/whatever at the first hop is behaving fine. I think this would seem to indicate that the issue may be isolated to my connection only and lies somewhere between my router and the first hop. Perhaps something is goofy with the ONT, the fiber drop, etc? I've already done a hard reboot of the ONT with no real results.
Well, I think I'll have to keep calling Verizon until I can get someone on the line who understands the issue. One tech I got understood the issue, but said he couldn't set up a ticket or anything with their systems being down. The second call, I had to explain what latency meant...that didn't go so well. I try to be nice, though, as I do tier 3 tech support for a large corporation as my day job, so I can see things from both sides.
Thanks for the continued input! |
|
guppy_fishPremium
join:2003-12-09
Lakeland, FL
kudos:1 Reviews:
·Verizon FiOS
| first hop can be your router or noise on the coax from your router to the ONT. Go into the router and look at your WAN statistics and see what they are.
You can try connecting the router direct to the ont ( dedicated run ) and see if that helps, but it sure sounds like a home wiring or splitter issue |
|
