| [Rant] Comcast Maintenance at 2am ends in null-routed subnet
Modem diag screen showing maintenance at 2:16am when Hector went offline |
It seems that during last night's maintenance, some idiot null routed my friend's subnet.
Addresses 71.196.204.1-71.196.205.254 inclusive (71.196.204.0/23) are ALL unreachable. Traceroutes below.
Small backstory: I'm the go-to network guy for some friends of mine in the Denver area. I've got custom OpenWRT running on their routers, with SSH open for remote access and each router has a dyndns address in case I need to fix something remotely.
I get a call from my friend Hector. He tells me that since about 2am last night (~02:00 December 14 2011) his internet has been dead.
I explain to him that Comcast often will perform network upgrades and maintenance at this time of night, as when I subscribed to their service, staying awake all night long, I'd lose connectivity around that time for a few hours every few months.
So I asked if it came back on later, and no, it was still dead as of 7am December 14 before he left for work.
So I traceroute from my house, which from CentQwesturyLink goes thru texas instead of _directly in town_ (i'll NEVER understand why the 2 main ISPs in denver screw their customers over by routing to DALLAS first. HLRN means Highlands Ranch, a suburb of Denver where, apparently, the DSL services are hubbed)
traceroute to hectors.dyndns.host (71.196.204.25), 30 hops max, 60 byte packets
1 Heimdall.mystica.lan (192.168.x.x) 0.163 ms 0.185 ms 0.220 ms
2 192.168.0.1 (192.168.0.1) 0.870 ms 0.884 ms 0.904 ms
3 hlrn-dsl-gw91-156.hlrn.qwest.net (207.225.112.156) 21.901 ms 22.038 ms 22.315 ms
4 hlrn-agw1.inet.qwest.net (71.217.190.209) 22.325 ms 22.900 ms 22.911 ms
5 dap-brdr-04.inet.qwest.net (67.14.2.162) 46.566 ms 47.289 ms 46.719 ms
6 ix-0-1-0-0.tcore2.DT8-Dallas.as6453.net (206.82.142.1) 48.026 ms 46.924 ms 45.620 ms
7 206.82.142.22 (206.82.142.22) 47.866 ms 47.925 ms 206.82.142.18 (206.82.142.18) 47.874 ms
8 pos-2-2-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.86.149) 46.524 ms 46.374 ms 45.685 ms
9 pos-2-10-0-0-cr01.denver.co.ibone.comcast.net (68.86.87.185) 60.631 ms 60.842 ms 60.801 ms
10 * * *
11 *^C
This traceroute is very unique, because of where it dies. It does not die on the 'ar' hop. Not on the 'ur' hop and not on the 'cdn' hop. It dies on the 'cr' hop, presumably meaning 'core router'. The IBONE HOP. Baffled at this, I tried from my cellphone.
traceroute to 71.196.204.25 (71.196.204.25), 30 hops max, 38 byte packets
1 10.170.205.48 (10.170.205.48) 701.783 ms 116.394 ms 118.317 ms
[10.x addresses in tmobile's internal network were removed for space ]
9 10.176.188.190 (10.176.188.190) 118.408 ms 96.497 ms 119.812 ms
10 te-9-3.car2.Denver1.Level3.net (4.53.14.73) 119.782 ms 96.710 ms 119.690 ms
11 ae-2-52.edge3.Denver1.Level3.net (4.69.147.104) 119.812 ms 135.681 ms 110.535 ms
12 COMCAST-IP.edge3.Denver1.Level3.net (4.79.82.54) 119.721 ms COMCAST-IP.edge3.Denver1.Level3.net (4.28.20.6) 100.037 ms COMCAST-IP.edge3.Denver1.Level3.net (4.79.82.54) 96.680 ms
13 * * *
14 *^C
/ #
No Ibone, but, again, the hop just before the Denver Comcast core router, this time on Level3's side.
Stumped at traces being blocked I had someone presently on Comcast try:
traceroute to 71.196.204.25 (71.196.204.25), 30 hops max, 60 byte packets
1 10.120.150.240 (10.120.150.240) 0.663 ms 1.440 ms 1.821 ms
2 96.152.122.1 (96.152.122.1) 24.303 ms 25.202 ms 27.116 ms
3 te-9-2-ur01.aurora.co.denver.comcast.net (68.86.105.81) 16.506 ms 16.629 ms 16.737 ms
4 * * *
5 * * *
So this seems to be a largish routing failure. It never reaches the CDN hop that a proper trace shows:
9 pos-2-15-0-0-cr01.denver.co.ibone.comcast.net (68.86.85.173) 60.671 ms 58.924 ms 60.328 ms
10 pos-0-13-0-0-ar02.aurora.co.denver.comcast.net (68.86.95.246) 64.588 ms 60.150 ms 64.050 ms
11 te-8-3-ur01.aurora.co.denver.comcast.net (68.86.103.38) 60.068 ms 59.640 ms 59.886 ms
12 te-17-10-cdn12.aurora.co.denver.comcast.net (68.85.221.190) 72.856 ms 61.788 ms 79.876 ms
13 c-71-196-255-xxx.hsd1.co.comcast.net (71.196.255.xxx) 68.618 ms 70.031 ms 71.013 ms
So I meet him at home after he gets out of work, and I find his router is PROPERLY DHCPING THE IP address. IT HAS PROPER DNS SERVERS. IT HAS THE PROPER GATEWAY. IT SIMPLY CANNOT ROUTE.
I try the laptop directly to the modem? works perfectly fine. Of course, its a different IP, as the IPs are based on MAC address and assigned by DHCP.
I CHANGE THE ROUTERS MAC ADDRESS to DHCP a new IP; IT ROUTES FINE!
How did you MESS THIS UP SO BAD COMCAST?!
Without me actively understanding IP networks, my friend would STILL be without internet.
I called in a trouble ticket and explained to the first level tech that the packets were not routing at all, gave the IP inside the subnet that was affected... I just hope that said trouble ticket gets to the right techs, else a few more people are likely without internet right now and DON'T KNOW WHY.
--
:wq!
Wrote dslreports_forum_post
mike@thor:~>_ |
|
|
|
Reviews:
 ·Comcast
| exact same thing happened to me this morning while i was in a skype group audio call. my logs match up exactly to yours line by line. Mine happend around midnight but it came back after about five minutes. maybe they broke something important in his area? maybe his modem got fried with a bad firmware update? |
|
| If your line goes out, but comes back within 5 minutes, thats a normal maintenance window.
If your line goes out, and stays out the entire day, and the entire subnet you are on cannot route *anywhere* FROM *anywhere* this is a wholly different problem.
Someone typoed a routing table update. The modem never flashed any new firmware; The internet works fine once I changed the MAC address on the router to force a new DHCP IP.
--
:wq!
Wrote dslreports_forum_post
mike@thor:~>_ |
|
| possibly but its unsual for that to happen and no one to notice in the company. broken hardware on comcast's end comes to mind, or something physical thats wrong. |
|
| Definitely unusual, but, it did happen, and its definitely on Comcast's end. If a packet to a subnet dies at the core router, yeah, the routing table is messed up.
--
:wq!
Wrote dslreports_forum_post
mike@thor:~>_ |
|
| well lets just hope it gets fixed soon before something else happens  |
|
netcoolPremium
join:2008-11-05
Englewood, CO
kudos:11 | reply to mystica
said by mystica:I called in a trouble ticket and explained to the first level tech that the packets were not routing at all, gave the IP inside the subnet that was affected... I just hope that said trouble ticket gets to the right techs, else a few more people are likely without internet right now and DON'T KNOW WHY.
Should be fixed now. |
|
| said by netcool:Should be fixed now.
Thanks for this. I'll keep my custom MAC in the router until I can get over there again and reboot the modem to clear its MAC cache.
--
:wq!
Wrote dslreports_forum_post
mike@thor:~>_ |
|
| reply to mystica
Well I am also experiancing something similar that occurred at the same time. I have a Windows Server 2003 that I am using as my router/gateway with comast. When I have just that machine connected to the internet it works great. As soon as I plug in the other NIC and add my LAN to the network I have internet across the LAN for about 5-10 min then I lose it. At that point I can only ping Comcast's gateway but no further. The tracert also dies at the gateway. After restarting the server box with the LAN disconnected it works again. Reconnect LAN and it dies. Is there possibly something in this upgrade that would be stopping me. I dont understand why, because I am just using a compter as a router in place of a typical one. Any suggestions would be great. |
|
Reviews:
·Comcast
| That sort of sounds more like a business setup rather then a residental setup with a router, so maybe something is detecting the server OS and raising a red flag? I know that Comcast has been working hard at eliminating businesses using residential service. |
|
 beachintechThere's sand in my tool bagPremium
join:2008-01-06
kudos:5
·Comcast
| reply to Reggie
The problem lies with your router, and not with comcast. If your connection works with just the main router box connected, it's good to go. You adding a nic with your LAN connected won't impact the first connection unless it breaks the setup on the OS.
--
Ex-Tech at the Beach.
I speak for myself, not my former employer. |
|
 Al_S
@comcastbusiness.net | Did anybody experience any problems over this past weekend in Chicago? I lost internet on Saturday and per Comcast it's my modem. I replaced it and it still didn't want to work. Had to call them a few times and now it's working but my router doesn't work. Replaced it with an old one I had and it kind of works. My PC in the bedroom doesn't want to connect. I thought I got a power spike or something but another coworker had a similar problem. |
|
