US Cable Support > Comcast HSI > [IPv6] TomatoUSB and Comcast IPv6 -- bugs found

[IPv6] TomatoUSB and Comcast IPv6 -- bugs found

Re: TomatoUSB and Comcast IPv6 -- bugs found

reply to koitsu
Unknown at this time. I haven't received a response back from Toastman yet (he's probably at work), and Shibby hasn't commented on the Linksysinfo.org post. So, at this time, no ETR.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

reply to koitsu
I wrote the following workaround which I've confirmed as 100% reliably working on my RT-N16 running Toastman's firmware (specifically tomato-K26USB-1.28.7499MIPSR2Toastman-RT-VPN.trx). TomatoUSB users can put this in their Administration - Scripts - WAN Up section:

#
# Workaround for modifying /etc/dhcp6c.conf to work with Comcast IPv6,
# requiring IA-NA DHCP option bit set.
#
# Note that this workaround intentionally messes up the spacing of
# the send ia-pd option; this is to ensure that if if the sed command
# is run multiple times it won't continue to append the send ia-na entry
# over and over.
#
# http://www.dslreports.com/forum/r27234575-TomatoUSB-and-Comcast-IPv6-bugs-found
#
sed -i -e's/ send ia-pd 0;/send ia-pd 0;\n send ia-na 0;/' /etc/dhcp6c.conf
kill `cat /var/run/dhcp6c.pid` && dhcp6c -T LL `nvram get wan_iface`
#
# Workaround for TomatoUSB bug where a spurious default IPv6 route is
# added for no justified reason, resulting in packets getting forwarded
# effectively to /dev/null.
#
# http://www.dslreports.com/forum/r27234575-TomatoUSB-and-Comcast-IPv6-bugs-found
#
route -A inet6 del default gw :: metric 1024 `nvram get wan_iface`
 

root@gw:/tmp/home/root# uptime
 15:19:26 up 0 min, load average: 0.15, 0.05, 0.01
root@gw:/tmp/home/root# ping6 ipv6.google.com
PING ipv6.google.com (2001:4860:4001:803::1010): 56 data bytes
64 bytes from 2001:4860:4001:803::1010: seq=0 ttl=56 time=56.647 ms
64 bytes from 2001:4860:4001:803::1010: seq=1 ttl=56 time=46.848 ms
64 bytes from 2001:4860:4001:803::1010: seq=2 ttl=56 time=14.746 ms
64 bytes from 2001:4860:4001:803::1010: seq=3 ttl=56 time=64.740 ms
64 bytes from 2001:4860:4001:803::1010: seq=4 ttl=56 time=15.879 ms
64 bytes from 2001:4860:4001:803::1010: seq=5 ttl=56 time=16.359 ms
 
--- ipv6.google.com ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 14.746/35.869/64.740 ms
 

root@gw:/tmp/home/root# ifconfig vlan2
vlan2      Link encap:Ethernet  HWaddr E0:CB:4E:C0:00:C5
           inet addr:67.180.84.87  Bcast:67.180.87.255  Mask:255.255.252.0
HERE -->   inet6 addr: 2001:558:6045:9b:114c:a6be:d64d:520c/128 Scope:Global
           inet6 addr: fe80::e2cb:4eff:fec0:c5/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:270068 errors:0 dropped:0 overruns:0 frame:0
           TX packets:88814 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:33258559 (31.7 MiB)  TX bytes:9207877 (8.7 MiB)
 

Didn't work for me. I am running an E4200 with Tomato Firmware v1.28.0499 MIPSR2Toastman-RT-N K26 USB VPN
and I have my IPv6 settings to DHCPv6 with prefix delegation with only the accept RA from WAN ticked.

reply to koitsu
Just a FYI, but you should be able to replace this line:

kill `cat /var/run/dhcp6c.pid`

with

killall dhcp6c

You might want to also add a check to make sure dhcp6c is even running before just starting it.

Oh and if the WAN connection goes down, the script will run again and add another "send ia-na 0;" line to the dhcp6c.conf file, even if that line is already there. Though I think that file gets generated on WAN UP so that might be okay.
--
The Comcast Disney Avatar has been retired.

kill `cat /var/run/dhcp6c.pid` && dhcp6c -T LL `nvram get wan_iface`
 

reply to LMoreland

can you send me your email address and I will send all of that to you. You can pm it to me from this or the linksys forum. I have replied to your thread pointing here in the linksys forum.

Thanks

reply to koitsu
I've heard that Toastman may be only communicating on linksysinfo and not tomatousb.org these days, so you may want to try to contact him over there.

No problem. Looks like someone also already posted in the latest Toastman build thread on linksysinfo: »www.linksysinfo.org/index.php?th···t-184391

reply to koitsu
Thanks for all the work you've put into this, koitsu! Using your script, my router is now assigned a native IPv6 address (2001:558:...), and IPv6 connectivity from the router to WAN works perfectly (ping6 via telnet to ipv6.google.com works, etc).

However, for some reason, none of my LAN devices are being assigned valid/working IPv6 addresses. The assigned address prefix is 2601:b:..., and there is no IPv6 connectivity at all. I'm new to Tomato (and IPv6), so I feel like I'm just missing a setting or otherwise doing something stupid. One of the lines from my router logs:

unknown daemon.err httpd[3398]: bind: [2601:b:a780:31:9644:52ff:fea6:cdfc]:80: Cannot assign requested address

Any ideas?

The "LAN ordeal", meaning getting actual IPv6 packets from a machine on one's local network to talk to the Internet, is a separate problem that I still cannot figure out. I have spent the entire day working on this and I'm no where closer to where I was before. Other people have IM'd me and sent me Email complaining about the same thing -- I can't help.

I'd love to wrap my hands around the committees/individuals that designed IPv6. The amount of obfuscation and nonsense going on under the hoo3 is almost embarrassing. It's about the most non-KISS compliant protocol I've worked with yet, other than, say, IPsec. :P

The httpd error you see indicates that httpd (that's the web server that runs on the router itself, used for configuring the GUI bits, etc.) cannot bind to an IPv6 address. The reason for this is that, if I remember right, the httpd binary on TomatoUSB is not built properly with IPv6 support. All this would impact is being able to reach your TomatoUSB router's GUI natively via IPv6. I get the exact same message, but for my Internet-facing IPv6 address. I'm not concerned about httpd in the least bit.

At this point I have quite honestly given up on trying to get this crap to work. I will paste what I've got so far though:

Comcast is handing out a /128 prefix (read: single IPv6 address) for the "public Internet IP" portion, and also handing off a /64 prefix to each customer. On my router, the /128 prefix portion gets assigned to vlan2, and the /64 portion gets to assigned to br0. vlan2 = Internet, br0 = LAN. Here's the relevant ifconfig bits:

br0        Link encap:Ethernet  HWaddr E0:CB:4E:C0:00:C4
           inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
           inet6 addr: 2601:9:4600:4f:e2cb:4eff:fec0:c4/64 Scope:Global
           inet6 addr: fe80::e2cb:4eff:fec0:c4/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:136169 errors:0 dropped:0 overruns:0 frame:0
           TX packets:140120 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:12825633 (12.2 MiB)  TX bytes:34201252 (32.6 MiB)
 
vlan2      Link encap:Ethernet  HWaddr E0:CB:4E:C0:00:C5
           inet addr:67.180.84.87  Bcast:67.180.87.255  Mask:255.255.252.0
           inet6 addr: 2001:558:6045:9b:114c:a6be:d64d:520c/128 Scope:Global
           inet6 addr: fe80::e2cb:4eff:fec0:c5/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:395630 errors:0 dropped:0 overruns:0 frame:0
           TX packets:130505 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:45129459 (43.0 MiB)  TX bytes:13136168 (12.5 MiB)
 

PING 2601:9:4600:4f:e2cb:4eff:fec0:c4(2601:9:4600:4f:e2cb:4eff:fec0:c4) 56 data bytes
From 2001:558:82:a4::2 icmp_seq=1 Time exceeded: Hop limit
 

10  pos-0-12-0-0-ar01.sfsutro.ca.sfba.comcast.net (2001:558:0:f697::2)  50.183 ms  50.184 ms  50.213 ms
11  te-9-2-ur03.santaclara.ca.sfba.comcast.net (2001:558:80:178::2)  46.067 ms  46.294 ms  46.421 ms
12  * * *
13  te-0-0-0-12-ur05.santaclara.ca.sfba.comcast.net (2001:558:82:87::1)  45.498 ms  45.569 ms  45.672 ms
14  * * *
15  te-5-5-ur03.santaclara.ca.sfba.comcast.net (2001:558:82:87::1)  45.777 ms  45.535 ms  45.909 ms
16  * * *
17  te-0-0-0-12-ur05.santaclara.ca.sfba.comcast.net (2001:558:82:87::1)  45.923 ms  46.036 ms  46.084 ms
 
...repeat indefinitely...
 

Wow, thanks for that exhaustive reply! I hope that people here smarter than I can pick up and run with it... and/or when you get a chance to sleep on it, some solutions may occur to you.

Tracing route to ipv6.l.google.com [2001:4860:800a::6a] over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  2002:473a:21c:1::1
  2    39 ms    19 ms    19 ms  2002:c058:6301::
  3    37 ms    51 ms    62 ms  ge-2-41-ur03.ndceast.pa.bo.comcast.net [2001:558:fe14:1::1]
  4    57 ms    29 ms    23 ms  te-1-3-ar01.ndceast.pa.bo.comcast.net [2001:558:e0:14::2]
  5    20 ms    34 ms    43 ms  te-1-3-ar01.philadelphia.pa.bo.comcast.net [2001:558:e0:3::1]
  6    48 ms    23 ms    79 ms  te-0-0-0-9-cr01.newyork.ny.ibone.comcast.net [2001:558:0:f745::1]
  7    23 ms    30 ms    23 ms  pos-1-5-0-0-pe01.111eighthave.ny.ibone.comcast.net [2001:558:0:f5d8::2]
  8    54 ms    39 ms    19 ms  2001:559::366
  9    20 ms    21 ms    21 ms  2001:4860::1:0:755
 10    31 ms    20 ms    20 ms  2001:4860::8:0:2fc6
 11    29 ms    26 ms    26 ms  2001:4860::1:0:9ff
 12    55 ms    39 ms    32 ms  2001:4860::8:0:3005
 13    55 ms    40 ms    39 ms  2001:4860::8:0:2f04
 14    36 ms    39 ms    40 ms  2001:4860::2:0:a7
 15    39 ms    48 ms    37 ms  2001:4860:0:1::10b
 16    47 ms    55 ms    57 ms  yx-in-x6a.1e100.net [2001:4860:800a::6a]
 

Thanks Ryan -- yeah, I had a feeling the 6to4 Anycast Relay would work. However, from everything I've read, the throughput is unbearably slow (I'm still trying to find the articles I read today on that problem; my browser history/cache is impossible to sift through). In fact, I can confirm that using the 6to4 Anycast Relay and pinging from the TomatoUSB router itself to ipv6.google.com intermittently sees packet loss, regardless of what address it gets back for that FQDN (it round-robins).

Given that Comcast is doing native IPv6 I would much rather use that, obviously. Otherwise I'm forced to set up 6to4 on all of my LAN machines (some are Windows, others are FreeBSD and Linux), so it's somewhat of an administrative nightmare.

Regardless of my situation, I hope the information you've provided is helpful to others who at least want to just get *something* IPv6-wise.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

reply to koitsu
For those following this thread (and I imagine there is many since I posted it everywhere relevant, haha ) I thought I'd provide a status update as to where we are, what's being discussed, ideas, etc... In no particular order:

1. Toastman and I have talked privately on the linksysinfo.org forum. During our discussion another user showed up and posted something that looked like a key piece of information (keep reading). Toastman at this point is willing to build firmwares for people to download/test, assuming there are folks who have good ideas as to what the bugs/issues are and what to change.

The sad part of our conversation is that IPv6 support in TomatoUSB in general is neglected big time. There was a really key person who was IPv6-savvy and part of the Tomato/TomatoUSB project who has since disappeared (hasn't responded to Emails, etc.), which is a bummer given his extensive knowledge of the protocol. (I looked the guy up myself and he even provided patches/code to the Linux iptables folks to fix problems). Toastman and I both wish we could find this guy.

2. An individual on the tomatousb.org and linksysinfo.org forums named armarkey mentioned that the "spurious default route" actually isn't so spurious after all, and that it might be related to metrics. Metrics define what route (or interface, depending on context) get preferred over another; think of it as a preferencing order. Two routes with an identical metric result in the route which got defined first taking precidence.

armarkey's theory is that because the IPv6 ICMP-based RA (route announcement) default metric is 1024, and that the DHCPv6 client's default route metric is 1024 (which is created the instant the interface is brought up), this causes a situation where the latter route was defined first, thus gets used when in fact the ICMP-based RA route should be being used.

I tried solving the problem by removing the metric 1024 route (same route you see in my first post) and inserting the same route with a metric of 2048, but to no avail.

So we're still working on that to figure out what the cause is, if the metric is actually responsible for the problem, yadda yadda.

3. armarkey also pointed out that the IA-NA workaround which was proposed by Comcast forum user morrildl may actually be incorrect in general. All this does is, in addition to Comcast handing out a /64 network to the DHCPv6 client in TomatoUSB, it also hands a /128 (single IPv6 address).

The theory is that the /128 isn't needed at all. That and the default route fix do make it so you can talk to the Internet via IPv6 directly inside TomatoUSB, but technically one should be able to do that anyway since br0 is assigned an address as part of the /64.

I am in agreement with this theory -- but of course, in practise, it doesn't work and we're still not sure why.

So for now, generally speaking, the IA-NA workaround is probably not needed.

I can absolutely confirm, hands down, no questions asked, that if you run TomatoUSB without the IA-NA workaround that you do in fact get handed a /64 network from Comcast. This means the DHCPv6 request in TomatoUSB is in fact working correctly, and that the problem may be elsewhere.

That leads me to the final item...

4. Most everyone I've been hearing from (privately and from reading public forums) has the exact same problem when using TomatoUSB's "DHCPv6 with Prefix Delegation" mode -- specifically it's that IPv6-capable machines on the LAN segment simply don't work.

I spent some time tonight poking at this (see my long post a few pages up) and didn't get anywhere. The IA_NA fix and default route fix don't resolve this problem. All I'm able to confirm at this point is that packets from a machine on the LAN segment do get sent to the TomatoUSB router correctly, and the TomatoUSB router does in fact send the packets out the WAN interface. However, no responses are ever seen. My guess is that the response packets never arrive because of the routing loop ordeal I mentioned a few posts up.

So we're still researching that issue as well.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

reply to koitsu
I can confirm that IPv6 via a 6to4 relay is, indeed, unbearably slow. I just ran the test at »ipv6-test.com/speedtest/ - my IPv4 speed is 16.4 Mbit/s, and my IPv6 speed is 148 Kbit/s... over two orders of magnitude slower. I haven't done any long-term testing, so I can't speak to the reliability of the connection.

You mentioned that you'd be forced to set up 6to4 on your LAN machines with a 6to4 router setup; that's not the case. From the perspective of LAN devices, they have a native IPv6 connection. The router has a LAN-facing IPv6 address (2002:473a:21c:1::1), and it assigns valid and functional IPv6 addresses to the LAN devices with the same 2002:473a:21c:1 prefix. Several devices on my network with varying operating systems, all set to retrieve addresses via DHCP, were successfully assigned their own IPv6 addresses without any configuration after setting up 6to4 on the router.

Obviously, native IPv6 would be better, and that's what I've been trying to do too.

With the combination of a working WAN interface using your script and a working LAN interface using the 6to4 relay, I feel like we're getting close to a comprehensive solution!

reply to koitsu
I wonder if this is related to why Native IPv6 isn't working on the Asus RT-N66U? I'm working with Netdog on it, and a person over at the Asus Forums, but so far it just refuses to work. The only IPv6 I can get on it is using Comcast's 6to4 Relay and 6in4 Tunnel to Hurricane Electric.

reply to rgolds