MrMoke
join:2002-06-06
Austin, TX
|  Bug in BEFSX41 fw 1.45.7
As I've mentioned previously, I can't get one of my XP machines to work with any firmware from 1.45.3 thru 1.45.7 when trying to use Internet Explorer 6/SP1. I turned firewall logging to see what might be happening, and need some expert advice on what the logs are saying, if anything.
Accessing MSN Home Page:
Using fw 1.44.11t
#Verson: 1.0
#Software: Microsoft Internet Connection Firewall
#Time Format: Local
#Fields: date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info
2004-02-21 09:41:15 OPEN UDP 192.168.250.100 192.168.250.101 62514 62516 - - - - - - - -
2004-02-21 09:41:20 OPEN UDP 192.168.250.100 192.168.250.1 1056 53 - - - - - - - -
2004-02-21 09:41:20 OPEN TCP 192.168.250.100 207.68.172.234 3100 80 - - - - - - - -
2004-02-21 09:41:20 OPEN UDP 192.168.250.100 192.168.250.1 3062 53 - - - - - - - -
2004-02-21 09:41:20 OPEN UDP 192.168.250.100 192.168.250.1 3063 53 - - - - - - - -
2004-02-21 09:41:20 OPEN TCP 192.168.250.100 63.211.236.247 3101 80 - - - - - - - -
2004-02-21 09:41:20 OPEN TCP 192.168.250.100 216.239.51.104 3102 80 - - - - - - - -
2004-02-21 09:41:20 OPEN TCP 192.168.250.100 63.211.236.247 3103 80 - - - - - - - -
2004-02-21 09:41:21 OPEN TCP 192.168.250.100 65.54.140.158 3104 80 - - - - - - - -
2004-02-21 09:41:21 OPEN TCP 192.168.250.100 65.54.225.156 3105 80 - - - - - - - -
2004-02-21 09:41:22 OPEN TCP 192.168.250.100 207.68.178.237 3106 80 - - - - - - - -
2004-02-21 09:41:22 OPEN TCP 192.168.250.100 207.68.178.237 3107 80 - - - - - - - -
2004-02-21 09:41:22 OPEN TCP 192.168.250.100 216.74.132.11 3108 80 - - - - - - - -
2004-02-21 09:41:22 OPEN TCP 192.168.250.100 198.5.148.2 3109 80 - - - - - - - -
Using fw 1.45.7
#Verson: 1.0
#Software: Microsoft Internet Connection Firewall
#Time Format: Local
#Fields: date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info
2004-02-21 10:05:36 OPEN UDP 192.168.250.100 192.168.250.101 62514 62516 - - - - - - - -
2004-02-21 10:05:38 OPEN UDP 192.168.250.100 192.168.250.1 1056 53 - - - - - - - -
2004-02-21 10:05:38 OPEN TCP 192.168.250.100 207.68.172.234 3188 80 - - - - - - - -
2004-02-21 10:05:49 OPEN TCP 192.168.250.100 209.246.136.72 3189 80 - - - - - - - -
2004-02-21 10:05:49 OPEN TCP 192.168.250.100 209.246.136.72 3190 80 - - - - - - - -
2004-02-21 10:05:59 DROP TCP 207.46.107.99 192.168.250.100 1863 3179 68 AP 4259121558 3132832248 17520 - - -
2004-02-21 10:06:00 DROP TCP 207.46.107.99 192.168.250.100 1863 3179 68 AP 4259121558 3132832248 17520 - - -
2004-02-21 10:06:03 DROP TCP 207.46.107.99 192.168.250.100 1863 3179 68 AP 4259121558 3132832248 17520 - - -
2004-02-21 10:06:08 DROP TCP 207.46.107.99 192.168.250.100 1863 3179 68 AP 4259121558 3132832248 17520 - - -
2004-02-21 10:06:08 DROP UDP 192.168.250.100 192.168.250.255 138 138 202 - - - - - - -
2004-02-21 10:06:08 DROP UDP 192.168.250.100 192.168.250.255 137 137 78 - - - - - - -
2004-02-21 10:06:09 DROP UDP 192.168.250.100 192.168.250.255 137 137 78 - - - - - - -
2004-02-21 10:06:10 DROP UDP 192.168.250.100 192.168.250.255 137 137 78 - - - - - - - |
|
sgkent
join:2002-04-15
Sacramento, CA
| Hi - I have several BEFSX41 and have had no trouble.
(1) When you say that you can't get 1.45.3 or later to work, is this for any site or is it MSN only?
(2) Do utilities such as Ping and TracerT fail as well or is it just IE 6 using port 80? |
|
MrMoke
join:2002-06-06
Austin, TX | No web pages load from anywhere.
Ping and tracert to outside destinations do work.
The dropped packet messages really bug me. The browser just sits there, and never times out. Switch back to 1.44.11t, and it goes away. |
|
Flogator
Premium,MVM
join:2003-01-19
Cantley, QC
 ·Acanac
 ·Videotron
| reply to MrMoke
I am assuming these logs are coming from the Windows XP firewall. Basically, these logs imply that the XP firewall has dropped one packet as well as all of its retransmission. Still unclear why the XP firewall made the decision of dropping them though??? Perhaps packet sniffing will indicate what is different in the packet to cause that drop.
You know, you might have very well found a bug in BEFSX41 firmware 1.45.X but it is still premature to conclude this before we get packet sniffing logs. You may use Ethereal to do packet sniffing. If you have trouble reading the logs, I can help you if you forward them to me. |
|
MrMoke
join:2002-06-06
Austin, TX | OK- I'll try that. Would it beter to run it from my redhat box or the XP box that's giving me the problem? I only ask because you know I don't want to read the manual:) |
|
Brano
I hate Vogons
Premium,MVM
join:2002-06-25
Burlington, ON | You have to run ethereal from your XP box to see all packets on the wire.
If you want to do it from your other machines you have to connect all machines which you want to sniff traffic for into a HUB (not SWITCH). |
|
MrMoke
join:2002-06-06
Austin, TX |  OK- Thanks, will download now. |
|
sgkent
join:2002-04-15
Sacramento, CA
| reply to MrMoke
To me I think you have a DNS error. It sounds like that in your verbal description. Also when you look at the logs, in the first request there are 3 DNS packets I believe (port 53)
In the failed one there is one port 53 connection and then a dropped connection.
Use IPCONFIG /all to compare DNS settings between computers that worka nd don't. I can't read packets worth a darn but I do know that port 53 is DNS and that your description also matches DNS errors.
Steve |
|
MrMoke
join:2002-06-06
Austin, TX
1 edit |  Update
Flogator has been gracious enough to help me with the strange problem I was encountering with 1.45.x firmware killing the browser on one, but not all, of my XP boxes. So far, we have discovered that something has definately changed with regard to how the SX41 handles MTU in the new versions, but we don't know whether it was something that got broken, or something that got fixed. The temporary fix for me was to lower the SX41 MTU to the value being used by the XP box that didn't work.
For the record, the XP box should have been using 1500, but I have Safenet VPN client and ssh secure client and Flogator figured that one or both were dropping MTU to 1446. Ethereal could see the shortened packets, and we assume that they were transmitted unaltered, but the responses from web sites were coming back at 1500. I can only suspect that the SX41 may have replaced the outgoing MTU request with it's own setting, but need further testing. Flogator is currently researching MTU errata for Windows XP.
More Later |
|
Flogator
Premium,MVM
join:2003-01-19
Cantley, QC
·Acanac
·Videotron
| reply to MrMoke
Re: Bug in BEFSX41 fw 1.45.7
Ok ladies and gentlemen, we have found the mystery. At first, let just say that indeed, something got changed between firmware 1.44.11t and 1.45.7. And that little something was enough to cause trouble to many people and will likely continue unless BEFSX41 users are aware of this. The changes is related to updating MSS (Maximum Segment Size) options.
I actually had to reproduce MrMoke issue at my main site and to sniff on the WAN side of my BEFSX41 to figure it all out. Before I spoil the result of the experiment, here is a bit of theory (hope I do not put too much there).
The MTU (Maximum Transfer Unit) parameter describes the maximum amount of information (aka packet) that can be exchange between two networks before another packet needs to be generated. The MTU is always local to the physical layer meaning that if you are going through a router, for example, it if possible that the MTU on the other side of the router be different. That's precisely where problems start.
In order to support the ability to travel on network with different MTU, the IP protocol came up with the concept of fragmentation and reassembly. Every network nodes on the internet (including your PC) needs to know how to reassemble IP packets. When a router forward IP packets from one network to the other, if the MTU is smaller, it will fragment the packets in smaller chunks. This operation is however very complicated and demanding on routers. Not to mentioned that cheap routers such as the BEFSX41 does not even support fragmentation  .
In order to minimize the need for fragmentation (thus attempting to improve overall performance), the creation of a mechanism was created: "Path-MTU Discovery". Unfortunately, this mechanism is still reactive. The way it works is you still need to send one big packet using a flag that says "do not fragment". If a router can't forward the packet because of a MTU violation, it will drop the packet and reply with an ICMP message to tell the originator, please lower down the IP packet size. The the originator uses this indication to send smaller packet to that destination thus eliminating the need for fragmentation in the middle. Still a reactive mechanism but way better than fragmenting all the time. Nice mechanism hein? Well, the BEFSX41 does not implement it .
Now the pro-active way. When you do optimization, you better be pro-active rather than reactive. Well, the TCP protocol came up with a nice mechanism called MSS exchange. When establishing a TCP connection, both ends may optionally specify the MSS it supports to help the far-end send packets of the right size. This works great because only the end point needs to worry about this. Most of all, all modern TCP implementation will always specify the optional MSS in their first TCP packet (aka SYN packet). This is precisely the reason why lowering down the MaxMTU parameter on your Windows-based system will solve MTU problems because the TCP stack will derive their MSS from the MTU (cleaver isn't).
Up to now, the BEFSX41 did not care about MSS exchange. If you had MTU problem, you had to change your PC's MaxMTU value to work around them. Well, the good news is that starting with firmware 1.45.7, the BEFSX41 finally do something useful with its MTU settings. What it does is it snoop in the SYN packet and replaces the MSS value by its own derived by the MTU you configured in the router. That's why changing the MTU in MrMoke's BEFSX41 solved his problem. The bad news is that Linksys should have used the minimum value between current MSS in SYN packet and what it would have normally put there.
The reason his setup worked using 1.44.11t, is that the BEFSX41 did not modified the MSS option traveling in SYN packets thus the servers at the far end saw the MSS associated to an MTU of 1446. The reason his setup did not work using 1.45.7, is that the BEFSX41 did modified the MSS option traveling in SYN packets from its original value up to the MSS value associated to an MTU of 1500. The far-end started to send packets back to his XP box but too big for its NIC configuration.
Bottom line, no matter if you're on cable or DSL, as soon as you are using application that reduces the MTU value (i.e. PPPoE, VPN client), it might be wise to figure out what MTU value it is and apply that value to your router's configuration. In the case of MrMoke, a simple packet sniffing demonstrated that his XP box was using an MTU of 1446. As soon as he applied that same value to its BEFSX41, most of his problem went away. The point to make here is smaller is the MTU, more chances you will see your packets to reach the other end without fragmentation. However, your overall throughput might go down as you need to send more packets to represent the same amount of information. That's why you want to set it to its maximum possible value for your application.
Please accept my apologies for this humongous message but I think in the end, it's worth reading. Also we must all thanks MrMoke for his perseverance in getting to the bottom of this. |
|
Jan Janowski
join:2000-06-18
Skokie, IL | reply to MrMoke
That accounts why I did not see anything like that here...
'Cause I'm on DSL, and Always set it to manual and 1492
Beautiful explanation, Flogator!!! Printed it out as a FAQ!!!
--
Looking for 1939 Indian Motocycle |
|
peaches28
join:2004-01-18
| reply to Flogator
That was a good bit of information flogator! Good work. Now for a little monkey wrench. I do not dispute your findings, however, I have a similar setup to mister moke, the parameters are:
I use Cisco VPN 4.03 tcp transparent IPSEC tunneling.
MTU 1300 set by the VPN software on the XP Pro machine.
Conequently all packets out of this machine are at 1300.
BEFSX41 1.45.7 FW
MTU setting is on default of auto 1500.'
I don't have the problem that he is having. And I wonder why? Not that I am complaining but the only real difference is that my mtu is set to 1300 and not 1446? Why does that matter ?
James |
|
Flogator
Premium,MVM
join:2003-01-19
Cantley, QC
·Acanac
·Videotron
1 edit | That's a good question. Don't know yet. In the case of MrMoke, his XP box had the MTU set to 1446 whether the VPN client has enabled the VPN tunnel or not. Not all client behave the same way. It could very well be that if your VPN tunnel is on, the MTU is 1300 but if your VPN tunnel is off that your MTU is back to 1500 (or whatever value it was before). Some other VPN client are so nasty that they overwrite the default route 0.0.0.0 to the VPN tunnel such that all traffic is sent in the VPN tunnel. Could that be your situation?
Basically, do the following commands with and without the VPN enabled and provide your results.
# test the MTU of 1500 and see if it works
ping -f -l 1472 www.dslreports.com
# test the MTU of 1300 and see if it works
ping -f -l 1272 www.dslreports.com
# test the route taken to reach the far end
tracert www.dslreports.com
Note that I personally am using a software VPN client (from 3Com to connect to the office's firewall). That software VPN client is not changing the MTU of the interface thus when I enable the tunnel, only the private IP addresses specified in the tunnel are subject to a lower MTU. The rest of the traffic is based on my 1500 MTU thus I do not experience the problem that MrMoke was having. |
|
peaches28
join:2004-01-18
| No the client sets it to the 1300 value all of the time. It is not a dynamic value. I have sniffed also to verify that this is so. My connectivity works with or without it on. It is intersting that this behavior is so different from what you have found. |
|
LawrenceD
@cox.net
| reply to MrMoke
FYI
I also am using the Safenet SoftRemote 8.0.0 (Build10)SonicWall VPN client to hook up to a SonicWall FW. I have no problems with the firmware v1.45.7 and my MTU setting at auto.
Since upgrading to v1.45.7 I am now having problems with frequent (once every other day or so) cold restarts of he BEFSX41... other than that, all is okay.
these BEFSX41s sure have been more trouble than they were worth for the last 14 months, that's for sure....... oh well |
|
MrMoke
join:2002-06-06
Austin, TX
| reply to peaches28
Conspiracy Theory
Let's start the rumor that The latest modifications to the SX41 were designed to properly support only Cisco VPN products.:D Nah, maybe not.
Now that Flogator has helped me past this stumbling block, I think I'll see what else I can tinker with in this 1.45.7 firmware.
Again, many thanks to my new Packet Protocol Professional for his invaluable help. |
|
Flogator
Premium,MVM
join:2003-01-19
Cantley, QC
·Acanac
·Videotron
| reply to MrMoke
Re: Bug in BEFSX41 fw 1.45.7
This message is for peaches28 and LawrenceD. One thing for sure, if your setup is working as you described it, that is wonderful (no problem is good news)  . Let's keep it up that way.
There is obviously a behavior change between firmware 1.44.11t and 1.45.7 about the MTU settings. I have seen the difference while doing packet sniffing on the WAN side. Seems like not everyone will be affected by this problem. At least you know that if you ever are getting strange behavior related to MTU, that this thread will be helpful . |
|
sgkent
join:2002-04-15
Sacramento, CA
| reply to MrMoke
this is only my two cents worth to add.
I have two VPNS right now using the BEFSX41 with 1.45.7
Using the ping -l (that is small L) (packet size) -f (do not fragment) I get different results and they can be weird.
Between two W2000P computers running through a BEFSX41 tunnel CA to FL if the MTU is set above about 1400 the packet gets fragged regardless of what the BEFSX41 are set at. (typically 1500 at each end)
Between a W2000P computer using native MS VPN behind a BEFSX41 NAT to a X-SENSE device at the other end similar to a BEFSR41, the MTU can go up to about 1350 before it frags. If I lower it on the W2000P computer to say 1300 to get it through to work, then the CA to FL VPN adjusts down and it will frag out then about 1250. If I lower my computer here to 1200 then both tunnels work without fragmenting. The software between work and here fails if the packet gets fragmented.
If the BEFSX41 was really establishing the MTU rather than my computer why would changing my computer be affecting both tunnels. Could our setting ICMP off to block WAN pings be affecting the routers ability to adjust MTU's? |
|
Batavia
join:2003-01-03
| said by sgkent  :
If the BEFSX41 was really establishing the MTU rather than my computer why would changing my computer be affecting both tunnels. Could our setting ICMP off to block WAN pings be affecting the routers ability to adjust MTU's?
The MTU setting available in the router only applies to TCP sessions, meaning that during TCP session establishment the proper (=maximum possible) segmentation is negotiated.
Some of the VPN clients out there are working on UDP. Here the router can't adjust the MTU and the system itself must have the proper settings.
Concerning ICMP blocking. It actually interferes with notifications to your PC. Normally ICMP is used to signal a too large packet somewhere on the way to the destination. The ICMP notification will tell the sender the correct MTU size to be used, if such notification never reaches the sender because it is blocked, then it stays unaware.
--
A market is never saturated with a good product, but it is very quickly saturated with a bad one. --Henry Ford |
|
Flogator
Premium,MVM
join:2003-01-19
Cantley, QC
·Acanac
·Videotron
| reply to sgkent
Batavia is correct. The MTU settings in the router is largely used to adjust the MSS (Maximum Segment Size) parameter that is being exchanged during TCP connection setup.
Furthermore, every software VPN client seem to have a different behavior when being attached to a particular IP interface. I find it normal to see different results when testing different VPN client.
Additionally, keep in mind that when you send data through a VPN tunnel, that data is not subject to the router adjustment. The reason is this data is encapsulated in a ESP packet, not TCP. Therefore, the BEFSX41 has no idea what so ever about what is being carried within the ESP packet. The behavior of the traffic going through the VPN tunnel is solely relative to the MTU of the ip interface where the traffic originate as well as the overhead used by the VPN tunnel implementation. Do not compare the behavior between traffic going through the VPN tunnel and traffic flowing outside the VPN tunnel as those do not rely on the same rules (one is using ESP and the other one may be TCP for example).
The results you are exposing to us seems to refer to traffic going through a VPN tunnel. As a result, I am not surprised at all of the results. In fact, it makes totally sense to me. You may repeat the experiment using traffic running outside the VPN tunnel (using www.yahoo.com for example) and you likely see different results as this traffic will now be subject to MSS adjustment by the BEFSX41. |
|
