Free Allegiance Forums

Ouch.

Like I said, I'll forward this to the devteam, but I still suspect your ISP. Its a shame that we can't do tracerts using UDP so we can find out where in hell these packets are getting dropped :/

Glad to know that you found a work around though. /smile.gif" style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" />

that is one weird workaround. Thanks for sharing it with us. /smile.gif" style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" />

ImmortalZ wrote:QUOTE (ImmortalZ @ Jan 6 2007, 05:22 PM) Its a shame that we can't do tracerts using UDP so we can find out where in hell these packets are getting dropped :/

Ah ha, but I can do traceroutes using UDP. /mrgreen.gif" style="vertical-align:middle" emoid=":D" border="0" alt="mrgreen.gif" />
*nix is good for something, eh?

Using "traceproto" with a 12 byte payload shows everything is OK:
QUOTE # traceproto -p udp 92.c9.5d45.static.theplanet.com
traceproto: trace to 92.c9.5d45.static.theplanet.com (69.93.201.146), port 80
ttl 1: ICMP Time Exceeded from inside (172.16.2.254)
0.89200 ms 0.69000 ms 0.65000 ms
ttl 2: ICMP Time Exceeded from me (xxx.xxx.xxx.xxx)
14.939 ms 14.859 ms 14.627 ms
ttl 3: ICMP Time Exceeded from 10.11.42.33 (10.11.42.33)
15.562 ms 15.200 ms 15.080 ms
ttl 4: ICMP Time Exceeded from 10.11.1.65 (10.11.1.65)
15.881 ms 15.184 ms 15.527 ms
ttl 5: ICMP Time Exceeded from 10.11.1.57 (10.11.1.57)
16.208 ms 16.824 ms 16.646 ms
ttl 6: ICMP Time Exceeded from 10.4.2.9 (10.4.2.9)
22.252 ms 23.083 ms 22.025 ms
ttl 7: ICMP Time Exceeded from 10.0.61.25 (10.0.61.25)
27.421 ms 28.593 ms 27.385 ms
ttl 8:no response no response no response
ttl 9: ICMP Time Exceeded from 10.9.192.6 (10.9.192.6)
27.389 ms 26.760 ms 27.482 ms
ttl 10: ICMP Time Exceeded from 10.0.128.182 (10.0.128.182)
27.714 ms 27.515 ms 27.095 ms
ttl 11: ICMP Time Exceeded from 59.128.14.21 (59.128.14.21)
30.218 ms 37.265 ms 34.137 ms
ttl 12: ICMP Time Exceeded from otejbb204.kddnet.ad.jp (59.128.7.2)
27.508 ms
ICMP Time Exceeded from otejbb203.kddnet.ad.jp (59.128.7.65)
26.773 ms
ICMP Time Exceeded from otejbb203.kddnet.ad.jp (59.128.7.1)
27.500 ms
ttl 13: ICMP Time Exceeded from 59.128.4.174 (59.128.4.174)
27.444 ms 27.735 ms
ICMP Time Exceeded from 59.128.4.162 (59.128.4.162)
27.316 ms
ttl 14: ICMP Time Exceeded from lacbb001.kddnet.ad.jp (203.181.100.158)
141.26 ms 140.96 ms 140.93 ms
ttl 15: ICMP Time Exceeded from tr-la6.kddnet.ad.jp (59.128.2.66)
150.06 ms 159.65 ms
ICMP Time Exceeded from tr-la6.kddnet.ad.jp (59.128.2.86)
142.74 ms
ttl 16: ICMP Time Exceeded from so2-1-0-622m.ar3.LAX1.gblx.net (208.50.13.17)
144.40 ms 159.13 ms 154.96 ms
ttl 17:no response no response no response
ttl 18: ICMP Time Exceeded from The-Planet.GigabitEthernet7-3.ar2.DAL2.gblx.net (64.208.170.198)
180.93 ms 187.44 ms 189.06 ms
ttl 19: ICMP Time Exceeded from te9-2.dsr01.dllstx3.theplanet.com (70.87.253.14)
179.66 ms
ICMP Time Exceeded from te9-2.dsr02.dllstx3.theplanet.com (70.87.253.30)
181.34 ms
ICMP Time Exceeded from te7-2.dsr02.dllstx3.theplanet.com (70.87.253.26)
191.20 ms
ttl 20: ICMP Time Exceeded from vl21.dsr01.dllstx2.theplanet.com (70.85.127.67)
193.96 ms 178.54 ms
ICMP Time Exceeded from vl22.dsr02.dllstx2.theplanet.com (70.85.127.76)
180.98 ms
ttl 21: ICMP Time Exceeded from po2.car03.dllstx6.theplanet.com (12.96.160.37)
193.08 ms 190.64 ms 183.59 ms
ttl 22: Port Unreachable from 92.c9.5d45.static.theplanet.com (69.93.201.146)
183.18 ms 181.46 ms 179.55 ms

hop : min / ave / max : # packets : # lost
-------------------------------------------------------
1 : 0.65000 / 0.74400 / 0.89200 : 3 packets : 0 lost
2 : 14.627 / 14.808 / 14.939 : 3 packets : 0 lost
3 : 15.080 / 15.281 / 15.562 : 3 packets : 0 lost
4 : 15.184 / 15.531 / 15.881 : 3 packets : 0 lost
5 : 16.208 / 16.559 / 16.824 : 3 packets : 0 lost
6 : 22.025 / 22.453 / 23.083 : 3 packets : 0 lost
7 : 27.385 / 27.800 / 28.593 : 3 packets : 0 lost
8 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
9 : 26.760 / 27.210 / 27.482 : 3 packets : 0 lost
10 : 27.095 / 27.441 / 27.714 : 3 packets : 0 lost
11 : 30.218 / 33.873 / 37.265 : 3 packets : 0 lost
12 : 26.773 / 27.260 / 27.508 : 3 packets : 0 lost
13 : 27.316 / 27.498 / 27.735 : 3 packets : 0 lost
14 : 140.93 / 141.05 / 141.26 : 3 packets : 0 lost
15 : 142.74 / 150.82 / 159.65 : 3 packets : 0 lost
16 : 144.40 / 152.83 / 159.13 : 3 packets : 0 lost
17 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
18 : 180.93 / 185.81 / 189.06 : 3 packets : 0 lost
19 : 179.66 / 184.07 / 191.20 : 3 packets : 0 lost
20 : 178.54 / 184.49 / 193.96 : 3 packets : 0 lost
21 : 183.59 / 189.11 / 193.08 : 3 packets : 0 lost
22 : 179.55 / 181.40 / 183.18 : 3 packets : 0 lost
------------------------Total--------------------------
total 178.54 / 77.089 / 193.96 : 60 packets : 6 lost[/quote]******************************

BUT: doing the same with a 500 byte payload shows where the droppage is happening:


******************************

QUOTE # traceproto -p udp -P 500 92.c9.5d45.static.theplanet.com
traceproto: trace to 92.c9.5d45.static.theplanet.com (69.93.201.146), port 80
ttl 1: ICMP Time Exceeded from inside (172.16.2.254)
1.0700 ms 1.0230 ms 1.0310 ms
ttl 2: ICMP Time Exceeded from me (xxx.xxx.xxx.xxx)
18.806 ms 18.943 ms 18.777 ms
ttl 3: ICMP Time Exceeded from 10.11.42.33 (10.11.42.33)
20.009 ms 19.632 ms 19.405 ms
ttl 4: ICMP Time Exceeded from 10.11.1.65 (10.11.1.65)
20.160 ms 22.880 ms 20.344 ms
ttl 5: ICMP Time Exceeded from 10.11.1.57 (10.11.1.57)
21.054 ms 20.762 ms 21.043 ms
ttl 6: ICMP Time Exceeded from 10.4.2.9 (10.4.2.9)
26.371 ms 26.565 ms 26.616 ms
ttl 7: ICMP Time Exceeded from 10.0.61.25 (10.0.61.25)
31.622 msno response 34.011 ms
ttl 8: ICMP Time Exceeded from 10.0.192.141 (10.0.192.141)
32.177 msno response 32.231 ms
ttl 9: ICMP Time Exceeded from 10.9.192.6 (10.9.192.6)
31.059 ms 31.596 ms 31.546 ms
ttl 10: ICMP Time Exceeded from 10.0.128.182 (10.0.128.182)
31.307 ms 31.008 ms 31.476 ms
ttl 11: ICMP Time Exceeded from 59.128.14.21 (59.128.14.21)
31.795 ms 42.731 ms 35.748 ms
ttl 12: ICMP Time Exceeded from otejbb203.kddnet.ad.jp (59.128.7.65)
31.954 ms
ICMP Time Exceeded from otejbb203.kddnet.ad.jp (59.128.7.1)
30.998 ms
ICMP Time Exceeded from otejbb204.kddnet.ad.jp (59.128.7.66)
31.435 ms
ttl 13: ICMP Time Exceeded from 59.128.4.174 (59.128.4.174)
31.800 ms 31.610 ms 31.317 ms
ttl 14: ICMP Time Exceeded from lacbb001.kddnet.ad.jp (203.181.100.158)
145.28 ms 145.15 ms
ICMP Time Exceeded from lacbb002.kddnet.ad.jp (203.181.100.174)
164.14 ms
ttl 15: ICMP Time Exceeded from tr-la6.kddnet.ad.jp (59.128.2.66)
154.15 ms
ICMP Time Exceeded from tr-la6.kddnet.ad.jp (59.128.2.86)
147.28 ms 147.67 ms
ttl 16:no response no response no response
ttl 17:no response no response no response
ttl 18:no response no response no response
ttl 19:no response no response no response
ttl 20:no response no response no response
etc, etc[/quote]

So, proof that it's happening on the border between Japan and USA, where it hits gblx.net. That is only two hops from where it hits Alleg's ISP though. Way upstream of my ISP too, we're talking Juniper routers that fan out to OC-3 links and the like. "Global Crossing" don't like the Alleg lobby server. /blink.gif" style="vertical-align:middle" emoid=":o" border="0" alt="blink.gif" />

Grimmwolf_GB wrote:QUOTE (Grimmwolf_GB @ Jan 6 2007, 10:05 PM) that is one weird workaround. Thanks for sharing it with us. /smile.gif" style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" />

No problem. Anything I can do to help, and advocate using Free Software (I couldn't do my hack without OpenWRT and its SSH command line). /smile.gif" style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" />

Looks like its hitting the router that is on the receiving end of the trans-ocean fiber link. Damn. I guess all we can hope for is some network administrator at that end is sensible and finds out soon enough that the router is dropping UDP packets like crazy.

Still, I'm happy that you found a way around it.

Funny thing is that the Alleg lobby server and the Planet servers are hosted by the same ISP. I'm at a loss on how you can't get to the lobby server, yet play on the normal servers. Did you see what packet sizes are involved once the game connects to a game server?

ImmortalZ wrote:QUOTE (ImmortalZ @ Jan 6 2007, 11:43 PM) Looks like its hitting the router that is on the receiving end of the trans-ocean fiber link. Damn. I guess all we can hope for is some network administrator at that end is sensible and finds out soon enough that the router is dropping UDP packets like crazy.

Still, I'm happy that you found a way around it.

Funny thing is that the Alleg lobby server and the Planet servers are hosted by the same ISP. I'm at a loss on how you can't get to the lobby server, yet play on the normal servers. Did you see what packet sizes are involved once the game connects to a game server?

Indeed, I think I'll have a moan to them, somewhere on http://www.gblx.net/company/company_contact_us.aspx

The whole lobby (300 MTU max) vs game servers (1500 MTU) thing is truly odd. I left a tcpdump going on a laptop while I played this afternoon, and there were lots of 1+KB UDP packets going through during a game, no problem at all. For example, I'd play on DN (at 1500 MTU) and then try dropping back to the lobby, which promptly gave me the "failed connection" error. It's completely reproducible, so I'll continue to gather evidence and try using traceproto on the game servers.

EDIT: Just to make it more interesting, do you have some IPs of non-Planet game servers?

Planet : 70.64.59.71
PookCore : 69.93.201.150
ThalServ : 12.96.40.27
FTX : 68.146.214.11

Thanks for the IPs, ImmZ. Most of the servers are OK for large UDP packets because my ISP has various upstream transits for the different destinations.

My traceproto results:
Planet: fine (KDDNet, then Shaw Cable)
PookCore: dead to me (KDDNet, then glbx.net)
Thalserv: fine (NTT, then AT&T)
FTX: fine (KDDNet, then Shaw Cable)

The oddest thing is that the DN server (47.3b.5446.static.theplanet.com) also gives me the failed result (500 byte packets) via traceproto, but here is a typical tcpdump snippet during gameplay:
QUOTE 08:56:35.669141 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 148
08:56:35.671792 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 494
08:56:35.681913 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 12
08:56:39.992190 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 67
08:56:40.003258 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 12
08:56:41.239661 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 24
08:56:41.425538 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 12
08:56:41.500625 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 24
08:56:41.512054 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 12
08:56:41.802508 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 51
08:56:41.814782 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 12
08:56:42.305607 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 16
08:56:42.317737 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 12
08:56:42.407065 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 776
08:56:42.420295 IP me.2303 > 47.3b.5446.static.theplanet.com.2302: UDP, length: 12
08:56:45.726549 IP 47.3b.5446.static.theplanet.com.2302 > me.2303: UDP, length: 494[/quote]

Large UDP packets are not blocked USA -> Japan, only Japan -> USA. What a glorious cock-up there must be in that router's configuration. As long as my client only sends small packets (i.e. during a game), it's OK. The lobby server requires that my client send a few large packets, hence I have to set a small MTU to split the packets into fragments.

I've sent my evidence to the Global Crossing NOC. Probably they'll ignore it since I/we aren't a direct customer. Oh well. /unsure.gif" style="vertical-align:middle" emoid=":unsure:" border="0" alt="unsure.gif" />

The only thing I can suggest is someone bitching at theplanet, to make them bitch to Global Crossing. My ISP isn't even a customer either, but theplanet probably is.

Very interesting indeed. aarmstrong runs the planet servers. But I do not know in what position he is in with regards to bitching at them /tongue.gif" style="vertical-align:middle" emoid=":P" border="0" alt="tongue.gif" />. I'll let him know that his servers are off limits to Japanese players (and God knows who else routed through Glbx). We'll see /smile.gif" style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" />

I got a reply from glbx:
QUOTE You may contact our Premier Support group at 800-404-7714 / Europe Toll-free +0 0080045004511 to open a trouble on this issue.

Thank you,

Global Crossing[/quote]

As expected; no support contract, no love.