Latency is a Killer
If you’re hosting your website overseas you might want to rethink. I’ve been doing some research on the true impact of hosting Australian websites in the U.S. and it’s worse than you might think.
I plotted the time it took to retrieve 200 files from 1-200KB from two locations. The first was a server in the same city as me (kindly provided by SCT.com.au) and the second is with Slicehost in the US. We’re comparing the effects of 15ms RTT vs. 250ms RTT on performance.
You might imagine the US server would simply add 250ms to the total download time. The reality is that it can take ten times longer to get files from the U.S. The TCP protocol includes provisions to avoid network congestion that were designed before we had broadband. I discussed these in my presentation ‘Speed Matters’ (slides here).
The stepping you see is due to TCP Congestion Control (RFC 2581) and Delayed ACK (RFC 813).
My investigation was inspired by a talk by John Rauser at VelocityConf last month.
Wow, nice write up Mike.
This will definitely come into play next time I’m choosing a host for an Aussie based application.
Thanks.
Comment by Scott Harvey — July 9, 2010 @ 7:41 am
It’s interesting to see the banding on RTT boundaries. A really nice visual representation of the effects of slow start, thanks!
Comment by James Healy — July 9, 2010 @ 8:00 am
Awesome graph. Thanks for compiling this.
Comment by Dr Nic — July 9, 2010 @ 8:06 am
Australia is difficult to service from traditional cloud providers hosted in the US. Even hosting at Rackspace out of Hong Kong accelerated by Akamai doesn’t cut it. If the majority of your customers are in Australia you almost have to host here.
Comment by Michael Baker (@cloudjunky) — July 9, 2010 @ 8:19 am
Thanks Dr Nic! I was inspired by John Rauser’s talk at #velocityconf, “TCP and the Lower Bound of Web Performance”. It was an awesome history of TCP.
http://en.oreilly.com/velocity2010/public/schedule/detail/11792
His slides are up but I think I have the only video of his talk. If enough people are interested I’ll see if I can get permission to post it.
Comment by mbailey — July 9, 2010 @ 11:47 am
These findings are even more potent when considered from the mobile angle- mobile telephones already have shockingly bad latency issues. The use of “the cloud”; services such as Amazon’s S3 (I’m not affiliated) should improve the situation by having servers ‘local’ to the end user, but in the meantime, until these services are more widespread, one can improve the situation by reducing the number of HTTP requests made to a server.
Comment by Gavin Laking — July 9, 2010 @ 1:13 pm
Great article, thanks for this. We’re based in Europe and there’s been a lot of talk about moving our hosting to the cloud in the US. We tend to have a 200-250ms latency to there as well. This will go a long way in convincing them to stay local
About the talk, can’t speak for anyone else but I’d love to see it.
Comment by lars — July 9, 2010 @ 1:47 pm
I’ve only just realized my ancient wordpress installation has been treating ever comment as spam. Time to upgrade!
Comment by mbailey — July 9, 2010 @ 2:56 pm
Do you find the effects as bad if you’re using a CDN like Amazon cloudfront out of Singapore?
I’d be very interested in results since we’re virtually convinced for cost and scalability reasons (as well as not great support responsiveness) we need to look at overseas hosting for the next version of our platform.
Comment by Daryl — July 10, 2010 @ 2:39 am
I was pretty excited when I heard Amazon were opening a data center in Singapore but the RTT from Melbourne isn’t much better than California.
Right now I’m getting a lower RTT to us-west-1 than ap-southeast-1
mbailey@island:~$ ping -c 3 ec2-184-72-17-79.us-west-1.compute.amazonaws.com
PING ec2-184-72-17-79.us-west-1.compute.amazonaws.com (184.72.17.79) 56(84) bytes of data.
64 bytes from ec2-184-72-17-79.us-west-1.compute.amazonaws.com (184.72.17.79): icmp_seq=1 ttl=53 time=188 ms
64 bytes from ec2-184-72-17-79.us-west-1.compute.amazonaws.com (184.72.17.79): icmp_seq=2 ttl=53 time=188 ms
64 bytes from ec2-184-72-17-79.us-west-1.compute.amazonaws.com (184.72.17.79): icmp_seq=3 ttl=53 time=188 ms
— ec2-184-72-17-79.us-west-1.compute.amazonaws.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 188.045/188.151/188.222/0.076 ms
mbailey@island:~ $ ping -c 3 ec2-175-41-171-87.ap-southeast-1.compute.amazonaws.com
PING ec2-175-41-171-87.ap-southeast-1.compute.amazonaws.com (175.41.171.87) 56(84) bytes of data.
64 bytes from ec2-175-41-171-87.ap-southeast-1.compute.amazonaws.com (175.41.171.87): icmp_seq=1 ttl=47 time=234 ms
64 bytes from ec2-175-41-171-87.ap-southeast-1.compute.amazonaws.com (175.41.171.87): icmp_seq=2 ttl=47 time=233 ms
64 bytes from ec2-175-41-171-87.ap-southeast-1.compute.amazonaws.com (175.41.171.87): icmp_seq=3 ttl=47 time=234 ms
— ec2-175-41-171-87.ap-southeast-1.compute.amazonaws.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 233.411/234.172/234.571/0.776 ms
# Compare this to my server in a Melbourne Data Center (15ms)
mbailey@island:~$ ping -c 3 mel.failmode.com
PING w-001.blocksglobal.com (116.240.200.168) 56(84) bytes of data.
64 bytes from 168.200.idc.iprimus.net.au (116.240.200.168): icmp_seq=1 ttl=57 time=14.4 ms
64 bytes from 168.200.idc.iprimus.net.au (116.240.200.168): icmp_seq=2 ttl=57 time=15.8 ms
64 bytes from 168.200.idc.iprimus.net.au (116.240.200.168): icmp_seq=3 ttl=57 time=15.4 ms
— w-001.blocksglobal.com ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.494/15.241/15.813/0.552 ms
Comment by mbailey — July 10, 2010 @ 8:50 am
Good to see this. You might want to try Linode, particularly their Fremont, CA datacentre, although I found Dallas was only slightly slower and not on a fault line. Linode generally performs faster than SliceHost and more consistently over time (can’t find the url for analysis of this) although I have noticed Linode slow down during US waking hours.
Comment by Gary — July 11, 2010 @ 2:02 pm
We use Edgecast as a CDN & it’s got some decent latency in Australia:
Pinging gs1.wac.edgecastcdn.net [68.232.44.19] with 32 bytes of data:
Reply from 68.232.44.19: bytes=32 time=22ms TTL=54
Reply from 68.232.44.19: bytes=32 time=22ms TTL=54
Reply from 68.232.44.19: bytes=32 time=22ms TTL=54
Reply from 68.232.44.19: bytes=32 time=24ms TTL=54
Comment by Stuart — July 12, 2010 @ 5:04 am
I try to consider what solution to reduce the time for big customers?
is the question I ask myself and that I did not answer.
I may have found a solution with “cedexis.com” using multiple CDN (it’s CDN load-balancing) to reduce travel time data when a american user charge my client website hosted in Europe.
I think the best answer is in the complementarity of choice: a good web host that has good transit operators, a website with an optimized code, http, powerful (LightHTTPD, nginx, etc.) properly configured, and why not, the Using CDN’s (with load balancing) to reduce the loading time of some elements.
Comment by Mickael — July 12, 2010 @ 9:01 am
This ratifies a lot of what I’ve been saying for many years about bringing Australian content back into Australia.
The effect you’re seeing is a result of something called TCP windowing, bandwidth-delay product or colloquially known as “long fat networks”. The Wikipedia article at http://en.wikipedia.org/wiki/Bandwidth-delay_product explains the complexities of the concept more concisely and far better than I ever could.
Something else to consider is the nature of international networking. International carriage is EXPENSIVE (in the order of $100+/Mbps) and is therefore FAR more likely to be heavily oversubscribed than local traffic.
Hosting in Singapore will make little difference: international capacity to Singapore (carried via the SMW-3 international cable) is orders of magnitude more expensive than capacity to the US (where competition between cable providers SXC, AJC, PPC-1 is fierce). As a result of this, most providers will route to east asia via the US, which will further decrease performance - Asian hosting is usually counter-productive!
Local IP traffic is often handed off via private peering or one of the many MLPA (multi-lateral peering authority) organizations in Australia, such as WAIX, PIPE Networks and Equinix. Most ISPs connect to these MLPAs at 100Mbps, 1Gbps or beyond, using a fraction of this capacity: as a result, it is not difficult to achieve line-speed data transfer, even interstate (especially if the traffic is carried via a reliable, well-peered network such as Agile, UEComm, Chime etc.)
The difference in performance between Australian data and overseas data is astronomical - I’m glad someone has taken the time to compile the evidence in a graphical format, somewhat easier to understand than the examples which are available on the Whirlpool forums at http://forums.whirlpool.net.au/forum-replies.cfm?t=1355570&ux=142444
Comment by Curtis Bayne — July 13, 2010 @ 1:50 am