Are You Testing in Realistic Network Environments?
Happy New Year!
Just before Christmas we exhibited at the TestExpo show, which specialises in tools for testing software.
Chatting with the testers and developers during the show I’m still amazed that when testing, and during the development process itself, the effect of the network on the application being tested/built itself is rarely taken into account at all.
Now, if the application is stand alone or will only ever run in a LAN this doesn’t matter much, as the test and build environments will match the final deployed environment. But if the application is due to run in a WAN (or GPRS, 3G, ADSL, DSL, Satellite or Radio network – I’ll call these all WANs for simplicity) the network really matters, and not just because of limited bandwidth.
Taking the road network as an useful analogy, using a LAN is like having your local town, with all the roads (even your local road) being high speed, multi-lane highways with on/off ramps, no traffic lights etc: fast, capable of taking lots of traffic, unlikely to get congested with minimal delay when getting to any other place. The WAN by comparison consists of a very wide range of roads: multi-lane highways (often congested), single lane highways, lanes, tracks and even muddy fields.
You’d never find a car manufacturer only prototype and test a car on beautifully flat roads if it is intended to go off road. A normal (non 4×4) car wouldn’t get 10 yards in a muddy field!
I think by now most everyone involved in IT understands bandwidth limitations. We’re forever being sold or touted faster home Internet connections, so that we can download faster etc. What most people don’t understand is that latency (delay), loss and errors are very important factors in data transmission across networks. Just as important as bandwidth.
We can get an idea of latency by ping-ing systems both in our LAN and on the WAN e.g. Google and we’ll quickly see that the ping times are very small (<1-3ms) in the LAN and large, sometimes very large in the WAN (e.g. 90ms UK to Eastern USA , >700ms for Satellite). This is the equivalent of a WAN being a long road, with lots of traffic lights, roundabouts, stop signs, muddy even. We can buy more bandwidth (effectively more lanes on the road) but we can’t do much about latency. For TCP based protocols (ftp – file transfer, http/https – web, smtp/pop3/imap – mail transfer, Microsoft DS – Windows file sharing and most custom sockets applications) the delay really matters, as acknowledgements of successful transmission of data must comeback from the other end.
So why don’t they build and test in these WAN (or GPRS, 3G, ADSL, DSL, Satellite or Radio) environments?
One reason is lack of knowledge that it matters. My mission is to change that.
A second is that all that equipment to recreate the WAN, GPRS, 3G, ADSL etc. etc. costs loads and is unmanageable,. which is why iTrinegy exists and produces WAN Simulators/Emulators.

Our company had this exact issue with our FTP infrastructure. Our file tansfers were only going a fraction of the speed they should have been. We ended up replacing it with a product called FileCatalyst (http://www.filecatalyst.com) which was actually designed to work with the high RTT we had on our WAN. However that never solved our other problem of how to test without going on our live network. We might be contacting you in the next few weeks about this.
Comment by Gordy — January 21, 2009 @ 8:25 pm
That’s the problem with latency, frequently this is the bounding limit of your application, and adding bandwidth doesn’t solve the problem.
What can work for ftp are download manager tools like Free Download Manager (http://www.freedownloadmanager.org/) which effectievly do lots of downloads (of the same file) in parallel by splitting the file into bits. Of course some sites hate this because you also circumvent any maximum “per user” bandwidth limits that they may be using to prevent their sites being overwhelmed…
Comment by Frank Puranik — February 4, 2009 @ 4:47 pm