Home > Reference > Articles

A 10 point plan for load testing - what to expect and what to insist on ...

Whatever you call it, applying an artificial load onto a website can tell you a lot about how well the website will cope under stress. Load testing, stress testing and capacity planning essentially refer to the same thing, and are typically carried out before taking a new system live, or in advance of a predicted peak of seasonal traffic or before a marketing campaign.

In some circles Load Testing would simply test up to a pre-defined traffic level, whereas Stress testing would test up to the limits of the system. In practice, with good test engineers both should be covered.

Many engineers see the prime purpose of this testing to find out if more hardware is needed to handle the desired traffic levels. But in nine cases out of ten the main benefit comes from identifying below par areas of the web application's architecture or coding. By fixing these areas there is often a substantial capacity gain (a several fold improvement) - far greater than can be achieved by simply doubling the hardware.

So when commissioning Load or Stress Testing, what should you insist upon, to achieve the most benefit?

1. Test the system as closely as possible to its real world state. Use the same server hardware, firewalls, ISP bandwidth as the final site. This is often best carried out overnight or whenever you have fewest visitors.
2. The virtual user traffic to load your website should be as near as possible to real traffic. This rules out forms of load testing that simply hit isolated pages or your home page. Your site needs to be loaded with virtual users performing tasks that real users would - following multi-page User Journeys.
3. User Journeys, one for each of the main services or transactions offered to users need to be carefully scripted, so that at each step they act like a real user - eg when filling in forms and making choices. Creating effective scripts for every step of each Journey can require expert testing staff.
4. It's not enough to simply have a page returned, it must contain the content that is expected as well as the link or button required to move to the next step in the Journey. Simple 'on-line self service' load testing approaches often prove to be disappointing - the level of scripting expertise initially required is not available.
5. Once five or six Journeys for your site have been agreed, check that the scripts generated have proven stable over a few days, usually by running them in single-user mode.
6. At this stage, a bandwidth test should be performed. This is to check that the bandwidth available is in line with what you're paying for (not always the case). Also, it is vital that the bottleneck due to bandwidth is already known, so that any performance slow downs due to bandwidth can be ruled out of the investigation as the testing rolls out.
7. Next, test each User Journey in isolation - increasing the load and observing the performance degradation. This stage is not just about numbers - it requires expert staff to 'reflect' on the pages that returned errors or were missing content. Expert analysis provides invaluable insight into the root causes of problems, and these can be quickly fed back to clients. The key statistic you'll get for each Journey at this point is the peak number of 'journeys per second' that the system can handle.
8. It's not unusual at this point, for experienced testers to highlight substantial performance gains. Testing can be continued after a short break with a much improved web system, with much higher throughput. This added intelligence is not available with automated test tools.
9. Testing of each journey should include a more complex test sequence - for example SciVisum's high simultaneity runs where many virtual users follow a journey in their own time, but all stop before a defined page and hit it concurrently. This is an aggressive test, and must be used carefully, but can identify a range of coding and configuration problems - known as 'race conditions' that are often down to coding errors, configuration oversights or database locking problems - including root causes that are extremely difficult to find by any other method. For client's who can provide data on their 'drop off' rates (what percentage of users drop off and at what point in say a Checkout journey), extra journeys can be scripted to model the same drop off ratios. These provide a more realistic load than Journeys where 100% of those starting finish, and often provide quite different throughput capacity figures.
10. The final stage of heavy testing, is running mixed User Journeys - this again helps identify more realistically how the system performs in the real world, by having a mix of User Journeys running which match the expected load ratios - e.g. 20% checkouts, 30% add to baskets, 40% product searches and so on.

After the heavy testing is done, some vital investigation work remains - analysing the delivered pages for more root causes and possible optimisation. Having the same engineers that scripted the User Journeys at the start to perform this final stage usually ensures best value.

At the end of the day, what's needed from a Load Test is a lot more than just a set of numbers per User Journey, vital though these are - it is the engineer's insight into root causes and the re-factoring and re-configuring that will provide the biggest performance gains for the lowest cost.