Six key steps to load testing

Whether it's a whole new site design, a new Content Management System, an on-line shop upgrading to use Verified-by-Visa, or simply looking ahead to traffic peaks, risk reduction is important. So what forward planning will a Web Manager need to adopt to address the 'Will it Work?' question?

News headlines like those resulting from this year's Glastonbury Festival ticket fiasco, continually reinforce the pitfalls that occur when a system is unable to cope with peak loads. This media attention plus the assumption that a new system will work for a single-user often leads organisations to implement Stress/Load Testing alone to see if a site can cope with real usage levels.

This assumption often stems from programmers testing as they go along; which isn't the same as covering the entire functional space. Prior to stress/load testing, a full Functional Test/User Acceptance Test should be the starting point for any new project testing process.

Following functional testing a system requires modelling and several preparatory stages before stress/load testing can begin.

Modelling your system

When modelling website usage patterns average traffic figures are of little use. At its simplest, you need to identify from your web logs the peak traffic levels on the site. 24,000 hits a day doesn't mean 1,000 hits every hour.

Find traffic levels during the busiest hour. Then use the average number of pages per visitor to calculate equivalent pages per hour, and work out the 'worst case number' of concurrent users on the site: for example, 360 visits per hour, at 10 pages per visit = 3600 pages per hour =1 page per second.
Stress/load Testing Process

So here are the six stages we advise you to adopt for stress/load testing:

1. Identify 'User Journeys'
Identify and build the multi-step User Journeys. These will emulate the activity of real visitors to your site. You will need 3 to 5 journeys; maybe more if you're incorporating functionality test.

2. Build 'User Journey' scripts
Building User Journeys requires a good understanding of your web system - how it handles session IDs, directory structures and product numbering conventions, how caching of page content and images is handled by the server, javascript usage and so on. Often this is the stage at which SciVisum testers find configuration flaws that can be quickly removed, reducing unnecessary load on the server.

3. Run scripts in single-user mode
This is done to measure the system's responses without any stress. Performed repeatedly, it provides data to tell you whether the system is running stably, consistently and error-free at low user levels. If your web system is already running out of hardware capacity, or the application design is flawed then inconsistent performance will be highlighted here. It is difficult to get reliable test data from a system where performance varies greatly, independent of the changing stress test load.

4. Identify system flaws
A range of system design flaws can contribute to site errors and slow page delivery times. These are usually random, difficult to pin down, and don't go away when hardware is upgraded. Flaws are identified by running the User Journey scripts at relatively low levels simulating 2, 5 or maybe 10 virtual users; and running them at high simultaneity - accessing the same page at exactly the same moment. A high percentage of errors here indicates that an application may have database locking flaws, or errors when handling shared application variables or similar.

5. Find bandwidth limitations
The last preparatory step is to identify any system bandwidth limitations - so that later testing can be kept within these limits. This ensures that results will not be distorted because of servers hitting a bandwidth ceiling. Bandwidth tests involve requesting multiple static objects (objects that require no CPU processing to build) from the server, from several well-connected (100Mbit/sec) remote Internet locations, then measuring the bandwidth limit.

6. Run tests under load
As the quantity of virtual users is ramped up, the effect on page delivery time and error rates is measured. Testing needs to be able to monitor and report on a range of defects that can arise as a system starts to 'creak' under load. Look out for missing page components (graphics, style sheets, javascript files), the appearance of error messages within pages or missing page content. A page may be delivered many times without any 'server errors' being reported, yet have obvious defects that can impact on users.

As the traffic volumes increase, and the performance of different User Journeys begins to decline, the test cases planned evolve to drill deeper and identify likely root causes. New test scripts are written to further expose the problem causes, and to rule out alternative explanations. Test team experience and expertise is vital here; as is a flexible and powerful test engine that can smoothly construct new test cases on the fly, and rapidly correlate data across a number of different runs and user journeys.

Never approach testing with preconceived assumptions about potential problems. In SciVisum's experience it's common for a team to have concerns about a specific issue as their 'prime suspect' - but when testing begins and real 'evidence' is produced, these may be shown to be wrong. The real bottlenecks will be discernible and unambiguous from analysis of the test data. It's sometimes a shock to find that perhaps the ISP is not the bottleneck after all, and what's causing the problem is actually an element in the application design.