| |
Whatever you call it, applying an artificial load onto a website can
tell you a lot about how well the website will cope under stress. Load
testing, stress testing and capacity planning essentially refer to the
same thing, and are typically carried out before taking a new system live,
or in advance of a predicted peak of seasonal traffic or before a marketing
campaign.
In some circles Load Testing would simply test up to a pre-defined traffic
level, whereas Stress testing would test up to the limits of the system.
In practice, with good test engineers both should be covered.
Many engineers see the prime purpose of this testing to find out if more
hardware is needed to handle the desired traffic levels. But in nine cases
out of ten the main benefit comes from identifying below par areas of
the web application's architecture or coding. By fixing these areas there
is often a substantial capacity gain (a several fold improvement) - far
greater than can be achieved by simply doubling the hardware.
So when commissioning Load or Stress Testing, what should you insist
upon, to achieve the most benefit?
1. Test the system as closely as possible to its real world state. Use
the same server hardware, firewalls, ISP bandwidth as the final site.
This is often best carried out overnight or whenever you have fewest visitors.
2. The virtual user traffic to load your website should be as near as
possible to real traffic. This rules out forms of load testing that simply
hit isolated pages or your home page. Your site needs to be loaded with
virtual users performing tasks that real users would - following multi-page
User Journeys.
3. User Journeys, one for each of the main services or transactions offered
to users need to be carefully scripted, so that at each step they act
like a real user - eg when filling in forms and making choices. Creating
effective scripts for every step of each Journey can require expert testing
staff.
4. It's not enough to simply have a page returned, it must contain the
content that is expected as well as the link or button required to move
to the next step in the Journey. Simple 'on-line self service' load testing
approaches often prove to be disappointing - the level of scripting expertise
initially required is not available.
5. Once five or six Journeys for your site have been agreed, check that
the scripts generated have proven stable over a few days, usually by running
them in single-user mode.
6. At this stage, a bandwidth test should be performed. This is to check
that the bandwidth available is in line with what you're paying for (not
always the case). Also, it is vital that the bottleneck due to bandwidth
is already known, so that any performance slow downs due to bandwidth
can be ruled out of the investigation as the testing rolls out.
7. Next, test each User Journey in isolation - increasing the load and
observing the performance degradation. This stage is not just about numbers
- it requires expert staff to 'reflect' on the pages that returned errors
or were missing content. Expert analysis provides invaluable insight into
the root causes of problems, and these can be quickly fed back to clients.
The key statistic you'll get for each Journey at this point is the peak
number of 'journeys per second' that the system can handle.
8. It's not unusual at this point, for experienced testers to highlight
substantial performance gains. Testing can be continued after a short
break with a much improved web system, with much higher throughput. This
added intelligence is not available with automated test tools.
9. Testing of each journey should include a more complex test sequence
- for example SciVisum's high simultaneity runs where many virtual users
follow a journey in their own time, but all stop before a defined page
and hit it concurrently. This is an aggressive test, and must be used
carefully, but can identify a range of coding and configuration problems
- known as 'race conditions' that are often down to coding errors, configuration
oversights or database locking problems - including root causes that are
extremely difficult to find by any other method. For client's who can
provide data on their 'drop off' rates (what percentage of users drop
off and at what point in say a Checkout journey), extra journeys can be
scripted to model the same drop off ratios. These provide a more realistic
load than Journeys where 100% of those starting finish, and often provide
quite different throughput capacity figures.
10. The final stage of heavy testing, is running mixed User Journeys -
this again helps identify more realistically how the system performs in
the real world, by having a mix of User Journeys running which match the
expected load ratios - e.g. 20% checkouts, 30% add to baskets, 40% product
searches and so on.
After the heavy testing is done, some vital investigation work remains
- analysing the delivered pages for more root causes and possible optimisation.
Having the same engineers that scripted the User Journeys at the start
to perform this final stage usually ensures best value.
At the end of the day, what's needed from a Load Test is a lot more than
just a set of numbers per User Journey, vital though these are - it is
the engineer's insight into root causes and the re-factoring and re-configuring
that will provide the biggest performance gains for the lowest cost.
|
