Why Do Test Teams Want To Avoid Load Testing The Live Website?

Date: 25th April 2013
Author: Deri Jones

Last night I spoke at the Agile Testers London Meet Up on: The Lessons We Learnt Load Testing The Website Of A Large UK Retailer.

The MeetUp is always a lively event. Last night in particular there were a lot of excellent questions from the floor, and some very interesting “compare and contrast” discussions with folks from a wide spectrum of organisations from Marks & Spencers, O2 and Sainsburys through to Aviva.

But one startling issue that arose is how few companies are load testing the ‘real thing’: their live site!

A show of hands suggested that Load Testing is not something that most testers are focused on. Only about 10% had done any at all in the last 12 months, and only a couple of percent had done any in the past month.

The tools being used when load testing had been carried out were mostly the old names, with their well know old strengths and weaknesses: Load Runner and jMeter.

But the really startling finding was: the level of resistance to load testing the actual live website.

For some it was the problem of getting Business Team sign off and ‘risk’ the live site.

For some it was the pain of getting set up to allow the quick ‘clean up’ needed after a load test, deleting from databases the dummy transactions for example.

For many, the load testing scripts being used were only checking lower level page events, as that sort of test had been less pain to put together.

And it was clear that in a number of cases the load testing done on staging environments, had become the ‘magic recipe’ that the organisation depended on as the performance Go/NoGo. This was even the case when several team members felt that actually what was being tested, and how it was being tested, actually no longer bore any real relationship to what happens for real users on the live site!  The number generated each time was just a “magic token”.

It’s hard to write meaningful Load test scripts that ‘Do what the Customer Does’

Chatting to folks afterwards it was clear that load test scripts are the unloved child in the testers world! They are a pain to keep up to date as the site changes, a pain to make realistic as sites become more richer with AJAX and HTML5.

So the SciVisum fully Dynamic approach to Load Testing prompted a lot of questions – and especially the benefit of load testing that does not throw false errors just because the live site does not change, in terms of especially retail sites where products can go out of stock between the start and end of a load test.

In conclusion it seemed like a lot of Business teams are not aware that the  Load Testing being done for them, is in fact not doing anything to reduce the risk of capacity limits being hit, and real customers, and revenues, being impacted.

It was a great MeetUp and I look forward to more in future.