Sometimes when planning a website load test project, the question comes up as to whether the live site should be tested, or the staging site.

ITIL's Capacity Managament and Capacity planning includes website load testing - and that too has encouraged many organisations to weigh-up live versus staging testing.

For most organisations that have been running load tests in house for while, the natural reaction has been to choose staging - but this is often for all the wrong reasons - ie reasons that try to make the testing easier, rather than maximise the business or technical value of the results. If you're going to do a load test in your site, confidence that it will have been worth the effort is directly linked to how widely the test results are understood and how accurately they prove to match later real world performance.

And if you are have not performed load testing on the final system for a while - it is vital to start a new season of load testing with one on the live site - to have the essential keynote benchmark, of the 'real world'  - your real site.  Any checkllist for your capacity planning should include :  when did we test the real site last?

The reasons that inhouse test teams, or consultants paid per day on site, often prefer to test the staging environment are usually:

• can't be blamed for any damage to the live site!
• it takes too much hardware to generate enough virtual user traffic on the live site
• it's cheaper - most load test software tools charge for the number of virtual users (unlike SciVisums price model)
• it's a safe sandbox environment - you can flush databases and reset then after each test
• you don't need staff to work out of hours - the staging environment is free for testing 9 to 5

There is one keynote benefit of web load testing a staging environment - that is it makes it easier to run Checkout Journeys that actually click the final Buy button: on staging no real orders will be processed, but on live, you'll need to involve coders to have modules to delete orders immediately they are made, to prevent real bank transactions or warehouse orders being raised!

It's impossible to build an accurate scaled down version of your live site

Load testing is about building confidence, confidence in your site, and confidence that you know what traffic it can handle.

So if you want to be confident that you know what your live site can handle - you must test it directly!

When testing your staging environment, although it can give useful benchmarks as to how performance changes occur from code release to code release: it can't give a certainty about the actual absolute capacity of your live site  - when you lose certainty in the  Capacity planning on your site, confidence in the whole process is lost.

Think about the analogy with other industries - when testing a new aircraft design - they'd start with a scale model, and test it in a wind-tunnel to see how it performs.

That's easy for an aircraft model - making a 1:100 or 1:20 model of an aircraft shape is easy.

But in computer server terms, it's hard or impossible.

If you live web site has 6 servers of 8GB RAM each - what would be a realistic 1/10 scale be?  6 servers with 0.8GB each!  Or 1 server with 4.8Gb?  And what about CPU's?

Clearly, any real world software application will not have 1/10th the throughput just by dropping the RAM by 1 factor of 10: it's not linear with RAM or CPU.

What about your load balancing hardware - or SSL accelerator... it's hard to predict accurately how it scales

Secondly, even if magically on Day 1 you had proven your 1/10 scale staging environment did perform with 1/10 the capacity of the live site for every possible user journey - in the real world, just days later, the live environment is changed - RAM is added, an old box is swapped for  newer one, another web server is added to the farm etc.

Each change destroys the accuracy of your supposed 1/10 model  unless you in parallel upodate your staging environment too.

Impossible.

Testing the live website - gives you accurate metrics of the live site capacity!

There's no need to multiply by 10, or to estimate the effect of the fact that the live site has 3 database servers in the cluster and the staging has 1.

The numbers are real, straight off the tin.

Off course you'll have to test out of hours - to avoid impacting any quantity of real visitors. There are some extra factors.

But they are outweighed by the certainty you can have in the final metrics.

When the business wnat to know how many Add_to_Basket_via_random_Search user journeys the site can handle right now - you can measure and tell them, hand on heart.

Another good idea when running your load test, is to have your website monitoring Users Journeys portal visible too, so you can track the effects on them, as well as on your own server monitoring or basic siteuptime tools and etc.

One last tip when testing the live site - don't forget to tell your third-party hosters, network providers, content providers etc , as you will off course be loading their sites as well - although with a flexible test platform you can have some leeway on this - eg if you're using a 3rd party to host all the images from your website -you can specify in the User Journeys that they do, or do not, actually grab those images.  So you could test 'without images', but that still is generating the exact same load on your own web farm and systems. Once you know the capacity there, you can then add images back in, and test again, and make sure the 3rd  party image systems can handle the same load.

Further:  Website Load Testing articles

Request Demonstration

View our client testimonials If you would like one of our service professionals to contact you to arrange a demonstration of our web testing services or discuss your web site performance goals/needs free of charge please contact us. To contact a service professional call us on +44 (0) 1227 768276 or request a telephone/email call back