Date: 25th June 2013
It’s amazing, in a painful way, the range of things that happen under the title of Website Load Testing.
I thought I’d seen everything: big company treacle approaches with numbers that no one understood; small company fly-by-the-seat-of-the-pants approaches that produced well understood numbers, that didn’t help any website capacity planning decisions anyway! Load testing of ATG, of Hybris, of WebSphere, of Drupal, of home-grown C#.
But this week, it was the Punch-up with the Web Analysts.
Let me first say, that although tempers were high, no literal punches were thrown.
The story starts as a common one – an organisation has called us in to help them think through how they can move their load testing to the next level – what best practise they can learn from, what tools they should drop or pick up, and most of all:
- How can we Load Test to get a Realistic measure of our online business
Not unusually, they knew the load testing approach from the past was no longer fit for purpose.
And not unusually, when senior business people were asking about future capacity planning, the eCommerce teams had not been able to provide a robust evidence base to work on, and so something better was put into plan.
This organisation was in the online gaming space, so unlike the case of a product retailer where the number of available things that can be bought by virtual users in a load test is large, but fixed: for an online gambling load test the range of bets that can be placed is wide, with various parameters:
- date of event
- type of event
- horses or team to bet on
- real-time changing odds
- complex set of betting and multi-bets and all the richness of bet possibilities
What went wrong…. is that the thought of all that complexity, and the need to reproduce it during load testing, was daunting. Understandably.
And hence historic load testing, done internally, had focused on what had been easy and direct to script and run, rather than what would give a true measure of ‘How Big is our Gaming Store Capacity’.
My team were to do the Journey scripting, and we had as always our unique and flexible SaaS load testing platform to do anything needed, literally anything. But – it was hard to work out what the realistic specification should be.
So a Specification was passed to us, that had come 2nd hand, via the process of internal NFT testing scripts, and had started some way back before that: based it seemed, from analytics data about the last traffic peak from 2012.
So we scripted the User Journeys, and this raised some questions – the mix of traffic in the Spec, was not one that real users could ever do! The website had new routes for users now, or maybe the Spec had always been unreal in that sense. But the mix of user actions and page types, was not realisitic.
So it was agreed to tweak things, to make it a little less unrealistic, and in our scripts we forced some things that real users couldn’t have done, so that the numbers matched the idealised Spec. And we ran the load test, running a Sunday morning 00 to 06 AM stint.
The numbers were OK: not un-expected given the test was first at a level not too much higher than last year.
But then, before the 2nd round could start, with higher levels: – the debate kicked off, when the Web Analyst team became directly involved.
They had of course access to the detailed, full web analytics data from the past peaks.
And it didn’t match the Spec, that had built up separately.
So they created a new more realistic Spec: that was based on the solid ground of the true analytics data.
So, to cut a long story short: we rebuild the matrix of realistic Dynamic User Journeys so that the overall traffic generated matched the mix pulled from real web analytics; the new Spec.
It was a rather crazy last-minute run round, but we managed it.
And ran the next online gaming load test with that set of Journeys.
Conclusion: it’s better to bring your web analytics data, and team, into your capacity planning before spec’ing the up the load tests.
It is the one thing we can’t help clients with on load testing: they of course have to bring the web analytics data from their own site, to the planning table.