Stress testing the Cloud

Date: 24th August 2009
Author: Deri Jones

The findings from the University of New South Wales, who have been running some web stress testing on the 3 main Cloud suppliers, has been interesting.

Monitoring over several months, they found as I’d have expected, that there were performance fluctuations as the vendors rolled out new features, and changed things.

The Cloud is not 100% stable yet, that’s to be understood.

More interesting, they found considerable variation in performance with time of day – with response times varying by up to a whopping 20 times.

This highlights one of the key considerations for any organisation starting to depend more on the Cloud: what happens when all your Cloud supplier’s customer have a traffic peak at the same time – what kind of a web load test is that?

If you’re an online retailer; then you probably have an evening visitor peak of say 6 to 9pm: and so do all the other retailers with your cloud company. So you’re all vying for capacity at the same time. Stack up the trafic even more with events like the last online shopping day before Christmas: and the scope for the Cloud to be unable to handle the load becomes clear. No Cloud company can have infinite hardware.

Your cloud supplier can probably handle the local peaks that you cause when you do a compelling mailshot to hundreds of thousands of your customers, who respond by visiting the site within a few hours. But how would they cope if several of their clients did a big mailshot on the same day?

Anna Liu, Associate Professor in services engineering said: “None of the platforms have the kind of monitoring required to have a reasonable conversation about performance,”

Which highlights the need for 3rd-party continuous monitoring of your online User Journeys, as you make the transition to use of the Cloud.

And the SLAs form this monitoring need to be defined and administered by the commercial teams in your organisation: technical based metrics alone won’t catch the changing user experience: that business based, User Journey web performance monitoring provides.

Their findings were based on web load tests on Amazon’s EC2, Google’s AppEngine and Microsoft’s Azure cloud computing services.