Web performance – sorry is (not) the hardest word

Date: 14th September 2010
Author: Deri Jones

It’s a challenge to every online business – when you get something wrong, when the site just goes too slowly, when your pricing errors get hundreds of customers excited and then you cancel the orders…

Do you say sorry? Or keep schtumm, hoping most customers won’t have noticed.

Certainly there was a very big Sorry out on the wires this week, from one significant ecommerce player, Steadfast.

But in your organisation – do your customers ever hear you say sorry? Or is it only the ones that phone in that find out?
Does someone in your team suggest an apology really should go out… but does it get delayed in the desire to polish the words, and eventually too many days have gone by and it is canned altogether? Maybe the discussion is just too heated internally as to which team is to be blamed, that there’s no stomach for a public apology.

As web load testers and web performance people, I guess we are see up lose some of the problems that the customers of the web systems we measure have to put up with.
We see the graphs that show the 10 or 15% error rate that occurs each night from 7 to 9pm for days in a row, when a code release on a website has unforeseen consequences on system loading, and it takes too many days to roll back.
Or our dynamic User Journeys that really do what customers do, and pick up the sporadic errors such as when one customer’s address and credit card number appears on the Confirm page of a different customer – even at 1% occurance that is a killer for your site’s credibility.
Or when the bandwidth supplier has to come clean when load testing shows that the infrastructure in place falls far short of what was contracted for.

Karl Zimmerman CEO of Steadfast Networks, sizeable hosting company, kicks off his Sorry email with:

First of all, I am very sorry. I cannot say that enough

His company had some scheduled maintenance due at the weekend, and it went wrong and a number of their clients were off-line for 12 hours and more.
Which is clearly not ideal.

I truly apologize….Simply put, we got in over our heads…We have let you down…We thought we had everything prepared and had spent weeks in configuration and testing, but it appears we were wrong…

He’s trying to come clean – and part of the reason is that a number of his clients felt that they didn’t get accurate up-to-date information on what was happening at Steadfast through the outage period: and even that they’d been lied too – statements that support staff had been onsite when they actually been working remotely.

Anyway, instead of Schadenfreude – we should all take it as a lesson that ‘there for the grace of God go I’ – as no-one can guarantee 100% error free when there are humans in the mix – all software has errors, all websites throw a percentage of errors, it’s just a a question of what the percentage is, and do you even know where they are occurring, or is it just your customers who find out?

So maybe we’ll all treads a little more carefully about making those web site code updates- especially in the run up to the Christmas shopping peaks coming soon.
Maybe it’s worth a little more time on the preparatory web load testing – do it well enough in advance to give time to safely fix things.
Having load tested your site, confidence in being ready for show time will undoubtedly increase.

None of us want to be sitting at home after Christmas, with an email like this in our Sent box and still ringing in our ears:

I am posting this here myself so we can accept our blame and take our shots. We deserve it for this one. You cannot understand how bad I feel for this and how sorry I am for the customers this has hurt.