UK eCommerce sites riddled with ‘invisible errors’

23, October 2007 - UK e-commerce sites are jeopardising more than £300m* in lost sales each year and risk driving customers to competitors because of ‘invisible errors’ that can’t be detected by web analytics.

One third of the consumer online journeys tested by SciVisum experienced more than three per cent error rates, while more than ten per cent demonstrated extreme inconsistencies in delivery speed of the journey.

"This is a worrying trend for eCommerce and IT directors and for consumers with Christmas just around the corner. Poor performance and web errors will mean lost sales," said Deri Jones, CEO, SciVisum. "Companies wanting to maximise their online sales this Christmas need to check the performance of their sites now. Those that fail to do so might as well include a link to their competitors’ site."

These are the key findings of the SciVisum Lost Online Sales Study, which investigated the performance of 40 online sites from the retail, finance, insurance and travel sectors, over a period of six months. The study confirmed that customers of e-tailers are being exposed to a significant number and range of problems online which prevent them finishing their desired journeys, but are invisible to existing tools and web analytics.

Invisible errors

"Invisible errors are not outages affecting 100% of users, but are problems that impact a percentage of users at any point in time. A problem that impacts say 1 in 100 random users on a particular journey is not reproducible by IT teams, and so frequently remains unresolved," said Jones.

SciVisum's testing adopts a mystery-shopper approach that actually visits the site and attempts to make a user journey every five minutes throughout the day. This allows the company to see what customers see, and makes it possible to identify a range of intermittent problems that impact real users, but that are invisible to any other analysis. These problems include:

• Session swap: where two users see each others' online sessions. Nowhere is this category of problem detected in server or analytic logs.
• Page not delivered errors: because the page is not delivered, there is no log of the error in web analytics.
• Jump back: the user is in error forced back several pages: the new page is itself a valid page, so no errors logged in analytics or tech logging.
• Page content incomplete: web analytics logs only that a page was delivered, not whether it showed the user what they expected.
• Shopping Basket errors: e.g. basket is empty after adding items. Nowhere is this category of problem detected in server or analytic logs.

Need for speed

The research also highlighted massive inconsistencies in the delivery speeds of the journeys that users undertake. More than thirty per cent of journeys experienced performance varying by more than 200 per cent, with one in ten varying more than 300 per cent, data averaging over a seven day period.

Variations in performances means that returning visitors will be frustrated on the occasions when the websites run slower, while first time visitors are likely to be driven to competitors. Because different technology blocks are used to deliver the different routes that customers follow, those journeys provide significantly different experiences, even though they run on the same website

Frustratingly for consumers, website performance was shown to be most commonly worst in the evenings between 8pm and 10pm. This often coincides with peak traffic levels, meaning that the site performs worst when it will inconvenience the most visitors. This is often invisible to the eCommerce manager, because simplistic measures of overall page-speed averages per day hide the fact that just one or two of the core user journeys perform really poorly for a couple of hours each day.

Unacceptable behaviour

"As a specialist web tester, we’ve come in to contact with invisible errors for some time now, but it was only when conducting this research that the extent of the problem became so apparent. The UK’s online landscape is plagued by these errors and as users continue to become web savvy and increasing numbers of people encounter them, they won’t remain invisible for long," said Jones.

"Interestingly, we often find that our clients Call Centre folk are aware that there are problems impacting users that are invisible to their own colleagues in other departments. It seems that a deaf ear is turned to their feedback; being repeatedly told that 'no problem was found' or that the problem 'couldn't be reproduced by the tech team'. That last phrase is a typical internal response to the kind of sporadic, invisible errors we found, and means that the underlying problems can never be addressed," added Jones.

Recommendations

Based on the findings, SciVisum made a number of broad recommendations for e-tailers to improve their performance:

• Adopt a Mystery Shopper approach and test the paths that real users take when making use of your company’s website. Simple uptime/downtime monitoring of your home page and/or a few main pages simply won't reveal invisible errors - 24/7 functional monitoring, running multi-page user journeys that mimic real users' product finding and purchasing transactions on-line is what is required.


• Focus on relevant performance data. There is a wealth of website performance data available to firms, and this has in part contributed to the continued prevalence of invisible errors. To help detection, companies must focus on using data gathered by simulating real user journeys and experiences, not data from internal servers or monitors


• Business people must take ownership of the issue. During the study we have found that it is business and marketing personnel that are most aware that there is an issue with their website. Yet when the issue is raised with the IT department, they are fobbed off with an avalanche of data and metrics that indicate the site is performing well from a metrics perspective. Business people must push back when this occurs and only accept data that is relevant to a user’s experience.

Methodology

SciVisum tested 52 multi-page user journeys for 40 different online organisations over the six month period April to September 2007. SciVisum’s multi-page journeys measured at five minute intervals and were defined with each client so as to follow the same core path that real users take when they make use of an online portal. These journeys measure the vital Money-Making or Customer-Servicing features of a website including Add-To-Basket: browse and select; Add-To-Basket: text-search and select; and Checkout and Pay. Five minute sampling equates to approximately 300 measurements over each 24 hour period.

Each page of a journey is checked to ensure that the correct and expected content has been served. Speed timings are also measured for the total journey, and for each page that makes up the journey.

Download your free copy of of 'SciVisum : Lost Online Sales Study 2007'