Black Friday analysis of website failures. What went wrong?

Date: 27th November 2015
Author: Deri Jones

Also see:

 

 

Whilst monitoring leading retail sites over Black Friday 2015 we saw a high level of website failures, read our commentary below to find out what went wrong.

Noon: Black Friday 2015 

Lost sales due to Marketing Conversion Tools/Trackers failing

As well as retailers losing sales today through lack of web capacity, it also looks like many retail websites will be losing sales on Black Friday, as their 3rd-party lead-generation tools are not working this morning, with resulting loss in CRO.

In addition some online Marketing Directors will be in the dark when they come to carry out their post-Black Friday wash-up next week – because a number of the 3rd-party tracking/big data services are failing today, and may not provide a true picture.

Measuring user experience across many retail clients, we are seeing failures. Our measuring approach is to use Mystery Shopper User Journeys, which follow realistic multi-page User Journeys many times an hour, across a site to  ‘Do exactly what the Customer Does’.

A typical  Journey may for example:

  • navigate from home/sale page, to basket via categories and sub-categories/filters
  • choose a different product each time
  • at each step: measure availability and the speed the user experiences

Apart from seeing awkward failures at critical moments in shopping Journeys, we are also seeing many 3rd-party errors today.

Example of failure at a critical moment; the user is in the basket and wanting to complete an order but a queue page is shown:

Black friday basket queue at Argos

Example of third party error; Scarabresearch.comlike

  • At Step 10 of a Journey: stepname: “In the basket the user makes Payment selection”
  • Error is:  “Http Error Code 503 Service Unavailable – The server is currently unavailable (overloaded/down) in Component:

http://recommender.scarabresearch.com/merchants/<anonymised>/

Scarab Research are widely used by retailers, their website says: Scarab Research provides highly personalized recommendations for customers of e-commerce businesses.

Other examples of failing 3rd-party tags in Retail pages.

Our shopping monitoring journeys are showing the following as failing/slow so far on Black Friday:

  • recommender.scarabresearch.com
  • abmr.net (Akamai)
  • yieldmanager.com  (running Yahoos’ advertising exchange, Right media ?)
  • bluelithium (aka Yahoo Advertising)
  • salescycle.com – On-Site Remarketing
  • https://ads.yahoo.com/pixel has been failing on and off all week and is very bad today

We also saw  HotUkDeals.com  tags problematic too.

If your retail site depends on these services, and your sales are not as you would have wanted, these suppliers may be the root cause of the lost sales as well as unreliable/confusing traffic data when carry out your post Black Friday post-mortem.

*5pm Update: Black Friday 2015*

Since the original post here at noon, a number of interesting things have happened in the worst cases some retail sites performed badly enough to hit the national news: John Lewis website crash – ITV report.

More interesting are some of the web performance lessons that can be learnt from other major retailers that offered less than ideal user experience this afternoon, as our monitoring of Top 10 UK retailers sites showed. Not all sailed through with a green-light: Green is good, purple is slow, red is significant level of errors, yellow less serious errors that don’t significantly stop online shopping.

black friday web performance 2015 last2 hours

The 10 bars here are measured by running multi-page ‘Do what the Customer Does’ journeys across the 10 retailers websites: running those measurements many times per hour, each time looking for and adding different products to basket, via navigating a different route through the website.

The chart above: the small mini-graphs on the right-hand side show the performance fluctuation over the period: and the last sample timings show a wide spread between retailers  from 20-70 seconds for the full journeys! Quite a variation.

The aim of this Dynamic Journey approach, is to more accurately represent the mix of real world traffic which will also highlight issues that may impact one part of the site, but not another.

NB:  All Journeys were run from Mobile browsers.

Praise for the best performing Retailers

The greenest bars above belong to retailers who deserve a pat on the back:

  • Amazon
  • Boots
  • Debenhams
  • Littlewoods
  • Very
  • Tesco

There were two clear fastest sites, for these multi-page shopping Journeys from mobile John Lewis (shame about those error periods) and Boots.

What the bars reveal is that many sites performed poorly. Yes, most sites didn’t get in the news, and most didn’t have extended periods of 10 or 20 minutes of poor performance but the majority were not serving every visitor, every time, with a fast, error free experience.

Sometimes a site that ‘keeps on limping along’  is more frustrating to thousands of shoppers, than one that is simply obviously down for 30 minutes.

There was a lot of evidence of system overload around. We are picking here some real-world examples; not to point the finger at specific retailers in particular. More than half were doing similar things for some users in some parts of the site for some of the time:

  • Argos:  in Step: ‘View Basket Page’

    • Classic HTTP 500 server error : for https://www.argos.co.uk/webapp/wcs/stores/servlet/ArgosProductPurchaseNoJSMobile
  • John Lewis:  for a while the shop front door was preventing Journey starts,:
    • http://m.johnlewis.com/h5/index  (500 error again)
  • Boots: some categories were empty of products, now and again.  Not offering products, means users cant buy them!

Black Friday - Boots plc - empty category page

To reproduce the error above (still present for some users at 17:30)

  • At Boots home page, click menu, Shop and select Electricals
  • select ‘Boots Kitchen Appliances’

To conclude

Peak days like Black Friday are not trivial, and there’s no value in pointing the finger at others! A lack of realism when pre-testing the capacity of web systems is often to blame giving false hope for performance at peak times.

Our monitoring picked up a number of other problems on sites today that may be considered less serious. For example a site where the menu navigation was a little flaky:  some times the user could not click on the menu choices at the bottom quarter of the menu, they did not respond to the action. The user could see the choices but the click failed however it could be made to work by scrolling back up and down again. Not a show-stopper, not a server error that can be pointed at infrastructure teams, most probably a front-end framework glitch.

*Black Friday Update: 24 hours on*

Since the reports here on Black Friday 24 hours later there are now a number of interesting web performance issues to report on. Overall, the top retailers have performed much better in the last 24 hours, a lot more green is good:

post_black-friday__looking_better

3rd party tags still showing problems

  • The speedtrap tracking at Very.co.uk

was unable to cope with the traffic load at 9pm on BlackFriday evening:  60 seconds is too long:

shopdirect_speedtrap

  • Javascript libraries struggling

Argos use of the dodjo.js framework, seemed to struggle under load:

argos-dojo

To receive a monthly roundup of SciVisum’s latest research, reports and blogs please sign up below.

      Fields marked with * are required.



Please leave this field empty.

Your information is confidential and will never be disclosed.

 Also see:

Top