Which is a great question, and one that everyone in eCommerce should be asking, but even more interesting is when a  Director level person poses the question:

“How do I build and manage a TEAM, to build the Best Retail Website ?”

Because it shows a much deeper, more sophisticated, understanding of the evoloution of online retail: technology can do great things – but it needs great people to achieve that.

Some retailers I meet are engaged on a fast track growth in their online operations:  either newer companies for whom online has always been the greater part of the business, or household names who are ramping up online strategically.The challenge that I see the more effective ones meet in is recognising that to build the team that will build the best website, it’s vital to get the IT wing and the non-IT (business) wing working together.  Retailers who have not strategically realised the level of focus needed internally to embrace multichannel retailing tend to find it harder to bridge the technology gap.  (Fair do’s, it wasn’t so long ago that IT were seen and not heard, running head office systems that did overnight batch processing! ).

Online retailing means that everything you do, everything, is dependent on the technology! Bringing IT into the circle in a new way.

It’s even clearer that the technology is central when you have in-store kiosks, web-sites, mobile websites, video delivery, social network features, Twitter integration!

What’s needed is to Unite the Tribes by having a common language that both sides can understand. This provides both sides of the vital double-sided coin approach, i.e. an approach that works for both your IT teams, and your marketing and business teams.

The double-sided approach supplies metrics and KPIs for both.

Provides Business level, Budget level proof to the E-commerce Director and the business teams that:

• Customer experience is being maintained centre-stage through the hectic page of feature change, site development and visitor volumes ramp-up
• It has been made easy to pass problems and tasks to IT because they are well defined without needing your input; and proof of success is available for free every day in the metrics
• There is no need to feel powerless, and fear that problems with technology-based root causes seem to avoid solution or genuine identification

Provides  IT teams (both Software Development and Operations):

• the details and specifics of the online problems that are hurting customer experience
• specifics right down at the tech detail level, so your teams can quickly find and fix the causes of the symptoms that Business teams say are top priority right now – no more looking through known error logs, hoping to work out which ones are the really important ones!
• No more feeling that the Business expects a magic wand! That they expect you to ‘fix the website’ without saying which bit and then, conversely, seem reluctant to provide budget for the system updates you request because they want more ROI proof than is easy to provide from an IT systems perspective.

The video of Chris Howell, Director at Dixons Stores speaking on this theme of Uniting the Tribes last year at an online retail event in London, is a worthwhile investment of 3 minutes for any online retail director.

The common language you need – it has to be based on 24/7 monitoring of the customer experience on your site – the vital Do What your Customer Does metrics.

The eCommerce Director directly benefits from these measurements by use of a common language.

No more inconclusive post-mortem meetings, after poor performance during a major marketing campaign:

• the users’ experience metrics are there in black and white, together with the detailed root causes down to the specific problem pages in the users’ journeys, the specific page components that caused the symptoms
• IT teams can go away empowered to fix and improve what is already highlighted, instead of being asked to look back and see ‘what happened’

Budget dilemmas around whether to spend software dev. time on new features, or address the worst of the known bugs are easily resolved:

• There are always bugs in a  website! The question is which bugs are losing you the most money? The KPIs answer that for you – invest in those and ignore the rest
• Budget spend for things IT want: hardware, upgrades in software packages,  instead of them being a cost you can demonstrate them into a ROI
• If it can be proven that a request for budget will speed up User Journey X, or reduce errors on User Journey Y, then they can have their money.

Every online retailer is different, but a shared aim seems to be focus around the one thing that success online requires: having IT and Marketing pull together on:  continual improvement of the customer experience 24/7.

That ensures a path towards the best multi-channel delivery:  the best retail mobile website, the best retail kiosk website and the best retail website.

I wrote earlier this week how thinking ‘page’ is not as effective as thinking ‘Journey’ – but putting that to one side  – in particular I was asked about Google’s Page Speed tool.

It’s a useful tool, I’m not knocking it, but I am saying that it’s scope and limitations need to be understood. You can entirely waste precious tech team resources if you don’t.

In the pursuit of “How can we make the website faster for end users” it has a role,  but first and foremost be clear: it is a tool that makes recommendations on good things to do, in general, to speed up a page.  Note I said ‘in general’ – because the issues today in your website platform that are causing poor customer experience online may be quite different to what Google Page Speed will recommend to you!

The analogy, is talking to a knowledgeable friend about a problem your car has cold starting, and they say your should change your air filter, it looks due.   It’s not wrong advice, air filters can be a source of problems, but for your car there are so many other specific places that could be causing your issue, that the generic advice is more than likely to miss the mark.  Of course it’s quick and easy for your friend to offer a generic solution that is know to work in many cases rather than to dig deeper and actually check performance of spark plugs, timing, ECU boxes,  petrol filters etc etc and find the genuine root cause!

So the tool is good at pointing out some useful things to do that, in general, are sensible things to do,  but be aware they make very little difference in practise to your specific site’s speed problems.

The second limitation of Google Page Speed, and it’s not a bug it’s designed this way’,  is that it intentionally ignores everything about your website performance, except the shipping of page content to the end user’s browser.   That is, of course, a sensible thing to measure and optimise and track over time, but on your website, your users may be getting a really nasty user experience every time you ramp up an online campaign for reasons quite outside that scope.

In fact, in the real world most of the performance problems that we find are causing our clients lost sales online are at the website datacentre.  It’s easy to add more features and complexity to a site in an attempt to increase the richness of the user experience and end up with unforeseen quirks and slowdowns due to effects in your software in your web platform.

These are the kind of typical multiichannel retail  user experience problems that Google Page Speed can never help you with:

• Login is sometimes slow
• Add to Basket performance is variable
• Checkout page sometimes slow

So, if your tech team spend a lot of time on Google Page Speed work, and achieve higher scores when they run them against the Google tool, be aware that it may not make the difference to your user experience that you’d like – it may miss the ‘make website faster’ objectives that you have.

It’s not a tool that sits close enough to your real user experience, to really belong in any kind of KPIs for your site – for that you need to measure by Doing What the Customer Does‘.

PS
Just to be really clear, the tech factors that Page Speed measures are in themselves good things, so yes we do support sensible use of:

• Browser cache optimisation
• Reducing network round-trip delay by reducing the number of objects per page
• Reducing the number of packets per page by minimising content (minify’ing)
• Avoiding HTML content ordering that delays rendering times
• Optimising for mobile  (we have monitoring services for m-web and mobile-apps)

The most interesting ones are with organisations that have evolved far enough along the eCommerce evolutionary path to realise that it’s no good IT and Business teams sitting in separate silos. Those that understand that all teams need to pull together to achieve better online sales for their company, that their functions are interdependent and that it is through focusing on continually improving the technology underpinning the user experience that they will achieve their biggest success.

Its at that point in an organisation’s evolution, that things online get exciting. The pace of change picks up, the common language of metrics derived from a ‘Do what the Customer Does’ approach to measurement reduces friction, and overall there’s a buzz and excitement growing as sales and conversions rise. Not only that but there is a new satisfaction for tech teams that their work has made a visible difference and they are no longer just the whipping boys for website shortfalls!

But there’s still scope for the momentum to be lost through some common pitfalls.

The downside to the tremendous pace of online change is that beliefs based on old knowledge are still hanging around in the dusty corners of boardrooms and server centres up and down the country.

Let’s throw a bit of light into some of these dark corners.

Page resolution speed is a vital element for good Sales Conversions.

Sure: new dialogue opens up with Tech teams around speed of User Experience, and early initiatives gain success through measuring the performance 24/7 of the multi-page Dynamic User Journeys that reproduce what real customers do.

However… someone, somewhere starts to get obsessed with the speed of individual pages in isolation to the user’s entire Journey. Complex multipart journeys need to be considered as a whole as well as their component parts.

Of course, once you are measuring 24/7 multipage User Journeys, one of the key metrics it will quickly provide, is which of your page types slow down the most under load. And of course, knowing that fact, its good to aim to speed up that page.

Speeding up a page is good of course – but once you’ve achieved some gains, you need to go back to the Customer level, to the user Journey: and see what the next issue is not stay blinkered down at the one page basement.

But, whether initiated in Marketing or in Tech the focus can shift from the customer, from the multi-page User Journeys that they follow: and can become all about ‘go faster stripes’ for just one page.

A page that resolves quickly but does not meet user needs as part of a journey in other ways will not perform well in terms of results.

A page is not a journey.

It sounds obvious, right? But it is very common for people to focus on an isolated step of a journey – for example the product page on a retail site – but this is often the just the first or second step in a functionally complex multipage interaction and as we know from a wealth of usability data it is often the later steps such as checkout processes where users are more likely to have a lower tolerance for problems.

But you’ll see forum discussions, full of requests on ‘making my page faster’, how to ‘minify’ my page, tricks to improve caching of my page and so on, but this shows a lack of understanding as to how sites work now.

Focusing on one page, is rather like going back to the early days of the Internet – when pages were simple, before CMS and inventory management systems became commonplace. In that world, the performance of the page was indeed not determined by any other page- it stood alone.

But in the modern world of rich pages dynamically put together in real time – that is no longer true.

To take a simple example let’s look at the CSS and JS files that a page needs: if you have already visited other pages at the same site, your browser will have those pages in cache already, so the next page is automatically lighter. It doesn’t just matter where the user goes, it matters how they got there, in other words, it’s the journey that counts.

One page is no longer just one page.

What’s more – with the advent of widespread AJAX and increasing HTML5 – the page actually changes as you move around it.

One of the most important links on any page on your site is at the end of Checkout, the ‘Confirm Purchase’ button. On many sites you cannot reach that button until you have filled in some previous fields; and in the process various parts of the page may have opened or closed in a “concertina”.

So you cannot optimise the performance of that page, without measuring all those steps within the page. All the little AJAX server connections that may happen as the user acts.

Increasingly many pages are made up of a number of disparate components, sometimes provided by, or displaying content from, external sources such a media players, ad servers, news tickers, twitter feeds etc. What appears as “a page” to the user is actually more like a “housing” for several “sub pages” that update at different frequencies, from different sources, hosted in different places. In the course of completing a user journey some of these components may be more important that others.

So optimising “a page” is, taken out of context, quite the wrong focus.

Realism is vital but it is not simple.

Yes, it can be a simpler world, when you only care about one page in isolation, and ignore meaningful Journeys – the web page optimisation route.

Yes – it can make Load testing projects much more bite-sized.

For data to be meaningful and actionable and all those other good things it needs to tell us important information about the real world. The snag is that the real world is not “easy” or “simple” it can’t be summed up by “one simple top line number” no matter how handy that would be for meetings and presentations.

Only the MultiPage Journeys reveal how Fit for Purpose your site is

If you ignore the Journey numbers and look only at page numbers: you are no longer measuring Customer experience, and can easily end up with a single page type that is 20% faster, but the overall User Journey no faster at all because some code optimisations have inadvertently slowed other pages!

Or a new sporadic error type may be appearing after a small tech change, so that for some of the boundary conditions that pages only experience when they called from other pages in certain paths.

Those errors are losing business from real people – but you won’t know if you only measure page performance.

A good number of the projects we pick up, clients have already some page-based website performance monitoring in place – some of the tools they have subscribed to and used before coming to us have been single URL focused, and given them a false sense of security that is shattered when they see the actual experience their users are getting when our Journey based measurements start to deliver data!

It’s not about the Page – it’s about the Customer Journey!

Contact us for more information or to discuss a demo or trial.

I’ve been very impressed with the team here, how they’ve been able to juggle such a busy period whilst at the same continue development of our testing systems, to allow us to model even better realism for our multi-channel retail projects.

There’s been some fascinating graphics of realism too, as we’ve evolved our approaches.

Firstly, our continual development of our dynamic User Journey based approach has not only benefited 24/7 monitoring of ‘Do what the Customer Does’ dynamic routes through a site: but has provided the same realism gain to our load testing too.

Just last week we helped nail down a problem that was losing sales – doing a search in the usual top right search box on this retailer site, would produce an error response for certain keywords chosen – words that whilst not in the top 10 search terms on their site, were words directly related to the products being sold.  It’s only because our dynamic, randomised approach was able to try so many combinations that some of the slightly ‘ under the water line’ holes were spotted.  And it helped that our technology is flexible enough to allow us to run Journeys against their website and their in-store kiosks too; that provides better realism of site conversion across all channels.

But the brainy engineers here just recently have been looking at ways to generate website Load tests with even more realistic spread of virtual users.

The work was triggered by the fact that more and more of our clients are using the Cloud to host parts of their online store, and we’re seeing a big jump up in the peak customer capacity of some clients, the ones who get their software properly Cloud ready.  Which means we are adding to our own load testing infrastructure: we run both dedicated and cloud based test servers.

So, controlling more and more servers during a load test, aiming for higher traffic peaks and handling the complexity of cloud servers variable capacity has thrown out some intriguing challenges.

One of the main strengths of the cloud model is also the cause of one of the major differences involved with testing on a a cloud based system, and that is that your servers are not just “your servers” anymore.  While there are many benefits to this what it means in testing and performance terms is that they cannot be relied upon to give you exactly the same amount of power and performance in each instance and will sometimes not be able to consistently deliver the amount of “oomph”required. However, this is exactly the realism you need, as this is exactly the user experience problem that can occur on the live site.

Once you then throw in some Think Time realism  per page we are starting to see load tests that need substantial ramp-up times simply to ensure a sensible mix of users across all stages of activity – to ensure realism.  You have to work harder to avoid ‘bunching’:  where at any one moment of time too many virtual users are active in one task or page, and not enough are active in other areas.

Imagine starting a 100,000 virtual user load test made up of a number of User Journeys that, although following dynamic randomised routes, all start at the same place, the home page.  Without ramp time cleverness your first couple of seconds would be 100,000 home page requests and nothing else! And then for the next period there’d lots of activity all over the site but absolutely zero on the home page, until the Journeys start to finish and new virtual journeys start the process again at the homepage !

So ramp-up is vital to get a realistic load on your site, to spread users across the journeys in use.

But… ramp-up time is wasted time as far as measurement of your sites ability to handle seasonal traffic peaks is concerned because for at least two thirds of the ramp up, your online store is not breaking out into a sweat.  Everything is smooth, no errors are thrown, no pages slow down, and all the lovely graphs of server utilisation show nothing much happening.   Wasted time for all the engineers on duty.

As most organisations want their load testing out of hours, to avoid impacting real users,  there is only a limited time window over night to get in as much measurement and as much evidence of things needing to be fixed as possible.

Some nights, we reckoned we were spending 20% of the time, thumb twiddling in ramp-ups.

So, our clever team have been experimenting with ways to shorten the Ramp up times, but maintain and even extend the realism of load testing, the user spread.

The problem of lost time during a traditional  extended ramp-up – is shown clearly in this graph: the ramp-up of virtual users in pale blue is clear, with the outcome that it’s nearly 300 seconds before results start to come in (the green line) – between 200 and 400 on the X-axis before all the bases are filled and Journeys start to finish .

With lots of think time at each step of the Journeys making up this test it added up to about 4 minutes before the first starts, got to the end.  This is a low volume example: on some major client projects complex think time needs sometimes involved 30 minutes ramp-ups!

But credit where credit is due, this Ramp-up did a  good job of setting up the users on bases so that a constant rate of about 80 users per minute (it’s a low volume test) are finishing Journeys, once the ramp-up was done.

Dynamic Start approach  -  this alternative, has no ramp-up time at all, but achieves a mix of virtual users spread realistically instead by dynamically moving and adjusting think time: optimising the spread of users within a very short space of time by modulating think-time for the first run of each virtual user in the 400 concurrent virtual users that are quickly fired off:

It’s clear that this time the green line showing completed User Journeys per minute, gets up to the target 80 level much more quickly – within 30 rather than 300 seconds on the X-axis.  Note: always in load testing whilst it’s easy to control what virtual load you put in, the most important metric is how many User Journeys can be completed per minute: that’s your capacity measurement that your merchandisers and sales team want to know can handle their forecast sales figures.

So the team were pleased by early experiments on that approach.

Dynamic Ramp approach

Results were already of interest, but it’s always good to have choice, and natural for clever software guys to makes things even better :<) so the team also experimented with a 3rd approach to the realism challenge – called for convenience Dynamic Ramping:

Looking at the graph, this has similar properties in terms of achieving rapid balancing the of virtual users, but as is just about visible on the graph, it does use a ramp-up time, albeit a very short one, and this time with a different approach to adjusting think time to fill the bases: based not on moving think time between steps, but on modulating it to best fit the first wave of 400 concurrent users across the journey steps.

Conclusion

As eCommerce is a fast changing technology world I’m always interested in how real business benefits can be gained from clever software guys – so apologies if this blog got a bit deep technically.

But if your company already use our services, or you have a colleague who’s worked before at a retailer who has, be sure to ask and find out what deliverables were gained. Although it may be angles that they’ll bring to mind other than Ramp Up optimisation like the above, it’s very likely to be realism based features that gained the benefits.

Certainly across all the new clients this last quarter who’ve upgraded from simpler approaches to trying out same of ours, the common thread has been their desire for more realism, so that they are prepared with better facts to optimise their website conversions.

A B2C website client, Inghams Travel, has been so impressed with our Dynamic User Journey monitor service over the last year and more, that they decided to put up our logo on their website footer, to let their customers know their experience online is important.

It’s a first for us – and extremely unusual for a B2C website to badge up with one of their B2B suppliers.

So thanks to the team at Inghams – hope you sell more ski holidays this year!

The travel sector does have more challenges online in terms of customer user journey monitoring – having a dynamic inventory, with packages coming and going very rapidly, a simplistic monitor approach that just gets the same list of pre-defined URLs each time, just won’t find the gotchas that real users are exposed to.

So thanks to Inghams for the recognition – and thanks also to the many clients who’ve been praising the team here, for how much our services have been able to impact your bottom line – you’ve kept us all busy  the last couple of months, notably with the travel sector and retailers’ pre Christmas website load testing.

To achieve full multi-channel understanding measurement of dynamic User Journeys on the public website have been complemented with Journeys on the internal  in-store  shopping Kiosks and ePos systems as well as with Mobile User Journeys on the m-web mobile website.

But things are more tricky for clients on shared eCommerce platforms – the likes of Fresca,  Venda, IBM WebSphere and so on – that make it tricky to be sure your website will be ready for big traffic peaks.

Hosted eCommerce solutions are not suitable for every retailer -  in theory they may be less flexible than a roll-your-own web platform, but the benefit is removing the maintenence and mangement software and infrastructure  from the retailer. Ideally this is supported with a software base that is already proven, and an infrastructure that has near perfect downtime, and no outages.

That kind of reliability promise sounds like a good strategy to avoid the damaging social media sentiment effect that errors and slow downs can cause throughout your multi-channel enviroment as a result of a negative user experience on any part of it, whether it is the main website, the m-web or the in-store Kiosks that underperform.

There are many hosted eCommerce solution names to choose from, such as Venda, BT Fresca, Web Sphere, Hybris, Actinic, Advanysy, Screen Pages, Snow Valley and TradeIT (redTechnology), all of which have their own specialities and focus areas.

However, there is one area of brand damage that can result from using a hosted eCommerce solution over which you have no control – and that is the issue of  Capacity or handling traffic peaks.

It’s not easy to measure Capacity of a shared platform.

A shared platform could be your organisation’s use of Cloud hosting; or a hosted eCommerce solution – but there are some fundamental challenges that are caused in their use.

Compare the situation of a regular eCommerce multi-channel site – with one on a shared platform.

With a regular eCommerce multi-channel site, you own (or rent) and are in control of a bunch of hardware – the web servers in your web farm.

If you want to do some Load Testing – to get hard evidence for your Business teams as to whether your systems can handle the upcoming traffic peaks,  Xmas or a sales season: then you can appoint a load testing team to provide the work.

And typically, so as to avoid upsetting real users on your site, you’d schedule any testing to be done out of hours – or at least at your quietest, least profitable periods – choosing a time when there as few genuine shoppers on your site as possilbe, so that the slow down during load testing has the least impact.

So far so good:  because you’re testing out of hours, you know there is essentially no one on the site, except your load test virtual traffic.  When the load testing finds the throughput capacity limits of your site on your own hardware then you know, that is the limit, there’s no doubt or uncertainty.  The Load testing will  show  the capacity of the site as a % of the ‘Last Sale Traffic’  from the last real peak period:  if you’ve designed your User Journeys well to reproduce the traffic cross section of Browsing, Searching, Adding to Basket and Checking-Out. It will also show the max throughout Journey by Journey for each.

Now compare the same Load Test, when your site is hosted on a shared eCommerce solution.

Testing out of hours – means that not only is there virtually no load from real users on your site – but there is also no load from visitors to the other retailers hosted at the same vendor.

Your provider surely has enough capacity to handle all their clients’ traffic in parallel: at least for normal, non-peak traffic levels!

So when testing out of hours, all that infrastructure which is enough to support their 10 or 20 or 50 clients; is available to your load testing alone.

So it should be very easy for your load testing to show the hosting can handle just your traffic peaks! Because you’re not competing with users from other clients of your vendor.

But… that doesn’t prove that there is sufficient capacity for your future real-user traffic peaks, when the other clients at your solution provider will also have their own traffic peaks too.

Another issue may be that another client of your provider decides to carry out their load testing at a time that is typically quiet for them, slowing down the site and taking up capacity, at a time that is normally very busy and valuable for you.

So there’s added complexity here when you put in trust in a supplier due to looser brand control, you are exposed to having  a nasty user experience in your most important sales seasons, with the brand damage, impact on sales and hurtful social media sentiment effect that can follow.

Don’t Panic

In the inimitiable words from the Hitch Hikers Guide to the Galaxy:  Don’t panic.

If you’re using a hosted eCommerce solution like Venda, Snow Valley or Fresca etc – there’s still a lot of value in load testing, to uncover technology blips and bottlenecks in advance of the start of your big shopping seasons.

A project we handled this month for one of the biggest UK Retailers, using a half-way house managed eCommerce solution proved of genuine value  when we showed hard evidence that one of their 3 major brand sites could only handle 1/4 the traffic of the other two on some dynamic User Journeys for vital shopping tasks.

The hoster, knowing that expected peak sales traffic was to be hightest for that brand was then able to work with client to plan for this and moved it to it’s own, better specification, web farm infrastructure. However, it’s dangerous to assume that just because you have discovered one issue that it will always be plain sailing because some small set-up errorswith the new set up meant that the resulting performance was actually lower in practice than on the older, lower spec system! Always test – even your newest kit!

Not discovering that until the actual Christmas sales rush would have allowed a nasty impact on sales across the multi-channel offering.

Your Multichannel systems never stand alone

Of course as multi-channel offerings provide the various interfaces for mobile users, PC users, in-store users  there is a trend that more and more of your page content is actually served by seperate 3rd parties to your main content.

So, for example, even with a hosted eCommerce provider, it is very common that your product images are being served from a seperate web farm managed by Scene7 or similar.

Or your social media pages.

Or even your m-web entirely: as some retailers have  sub-contracted their m-web out entirely to specialist m-web providers.

So all those reason – the User Experience that your technology provides when shoppers reach one of your multi-channel sites depends on several web farms:  so capacity planning  and load testing needs to cover those bases too: not just your main eCommerce solution.

The panel ran an interesting group session after lunch, involving a bit more energy as people moved round the room to reflect where they saw themselves on the continuum from being pure QA folk, or pure software Testing, or a bit of both. More of that later.

The Keynote from Jason Tylor at Spec Savers was fascinating.  It seems that off-shoring there, and often elsewhere,  results in a pulling back from an agile approach. Jason was talking on Managed Testing, using 3rd parties, and showed some major benefits from the approach, cost savings not being the least of them.

This recalled the session earlier in the afternoon on the topic of where does testing and QA really fit together.  Consensus seemed to be that:

• Testing is a  vital stand-alone activity,  what it achieves for the business is dependent on the effectiveness of the spec that the software is being tested and validated against.

• QA (Quality Assurance) was felt to be a wider role that involves trying to engage the rest of the business in the need to think  about quality and fitness for purpose. This is needed from right back at the early product concept stages and all the way through development,  not just at the final stage of software testing to prove that the new software does indeed do what the spec says it should.

This led on to – what is “Quality” when you are trying to encourage Quality thinking to the non-QA teams across the business.

Quality should sure as hell not be something that only QA care about!

The session I did with Den Fitzpatrick from Dixons Stores in the morning had also hinged on this issue of trying to get the whole business to engage. The header was ‘Uniting the Tribes: Bringing IT and Business together through Testing’ .  It showed how Dixons website is managed via a complex mix of different teams at different countries, and how the use of Dynamic User Journeys as a common language, provides a unifying force that cuts back the usual inter-team friction andgreatly reduces the difficulty of getting hard facts about what is happening on the website.

Above all  it answered the same questions raised in the afternoon:

• “What is Quality?”
• “What is the definition of quality that Business and IT can agree on?”

The Dixons answer to these issues is that “Quality”  on a Multichannel retail website is a necessarily a much wider concept than each individual component matching its specification. It is whether or not every aspect of the site and technology works in concert to provide the best user experience 24/7.

Dixons use SciVisum monitoring and measurement across a number of different multi-page Dynamic User Journeys, that ‘Do what the Customer Does‘.” to understand how performance is affecting user experience. There were lots of screen-shots of our Portal showing how the customer experience measured this way really does make an easy to understand measure of Quality that makes sense for the most senior Director, down to the most-hands on tech team on support.

Even better, it supports the QA Manager in his role to get the interest of the software development  team to extend beyond just releasing the code over the wall and moving on to the next project.  Often 90% of a software tester’s time is involved with the developers, and the one area that is really hard to measure in any meaningful way before release is areas of non-functional performance, for example  just what difference with the new software release make to speed of delivery when the site is busy?

Now, with User Journey performance being plotted on large plasma Wall Boards, all staff can see at a glance if today’s release made anything worse (or better)!

So, it was a good days outing at Iqnite this year. Overall the message rang out  that QA guys want to get more traction from across the business, and a common language of user experience measured 24/7 by dynamic User Journeys provides just the glue needed between teams – everybody knows that when all Journeys go green, that’s the definition of Quality for a multi-channel retailer!

But this week two specific instances really prove just how important that appraoch to monitoring methodolgy really is.  In fact, now that I think of it there are actually three instances if you count our remarkable Microsoft Azure performance testing results.*

New clients obviously don’t have experience of working closely with our Client Support Team, beyond the mid trial data exploration session at the time they are making their initial supplier choice,  so the key factors for them are the benefits of our approach, the insight into what’s really happening to their users online, and our intuitive portal interface that providers richer data for a wider audienceof  both Business and Technical users.

We do mention at planning meetings with new folk that our clients rate our Customer Support as one of our strengths, so it’s always nice to hear more proof.

Firstly, an existing client renewed a long term contract with us – and so impressed are they with how our services have helped them improve their website effectiveness, that they offered to put our logo on their website, with a ‘monitored by SciVisum’ message.

That was astounding! End users shopping on retailers websites have most likely not heard of SciVisum, we are the invisible protectors of thier user experience.

However, if they come to realise that many of the sites they use that provide that level of service and reliability all have an underlying point in common, maybe they will start to look out for it. In any case, all these consumers may well be people who work in IT, or Operations, or eCommerce or Online Marketing   – in fact it’s getting harder to find a job these days that has no relationship to online services at all!

So for a client to be so proud of us that they want to put our logo on their B2C site – that’s an wonderful accolade. And 100% unexpected.

The 2nd proof this week of happy clients, was this email exchange I was forwarded today.

This client has been with us, oh maybe 5 years.

Our comprehensive monitoring approach had flagged up to them a problem that might be affecting just a small percentage of their users – based on some funny browser compatibility issues.

But our fanatical about support team (with nods to rackSpace!), went the extra mile, and gave the client chapter and verse.

If we’ve helped them to plug a hole that was leaking 2.5% of their users – they join the ranks of other website monitoring clients where we have achieved similar gains.

This is what the client emailed back.

Thanks Sarah,
This was actually a  good catch for us. We have our developers working on a fix for this now and I think it was affecting about 2.5% of our users. Well done to your team for catching it and thanks for the extra info!

John

On 23 September 2011 10:00, Sarah wrote:

Hi John,
I understand you’re interested to know if the redirect is an issue occurring on specific versions of Firefox -
the tech team have done a  little more research:-

and have managed to test the site on multiple versions of Firefox (1.0, 1.5, 2.0, 3.0, 3.5, 3.6, 4.0, 5.0, 6.0 and 7.0Beta), any version of firefox pre Firefox 4 suffers the same redirect problem. However, if one runs the earlier versions of Firefox with javascript disabled we can avoid the problem as well. Firefox 4 and upward used a new javascript engine as compared to Firefox 3.6 so it’s entirely possible that something in the javascript and the way its being handled is causing this problem with the older firefox javascript engines, where the more recent versions are able to deal with it better and not force an unexpected redirect.

Another point of difference between the two is all previous versions of Firefox were using Gecko 1.x as their underlying engine where Firefox 4.0 and up have upgraded to Gecko 2.0 and up. However as this is only the layout engine the guys  don’t suspect it could be causing this issue.

If you need any more information give me a shout.

Coming soon we’ll have a fantastic account of how we sorted out a significant issue for a client this month with a site that uses the  Microsoft Azure cloud.

For now suffice it to say that it was only once our website load testing against the Azure cloud had conclusively proven the performance issue was with the specific API call used that Microsoft agreed to pile in lots of free engineers to look into the issue with the site. Confidence, or lack of it,  in your cloud supplier is an increasingly important issue. The cloud being as elastic as it is, performance under load is not always what you’d expect.

Contact us if you are experiencing your own Azure issues at the moment, or suspect that Azure may be part of a problem, and we maybe able to give you specific tips or carry out some testing to conclusively prove the case either way.

When talking to organisations about what SciVisum does, and what makes us different, we are always keen to get a live trial of our monitoring suite or testing services running on their own sites, using their own data.

Proof of Value as it’s sometimes called. Show, not tell, if you prefer. We can tell you how it works, and what it does, and why and how it does it, but in the end it’s like the Matrix. To misquote Morpheus “No one can tell you what what SciVisum does, you have to see it for yourself.”

Sometimes, when we suggest this, people  think that the return on their time may not enough to justify the hassle. People are familiar with the tools they currently use so, at first, they understandably assume that “website monitoring tools must all be pretty much the same”.

If they get little added value beyond alerts and basic “up or down” information out of the tool they currently have, they assume any other similar sounding tool would deliver much the same. It would be useful to have, but not business critical. It would provide some numbers for the weekly report, but not strategically vital business intelligence.

It’s the Black Swan Syndrome*.

So in some cases they’ll politely turn our offer down, saying, ‘Thanks but no thanks, I’m sure your tool is similar to what we have, but it’s not an area I’m looking to review right now’.

That’s probably why, if you take a look over our client list, you’ll see that it’s made up of people who really care about their online properties. The innovators, the forward thinkers, those out there on the cutting edge. They come in all shapes and sizes, from all kinds of industries, and with all kinds of brands and business and models; but what they share is an understanding that decisions based on data, on actionable intelligence, are what will keep them out in front.

Henry Ford once said: “If I had asked my customers what they wanted, they would have said a faster horse.” Which is to say that if you don’t know something exists how can you know if you want it?  Imagine you needed a way to travel 100 miles and were expecting a fast horse and then someone gave you a Ferrari…

Now imagine you were expecting an alerting tool that does the kind of basic monitoring that checks through a list of pre-defined URLS and tell you if it can reach them…and then someone showed you  SciVisum’s Dynamic User Journey Monitoring!   Thinking in terms of horses and sports cars, then, they’re both perfectly adequate modes of transport, and there are valid reasons for having one or the other – but they are not equally appropritate in all situations!

Mostly people are aware of, if not actually using, website monitoring  of some kind now. A few years ago a handful of techies might have been aware of the arcane sciences of measurement, but no one else was concerned.  As online grew into eCommerce and then into complex multichannel activities it has quickly become of concern to the entire business.

Of course sometimes a person has heard of what we do and comes straight to us, sometimes an individual who has used our monitoring moves to a new company and encourages their new team to trial it there as  being an improvement on what they have, but most people we do trials for are currently using another, more traditional, system.

Looking back at the new clients we’ve started working with this year, I noticed that most months we had a significant number of organisations that moved from their existing monitoring tools and approaches to embrace ours.  Inquiring deeper, I found some reasons, the unexpected unexpected, that these clients had in common.

Looking at the most recent such new user,  their proof of value started out in a common way with a company wide initiative  to review Best Practice around all the suppliers. An IT Change Manager had pushed the Operations Team, as the main users of the website monitoring tools, to have a look around to see how their current monitoring arrangements compared with the best out in the market.

Something obviously happened  during the trial with us, because they ended up signing a multi-year deal, with several months overlap on their existing simple monitoring contract.  There had been some kind of a Eureka moment.

This Eureka event, a shifting of expectations of monitoring, seems to be a shared theme amongst the clients upgrading to use our services.

For this client doing a Best Practice review,  the trial was running against their site a couple of  the dynamic user journeys we’re best known for.   There were a number of problems occurring on the client’s website that our tools found, problems they were aware of.  Known unknowns.  Things they’d not fixed, but were still tracking down.  That didn’t excite them.

But more importantly, our service highlighted problems that were new to them: unknown unknowns.

We found real errors that users would suffer, but their existing monitoring services and tools were, in contrast, showing all clear.  One thing in particular, was a recurring problem, that didn’t happen every time, maybe 1 in 30 or 50, was our virtual user being logged out and sent to the home page, when they were actually 5 or 6 pages into a purchase journey, with items in the basket.

Suddenly the team who’d initially just been looking for a comparison of simple alerting tools, was confronted with a problem in their technology, that they’d not expected – an unknown unknown.

That was the eye-opening moment – it can be a shock to discover that there are identifiable patterns of errors occurring on your site, that you were not aware of!    But this client wasn’t tempted to go into denial; they immediately saw the benefit of getting visibility of these sporadic errors, how the saved HTML and JS data for them that our service provides would help rapid trouble shooting, so they could fix them with least hassle.

Across the board, all the clients who moved to our more sophisticated monitoring model had a Eureka moment concerning our Journeys. Like the users above they realised that it was the dynamic user journey approach which was bringing to their attention these real errors that were impacting real users.   They realised the benefit of journeys that look in real time into the page as served in real-time, and choose from within a choice or link – as being better able to find the real issues that do impact real users.

Talking to the Client Liaison team, they reported other common Eureka moments when clients are trialling:

• Non functional user experience problems -  the journey works but there are say missing images of products, but only a few products from the total inventory. If nobody can see it, nobody will buy it!  (see our focused inventory monitoring approach)
• Consistency of user experience – “so you mean sometimes it is really fast but actually 5% of my users can take over 2 minutes just to add something to the basket?”
• Realistic catalogue coverage with random selection – if you monitor with only a simple, static URL ‘journey’ which by definition therefore always buys the same product, say tennis shoes, then you only know for that the processes and content involved in buying that one item are working/not working correctly.  You optimise it page by page and anyone buying tennis shoes has the most wonderful e-commerce experience.  But, for all the other products on your site where there is a choice of  colours, for example, for each product (that tennis shoes does not have)  the user experience may be poor. The system may have a bug only for certain product SKUs, or a problem with certain types or categories, or with content and data uploaded at a particulary point in time, it may be that in another section  the “Add to Basket” button does not show! Taking live, random samples from across the product catalogue to show a realistic, authentic, experience provides much more valuable data than an artificial reproduction of a journey stuck at the time it was created.
• Realisation that current, static URL monitoring, is actually following a series of URLs that not a single real customer follows! Missing Redirects is the root cause.  A static URL list may be accurate when first set up, but as the site evolves, it’s common that redirects are added behind the scenes.  So whilst real users on your site, may have their browser redirected 2 or 3 times between 1 page and the next (redirects are invisible to the user), the basic monitoring they have in place is missing out key redirect steps, so no longer doing what users do.   Errors in the code behind those redirect steps will impact real users, but won’t be picked up by static URL approach. Similarly for Ajax sections within a page – that Static URL monitoring will often overlook entirely.
• Non user-affecting issues, that upset analytics accuracy. For instance “oh, it’s just a missing third party tracking script” – but if that keeps breaking the business has no reliable analytics/conversion data to use!!

The Black Swan problem

“Not long ago people could imagine only white swans, because white swans were all they had ever seen. And so people predicted that every next swan they would see would be white. The discovery of black swans shattered this prediction. The black swan is a metaphor for the uselessness of predictions that are based on earlier experiences, in the presence of unknown unknowns.”

Also see:

• Wikipedia – Black Swan theory
• Book:  Taleb, Nassim. The Black Swan.

It’s a no-brainer ROI wise, that eCommerce retail sites need to prepare for the Christmas rush, there is simply such a high value of sales at stake for most players.

Getting the IT eCommerce technology ready is therefore a common activity – but sometimes we see Load Test planning activity that is missing the boat – especially if the business teams have rather left the IT teams to it, and to have not provided helpful targets as to what is required.

First Tip: All too often, eRetail web site Load testing is done at too late to be properly prepared for Christmas.

You need to allow enough time after the testing is completed, to work out if any remedial work is needed, and fit that work in before the Christmas traffic starts to ramp up in November.

Hence why August and September are our busy months for pre-Christmas load testing projects.

So maybe your organisation is well organised, and does run website load testing before November.

Are you suffering under a false sense of security -  due to the fact that the testing is done a very basic level, such that it does not represent real usage at all?

So our second tip, is to ensure that when you check your site early, you are genuinely checking what the users do – otherwise you’re missing out the first check, and the second check will be your real users finding it hard to part with their money on your site in the real Christmas sales!

The most common form of, let’s call it ‘bare minimum’ Christmas load testing we see is where the IT team have been given no specification instructions at all, so have understandly written a test spec from their perspective, not the end users.

So they will have defined 3 or four different kinds of pages:

• product page
• category list page
• home page
• etc

For each page type they’ll list a few URLs – maybe for product pages, they’ll make a list of URls for hundreds of products.

And then load tested by generating traffic that hits those pages:  probably in the percentage split between them, that is based on some web analytics data of the real world split between the different page types.

The IT team may even have done testing using free software tools, as the Business didn’t quite allocate a clear budget – credit to the IT team for the doing the best they could with the resources available to them

The problem of this approach is two fold:  firstly that it misses out key blocks of functionality on the site, such as Search, and the vital CheckOut functions of Address lookUp and etc.

But more importantly still, no real user just hits a few URls at random, they follow a User Journey route through the site, to achieve their goal, and click on links from each page, to get to the next.

So your load testing also needs to have virtual User Journeys that click through the site, in the same way.

That requires your load test technology to look into the page contents at every step, work out the links available, and follow on as defined in your test specification.  E.g the most obvious example of dynamic choices is that after a product search, the virtual user should choose a product at random from the list offered, and not choose hit a pre-determined URL  for a product that may or may not even show in the search!

To kick off discussions between Business teams and IT teams:   these 3 Journeys are a good place to start agreeing a specification:

• Find a product and Add to Basket – (using the navigation menus – and choosing random categories and products each time)
• Search for a product and Add to Basket – ( not using navigation this time: the search phrases to be randomly chosen from say 100 real world examples)
• CheckOut – exercising Address Lookup (so using a set of random port codes and etc).

In conclusion, if well before Christmas you check your site, confidence across your company in the ability of the technology to support your sales targets will grow.