Magento load testing has become an interesting area, and one where the lesson of ‘premature optimisation’ does matter

Magento Open Source eCommerce

Premature Optimisation is the principle, in software development, that you should not optimise your code until that part of your code has already proven to be the bottleneck.  It’s kind of a scientific, evidence-based, approach: don’t try to guess in advance where your code will need lots of optimisation, and where it will need a little.  Just build it and find out. That way you can optimise what’s needed and be sure you didn’t waste time on optimising something that was not an issue.

In the eCommerce space Magento is a widely used open source platform with a vibrant market of extensions and community of developers to build custom features for you.

It means that, although there are loads of sites all using Magento, that they are mostly all different due to having different combinations of extensions and customisations.

They will all have different Magento performance problems!

So it is that keen tech teams will often, with the best of intentions, spend time making certain changes or bolting on extras such as varnish or cool caches etc with the aim of improving performance.

Only to find the performance may still not be ideal.

A company like us is then sometimes called in to get some new evidence, by doing some genuinely realistic load testing. When we use our classic approach of ‘Do What the Customer Does‘, rather than just an artificial website load test, the load testing will often identify a bunch of issues in components  that ‘in theory’ are there to make things faster, but in reality are making them worse!

This could be because the component:

• addresses a problem that is not such an issue on newer Magento versions,
• is configured  in a way as to undermine its effectiveness  (a way that may have worked a year or two back),
• conflicts with custom work or bought in extensions.

These issues are particularly crucial when planning a Magento migration. It’s so easy to introduce Magento performance problems when making the kind of changes that are natural in a  migration:

• upgrading an extension without thinking about it,
• doing database memory allocation differently,
• etc.

Magento performance consultancy, we find, centres round the Best Evidence: the Magento load test.

To conclude: the fastest way to optimise your Magento performance, is about a journey that starts, and continues, with load testing.

And whilst this is true for ATG performance consultancy, or .Net performance consultancy or whatever, it is particularly true in the Magento space due to the mirage that it is a standard platform, when, apart form the simplest site, it never is.

Appendix:  Lists of Magento Tuning  Tips and Tricks

In the car tuning world, there are hundreds of things you can do, from fuel additives to new shocks for faster corners.

But for your specific car, engine, model year, customisations etc you wouldn’t google for them all and try to work your way through one by one!  No, you’d take the car to a race track, and try it out, then you’d put the car on a  ramp and measure raw engine directly, and only then would you start to plan what specific upgrades make specific sense for your specific situation.

The same approach applies to your Magento performance optimisation too, because look how long the list of possible performance improvements are:

• 3 dimensional: Process/ Transfer / Render
• OS optimisation
• Web server config optimisation
• HTTP header, eTag, cache,expiry
• database
• accelerator extensions
• Magento cache  /clearing
• Magento CSS/JS merging
• enable parallel downloading
• remove unused Magento modules
• consider flat catalogues
• PHP compilation and options
• zero tolerance of 404s
• Magento Compiler module
• use a CDN (or homegrown one)
• Use varnish
• HTTPS audit
• SSD or memory file-system usage
• Apache or nginx or Litespeed
• Lazyload
• Log less !  (ouch, some folks  risk making magento trouble-shooting very hard, if they do this)
• load balancing
• database replication
• magento clustering

If you are currently wrestling with a broken down Magento installation then give us a call. We know a man who can!

Putting in your own weekend time means that folk are there to be active, participate and learn from each other, and the variety of talks on offer reflected that enthusiasm.

A proud moment too when Shish from my SciVisum team did one of the most popular talks – on Web Performance optimisation – and there were some other good snippets.

The event brings in folks from a range of disciplines and backgrounds. There were quite a few photography sessions, such as Macro Water Droplet photography, as well as Chris Coopers great session on web security. There were “Deep Tech” sessions on Field Programmable Gate Array programming alongside some wider societal discussions such as ‘The End Of The University’ and ‘How To stop Seething And Love My Customers’ !

The entire list of sessions can be seen here.

For me a personal high spot was bumping into an old friend from over a decade ago, Tom de Havas. We’d once been neighbours in the same office office block, running software companies and discussing  software languages, R&D management, meta physics and more over long coffees!  The company he founded, Nexus Alpha, played a key role in bringing railway timetables, or rather real-time changes due to incidents etc, into the 21st century!

The SciVisum Team At Barcamp

Comments from SciVisum Team Members who attended Barcamp3.

• Shish (mentioned above) did a talk about keeping website performance high while scaling from a joke between friends to being one of the world’s largest online communities (without a similar increase in budget). The slides he used can be downloaded here.

• Greg says: “I saw a wide range of interesting talks on macro photography, multicast video streaming, insane website scaling, security, retro games and more. Was an inspirational weekend which has made me want to pick up my camera and soldering iron again (on top of the usual keyboard mashing ).”
  He also did a talk on his (ongoing) port of X11 to Javascript and all that entailed and you can see his slides here.

• David says: “I did a discussion about side coding projects and coding challenges.  It turned into more of an open discussion into ways to volunteer time to do coding challenges, codesprints, community projects, etc I also went to talks about the raspberry pi cat feeder and catcams with arduinos, both of these are actually web apps! Sessions I attended ranged from hardware focused to web and business concepts (Why The Semantic Web Will Fail by Colin Johnson was particularly interesting as it had some good questions about the way advertising works – or perhaps doesn’t work – on the net).  A good talk on talk on security on hacking focused on social engineering rather than breaking computers.Shish’s talk was one of the best.“

• Debora says: “I volunteered this year in Barcamp 3. I enjoyed the mix of techie and not techie talks. Also good mix of useful stuff that you can apply to your work/life and crazy stuff that you should never apply! But the best, talking to new, learning from new and different points of views, about things you have never seen before, a really collaborative space, crazy philosophical discussions about programing languages, pizza and beer. Really good experience!“

Silicon Canterbury

Canterbury is becoming more and more a great place for software development and internet projects. As well as the annual Bar Camp there is the monthly Digibury.co.uk event, which has been running for a year or more.

And last month,  the enthusiasm generated by the software industry here spilled out into a go-kart challenge,  organised by Chris at Kempt.co.uk:  sadly the three SciVisum teams didn’t make it to the podium. :<( But there is always next time.

The MeetUp is always a lively event. Last night in particular there were a lot of excellent questions from the floor, and some very interesting “compare and contrast” discussions with folks from a wide spectrum of organisations from Marks & Spencers, O2 and Sainsburys through to Aviva.

But one startling issue that arose is how few companies are load testing the ‘real thing’: their live site!

A show of hands suggested that Load Testing is not something that most testers are focused on. Only about 10% had done any at all in the last 12 months, and only a couple of percent had done any in the past month.

The tools being used when load testing had been carried out were mostly the old names, with their well know old strengths and weaknesses: Load Runner and jMeter.

But the really startling finding was: the level of resistance to load testing the actual live website.

For some it was the problem of getting Business Team sign off and ‘risk’ the live site.

For some it was the pain of getting set up to allow the quick ‘clean up’ needed after a load test, deleting from databases the dummy transactions for example.

For many, the load testing scripts being used were only checking lower level page events, as that sort of test had been less pain to put together.

And it was clear that in a number of cases the load testing done on staging environments, had become the ‘magic recipe’ that the organisation depended on as the performance Go/NoGo. This was even the case when several team members felt that actually what was being tested, and how it was being tested, actually no longer bore any real relationship to what happens for real users on the live site!  The number generated each time was just a “magic token”.

It’s hard to write meaningful Load test scripts that ‘Do what the Customer Does’

Chatting to folks afterwards it was clear that load test scripts are the unloved child in the testers world! They are a pain to keep up to date as the site changes, a pain to make realistic as sites become more richer with AJAX and HTML5.

So the SciVisum fully Dynamic approach to Load Testing prompted a lot of questions – and especially the benefit of load testing that does not throw false errors just because the live site does not change, in terms of especially retail sites where products can go out of stock between the start and end of a load test.

In conclusion it seemed like a lot of Business teams are not aware that the  Load Testing being done for them, is in fact not doing anything to reduce the risk of capacity limits being hit, and real customers, and revenues, being impacted.

It was a great MeetUp and I look forward to more in future.

There were some great presentations during the day including one stand out slot by Julian Burnett, Head of IT Architecture at John Lewis, where they are embracing change at quite a pace for such a large organisation.

Online retailing technology is fast changing at the moment – and there was one startling message I picked up during the day…

The journey that retailers have followed online, was painted in quite a startling light, one that emphasises once again the important of website performance on profitability and brand strength.

In the beginning of retailing on the Internet, it was a world dominated by Google. Regular consumers would start most  Journeys at Google – whether looking for information on a BBC programme by typing ‘BBC’ into Google, or whether beginning a shopping journey by typing in eg ‘digital camera’ .

The retailing marketing teams could use SEO and Google Adwords to get in front of possible buyers, but the rules were set and managed by Google. While this was “fair” in a broad sense what it meant was that no retailer had any unique advantage as the techniques for best use of Google advertising became more and more widely known.

That phase has ended, and we are now in a  Social Networking era – this time dominated by FaceBook. Consumers are this time likely to start their shopping Journeys at Facebook or another social media site.  Facebook, and increasingly others, do provide advertising opportunities and such but it’s still an an eco system, just controlled this time by FaceBook not Google. More importantly it still leaves retailers with no unique competitive advantage.

So the routes to bring visitors to your site have less and less ability to offer a unique brand experience.

This leaves the one place, where a unique brand experience can be built: the retailer’s own website.

Thus the focus is shifting back again to building and maintaining a website that offers the best Usability, combined with fast performance and smooth, error free, delivery.

Of course this means that the technology that underpins all the features online is increasing in the spotlight. Savvy retailers are recognising the value of having excellent tech teams and of measuring user experience online  in newer, more accurate and realistic ways.

After all, after spending the marketing budget on generating traffic, every visitor who is disappointed by the website speed and responsiveness is a double loss of ROI!

There was some fascinating discussion, with some tips and tricks in the “Monitor Everything” and the “Log Everything” sessions.

DevOpsDays started in Europe just two years ago. Events have now been held at a number of cities round the globe and this was the first in London.  The events are effectively  community led and have a model of a morning of key speakers, then the afternoon having an Open Space Technology  approach to (un)conference. This was my first exposure to Open Space.

The mechanism was very effective in ensuring that, of the variety of participants who proposed themes on the day, it was those with the most votes that were held.

The DevOps concept itself is still quite young. It was nice to chat with some of the more senior guys like me who were there, sprinkled among the mostly young crowd, to get the big picture so as to better understand their teams, and how they can facilitate the closer integration of software development and live operations that is at the heart of the agile-focused DevOps approach.

I was reminded of something from 2 years back.  Speaking at a Retail Technology conference  Chris Howell (then IT Director eCommerce at Dixons Stores) talked about ‘Uniting the Tribes‘ within an organisation. Specifically he meant using monitoring of the user experience on the live site as the common language of data for the organisation that determines whether the tech guys deliverables are fit for purpose.

Of the presentations  ‘DevOps For Dinosaurs‘ by  Niek Bartholomeus was particularly good. Moving a large organisation to new, faster ways of working takes some people skills, and the ability to decide which new practises can wait until later, and which the organisation is ready for first.

Overall, the discussions were first rate. There were two or three hundred folks there! I guess the fact they’d booked a conference that straddled across to a Saturday meant that they were highly motivated, proactive guys!

The Monitor Everything discussion open space highlighted the fact that tech guys, no matter what flavour, do want more data to feed their analytical inquisitiveness!  However, in reality, not all monitor data is equal: the low hanging fruit is the data that is closest to the hardware but sadly furthest from the end goal – the customers experience online.

The extra effort to monitor close to the customer was for some too much effort (hard line techies!) and for others was a goal they had in mind, but progress was hard. This reflected a lot of my experience doing website monitoring each day: the most useful, actionable monitoring is from ‘doing what the customer does‘.

The Log Everything session also appealed to the need for data to troubleshoot.  A good part of the discussion was questions being put to some of the guys who have been using logStash in earnest.  It’s a tool we’ve just started looking at and it makes a convenenient, easily searchable, ‘bucket’ to put all log data into: from server hardware level, right up to application level unique logs.

For some, just the power as a troubleshooting tool, the ease of with which you can build  powerful filters, was the main win. A couple of advanced users, though, were starting to log actual data so that they could track a user’s transaction through various systems with relative ease.

I must have a chat with our guys here and see how deep we have gone with logStash.

View video of the presentation.

So it was a warm feeling this week, to be told openly:  ‘SciVisum, you’re the market leader‘.

We’ve built a good list of clients, including Tesco, Debenhams, Clarks, Inghams Travel and many more. Our clients tell us they come to us, and stay with us year on year, due to the genuine technology edge of our services and the flexibility and customer focus of our Managed Service approach that means we listen to client needs and re-shape our deliverables to fit them.

We’ve been kind of obsessed with realism in the web performance tools we’ve built, and in our unique Dynamic Journey approach to achieving realistic and actionable website performance measurements no matter what the website or application in question.

Most of our clients have used other solutions before ours and simply made choices in the normal way based on their specific situation and needs.

But it was a warm feeling this week, to be told openly:

‘SciVisum, you’re the market leader’.

It is true that, looking back the last year or two, that it’s become am increasingly common,  occurrence for users of our solutions to be willing to explain to other prospective buyers,  why they have chosen our approach, and how it compares and contrasts with what they have used before.

So those kinds of client referrals are not news exactly.

Thinking about it, I’ve also noticed that I’m being increasingly invited to speak at conferences and provide articles for magazines. Just last week I was speaking at the DevOpsDays conference in London, presenting in both the Innovation theatre and Pecha Kucha sessions at the Retail Business Technology Expo exhibition and at the local Digibury evening. When I returned to the office it was to find that the piece I had written on Using Website Monitoring In Your CRM Strategy for Retail Technology Review had gone out!

But this week another client referral came along and surprised me – in a nice way! It made me realise that we’re  now being seen by some as an “emerging market leader” in the web performance space.

It started as one of those regular situations where a company has a project, and is getting a couple of proposals in, and is trying to compare and contrast the various solutions on offer.

In a phone call on Tuesday, after talking technology and details, the company asked for a reference as the decision time was getting close – and wanted to hear from a particular source.

So, as usual we asked our contact there if, as he had experience of using both our services and others, and was well placed to give an informed comparison, would he mind giving a reference?

This is what he emailed:

“I am happy to share my thoughts.  In all honesty I  prefer the SciVisum solution to the other website monitoring solutions I’ve used … particularly impressed by its dynamic search algorithm tailored to our key destinations, routes and passenger mixes.”

A nice little referral, nothing so unusual there.

But,as a result, we were told ‘So, looks like you guys are the Market Leader‘.

It was one of those warm, fuzzy moments.

When I heard what had been said about us I flagged it to the internal staff  email list, to spread the love around!

In reality of course, whether a company is ‘Market Leader’ or not, is not of prime concern when choosing a solution: the decision must be to choose the one that best fits your own unique needs, your situation, your risks.

A Market Leader slot may indeed even be misleading!  Just how much criticism a ‘Market Leader’ can get can be observed from the current media and blogging on Microsoft.  It’s clear that although they have been a market leader for so long, that many savvy buyers are no longer planning to stay with them but to move gradually away – whether it’s moving Internet systems off Silverlight , or moving mobile to Android, or putting question marks over long-term dependency on SharePoint – because the market that they were leading has changed in moved on in terms of its requirements.

At the end of the day each client makes their unique buying choice based on their unique situation and pain and risks. Whichever supplier they choose is, for them, the ‘Market Leader’.

So the challenge for the team here is :to listen to each new prospective user, understand their needs, and see if their  gap, can be addressed with any elements of our approach.

Mike Brown of uTest jogged my memory recently when his blog referenced the timeless  ‘Classic Testing Mistakes‘ by Brian Marick, from back in 1999. One of Brian’s points still stands out as a very powerful approach that directly aids eCommerce Managers in today’s world,  more, in fact, than it has any previous generations of technology management.

Failing To Correctly Identify Risky Areas: Load Testing The Live Site!

In a resource constrained world not everything can be tested before a project is launched,   so there is considerable scope for mistakes in judging where to focus test effort and time.

Until you test the engine in a real car, out on a real drive, all other results are guidelines only.

In the arena of website load testing  there is a set of common blind-spots that crop up repeatedly as our  team work with clients on planning Load Testing projects.

• Load testing an environment that is not the live environment

It is often a preferred choice of tech teams to load test a pre-production environment, rather than the real live one. This is usually on the basis that there is less hassle as there is no need to care about upsetting real customers if problems occur, and no need to tidy-up production databases afterwards.

However, unless you test the live site any metrics you get from the pre-production site are more likely to be misleading than helpful. Performance on the live site will not be a simple X% bigger than on a smaller pre-production one. There are too many places for small differences in configuration, impact of resource sharing etc to cause big differences in performance.

Load testing your live site should at minimum be an annual process, with testing on pre-production if nothing better can be arranged used between times and benchhmarked back to the full live test.

Paying More Attention To Running Tests Than To Designing Them

Recently, we started to work with a major client that had a load-testing regime that they followed per site iteration.

At planning meetings where we looked at our tools being used to take on their website performance in general, the details of the client’s load testing specification was looked at.

When it was put on the table and examined it became clear that the User Journeys used were ones that no real customer could follow. The test script was a list of URLs, in an order that was impossible to follow on the live site by using links on the actual pages!  Yes, all the URLs did work in isolation, but could not be followed by using a browser and clicking on available things in each page.

What had happened was that the live site design had changed in the last couple of years, but the load test script had not. This change in site would have been spotted if they had not been using the poor approach of a list of static URLs being considered as a User Journey – but that’s another story.

Sometimes there seems to be a psychological discomfort for tech teams in giving enough attention  to the design of load tests.

They are predisposed to wanting the spec to be as concrete as the lines of code in the program, and working with the business team to understand the end user and the complex data from Web Analytics logs etc can be too fuzzy. It can be less cognitive effort to spec up simplistic lists of page URLs than to embrace the complexity of real usage.

This is where eCommerce management has a vital role to play: counter-balancing this tendency and ensuring that the end-user experience is the starting point for designing load testing projects.

Not Reporting Usability Problems -  All eCommerce Managers Should Read This!

This is a real gem that Brian drops in among the excellent advice in his document.

He relates how, in the course of testing, the tester becomes familiar as a user of the software and may spot an issue of usability: something that could be much smoother / more intuitive for the end user.

In many organisations there is no mechanism for the testing team to report back such an issue, somehow UI improvement is not what the testers are assumed to be doing.  They are assumed to be just testing if the releasee worked exavtly as designed and to report back the important bugs in that.

This is even more relevant today, where our websites are more complex than ever, with HTML5 functionality, off-line web applications and so on.

The lesson to be learnt is that usability is at the heart of the customer experience, and all the testing we do, including website load testing and performance testing, must be based on the customer experience.

We have a phrase at SciVisum we use when planning web performance projects:

• User Journeys should “Do what the Customer Does“

Make that the heart of your load testing and you’ll end up with load test performance metrics that make sense to the Business teams and to the senior management. This also, back to lesson one (failing to correctly identify risky areas), won’t leave you vulnerable to unexpected performance problems later on due to the fact that the load testing was not accurately doing what customers do.

But the edgeconf this weekend looked like it was going to be worth it: with panel discussions not vendor keynotes! It also featured some of the actual hands-on guys doing the leading edge stuff, from the major browsers Chrome, Firefox and Opera, as well as some of the sharpest developers who are delivering business value from the least technologies.

The conference was organised by the guys from FT Labs, those responsible for the FT Newspaper’s HTML5 site that allows readers to read the paper on and off-line. Offline is when your device is no longer connected to the Internet, which for a normal website would mean you can’t access any more content.  However an Offline Web Application can carry on working and serving page content it has grabbed in advance so that the user experience is seamless.

My specific interest in Web performance cropped up in several of the slots, with some interesting new tips to take away.

The structure was perfect for learning: each 60 minute slot started with a 10 minute chat and then a  panel of 4 discussed questions from the floor.

Engineers were there from most of the major browsers (shame about MS and Apple… I’m told that apparently they are less keen to send send engineers to events like this).

With the discussion at the leading edge of technology, it was interesting to hear how the browser guys are watching what developers do with the new features of HTML5, and how they build new features on top in Javascript to address additional goals.  As good usage patterns evolve, the browser guys will plan to bring them into the core Browser itself.

Here are some of the website performance optimisation tips I picked up.

Rethink the network’s effect on your visitor
As users do more and more of the browsing whilst on the move it means that the site developer has to think again about the network properties. Fast broadband at home lulled us all into a false sense of security because our pages now need to work nicely (fast) for users browser via 3g (4g…2g..!).

The problem for the mobile networks is that they are extremely variant and transient.  The fact that your visitor had a fast connection over the last 5 seconds is no predictor that their connection will be equally good for the next 5 seconds. Moving a device just 10 or 20 cms can change the network connectivity from great to terrible!

So the consensus across the conference was that there is no value in the browser recording the network speed of the visitor and passing it to the website, as there’s nothing useful the site can do with data that doesn’t say how the the user’s network connection will perform over even the next 5 seconds.

There’s an interesting warning here to RUM (Real User Monitoring) based approaches:  any performance data calculated within the browser and sent back to the site is also hugely variable, because it’s made of data from each user that itself varies widely.

For example: RUM data may show that your site slows down between 8 am and 9am, and then speeds sharply. This may be quite untrue. The slowdown may be caused by visitors browsing on mobile devices as they travel to work and then from 9am onwards the site speeds  up, as they are now on well connected wifi network in the office!

Top Tip
Two years ago the advice to website developers, was to structure pages so as to send initially as little as possible to the user to get a  usable page fast and then send the rest after page load.

But this advice has now been reversed!

Instead, it’s better to send the whole page to the user and then nothing more! The reason is down to the properties of mobile networks whereby although bandwidth increases. latency is not reduced. This results in a delay between batching enough data to send and then sending it, and then waiting to batch again.  The more you send your page in small pieces, the more prone it is to being held up.

The second reason is battery life. In mobile devices the biggest battery hogs are the screen first and the radio second. Mobile devices will allow the radio to power down, after a period of mis-use: thus saving battery.  But if your page is chattering to the user over a long period of time the radio can never power down.  In one test a site discovered that traffic being sent between pages was only 0.1% of the network traffic, but when disabled caused 40% drop in  battery drain!

We don’t want customers to be the first to tell us about performance problems on our site!

Extremely rarely, a client will be worried about their problems getting onto the local TV news, or national news.

But the BBC website was very open about problem on February 6th.

This is what their site was showing:

It’s not the first time they’ve been equally honest:

• June 2012: ‘ we suffered a major failure of BBC Online last night. ‘
  http://www.bbc.co.uk/blogs/internet/posts/website_outage_june_11_2012

• Mar 2011: ‘the whole of BBC Online was down last night for an hour from 22:40′
  http://www.bbc.co.uk/blogs/bbcinternet/2011/03/bbc_online_outage_on_tuesday_2.html

The lesson to be learnt is:

Not that major outages can affect any site, unexpectedly and that we should design-in resiliency to all possible technical root cause failures so that the site itself can stay live despite those problems (which is true).

It is rather:

That all our sites suffer minor problems, at frequent intervals, affecting a few visitors, or just some of the online functionality.

And that we all need:

• a set of metrics that find and report 24/7 user experience and hence report these non-catastrophic problems
• an agreement that both business and tech teams, understand and can act on the metrics
• less time spent in after the event analysis and finger pointing: and more on finding tech root causes and solutions

The Impact Of Sporadic Errors

These problems may only be brief: so brief that often normal tech alerting may not have time to fire off.

At other times a problem may only effect a percentange of users, say 5% and so will never trigger a tech-based monitoring solution.

It maybe problems that simply are not visible to the tech systems: eg the ‘Buy Now’ button for certain holday dates and packages, does not work user side and no page request is sent to the servers at all.

Hence Sporadic Error reporting becomes vital to the website team that have already got their site to a point of stability so that  that outages long enough to get a warning page like the BBC one above just never occur.

We have been worked with several  clients in a number of industries over the past few months to investigate the real incremental cost and impact on KPIs  that can result from sporadic errors, little “blips” in performance that either do not last for a sufficent amount of time to trigger a warning, slow down or cut off alert, and are not immediately replicable.

In some cases these “blips” are due to the natural ebb and flow of network traffic, browser peculiarities, local conditions and the like and the thinking has typically been that such things cost more to try and investigate and control than resolving every last anomaly is worth to the business. In a lot of cases this assessment in correct, but what we have been working on is how do you identify the ones that actually would be worth the time and effort to analyse further.

Short term errors, while often insignficant in themselves, can build up to a substantial total over time. Their very nature means that they can be easy to overlook, or to miss underlying patterns and probable causes, that would be more readily apparent in errors and issues that last for longer periods of time and are seen by more of the operations and content teams.

With traffic volumes so high on many sites now, and content provision often so complex, a total of a few minutes problem every day can quickly add up over the course of a week or a month.

Last year SciVisum released a Sporadic Error Report.

This is one way to report on Sporadic errors: based on the vital Customer User Journeys measuring the routes that business and tech teams have agreed are fair routes to gauge genuine user experience.

Today it was something that Stephen Nelson-Smith had written, and that linked back in my mind to what Chris Howell (then online Director at Dixons Stores) had said on the speakers podium at Retail Business Expo 2011 (see video of presentation).

Stephen wrote:

This might seem incredibly obvious – of course we monitor our systems because downtime costs money. But if we really think like this, why are so many of our systems so woefully monitored? And why are we not monitoring business metrics alongside technical metrics?

This encapsulated succinctly exactly the reason why the monitoring services we’ve put together not only show the under-lying technical issues and root causes for problems (so tech teams can fix them fastest) but overlay on the top the underlying business truth:  what is the experience of the website  for Customers? Customers, the source of our money!

Chris Howell talked about this in terms of the monitoring Common Language a common language that can unite the tribes known as tech teams and business teams and describes how good user experience is,  24/7.

Note to self: must track Stephen down for a chat!

Related Articles

• User Experience Data: The Rosetta Stone Of Business
• Downtime Is Money