Over the years of testing websites using Clouds and Content Distribution Networks there are a number of pitfalls that we’ve seen organisations struggle with, so it’s nice now to be able to provide some tips to help people get the best from this sort of load test.

Pitfall 1: Testing as if there is no CDN in use

At first glance this seems sensible: a good load test will be a good load test no matter how the target web site is set up. But when a site is using a CDN there are actually a number of extra complexities that need to be planned for.

Firstly, if you test out of hours, to avoid inconveniencing the users of your own website  then chances are that this means it is out of hours for your CDN provider.  If you’re unlucky, you’ll hit a time when they are doing nightly backups or other background tasks  and thus you’ll see an unusually high error rate!

Conversely, as it is out of hours for all their other clients too, the CDN  will have huge swathes of free capacity, all available to you on your one load test.  So you’re likely to find the CDN delivered parts of your website perform  pretty well!

It’s easy to end up with  unsatisfactory test. For example, a major UK retailer ran with a load test supplier who repported back that the target traffic levels had been hit but alot of errors seen.  This contradicted the evidence of the client’s own servers, which had run at very low % utilisation during the overnight load testing, and were error free.  It was also contradicted by the CDN supplier who disputed the errors had occurred and had logs to support their claim.

Needless to say that retailer didn’t continue with that load test supplier.

The important thing to get from a load test is realism. Without that the test results are worthless for decision making, even if they look good on the presentation!

Pitfall 2: Testing your website without the CDN elements

The pitfall above, may suggest to you the idea that testing of a CDN supplier is a pointless exercise, unless you can generate enough traffic to reproduce the load of all the clients’ peaks  together.

That’s a lot of traffic!

If you’re buying your load test from a supplier who charges based on the amount of traffic you generate then that consideration alone means you may think about testing the website without the CDN elements.

This approach may not be 100% wrong, but it can easily lead to load test specs that are no longer ‘Doing what the Customer does’  and so are becoming unrealistic. Worse, depending on how your pages are constructed, it may be easy or very difficult to seperate out the CDN.

If  you’re a retailer and you put top-right on every page the status of the visitor’s shopping basket, then it is tricky.  Every product page contains both ‘standard’ content (the product info and image) and user-specific content (what’s in that specific user’s basket). Many sites now cleverly seperate the two into effectively two pages,  placed together as one in front of the visitor using AJAX and javascript.  This approach can allow more widespread use of the CDN for the standard product page content, making it easier to separate out the CDN in your load test.

Pitfall 3: The Cloud /CDN is so big and reliable, we don’t need to worry about it !

In the real world, of course, Cloud systems have bugs and problems, CDNs have less than perfect load distribution models and human errors mean your website performance may not always be as well supported as you think you’re paying for.

We had that exact case just a few months ago, and as a result helped a major site, built on Microsoft’s  Azure cloud, to track down a nasty performance bottleneck that caused them to have terrible user experience on the big first night after launch.

The root cause was the fact that major performance differences resulted from minor changes in subtle API usage, between default and tweaked settings in the Azure platform.
In that case,  the whole application was in the Cloud  so it had been tempting to ‘assume the cloud is infinite and don’t bother load tetsing’ .

It was the major fail on the first night, that forced a load test:  the system clearly was in trouble, and the bottlenecks needed to be found quickly.

Pitfall 4: Ensure you Keep it Real

As part of the speed and effectiveness of pages helped by CDN the levels of caching are necessarily much higher.

If the User Journeys you specify are simplistic, if they only include a small percentage of all the products on your site, then the results you obtain will be very wide of the mark. This is inaccuracy from over simplistic scripts is amplified to much greater degree than on a  non-CDN site where the caching efficiencies may not be so pronounced.  After all,  at the core of the CDN ROI is that it’s a caching technology that helps keep traffic off your own systems.

To measure meaningfully you need to ensure you’re using Dynamic Users Journeys that don’t just choose and buy products from a  pre-determined short-list of, say, 50 products. Instead, dynamic Journeys look into each live page during the load test and actively pull from the page any of the choices possible, choosing one at random.

For example:

• start right at the home page,
• find the top level Categories and choose one at random (maybe on your site it’s Womens wear, Menswear and KidsWear)
• at the next level down, choose a  random sub-category
• within that choose a random colour, or style or et
• finally choose  a random product from those offered in the final page.

That way, your load test will visit potentially every product in your store.

The Out of Stock Load Test Problem: Solved

If during the load test, certain products go out of stock as a result of the ‘purchases’ made during the tes,t  no worries, your site will no longer offer those products within the pages, and so the dynamic Journeys will not be able to find them and select them.

Another “keep it real” factor to bear in mind is the need to be able to cope with the subtle complexities of your site. For example, on some retail sites, as the visitor searches and narrows down their choices by factors such as style, colour, etc  the logic is such that whenever the search only matches one product the visitor is taken directly to that product. They don’t ever see a search result page saying ‘ 1 product matches your search’. Whereas if the search finds 2 products, the visitor is shown a ’2 products match’ result page.

You need to ensure that your dynamic script can handle that subtle complexity and detect the two equally valid search results. Not to mention handling all the other subtle differences that you’ve designed into your site that make it a better shopping experience, but a harder platform to test with simplistic tools.

Of course all of that is before we even start to think about how we should weight the different journeys that take place within a test to reflect an even great realism.

In Conclusion

Websites underpinned by Content Distribution Networks, or built with their systems half inside the Cloud, are still web based applications.

To test their capacity meaningfully, to get performance measurements that are valid in the real world, can be achieved, but needs a little more thought and care in the planning, and a little more subtle power in your script running technology.

Your old load test tools and suppliers may no longer fit the bill.

In today’s environment where the rush to launch is getting faster, with less time to test prior to going public, it’s also a  good time to bring in specialist help for those focused, furious periods just before go-live and to free up time for your own staff to use their unique knowledge, that no supplier can ever have, as to how they built and architected and coded your website. This way they are free and ready to fix the bottlenecks that load testing reveals, rather than tied up in the load testing themselves.

“Whanne that April with his shoures sote
The droughte of March hath perced to the rote…

Thanne longen folk to goon on pilgrimages. ”

Or at least to BarCamp!

Canterbury may be more famous as a Cathedral town, and have some of the oldest churches in the country (1,500 years old or so) – but it also has a prestigious recent history as the birth place of the  Internet in the UK back in the late 1980s early 1990s – and indeed it was on campus at The Univeristy Of Kent that the Bar Camp was held, where I was based with the UK’s first ISP back then.

That technology tradition continuous, as evidenced by the Bar Camp folk.
There were several familiar faces from the monthly Canterbury event Digibury – a hats off to Tim and the team at Deeson who manage that event – they are smart Drupal guys!

We sponsored Bar Camp’s first run last year too, but for Bar Camp Canterbury 2012 the new organisers had been able to pull in more sponsors, and get a bigger venue. The venue was also a better location for an event like this, with all the seminar rooms off a central space with chill out, snacks  and Crafts Building space in the middle. More sponsors meant  it could be run over 2 full days, rather than the one day like last year, so there was more time to meet people.

I led a seminar/workshop  on doing a Software Start-up and the discussion turned out to be productive as among the group of about 20 there were 3 start-ups represented. These were at various stages of growth and fund-raising, and addressing different types of market, so there was lots of practical wisdom!

Here is a round up of the key points:

• keep your costs pared down,
• be ready to work 50 hour weeks and not take home a regular salary,
• raising investment money will involve large time commitments and ironically take you away from growing your business!

As an ad hoc mentor to Kent software companies, I’m planning to run some future sessions, where those already down the road can share experiences, and address specific issues.  One in particular that came up in the workshop was the challenge of how to get a start-up to “the next level”.

The definition of “the next level” changes as a company grows.  One team wanted to go above 2 staff but not keen to commit funds to rent and overheads; another was stuck at 4 staff and finding it hard to find more people of the right profile, in their West Kent location with the pull of ‘exciting’ jobs in London so close by;  another wanted to move from selling consultancy to their own products but weren’t sure how to do the product planning and market planing needed.

The good news was that all were finding it easy to generate business in their niches, so there is obviously something good in the Kent air for software companies!

I benefited from a number of the sessions:

• picking up a Firefox speed-up tip from the session led by Mozilla,
•  learning about some of the  esoteric challenges when trying to design virtualisation platforms for apps where you consciously plan for a 25 year software lifetime,
• getting some Java tips on clever use of Generics with some debate as to whether it felt more like a deep maths lesson than coding, and whether the trickiness introduced outweigh the potential coding-brevity benefits,
• enjoying a thought provoking Gamification session
•  seeing some cool presentation platforms built entirely in HTML5, to put Powerpoint in the shade.

It was also interesting to swap notes with other software dev. managers. It turns out some of the Agile stuff we’ve put in process at SciVisum over the last year or two is actually pretty good, and it’s easy to forget how far we’ve come when we see it every day.  So nice  to chat with other teams at earlier stages on the same path!

Sunday the 2nd day started late, the free beer the night before at the nice new FruitWorks office space in Stour Street had been an irresistable temptation too for some, and there just weren’t enough folks to empty the generous quantity of barrels donated! But after the slower start, there was another round of seminars until some fun and silly Karaoke Powerpoint games brought eveything to a happy end at 4pm.

My only criticism is that it was a shame the tea shop downstairs was closed.  Free soft drinks and sandwiches is not to be sniffed at, but for me a cup of tea was sorely missed!

Definitely worth a visit next year!

When  teams have been used to the methodology, alert formats and reports of another website monitoring supplier it is important to help make the transition as simple as possible for the hands on Opps teams.

Of course it helps if you have the kind of focussed client liaison support all of our customers enjoy, but here are 4 top benefits gained from a managed transition collected from a number of clients that have moved to our platform quickly and easily.

Extra Realism  opens the way to an easier dialogue between Opps and non-Opps teams

Dynamic journeys provide a measurement more realistic to different users choosing different routes and products  – and that realism allows easier meetings between technical and business teams.  With artificial or static monitoring, Operations teams are often not able to be certain of the extent to which customers were impact, and by what percent, by a specific technical issue.

With newer dynamic journeys, meetings can be more focused, Opps can be more confident that a green bar means that Journey really was working, even if the call centre had extra callers compared to normal.

Or conversely, if a problem affecting real users is picked up and confirmed by the monitoring more information can be provide to the customer facing or eCommerce teams more quickly and the fact that sampling speeds up to 30 seconds on an error gives better precision as to when the problem was resolved.

Less often will the support or operations team be asked in a meeting to go away and drill down to find more information -that’s a  time saver.

Take the chance to rethink alerting, don’t just copy it.

There are a range of fine-grained alerting choices in the scivisum portal that allow the total number of alerts send out to be reduced per person to ensure everyone gets relevant communications. This can range  from the simple end of sending alerts due to DNS problems only to the guys that can fix DNS through to the option of dialling in specific URL patterns that relate to specific known bugs you have and sending those to the one software developer who knows how to fix them there and then!

In addition, Alert emails themselves contain more intelligence, to save time trouble-shooting such as stating which product had the specific problem. With a click you are taken to the detail-rich Single Sample view for the error that saves trouble shooting time by calling necessary technical detail right into the page. This includes which server in your farm was involved (by serverID),  average TTBF per host during the journey,  specifics of the errors that occurred and one click to Reply to see the raw HTML, or the actual served page that the user saw.

Make the most of this right away and configure what you need, not just what you had. If you do that you will get the results you got before, and they must not have been what you needed or you wouldn’t have changed supplier!

Put your Wallboard view on a  Plasma screen and immediately show  when website problems are due to 3rd-parties

It’s worth getting a large screen on the wall to display the SciVisum wallboard. This will give visibility to the whole team as to which Journeys you are now monitoring and their status.

Added intelligence, such as visible flags that state when a problem root cause is not your own systems, nicely reduce the occassions when the technical team get blamed for things outside their control!  If the root cause is actually down to a 3rd-party supplier, (such as an Image Content 3rd-party or search supplier)  the ’3′ icon shows it, with no live intervention needed from the Opps team.

Sci Visum Live Status Wallboard

Don’t be scared to re-spec your Journeys

A common theme from those who moved to us in the last 6 months, is the speed in which we make Journey changes as requested.

Firstly, our own team are watching your journeys  anyway, and if you change your site so much that a journey rewrite is needed, will do that proactively and let you know, typically 2 working hours later.

Of course the dynamic Journey approach, means that our Journeys don’t need updating as often as static ones.  If the  monitor is statically  hard-coded to hit an exact URL for an exact configuration of the same specific product, then it will of course fail when that product drops out of stock,  or when your site changes slightly and that old URL is no longer valid.  The dynamic approach looks into the live site, to find the list of products offered, and chooses from those actually available so never tries to buy a  product that is no longer even visible on your site!

But feel free to go beyond that and experiment.  If say a particular problem is being addressed by your development team,  then change your journey spec to ensure that feature is included and we’ll be able to pick up the errors that it produces, and give you proof, when no more errors like that occur, that the software fix has been a complete success.

So a change in website monitoring suppliers, with a new, better common language of user experience can be a key component in helping an organisation move together higher up the eCommerce maturity curve.

Get in touch if you’d like to discuss how we can help manage change, provide support and training and provide trials of our product to run concurrently with those from your current supplier.

It’s just my anecdotal evidence, but the further down that maturity curve organisations are, the better they are able to reduce Customer Struggle. It’s all about Marketing teams getting the ROI they want -and how they work with other teams to really make that happen, even when often they know it isn’t happening yet.  Let me explain how it works.

First off – Customer Struggle is what I mean by the scenario where successful marketing and online campaigns increase your online foot-fall, but your ROI goes down.  Lots of extra visitors to your site compared to the same campaign last year, but they seem to be spending less.  Maybe the reason is the fact that your site causes too much user pain, too much struggle and so many new visitors don’t become customers.

The struggle could be that your usability has gone down since last year – but in terms of the Maturity Curve I’m talking about, the focus is the relationship between Marketing and the team that deliver their desires:  the online IT and technology teams.

The best marketing campaigns in the world, all end up being delivered by technology, by technology teams in your organisation, and by the technology in your visitors’ hands, or on their desktops.

So the kind of technology gaps that cause successful marketing campaigns  (visitor targets reached) to fail (ROI less than expected) are things like:

• certain parts of the key add-to-basket or check-out journeys are just too slow
• most visitors may see say 2 second pages, but some see 30 seconds or more: at the same time
• errors are stopping the shopping trip – again not for all potential purchasers at the time

So what are the stages along the eCommerce Maturity Curve, that will increase the ability of Marketing and Business to achieve target ROIs online?

Stage 1: “Only we in Marketing really care about the website”

(aka Unconscious Incompetence (see footnote) – the organisation doesn’t even realise that the approach internally is broken)

This is the starting level, this is where everyone except pure-plays were 10 years ago. Many are still here today.

Marketing see the online IT team as just an extension of the team that keep the printers working, and sort out problems on your laptop.   They think they don’t understand customers and the market enough. So tech guys are not brought into strategy discussions much, not until it’s planning  time:  when they are told: we need feature X delivered by date Y, and we’re writing the spec this afternoon so you can start Monday, as this is a priority!

So any changes and new features wanted are not very fleshed out:  mostly they are very brief, half-page documents saying ‘can you just add on this feature XYZ please…’.

When the dev team come back with questions about choices in the feature, “should it do this or that, what about in this case, and that case”:  the requests are treated rather off-hand, as if the dev team should intuitively know without needing to ask!  Sometimes the questions and tricky and detailed, and the Marketing team don’t really have a good answer: they are reluctant to make a trade off where one is needed.  They are reluctant to split the feature into say 2 stages, and have the simpler stage 1 released first.

So the dev team learn over time not to ask questions, to get on and ‘get the job done’:  they learn that the Business teams seem more concerned with when the feature goes live, as to how nicely build and integrated it is.

With no common KPIs, measurements, shared language, understanding or frame of reference communication of ideas, goals and needs is practically impossible.

The result, for an organisation that stays at this stage is a website that becomes less and less usable, increasing customer struggle,  reducing ROI, it becomes harder to bolt on new features and the speed of progress slows down:  and overall a general malaise across the board that the website doesn’t quite cut it.

An organisation cannot be pulled forward out of this stage by the tech teams – only marketing and business people can do it!

If you hear phrases like this – then the chances of the organisation moving on are slim:

• yes we know from the call centre that there is lots of customer struggle, slow downs, occasional errors – but that’s not us in the marketing Departments’ responsibility!  That’s IT’s fault.  they had a project last year to fix that! No, we don’t want to measure customer struggle ourselves  – talk to IT will you.
• yes the site is a bit untidy and ugly now – but it’s so hard to get IT to add features that work nicely
• I know the site can be hard to use, but it has to look pretty
• it always takes too long to get new features live, not sure what it is with our technology – I heard a colleague at another company say their IT team use a better platform than ours – maybe we should move everything to a new platform?  Or maybe our IT team are the problem…

Stage 2: There is a problem – lets work together

(aka – conscious incompetence – the organisation knows it needs it can’t increase it’s ROIs without doing the Business<->Technology interface better)

There needs to come a Eureka moment, to kick off this stage,  something like this:

• Nearly everything we do with customers – depends on technology !
• Yes our product design, merchandising and marketing must be good, but
• Technology is just millimetres below the surface, for :
• every campaign email and SMS
• every mobile App we put out
• every interaction on the website that turns a Visitor into a Customer
• our own technology
• and our supplier’s technology too
• eg 3rd party Image servers, Search servers, Cloud or CDN
• every interaction on our partner websites, depends on our API technology

It may be phrased like the above, or maybe the result of a more down to earth realisation that ‘Our online ROI and website in general are poor, we know competitor X are doing it better, we can see that: how do we work together to plan to improve’.

What in concrete terms changes at this stage?

What happens to start “uniting the tribes”?

Centrally,  the marketing  business teams try to engage more closely with their technology teams.

And specifically – they start to realise that there is work to be done, by the non-IT teams themselves, to improve things.

There is a realisation that Dev Teams are different to Marketing teams:   when a new feature is in discussion Marketers are comfortable with the big picture, helicopter view of it.  But that view though useful to IT, is just not enough.  IT can’t build a new thing without getting down to the nitty gritty detail, to the blue print level.

Until the new feature has been fleshed out with detail, it’s unfair to ask IT to estimate how much work it will be.  The devil is in the detail!

So marketing need to be “commited to further conversation” with the developers, to flesh out the desired feature.  It’s not about writing a huge spec document, it’s about writing a short User Story, and then being willing to have a conversation, or a few, until the developers say they have enough detail to plan it.  And when the developers ask about the detail of how it should look, a proper UI person needs to be there too, to help the look and feel to be as attractive and intuitive as an iPhone, and not a Nokia phone from the late 1990s!

Of course this is not just “any conversation”. It needs to be true communication where both sides are “speaking the same language” and have a shared interpretation of the meaning of any results.

This ideal world can only be realised once an organisation moves into stage 3…

Next Time: Stages 3 and 4 and how to Unite The Tribes Through A Common Language and a Single Point of Truth.

Footnote
Read about the Four Stages of Competence at Wiki

Website ‘Last Mile’ monitoring is one – there are several limitations in practise, and we are finding fewer clients  getting value from it within their overall end-user experience monitoring.The phrase Last Mile came originally from the Telco sector and refers to the  physical wiring between a users premises and the local exchange.

In the realm of Website monitoring, Last Mile has the promise of  getting better real-world insights into the experience of customers visiting your site.   That’s a good aim and we are always striving for more Realism in our projects here.

There are a few ways to  monitor the Last Mile  and for a few clients we have provided a dedicated test system with dedicated, domestic quality, broadband lines installed at key locations round the UK with the same User Journeys running at each so that comparison between them is possible.  This is called CityWatch

For clients with particularly focused geographic needs, this can be effective, for example if, say, 30% of their user base is connecting from Manchester it is vitally important to have specific data from that location.

But for most clients, it’s impossible to get a statistically significant picture of the last mile unless you’re measuring from all of the top 5 UK Broadband providers, and from scores of UK locations. Clearly that’s an impossible task!  However, without that level of coverage, the data points are just too few to be statistically significant.

Another issue is that even if a huge set of data points could be gathered, there is little actionable data to be had. Even if you know BT is having a problem at city X for an hour there’s nothing you can do you to change that.  There’s no one at BT you can call to ask to fix it! The most you can hope to do is pass on that information to any customers as the reason for a slow down.

An alternative and widely publicised approach to Last Mile has the promise to get a huge set of  data points but in reality the data it generates is flawed. It uses the concept of putting software into real, consumers devices at home or on the move (in exchange for small payments)  thousands of them, connected to thousands of local ISPs  and having monitoring scripts run from those.

Is the flaw obvious to you?   I have teenagers at home, so it’s clear to me.   The performance of any website from my laptop at home is determined to a great extent by just what my teenagers are doing online!  If they are streaming audio, iPlayer, downloading, gaming… all of those will change the network performance I’m getting and have the effect of changing the performance of my broadband at home from fast, down to old-style modem speed!

And you can imagine the variability you get on a  wifi network in cafe, where the user base contending for bandwidth varies hugely, so mobile Last Mile monitoring is particularly inaccurate.

So, statistically, there is just way too much “noise” on the data to have any value at all in making evidence-based decisions on your monitoring website: Last Mile doesn’t help.

At the end of the day – that is the most important thing – do not make decisions about your website 24/7 user experience unless it’s evidence based.   The most important Realism, is to have dynamic Journeys that Really Do what the Customer Does – whether they are monitoring mobile journeys, iPad/tablet Journeys or regular browser journeys.

Which is a great question, and one that everyone in eCommerce should be asking, but even more interesting is when a  Director level person poses the question:

“How do I build and manage a TEAM, to build the Best Retail Website ?”

Because it shows a much deeper, more sophisticated, understanding of the evoloution of online retail: technology can do great things – but it needs great people to achieve that.

Some retailers I meet are engaged on a fast track growth in their online operations:  either newer companies for whom online has always been the greater part of the business, or household names who are ramping up online strategically.The challenge that I see the more effective ones meet in is recognising that to build the team that will build the best website, it’s vital to get the IT wing and the non-IT (business) wing working together.  Retailers who have not strategically realised the level of focus needed internally to embrace multichannel retailing tend to find it harder to bridge the technology gap.  (Fair do’s, it wasn’t so long ago that IT were seen and not heard, running head office systems that did overnight batch processing! ).

Online retailing means that everything you do, everything, is dependent on the technology! Bringing IT into the circle in a new way.

It’s even clearer that the technology is central when you have in-store kiosks, web-sites, mobile websites, video delivery, social network features, Twitter integration!

What’s needed is to Unite the Tribes by having a common language that both sides can understand. This provides both sides of the vital double-sided coin approach, i.e. an approach that works for both your IT teams, and your marketing and business teams.

The double-sided approach supplies metrics and KPIs for both.

Provides Business level, Budget level proof to the E-commerce Director and the business teams that:

• Customer experience is being maintained centre-stage through the hectic page of feature change, site development and visitor volumes ramp-up
• It has been made easy to pass problems and tasks to IT because they are well defined without needing your input; and proof of success is available for free every day in the metrics
• There is no need to feel powerless, and fear that problems with technology-based root causes seem to avoid solution or genuine identification

Provides  IT teams (both Software Development and Operations):

• the details and specifics of the online problems that are hurting customer experience
• specifics right down at the tech detail level, so your teams can quickly find and fix the causes of the symptoms that Business teams say are top priority right now – no more looking through known error logs, hoping to work out which ones are the really important ones!
• No more feeling that the Business expects a magic wand! That they expect you to ‘fix the website’ without saying which bit and then, conversely, seem reluctant to provide budget for the system updates you request because they want more ROI proof than is easy to provide from an IT systems perspective.

The video of Chris Howell, Director at Dixons Stores speaking on this theme of Uniting the Tribes last year at an online retail event in London, is a worthwhile investment of 3 minutes for any online retail director.

The common language you need – it has to be based on 24/7 monitoring of the customer experience on your site – the vital Do What your Customer Does metrics.

The eCommerce Director directly benefits from these measurements by use of a common language.

No more inconclusive post-mortem meetings, after poor performance during a major marketing campaign:

• the users’ experience metrics are there in black and white, together with the detailed root causes down to the specific problem pages in the users’ journeys, the specific page components that caused the symptoms
• IT teams can go away empowered to fix and improve what is already highlighted, instead of being asked to look back and see ‘what happened’

Budget dilemmas around whether to spend software dev. time on new features, or address the worst of the known bugs are easily resolved:

• There are always bugs in a  website! The question is which bugs are losing you the most money? The KPIs answer that for you – invest in those and ignore the rest
• Budget spend for things IT want: hardware, upgrades in software packages,  instead of them being a cost you can demonstrate them into a ROI
• If it can be proven that a request for budget will speed up User Journey X, or reduce errors on User Journey Y, then they can have their money.

Every online retailer is different, but a shared aim seems to be focus around the one thing that success online requires: having IT and Marketing pull together on:  continual improvement of the customer experience 24/7.

That ensures a path towards the best multi-channel delivery:  the best retail mobile website, the best retail kiosk website and the best retail website.

I wrote earlier this week how thinking ‘page’ is not as effective as thinking ‘Journey’ – but putting that to one side  – in particular I was asked about Google’s Page Speed tool.

It’s a useful tool, I’m not knocking it, but I am saying that it’s scope and limitations need to be understood. You can entirely waste precious tech team resources if you don’t.

In the pursuit of “How can we make the website faster for end users” it has a role,  but first and foremost be clear: it is a tool that makes recommendations on good things to do, in general, to speed up a page.  Note I said ‘in general’ – because the issues today in your website platform that are causing poor customer experience online may be quite different to what Google Page Speed will recommend to you!

The analogy, is talking to a knowledgeable friend about a problem your car has cold starting, and they say your should change your air filter, it looks due.   It’s not wrong advice, air filters can be a source of problems, but for your car there are so many other specific places that could be causing your issue, that the generic advice is more than likely to miss the mark.  Of course it’s quick and easy for your friend to offer a generic solution that is know to work in many cases rather than to dig deeper and actually check performance of spark plugs, timing, ECU boxes,  petrol filters etc etc and find the genuine root cause!

So the tool is good at pointing out some useful things to do that, in general, are sensible things to do,  but be aware they make very little difference in practise to your specific site’s speed problems.

The second limitation of Google Page Speed, and it’s not a bug it’s designed this way’,  is that it intentionally ignores everything about your website performance, except the shipping of page content to the end user’s browser.   That is, of course, a sensible thing to measure and optimise and track over time, but on your website, your users may be getting a really nasty user experience every time you ramp up an online campaign for reasons quite outside that scope.

In fact, in the real world most of the performance problems that we find are causing our clients lost sales online are at the website datacentre.  It’s easy to add more features and complexity to a site in an attempt to increase the richness of the user experience and end up with unforeseen quirks and slowdowns due to effects in your software in your web platform.

These are the kind of typical multiichannel retail  user experience problems that Google Page Speed can never help you with:

• Login is sometimes slow
• Add to Basket performance is variable
• Checkout page sometimes slow

So, if your tech team spend a lot of time on Google Page Speed work, and achieve higher scores when they run them against the Google tool, be aware that it may not make the difference to your user experience that you’d like – it may miss the ‘make website faster’ objectives that you have.

It’s not a tool that sits close enough to your real user experience, to really belong in any kind of KPIs for your site – for that you need to measure by Doing What the Customer Does‘.

PS
Just to be really clear, the tech factors that Page Speed measures are in themselves good things, so yes we do support sensible use of:

• Browser cache optimisation
• Reducing network round-trip delay by reducing the number of objects per page
• Reducing the number of packets per page by minimising content (minify’ing)
• Avoiding HTML content ordering that delays rendering times
• Optimising for mobile  (we have monitoring services for m-web and mobile-apps)

The most interesting ones are with organisations that have evolved far enough along the eCommerce evolutionary path to realise that it’s no good IT and Business teams sitting in separate silos. Those that understand that all teams need to pull together to achieve better online sales for their company, that their functions are interdependent and that it is through focusing on continually improving the technology underpinning the user experience that they will achieve their biggest success.

Its at that point in an organisation’s evolution, that things online get exciting. The pace of change picks up, the common language of metrics derived from a ‘Do what the Customer Does’ approach to measurement reduces friction, and overall there’s a buzz and excitement growing as sales and conversions rise. Not only that but there is a new satisfaction for tech teams that their work has made a visible difference and they are no longer just the whipping boys for website shortfalls!

But there’s still scope for the momentum to be lost through some common pitfalls.

The downside to the tremendous pace of online change is that beliefs based on old knowledge are still hanging around in the dusty corners of boardrooms and server centres up and down the country.

Let’s throw a bit of light into some of these dark corners.

Page resolution speed is a vital element for good Sales Conversions.

Sure: new dialogue opens up with Tech teams around speed of User Experience, and early initiatives gain success through measuring the performance 24/7 of the multi-page Dynamic User Journeys that reproduce what real customers do.

However… someone, somewhere starts to get obsessed with the speed of individual pages in isolation to the user’s entire Journey. Complex multipart journeys need to be considered as a whole as well as their component parts.

Of course, once you are measuring 24/7 multipage User Journeys, one of the key metrics it will quickly provide, is which of your page types slow down the most under load. And of course, knowing that fact, its good to aim to speed up that page.

Speeding up a page is good of course – but once you’ve achieved some gains, you need to go back to the Customer level, to the user Journey: and see what the next issue is not stay blinkered down at the one page basement.

But, whether initiated in Marketing or in Tech the focus can shift from the customer, from the multi-page User Journeys that they follow: and can become all about ‘go faster stripes’ for just one page.

A page that resolves quickly but does not meet user needs as part of a journey in other ways will not perform well in terms of results.

A page is not a journey.

It sounds obvious, right? But it is very common for people to focus on an isolated step of a journey – for example the product page on a retail site – but this is often the just the first or second step in a functionally complex multipage interaction and as we know from a wealth of usability data it is often the later steps such as checkout processes where users are more likely to have a lower tolerance for problems.

But you’ll see forum discussions, full of requests on ‘making my page faster’, how to ‘minify’ my page, tricks to improve caching of my page and so on, but this shows a lack of understanding as to how sites work now.

Focusing on one page, is rather like going back to the early days of the Internet – when pages were simple, before CMS and inventory management systems became commonplace. In that world, the performance of the page was indeed not determined by any other page- it stood alone.

But in the modern world of rich pages dynamically put together in real time – that is no longer true.

To take a simple example let’s look at the CSS and JS files that a page needs: if you have already visited other pages at the same site, your browser will have those pages in cache already, so the next page is automatically lighter. It doesn’t just matter where the user goes, it matters how they got there, in other words, it’s the journey that counts.

One page is no longer just one page.

What’s more – with the advent of widespread AJAX and increasing HTML5 – the page actually changes as you move around it.

One of the most important links on any page on your site is at the end of Checkout, the ‘Confirm Purchase’ button. On many sites you cannot reach that button until you have filled in some previous fields; and in the process various parts of the page may have opened or closed in a “concertina”.

So you cannot optimise the performance of that page, without measuring all those steps within the page. All the little AJAX server connections that may happen as the user acts.

Increasingly many pages are made up of a number of disparate components, sometimes provided by, or displaying content from, external sources such a media players, ad servers, news tickers, twitter feeds etc. What appears as “a page” to the user is actually more like a “housing” for several “sub pages” that update at different frequencies, from different sources, hosted in different places. In the course of completing a user journey some of these components may be more important that others.

So optimising “a page” is, taken out of context, quite the wrong focus.

Realism is vital but it is not simple.

Yes, it can be a simpler world, when you only care about one page in isolation, and ignore meaningful Journeys – the web page optimisation route.

Yes – it can make Load testing projects much more bite-sized.

For data to be meaningful and actionable and all those other good things it needs to tell us important information about the real world. The snag is that the real world is not “easy” or “simple” it can’t be summed up by “one simple top line number” no matter how handy that would be for meetings and presentations.

Only the MultiPage Journeys reveal how Fit for Purpose your site is

If you ignore the Journey numbers and look only at page numbers: you are no longer measuring Customer experience, and can easily end up with a single page type that is 20% faster, but the overall User Journey no faster at all because some code optimisations have inadvertently slowed other pages!

Or a new sporadic error type may be appearing after a small tech change, so that for some of the boundary conditions that pages only experience when they called from other pages in certain paths.

Those errors are losing business from real people – but you won’t know if you only measure page performance.

A good number of the projects we pick up, clients have already some page-based website performance monitoring in place – some of the tools they have subscribed to and used before coming to us have been single URL focused, and given them a false sense of security that is shattered when they see the actual experience their users are getting when our Journey based measurements start to deliver data!

It’s not about the Page – it’s about the Customer Journey!

Contact us for more information or to discuss a demo or trial.

I’ve been very impressed with the team here, how they’ve been able to juggle such a busy period whilst at the same continue development of our testing systems, to allow us to model even better realism for our multi-channel retail projects.

There’s been some fascinating graphics of realism too, as we’ve evolved our approaches.

Firstly, our continual development of our dynamic User Journey based approach has not only benefited 24/7 monitoring of ‘Do what the Customer Does’ dynamic routes through a site: but has provided the same realism gain to our load testing too.

Just last week we helped nail down a problem that was losing sales – doing a search in the usual top right search box on this retailer site, would produce an error response for certain keywords chosen – words that whilst not in the top 10 search terms on their site, were words directly related to the products being sold.  It’s only because our dynamic, randomised approach was able to try so many combinations that some of the slightly ‘ under the water line’ holes were spotted.  And it helped that our technology is flexible enough to allow us to run Journeys against their website and their in-store kiosks too; that provides better realism of site conversion across all channels.

But the brainy engineers here just recently have been looking at ways to generate website Load tests with even more realistic spread of virtual users.

The work was triggered by the fact that more and more of our clients are using the Cloud to host parts of their online store, and we’re seeing a big jump up in the peak customer capacity of some clients, the ones who get their software properly Cloud ready.  Which means we are adding to our own load testing infrastructure: we run both dedicated and cloud based test servers.

So, controlling more and more servers during a load test, aiming for higher traffic peaks and handling the complexity of cloud servers variable capacity has thrown out some intriguing challenges.

One of the main strengths of the cloud model is also the cause of one of the major differences involved with testing on a a cloud based system, and that is that your servers are not just “your servers” anymore.  While there are many benefits to this what it means in testing and performance terms is that they cannot be relied upon to give you exactly the same amount of power and performance in each instance and will sometimes not be able to consistently deliver the amount of “oomph”required. However, this is exactly the realism you need, as this is exactly the user experience problem that can occur on the live site.

Once you then throw in some Think Time realism  per page we are starting to see load tests that need substantial ramp-up times simply to ensure a sensible mix of users across all stages of activity – to ensure realism.  You have to work harder to avoid ‘bunching’:  where at any one moment of time too many virtual users are active in one task or page, and not enough are active in other areas.

Imagine starting a 100,000 virtual user load test made up of a number of User Journeys that, although following dynamic randomised routes, all start at the same place, the home page.  Without ramp time cleverness your first couple of seconds would be 100,000 home page requests and nothing else! And then for the next period there’d lots of activity all over the site but absolutely zero on the home page, until the Journeys start to finish and new virtual journeys start the process again at the homepage !

So ramp-up is vital to get a realistic load on your site, to spread users across the journeys in use.

But… ramp-up time is wasted time as far as measurement of your sites ability to handle seasonal traffic peaks is concerned because for at least two thirds of the ramp up, your online store is not breaking out into a sweat.  Everything is smooth, no errors are thrown, no pages slow down, and all the lovely graphs of server utilisation show nothing much happening.   Wasted time for all the engineers on duty.

As most organisations want their load testing out of hours, to avoid impacting real users,  there is only a limited time window over night to get in as much measurement and as much evidence of things needing to be fixed as possible.

Some nights, we reckoned we were spending 20% of the time, thumb twiddling in ramp-ups.

So, our clever team have been experimenting with ways to shorten the Ramp up times, but maintain and even extend the realism of load testing, the user spread.

The problem of lost time during a traditional  extended ramp-up – is shown clearly in this graph: the ramp-up of virtual users in pale blue is clear, with the outcome that it’s nearly 300 seconds before results start to come in (the green line) – between 200 and 400 on the X-axis before all the bases are filled and Journeys start to finish .

With lots of think time at each step of the Journeys making up this test it added up to about 4 minutes before the first starts, got to the end.  This is a low volume example: on some major client projects complex think time needs sometimes involved 30 minutes ramp-ups!

But credit where credit is due, this Ramp-up did a  good job of setting up the users on bases so that a constant rate of about 80 users per minute (it’s a low volume test) are finishing Journeys, once the ramp-up was done.

Dynamic Start approach  -  this alternative, has no ramp-up time at all, but achieves a mix of virtual users spread realistically instead by dynamically moving and adjusting think time: optimising the spread of users within a very short space of time by modulating think-time for the first run of each virtual user in the 400 concurrent virtual users that are quickly fired off:

It’s clear that this time the green line showing completed User Journeys per minute, gets up to the target 80 level much more quickly – within 30 rather than 300 seconds on the X-axis.  Note: always in load testing whilst it’s easy to control what virtual load you put in, the most important metric is how many User Journeys can be completed per minute: that’s your capacity measurement that your merchandisers and sales team want to know can handle their forecast sales figures.

So the team were pleased by early experiments on that approach.

Dynamic Ramp approach

Results were already of interest, but it’s always good to have choice, and natural for clever software guys to makes things even better :<) so the team also experimented with a 3rd approach to the realism challenge – called for convenience Dynamic Ramping:

Looking at the graph, this has similar properties in terms of achieving rapid balancing the of virtual users, but as is just about visible on the graph, it does use a ramp-up time, albeit a very short one, and this time with a different approach to adjusting think time to fill the bases: based not on moving think time between steps, but on modulating it to best fit the first wave of 400 concurrent users across the journey steps.

Conclusion

As eCommerce is a fast changing technology world I’m always interested in how real business benefits can be gained from clever software guys – so apologies if this blog got a bit deep technically, into Test Performance Consultants territory.

But if your company already use our services, or you have a colleague who’s worked before at a retailer who has, be sure to ask and find out what deliverables were gained. Although it may be angles that they’ll bring to mind other than Ramp Up optimisation like the above, it’s very likely to be realism based features that gained the benefits.

Certainly across all the new clients this last quarter who’ve upgraded from simpler approaches to trying out same of ours, the common thread has been their desire for more realism, so that they are prepared with better facts to optimise their website conversions.

Category Archives: Website Load Testing

Load Testing MCR Retail websites

A B2C website client, Inghams Travel, has been so impressed with our Dynamic User Journey monitor service over the last year and more, that they decided to put up our logo on their website footer, to let their customers know their experience online is important.

It’s a first for us – and extremely unusual for a B2C website to badge up with one of their B2B suppliers.

So thanks to the team at Inghams – hope you sell more ski holidays this year!

The travel sector does have more challenges online in terms of customer user journey monitoring – having a dynamic inventory, with packages coming and going very rapidly, a simplistic monitor approach that just gets the same list of pre-defined URLs each time, just won’t find the gotchas that real users are exposed to.

So thanks to Inghams for the recognition – and thanks also to the many clients who’ve been praising the team here, for how much our services have been able to impact your bottom line – you’ve kept us all busy  the last couple of months, notably with the travel sector and retailers’ pre Christmas website load testing.