Web site failures – expect the unexpected

August 25th, 2010 by deri

Add to: Facebook | Digg | Del.icio.us | Stumbleupon | Reddit | Blinklist | Twitter | Technorati | Yahoo Buzz | Newsvine

As web performance testers, we regularly see subtle technical issues cause substantial failures in website user journeys -we expect the expected.

Whether in 24/7 monitoring, when our multi-user-journey approach routinely highlights that all areas of a web site may be functioning fine, apart from the one crucial  journey, maybe CheckOut: or Add to Basket.

Or if we’re running a web load test, it can sometimes result in a a whole site stalling: fortunately this normally happens only for brand new sites being load tested well before say a Christmas rush *.

Expecting the unexpected **

Often, a website problem that is visible and obvious to the site visitors at the time, is never mentioned online at all, or only days later. ***

But what we don’t often see, is a web site manager have a problem and then make public the issue, let alone explain the root causes.

Hats off then to IT director Chris Waite at TravelRepulic.

Waite first explained that they’d seen an unexpected traffic spike: 30% higher than the week before, and the best day’s visitor numbers for “several months”.

Read the rest of this entry »

Posted in Uncategorized | No Comments »

The Marketers new badge of success – the site crashed!

July 22nd, 2010 by deri

Add to: Facebook | Digg | Del.icio.us | Stumbleupon | Reddit | Blinklist | Twitter | Technorati | Yahoo Buzz | Newsvine

Interesting bit of marketing spin this week from Marie Claire magazine – who reckon that their latest recommended must-have beauty product was so much in demand that it crashed the Boots.com web site!

Confidence in Boots takes a knock from that, doesn’t it? And today (3 days later) , the product is in stock.

Humm, kind of bad manners of them to tell tales on Boot’s website problems.

And actually…today July 22nd I read that:

‘…a Boots spokeswoman told Web User that the Boots site didn’t crash at all…”

So was it purely  a journalists fevered desire to spin a story and truth was the casualty?

Read the rest of this entry »

Posted in Uncategorized | Comments Off

Firefox 4 – New features for testing shown in London

July 22nd, 2010 by deri

Mozilla came to London this month, to explain some of the new features in the upcoming version 4 browser.

The web testing teams here at SciVisum have been Firefox users since way back 1.0 days, and it looks like the new features of version 4 will continue it’s popularity with us and web monitoring and web load tests worldwide.

Of course there are the speed improvements: 27% faster javascript it looks like.

But there are other nice things that will help our daily tasks as testers, comparing the underlying technical issues that impact web performance on each client site.

Read the rest of this entry »

Posted in Uncategorized | Comments Off

Less Flash – More user engagement

July 15th, 2010 by deri

It’s true that over time, the pain that Flash causes when trying to measure meaningful user journeys on an ecommerce platform, I’ve learnt to live with.  It just does make the web test engineers task more demanding, especially when clients have bespoked up clever encryption or data pushing implementations, like some of the recent flash project testing we’ve seen.

But it seems that we’re not the only ones with a Flash issue.  Whilst Scribd, the social document sharing players were willing to drop their 3 year investment in flash coding on their site, confidence in just how well it turned out for them is much higher than any could have expected.

Back in May they announced their intention to drop Flash, and move ship to HTML5, committing themselves to recreating tens of millions documents to HTML5.

The outcome, the same 50 million unique visitors each month: but they are staying longer and sharing more with friends.

In parallel of course, Scribd have added Facebook  integration and the launch of iPad has helped them too – so like any before and after comparison in ecommerce  it’s hard to be sure of any one root cause.

CTO Jared Friedman  said that user engagement had doubled recently, indicating some serious web acceleration.

I don’t suppose the stream of web testing we see for Flash (and for Flash video streaming testing  too of course) is going to dry up any time soon, but it is going to be interesting to see how the HTML5 versus Flash debate develops.

Posted in Monitor Video Streaming, Web Flash Monitoring, Web monitoring | Comments Off

Monitoring Video Streaming Performance – HTML5 in the wings

July 5th, 2010 by deri

Speaking as web journey monitoring people, there are a number of reasons why a move away from Flash for delivery of streaming video, to a slightly more standards standard like HTML5 is attractive.

In a nutshell, writing user journey scripts to benchmark or load test the performance of a video streaming site (or any non-trivial web site for that matter) can be quite tricky when you have to either emulate  ‘doing what the browser does’  – or else control and manage the browser to do it – and the nature of Flash’s streaming with RTMP under the bonnet, and the layered intricacies of how folks add authentication, encryption, DRM and etc: can make for a major effort.

Whereas the promise of a standard that is open and a standard from the start, could make that much easier building tests.

But it’s interesting that Youtube engineers  have come out this week and said they’ll not be dropping Flash for a while – or ever for certain cases: despite them being early adopters and advocates of HTML5 – they already have their own YouTube streamer for HTML5 for example.

Read the rest of this entry »

Posted in Cloud web testing, Monitor Video Streaming, SaaS monitoring, Web monitoring | Comments Off

Using the Cloud – Big Name outages hint at Cloud Cuckoo Land

July 2nd, 2010 by deri

We’re seeing a growing number of clients using Cloud technologies, whether public or private, to support their delivery online. And whilst web monitoring and testing user experience is a bigger challenge on the cloud – it is also even more important to do so – due to the wider range of technical problems that can occur, and because of the greater difficulty in mapping actual user experience from data from traditional server monitoring.

Just look at the Hall of Shame of big names caught out on line with serious outages recently:

  • Inuit’s online accounting sites QuickBooks, Quicken, Intuit.com and TurboTax were out for hours:
  • NetSuite – leaving companies worldwide who run their business from their CRM portal offline for 30 minutes and slow for some time after that
  • Sage USA was unstable for around 24 hours at the start of June

I’m not sure the customers out there would agree with Steve Jones of Explore Consulting (a NetSuite advocate), that “occasional minor outages are just part of the reality of cloud computing today.”

Whilst there has been increased discussion of SLAs to use with Cloud suppliers, the best SLA is only paper if it is not backed up with a meaningful monitoring program.

We’re finding an increasing number of London clients and wider who are using our in-depth User Journey monitoring services, are utilising the service to firstly, ensure their money-making routes through their site are performing, and secondly to build these metrics into SLA terms.

Whilst monitoring the cloud is not trivial for a site confidence in your cloud suppliers will grow if you have hard evidence of the impact on user experience 24/7 of all your technology including public cloud monitoring or private cloud or conversely, tricky negotiations with under-performing cloud suppliers will be based on hard evidence, not just anecdotal stories of problems.  Building the right highly-dynamic Journeys for your site is therefore vital – to have that hard evidence.

Sage VP Paul Johnson wrote:

“We have been experiencing instability and outages with some of our online systems over the past week”.

Ouch. However, it’s highly likely that Sage users had experienced much shorter, sporadic stability problems in the months up to June – with better monitoring maybe the Sage cloud team would have spotted those early warning signs and been able to get remedials in place before it got out of hand.

Web monitoring is not rocket science, it’s about thinking through how your clients use your site and the multi-page User Journeys that they follow: designing in the dynamic selections, and then building a 24/7 framework that can exercise those Journeys.

There can be difficult subtleties, such as when a Journey that is intended to finish with an ‘Add to Basket’ step finds at a later page this product is out of stock and cannot be added to the basket…. What should the User Journeys script do?  Maybe start again from scratch, or maybe go back to the search page and choose a different product offered?  How many times should it try?

With the right web monitoring tools or supplier, all those options are available depending on the design of the company website. However, with the wrong tools, there is no way to handle such real world cases.

Thinking through such subtleties of a meaningful user journey monitoring programme may take a little time: but will pay dividends when you have the web monitoring SLA on the table to negotiate with your internal tech teams and various cloud suppliers.

Posted in Cloud web testing, SaaS monitoring, User Journey Monitoring, Web monitoring | Comments Off

Online Retailing – cautious about the future?

June 30th, 2010 by deri

“We won’t see retailers that don’t have transactional websites in 10 years time”

Internet Retailing Magazine write in The Key To Retail 2010.

10 years? Why so cautious?

Any day now – would be my take on that.

Of course, as other folk have recorded, despite market growth, the credit crunch is impacting retailers online.

So it’s clear that the online commercial environment is set to become far more competitive in the near future, and as a result the web customer is likely to have higher expectations regarding their online experience. Respectively we can expect to see an increase in so called ‘web rage’ as users display even lower tolerance levels with site errors and slow page delivery; when there are so many other websites a mouse-click away vying for their attention, and more than willing to take their business, customers are under no obligation to remain brand loyal.

Having a competitive edge in what is set to be a frenetically congested market place will depend on the e-tailors’ ability to provide a consistent, positive online experience on site. As online stores pile on the extra goodies of videos, RIA, AJAX and the like to create a richer user experience: they are also increasing the likelihood of things not working on site!

Confidence despite the new complexity is needed.

The only way to achieve consistently high standards (customer confidence) and ensure all your applications are performing is to have a clear view of your website by monitoring from the customers perspective. Website monitoring is set to become an essential part of IT management and digital marketing activities as an increasing number of businesses will be forced to face harsh realities when company ROI is negatively impacted due to poor performance testing.

Companies need to wake up to the fact that web site monitoring is not just hitting three URls in sequence: homepage, category page X, product page Y.  It needs to be true User Journey monitoring: doing what real customers do: with dynamic searches, choosing products at random, adding to basket and so on. It is surprising how even big names, who have web monitoring in place, are still not covering the money making User Journeys they need to  – all too often leaving it up to the contact centre to find out first about the customer’s online problems.

Customers encouraged onto a website will quickly retreat when online performance expectations are not met, or they simply cannot access the site due to high demands generated through marketing efforts.

Even though we can expect both retailers and customers to flock to the online market place, it’s clear that without thorough testing of IT infrastructure, the customers may come – but they certainly aren’t guaranteed to stay.

Posted in Browser monitoring, Lost Sales monitoring, Missing Product Monitoring, Uncategorized, User Journey Monitoring, Web monitoring | Comments Off

Monitoring video streaming – the BBC set a benchmark

June 24th, 2010 by deri

I picked up the discussion at econsultancy.com about the quality (or otherwise) of the two main UK  TV channels video streaming online:  BBC versus ITV.

As someone in London noted:

“…However, the worst thing about the online coverage is the quality. The small screen shown above isn’t quite big enough to watch the game properly, but if you enlarge the coverage to full screen the resolution is not good enough and a blurry mess ensues”.

“Worse still, the coverage is jerky and needs to reload / buffer every few minutes…

Video streaming monitoring is something we’ve been doing more of lately: and it’s interesting the variety of different ways in which people do it.

Some run streaming servers in-house, others use specialist streaming 3rd-parties: or roll their content round a CDN network.  Some wrap their streams in encrypted channels, or DRM protect them; some use industry standard protocols and some wrap their own bespoke things round complex home-grown streaming server software on non-standard  TCP ports.

Whichever way, if there’s streaming on your site, confidence that users are getting a fast and smooth experience is not something that can just be assumed.  But of course, given the widely varying and potentially complex technology being used to deliver what the management may think is ‘just a video stream’, it’s often impossible for organisations to measure and monitor the user experience of their own streams, with the tools available to them

An example of the effort needed, for the video streaming monitoring we set up for the BBC over the world-cup, we had to create multi-step User Journeys that handled the first few steps of a user’s journey, navigating from the homepage through to choose the game to watch – and then monitored the streaming.

Fortunately, our existing web monitoring service is designed for handling the most rich multi-page User Journey, with log-ins, dynamic random selections in real time and so on:  so the pre-steps in the journey before the actual streaming starts were straightforward.

As with all project plans it was a last minute dash as more in depth monitoring was required than anticipated: but they were successfully set up in advance of the first game, and were able to provide the BBC team with the real time picture of performance.

Our journey script was also able to handle the fact that there are not live matches being streamed 24/7: between matches, the Journey looks for a live stream and doesn’t find it of course: so to avoid false alarms we simply coded it so that this didn’t throw an error .

And on Wednesday’s England game in the afternoon, at 1:0 at half time the graphs from our monitoring showed the streaming was as good as the mood of the English fans!

Posted in Monitor Video Streaming, Web monitoring | Comments Off

Web performance tuning = 10 % profit increase

March 26th, 2010 by deri

OK, I exagerated, 9% is the actual number.

Sure, we all know our sites could be a tad faster, but its maybe a shock to realise that the evidence all points to the fact that getting your money making User Journeys a little faster, will help your company bottom line so much.

For starters: Looking at one of the major business analysts report , the Aberdeen group – they found 9% was the profit gain from a little hard work in the load testing and performance tuning space across the 158 companies they talked to.

Then there’s the folks like Shopzilla’s Phil Dixon – who made public these findings:

  • redesign resulted in a 5 second speed up
  • speed improvement allowed a 25% increase in page views
  • revenue up by 7-12%
  • 50% less hardware was feasible, because of the code changes

This isn’t really news to us all – as Aberdeen pointed out: less than half (42%) of organizations were satisfied with the performance of their business-critical applications.

Many of those unhappy companies probably have some measure of performance in place.

(If you’re interested, this week there’s a LinkedIn poll on Networking monitoring tools).

If their performamnce measurement is to use a supplier with only a simple non-dynamic approach to hit a few URls to load test the site, confidence in knowing exactly what slow downs impact real users is obviously going to be weak.

But web load testing of dynamic joruenys generates extra profit margin – that’s good.

It also saves a bunch of everyone’s time.

We have London clients, where the whole team are working flat out – and now that they have hard facts of User Journey perfroamnce on the table, they save a whole bunch of time that would have been spent in the past working out what to do, based on incomplete data.

The evidence per User Journey load test makes decisions for them! As we measured the site, confidence in the facts of what and why was hurting their visitors increased.

It gives the C-Level decision makers the evidence that it’s a sound business choice to spend money on improving Journeys 1 and 2 but not on Journey 3 and 4 which are OK.

So there are no vague requests to make the whole website faster- but focused ones that the tech and business teams can agree together, looking at the Journey performance graphs round the table, and pass to the purse-string holders as a combined budget request.

And if the business case doesn’t satisfy you – how about some pyschology?

“The bottom line is that people in flow are having fun, and truly enjoying themselves. Interactive speed is a significant factor in all models of user satisfaction. Make your pages load quickly and minimize the variability of delay. Be especially careful to avoid sluggish response after your pages have loaded.”

wrote: Dr. Mihaly Csikszentmihalyi, (professor at Claremont Graduate University and former chair of the Department of Psychology at the University of Chicago) – in his book ‘Website Optimization’.

So however you look at it, uk companies across the board should be looking at the ROI of bringing their web performance activity up to date.

Posted in Web monitoring | Comments Off

User Journeys: How not to do web monitoring

March 3rd, 2010 by deri

It’s funny sometimes how much effort is spent on trying to make a web brand handle the traffic peaks associated with major marketing campaigns – and yet how little real progress can be seen.

Poor old Nat West / RBS hit the papers last week, when their online banking was down for over half a day. It can happen to the biggest of the best of London. (well the biggest anyway).

Heard about this apocryphal Grimms Fairy tale of an eCommerce story this week.
For these big brand guys’ web site, confidence that it could handle the expected future traffic hammering was vital.

So the Business Team rightly called for a Web Load test. All agreed it was good thing to do. And the Tech team were tasked to arrange it.

A month later the testing was said to be done and satisfactory. It took a few asks and some nagging before the actual test report managed to hit anyone’s desk. What it said was, that the site could handle up to 30,000 concurrent users.

At first, this gave the eCommerce director of the site confidence that it would be OK. It was a big number, he thought.

But when he asked how many Orders per minute that would mean – nobody could tell him: the testing hadn’t been at that level he was told, it was more abut CPU and page hits.

He asked for more: what would the tech team do to handle more throughput – their response seemed fairly generic: they would have to go away and look at new hardware. Any software tweaks possible? Not really it seemed.
So no concrete plans to improve capacity came about.

And two months later on, the big marketing day came, and all the marketing money that was planned got spent, and the visitors came and the site was busy.

But sadly many visitors found the site slow and error prone that week.
It never crashed as such, but was less than smooth. With the problems on the site, confidence ebbed and the Marketing team at one moment even pondered a mailshot that kind of said sorry. But canned that idea anyway.
Certainly the Conversion ratio was considerably lower than average. Even the Chairman said his wife had found the site a problem at the weekend.

The eCommerce director asked his team how bad the site had been that week. It must never happen again: “so what actually did happen” he asked.
Someone said that there was some performance web monitoring in place, that would show the numbers. Nobody had the details to hand. Anybody was willing to talk to the guy that was supposed to have control over the outsourced web monitoring.
“If we don’t know exactly what happened on our site, confidence in the next campaign is going to be hard to find” they all agreed.

Finally, the details and numbers of the web monitoring they had in place for that week were found, and the graphs were pored over.

Oh dear, why is this Journey’s graph yellow all week? Ah, it seemed that that Journey wasn’t really following a journey like a site customer would: it was just a series of URLs, pre-fixed and hit in sequence. And the product ID set in the URLs, ran out of stock on the 2nd day of the big week, so the Journey couldn’t finish, so it marked up as a yellow for Warning

And this is strange – this other Journey didn’t show any extra errors or slowdowns at all that week? How come. The other two graphs do show noticeable slow downs.

Another bit of digging around with the monitoring suppliers. It seemed that although this Journey was called ‘Add to Basket via Search’, that it also was just a list of fixed URLs. And that list had been put together 6 months ago : i.e. before the site upgrade earlier in the year. The old URls that customers six months ago would have used, did still work. But they were not the URls that customers today follow: the Search button on www.company.com used to be handled at a URL of www.our2ndBrand.com/search.do; but on the site now, the Search Button actually takes users to www.company.com/newSearch.do.

Oh dear, the Add to Basket via Search Journey was not even hitting the major brand site any more! It was not a dynamic Journey, looking into the pages that the users sees at all.

That’s why it was showing all clear during the week when the real site was struggling.

The eCommerce director said, “when it comes to knowing 24/7 what User Experience is like on our site, confidence and words fail me. It’s been the wolf in bed all along, not Grandmother, and we really didn’t realise what our web site performance was like.”

And the Chairman asked the wood-chopper to come out of the forest, and help the eCommerce Director negotiate his redeployment with HR.

The Motto of the tale: next time someone says you have User Journey monitoring in place – ask them exactly what you’re getting: whether it is dynamically following what users do, or just a pre-cooked static list of URLS.

Posted in User Journey Monitoring, Web monitoring | Comments Off

« Older Entries