Testing Flash Websites – I no longer hate them

Date: 1st December 2009
Author: Deri Jones

We’ve been testing quite a flurry of Flash based websites the last few weeks. Funny, a couple of years back I didn’t much like the use of Flash that I saw – but the web load tests on Flash sites recently have put me in front of some pretty clever portals.

Load testing flash web sites is a different ball-game than testing plain HTML ones. Proper Load Testing of RIA web sites is still not mainstream.

If it’s just a flash page here and there, then that’s not too bad, but some of the clever new sites are making much wider usage -and your average load testing tools and software don’t cut it.

Flash like any othr RIA technology can be used in standards compliant kind of ways (ish), or in line with some decent frameworks, or in total freestyle…  I’ve been in some heated debates in London about the pros and cons of these approaches!

One site we web load tested, was in the middle: a games site and had used the AMF (Action Message Format) framework, to manage the syntax between flash games and the back end servers.
Bolting an AMF layer into our load testing framework was feasible – because it’s all written by us, and designed from the ground up to be flexible.

This client hadn’t done a huge amount of unit load testing, due to the lack of flash load testing software to hand: our results were therefore their first glimpse of where their botlenecks where: and proided the vital capacity metrics for the bosses and tech team of ‘User Journeys completable per minute’.
Useful lessons were learnt, and tweaks enabled.

Compare that to another load test project – these guys were definitely in ‘freestyle’ mode in my categories above!
A very clever financial website: using flash finance feeds for a real time pricing service.

They had written a very neat engine, that used it’s own encryption framework, setting up connections over a non-HTTP port, and implementing their own cairo style push to get prices down the wire.
Lovely lightweight protocol on the wire -small messages , no reams of HTTP headers to pad things out – great for a fast real-time service.
It also had hooks to and from a regular HTTPS site, to do the logins, and similar, but their own flash protocol was doing all the share trades too.

Our guys had a quite a week of late nights, coding up a test harness to do the bespoke encryption and protocol handling and etc – did I already say our core web load test engine is flexible :<) ?…It is, but it still takes clever guys a good few hours of thinking to implement a new harness like this and not reduce the capacity of our load test farm to generate a healthy storm of traffic. Plus some healthy reverse engineering when the protocol on the wire doesn’t quite do what the client’s API documents said it did. Very tricky with bespoke encryption to muddy the water too (no chance to just read the ethernet traffic on the wire).

The testing and retesting cycle with this client was a joy to behold: their guys were as sharp as they get and improvements were made quickly, they understood immediately code root causes behind the detailed bottleneck symptoms we we provided: and achieved every spectacular performance gains in quick time. They had previously done a fair bit of unit-testing, so were kind of surprised that we were able to pinpoint some areas where great improvements were possible.

So, I’m not so anti-flash as I was. Yes, I know there are still accessibility issues and SEO and a bunch of other stuff you can get wrong with flash: but for highly visual, RIA web sites: if you’vea good team, then building even the freestyle variety is fesible, and effective load testing of flash web sites is well worth the effort.