Automated validation testing - helpful or burdensome?

Validation testing can pinpoint areas for improvement and help focus scarce technical resources. But should you choose Automated or Expert-based validation testing?

Most ‘off the shelf’ Automated test kits use the World Wide Web Consortium (W3C) set of standards as the core basis for automated testing – perhaps with a few extra checks on top for basic accessibility or page speed.

Where human interpretation is not required to check dead-links or for basic html errors, it’s a valuable approach. But there are real drawbacks. Fine, there may well be areas of non-compliance with the W3C, but in the real world they simply don’t cause a problem to the user. Automated testing of thousands of pages on a site may throw up many lines of problems. Yet these might boil down to perhaps just half a dozen real root causes.

In an ideal world, most busy designers will want to know what to fix, not what might need fixing.

Automated reports, often quite thick, may hold relatively little useful content - sometimes hidden like a needle in a haystack. For example, what’s the value of listing the slowest 10 pages on the site when the web manager probably knows in advance that they will be his big PDF files with graphical data in them; or the chairman’s multi-megabyte corporate video clips!

A More Expert Approach

Contrast the above with the approach of expert test staff using a well-defined test methodology. The expert staff, fed by data collected by a semi-intelligent engine, is able to provide a far more comprehensive service than auto-testing. The end report, specific to your website, will be far more to the point. Its focus - issues that undermine the user’s experience on the site - is not just blind compliance to standards.

The tester is able to manually analyse how the core technology of the site is working: caching, load-balancing, networking issues such as packet size, use of compression, inefficient java script code, cookies, session IDs, redirection, search forms, data entry fields etc. All these are vital in making the site deliver the best to visitors.

Testers with a thorough knowledge and a clear understanding of the range of web site technologies and products in use will be able to find root causes for poor performance or errors that automated tools probably won’t even have spotted, never mind determined the root cause.

A site-wide audit enables the tester to categorise down, to reduce the list of findings to report. The web designer needs only to address a minimum list of actions, all of which are clearly worthwhile.

Such reports will include expert interpretation and concrete recommendations for improvement – expert input without the lack of focus that sometimes results from bringing in generalist web consultants.
The use of a third-party testing service can often provide cost savings over in-house expert-based testing – particularly where it is difficult for in-house staff to gain and maintain the necessary breadth of test expertise, when they have only the one site to focus on.

Third-party costs will obviously depend on the scale and complexity of your web application, but won’t break the bank. The return on investment can be very quick, as such testing can commonly provide performance improvements of 20 to 30%.