Date: 11th January 2013
The news was bad this week that there are 2 nasty Ruby on Rails security flaws that will affect all the estimated 240,000 Ruby websites until they all update to the latest version which patches the problem, released 2 days ago.
But crisis response problems could cause problems for any of us – there is a lesson to be learnt here.
Whilst those of us not running Ruby can only feel sympathy for the Ruby guys rushing to patch their sites – which may be more effort for those who are a version or two behind, (having not needed all the latest features perhaps) – we can think about what the implications would be should the same thing happen to our own software and systems.
There’s a lesson for anyone running a website – not only will all Ruby sites need to be updated as a matter of utmost urgency, but they will then lose their support team for a number of days as they undertake an unplanned and time-consuming investigation to see if is there any sign that their site has already been comprised, and that everything still works correctly after the patch rollout.
How many days will this Dutch Government website outage last, in doing this exercise? What will the full impact of the downtime be?
Suddenly they will be short of sys-admin time to mange the site and investigate any performance issues or other bugs etc that may arise, and any planned sales and marketing campaigns cannot just be put on hold while this is done.
This lack of support resource means that even sites that were not directly impacted may lose money, opportunity or customer loyalty due to slow responses to the normal performance and error glitches this week!
This problem, of a sudden urgent task wiping out an existing team for a week, can arise from all sorts of sources, such as:
- unexpectedly big problems after a software release
- a 3rd-party supplier makes a small change to their service and causes your site to fail big time, until your system can be analysed to find the sensitivity
The Lesson: be prepared for the unexpected Crisis that stops all normal, vital, website support being done.
To ensure your site can run smoothly with a sudden drop in support team time make sure you have adequate, 3rd party 24/7 performance monitoring, and user-focused journey testing that will highlight errors clients experience, so your team can take a back seat from baby-sitting your site whilst they respond to another major, unexpected crisis issue.