Management Toolkit

Dealing with the worst-case scenario

Mike Talon Tech Republic

Published: 28 Nov 2005 11:50 GMT

"The Worst-Case Scenario", or a total loss of all data-centre structures and systems, is the last thing any of us wants to think about. However, it is entirely possible that such a disaster could hit your organisation, as was shown in the US during September with Hurricanes Katrina and Rita.

During this type of disaster, the idea of Disaster Recovery (DR) is going to take on a whole new meaning, as your organisation scrambles to correct for everything that has gone wrong in such a short period of time. Major floods, fires, hurricanes and man-made disasters such as terrorist attacks fall into this category.

Though the largest of organisations will be preparing with availability solutions allowing them to fail over quickly to another data centre outside the scope of the disaster area, most companies will find that a response to a Level 5 disaster is truly a recovery effort instead of a fail-over exercise. The vast majority of organisations will not be able to afford or manage DR data centres that lie far enough away from the primary facility to be helpful in this type of disaster, meaning that your DR systems will be impacted by the same event that disrupted service at the primary site.

Even if you cannot afford to keep full-fledged systems up and running at another location, you can contract to keep backup tapes and other copies of your data in far-flung locations. Many companies specialise in just such recovery services, allowing you to find one that fits both your needs and your budget. This will allow you to deal with the immediate impact of the event and then recover your data to new systems from the copies warehoused off-site after they are returned to you by the contractor.

It is at this level of disaster that you will also have to deal with non-technical issues well before your technology plant can come back online. Level 5 disasters almost always include loss of physical space and — unfortunately — loss of life as well. When your employees are no longer available to enact a DR plan, you will need to act as quickly as possible, given the situation, to find new staffers, train them, and get things up and running again. Also keep in mind the immense psychological impact of these kinds of disasters. Employees have probably just lost their homes, possibly their family members and friends as well. Attempts to coerce such employees to immediately report back to work is both unfair and in many cases unethical, which could leave some large gaps in your DR efforts. Temporary staff may be available in some cases for you to use in the short term, but for the majority of cases you will simply have to redefine your DR plan to take the extra recovery time into consideration.

The best planning you can do for a Level 5 emergency is to prepare everyone for what they can expect and hold firm if executives try to make you commit to anything unreasonable. Set up phone chains and other alerting structures ahead of time, get your data out of the scope of potential disasters that may impact your production environment and be ready to deal with the harsh consequences of a massive disaster. The best you can do is to prepare: Level 5 disasters will find every hole your DR plan has to offer.