Wikibon Blog

Avoiding Disaster Recovery Disasters

Friday the 13^th is coming in four days so I thought I’d kick off the week with thirteen questions CIO’s should be asking about their critical IT facilities.

Information technology is a fundamental ingredient of most businesses and is intrinsically linked to the operations of an organization. As it relates to IT, CEO’s and Boards of Directors generally evaluate risk in two high level dimensions:

• What is the likelihood or probability of a disaster?
• What is the impact of a disaster on our business?

Ultimately, these executives are trying to determine what actions can be taken to mitigate risk and how much they should spend on disaster recovery and business continuance. CIO’s and IT practitioners need to understand, document and communicate the business processes that are dependent upon technology and dig deeper to evaluate a range of scenarios and business technology impacts in order to support the risk mitigation needs of their operations.

One often overlooked aspect of disaster recovery is facilities themselves. Discussions about facilities are frequently treated by IT as someone else’s problem or a ‘white elephant’ that is pushed off for a discussion at a future date. However facilities often represent a weak link in an organization’s disaster tolerance chain. CIO’s and IT executives should ask the following questions about their disaster recovery generally and their IT facilities specifically:

1. Have we conducted a proper business impact analysis and do we understand the interdependencies across our application portfolio, the risks associated with our critical applications becoming unavailable for long periods of time and the linkage to facilities?
2. Do we understand our recovery point objectives (RPO) and recovery time objectives (RTO) for our major applications, do we guarantee these metrics as part of service level agreements (SLAs) and do we understand how our facilities factor into this equation?
3. Are the data centers which house our critical application data designed for today’s equipment needs? Specifically, today’s server, storage and communications equipment should be designed for 50kW per rack whereas many data centers were designed for much lower power per rack (e.g. 2kW per rack).
4. Are the facilities that house our critical data designed for chilled water cooling to handle the increased heat densities of today’s servers?
5. Do we take a holistic approach to facilities? In other words, are our hot site, cold site, work area and off-site data storage facilities connected via a carrier-class network?
6. Will our electronic data vaulting and transportation scenarios allow us to recover large amounts of data in a manner consistent with our RTO guarantees?
7. Can we certify to the Board of Directors that we’ve adequately tested DR in a true failover and failback scenario?
8. Have we conducted a proper threat assessment and scored our facility against that evaluation?
9. Are the facilities that store our major application data designed to withstand blasts as published by the GSAs Interagency Security Committee?
10. Is our electrical up to speed? For example, do our facilities use dual underground power feeds via dispersed entrances, do we have an adequate reserve fuel supply and do we have reserve power generation capabilities? Here’s a decent checklist courtesy of Recovery Point Systems.
11. Is our network designed to withstand a disaster (i.e. do we use dark fiber, is it self-healing, etc.)?
12. Is our mechanical infrastructure (e.g. chillers, boilers, CRAC units, etc.)  redundant?
13. Is our physical security adequate? For example, are our facilities non-descript with limited public access, do we use sophisticated access controls, do our policies align with our security objectives, are we near terrorist targets and major cities?

Ask tough questions and you may help save your company. Don’t ask questions and you may be exposing your organization to disastrous consequences.

Got any questions?