At the April 3 Peer Incite the Wikibon community had the good fortune to learn about how Mike Adams, a storage specialist at Lighthouse Computer Services designed data protection for his company's customer-facing cloud storage offering. The discussion made it clear that the organization as a whole — not just the CIO — owns the data protection decisions. Senior executives need to pay close attention when it comes to data protection and recovery, particularly as it relates to full disaster recovery.
Imagine you’re the CEO of a medium-sized business. Now, imagine that you get a phone call at 3AM from the police with a message that the fire department is at your business, which has just burned to the ground. Your first thought will be, "Is anybody hurt". Your second will be "How do I get operational again, and how long will that take."
But We’ve Always Done it This Way
Traditionally, the CIO or the IT department as a whole has had primary responsibility regarding data backup and recovery decisions. Personally, I’ve seen a lot of IT departments that have ‘’’sole’’’ responsibility for these critical services. At some, primary decisions regarding backup and recovery have even fallen to the administrator responsible for handling the tape backup process. It was left to this single individual to determine what could be the fate of the organization.
This has most often been true when it comes to protecting data at the local level. After all, it’s IT’s responsibility to recover quickly from what could be considered minor issues, such as accidentally deleted files, failed servers and the like.
Going back to the scenario, if you’re a CEO who has left responsibility for backup retention policies and methods to someone in IT, you may be disappointed to learn that it will take five times as long to recover from this disaster as you expected and some data will have to be pulled from last week’s backup set, because this week’s was lost in the fire.
It’s at this point when you find out that “we’ve always done it this way” is going to mean major problems for the company.
Data is Only the Beginning
As a CEO or other senior executive, never forget that your IT department is working diligently to protect your company’s data. Unfortunately, the data is just the tip of the iceberg when it comes to recovering from a disaster. I see disasters as a spectrum. At the left hand side of the graph, you have low impact “disasters” such as accidentally deleted files. These kinds of issues are generally really easy to recover from. At the right hand side of the graph, you have high impact disasters which are extremely difficult from a recovery perspective.
Really, all of this is disaster recovery. To the person who accidentally deleted a file, a disaster has occurred. That’s why I consider the full spectrum a disaster scale.
When it comes to recovery, you need to think about more than just data, however. For each successive move to the right on the disaster scale, recovery becomes more difficult. In the full-on disaster of our scenario, which is close to the right end of the scale, you first need to reimplement business processes, then find a new physical location for your office and data center, then buy and install everything from office furniture to servers, before you have anything to restore the data to. In other words, you need to consider how you’re going to resume business once you’ve recovered your data.
While your IT department will be able to master the data recovery element, it takes a group effort to ensure that business can continue as usual. Sure, you might choose to have your CIO lead this group effort, but this individual will work with a team that includes a broad array of people from across the organization. I’ve written previously about why IT governance is so important in an organization. When considering the range of options that are available for protecting data and the business, your existing governance structure may provide a ready-built framework for this collaboration.
How much insurance do you want?
Every organization needs to come to grips with a couple of key questions:
- What is your tolerance for data loss?
- What is your tolerance for business interruption?
Bear in mind that saying “None” to both of these questions will mean spending vast sums of money to put into place systems and processes that can achieve your zero data loss, zero interruption goals. Data protection and disaster recovery are insurance policies, so think of it in those terms. As you add features to the insurance policy, you also add new costs.
If you can determine how much downtime costs on an hourly basis, you can quickly determine how much insurance you want to buy to minimize interruptions of any kind.
CIOs: Don’t wait… act
If you’re a CIO reading this article, here’s my advice: Engage your executive team as soon as possible and make sure that every single one of them is aware of exactly what data protection and disaster recovery mechanisms you have in place right now. Further, work with that group to determine how much insurance the company wants and start creating solution options that meet those marks.
Action Item: As a CEO or other senior executive, make sure that you take an active role in understanding how the business protects data and the business itself. Working with the CIO, take steps to understand the range of options that are available. Then, if that dreaded 3AM call does take place, your second thought can be “we’ve got a plan for that”, and, even if you don’t necessarily rest easy, at least you know that the business is well taken care of.