True integration of deduplication into the IT infrastructure would allow data to be held in deduplicated format except when it is being processed. The data could be created, copied locally, copied remotely, migrated, backed up and recovered in deduplicated form. Applications could know about and control whether deduplication is evoked, and control any parameters.
This vision of deduplication as a storage service evokes three key questions for IT executives:
- Is it technically possible to create a deduplication standard?
- It is technically possible to create a standard and retain plenty of room for innovation. Deduplication has two parts. The hard part with plenty of room for innovation is creating the algorithms that detect common strings. The easier part is creating a schema that connects where duplicates have been found with the duplicate strings, lays out how the data is held within the same data volume, and allows protection of and controlled access to the data. This schema would be the standard, not the algorithms.
- Can the market create and sustain such a standard?
- Dave Vellante nicely addresses a potential win-win-win scenario between tape vendors, disk vendors and users in his piece “Data deduplication standards and the domino effect”. The market would effectively stall any attempt of a single company to establish a de facto standard. De jure is the only way.
- What is the degree of adoption that should be tried with and without a standard?
- Adoption of deduplication becomes an order of magnitude easier if a deduplication standard is in place. Without a standard it is necessary to deduplicate the data every time it is moved, and software vendors would have little incentive to support deduplication.
Action Item: IT executives should wait until deduplication standards that connect disk, tape and applications are in place and supported by major vendors before embracing the full integration of deduplication into the IT infrastructure. In the meantime, restrict implementations to point solutions for specific applications that have clear and high returns, and document clear exit strategies. Oh yes, invite Bill and Scott to lunch together.