In response to the necessity to manage the barrage of data and content inundating the enterprise more effectively, CIOs and CTOs have increasingly turned to vendors offering de-duplication (dedupe), compression and single instancing technologies in order to boost storage capacity and reduce costs.
With vendors touting dedupe ratios as high as 50:1 on back up and archived data and the availability of abundant horsepower to process up to 3TBs of data per hour, the case for bringing in a stand-alone appliance seemed clear. Vendors such as NetApp with A-SIS and EMC with Celerra also targeted primary storage capacity optimization. Today, dedupe and compression features and functions are an essential part of any storage or information management (IM) vendor’s list of capabilities and go to market strategy because they have become an essential check list item for buyers.
Key Dedupe Trends
As the dedupe market matures, two key trends will emerge by the end of 2010:
- A focus on unified or global storage optimization to include backup, archive as well as primary storage.
- The inclusion of dedupe as an embedded feature in existing storage and information management solutions.
Raising the Dedupe Bar for Primary Storage
On June 7th, Permabit announced its Albireo High Performance Data Optimization Software to address primary storage optimization. According to Permabit CTO and Founder Jered Floyd, “Companies have been wrestling with the concept of primary data de-duplication for some time now but have been unwilling to sacrifice performance in order to achieve this benefit. In order for de-duplication to operate effectively at the primary level it must have no performance impact, no feature set loss, and no risk of data integrity. Albireo makes this possible.” Floyd shared his thoughts on dedupe and vision for the Albireo platform with the Wikibon community during a June 8th Peer Incite research meeting.
EMC, NetApp and Permabit believe dedupe will become an embedded system feature, and even a commodity feature with affordable price points with little impact on overall performance.
Floyd, addressed participants concerns regarding what he referred to as “Bump in the Wire” technologies that slow processing speeds. “Albireo Scalable Data Reduction architecture supports both fixed and variable block deduplication with fixed block sizes down to 4k are supported by Albireo. In variable block deduplication, data is analyzed more intelligently, in a “content aware” manner, resulting in more efficient deduplication. For variable block deduplication, Albireo utilizes specific content “scanners” to identify and optimize deduplication of objects within files (e.g., Office® documents, images, zip files, backup files, VM.”
Futures and Concerns
How quickly storage and IM vendors or OEMs will embed deduplication functionality into their solutions remains to be seen. Stand-alone solutions such as Data Domain, now EMC, have sold extremely well and according to IT industry research sources, adoption of existing deduplication and compression technologies is yet to reach even 50% of enterprises, leaving the market wide open for a variety of approaches (in-line, post-processing, unified, etc) with primary storage dedupe adoption even lower. Meanwhile, other storage optimization vendors such as Ocarina Networks today sell appliances but one of their blog posts suggests the company sees dedupe technology as an embedded capability in the future.
Despite the immaturity of the dedupe market in general, thus far the potential achievable results are compelling enough to drive a robust market populated with many innovative products and advanced features for which earlier adopters have been more than willing to pay a premium. However, as more vendors adopt and develop de-duplication functionality and more competition enters the market, prices will level off and more products will include embedded storage optimization capabilities. By the end of 2011, storage vendors in particular will have to include embedded dedupe for primary storage as part of their storage optimization offering, or they will not make buyer short lists.
Action Item: As de-duplication technologies become more widely available by the end of 2011, storage optimization will be a fundamental offering embedded into storage arrays. CIOs and CTOs should expect it and demand it from array suppliers.