Four technologies are working in parallel to reduce the cost of storage:
- Basic technology that is doubling the bits that can be stored in the same place every 18 months;
- Storage efficiency that is moving the utilization of that storage from 20% to 90% with technologies such as virtualization, thin provisioning and now techniques for identifying deleted data;
- Storage Management with fully automated systems that allow performance, allocation and security management to be done with the minimum of effort;
- Compression and deduplication techniques that reduce the amount of data stored.
Video and pictures are the fastest growing data segments; just witness the growth of set-top boxes and TiVo. And this is not only in the consumer world. Businesses are using security cameras and doing facial recognition to help identify potential threats. Grainy cell-phone pictures from insurance adjusters are being replaced by high-definition videos. One picture on a real estate web site is being replaced by high definition video. Data from medical diagnostic stills and moving images is growing dramatically. Technical manuals are being supplemented by pictures and videos of demonstrations.
Two companies (among many) have interesting compression technologies which help to reduce the amount of storage and the amount of power to run the storage. They are:
- Qbit: a start-up that has an impressive array of mathematical talent and success in the Hollywood arena. They are focusing in on real-time lossless compression, achieving very high compression with highly parallelized software and specialized cards. The algorithms for each type of data are different; an interesting development would be specialized and adaptable cards or software within the array to identify and compress data without data loss and with a potential 6:1 compression ratio.
- Storwize: who have applied their mathematical talents to provide real-time lossless compression of NAS files through an appliance that is shipping now.
De-duplication has a powerful niche in backup, where time series data with little change makes this technology very effective and achieves consistent figures of a ten-to-one reduction. However, de-duplication vendors are pushing their technology into general purpose storage reduction, and the typical reduction figures of 2:1 or 3:1 are not nearly as impressive. The one big disadvantage of de-duplication is that there is a large amount of metadata that needs to be associated and cared for in order to restore the original object (and not all are mathematically lossless). With smart compression techniques that select the best algorithm and store the results together with the codec software, customers can restore the data on any system, reducing the degree of lock-in.
Action Item: CTOs should be cautious in deploying de-duplication technology on disk-based solutions outside backup. Compression technology is well established in tape handling. New compression technologies for disk look encouraging and offer the promise of more robust implementations.
Footnotes: