Originating Author: Fred Moore
In 1994, StorageTek delivered the industry's first virtual disk array, originally named Iceberg, providing the disk industry with the first implementation of outboard data compression, compaction, log structured files, and RAID 6+ architecture. Iceberg was designed for primary storage, mission-critical, enterprise-class data applications. The system's logical view of the Iceberg storage pool appeared to be one of a much larger storage subsystem containing more disk capacity than the physical storage subsystem actually offered. Disk data compression was the key to this concept and compression ratios of 4 to 1 or better were common for primary storage applications. Therefore customers could buy less hardware capacity than they had data to store, quite an economic benefit.
Tape compression has been a widely popular feature for tape drives since the mid-1980s. Tape compression ratios typically range up to 3 to 1. Disk compression never really caught on, despite its successful introduction in the 1990s. The perceived low acquisition cost of disk made compression seem less important. Nonetheless over the past 10 years the need for disk compression has increased as primary storage pools have experienced unlimited growth. For non-mainframe disk storage, typical utilization rates average 40% or less, meaning most disk drives have excess storage capacity, creating considerable wasted expense.
Compressing these drives theoretically could reduce the drive utilization well below 40% and potentially drive up the IOPs requirement as additional workloads are then allocated to the drives.
High performance applications are often allocated on “short-stroked” or limited allocation disks, further decreasing allocation levels in order to maintain performance. With flash drives starting to move into the disk market to handle many high performance applications, the remaining disks will typically have moderate I/O activity and room to handle the higher allocations and increased workloads without degradation with the use of disk compression. This will reduce the number of drives needed, reduce footprint and energy requirements, and reduce operating costs allowing for data growth without adding additional under-utilized disk drives.
Action Item: The storage industry continues to be “Thriving Tomorrow on Yesterday's Problems." Thin provisioning is a good example of this as it first appeared in 1965 on mainframes and then again in the early 2000s for non-mainframe systems. Expect compressing primary disk storage to make a more successful appearance than it did in the 1990s. Today’s economics mandate optimizing storage at every opportunity.
Footnotes:
