The speed of disk systems (measured in milliseconds) has not changed significantly in the last 20 years, while flash devices can access data in nanoseconds. New architectures have emerged that avoid putting production data on disk. Wikibon co-founder and CTO, David Floyer had an animated discussion with the community about flash technology's future on a Peer Incite call, December 13, 2011. The flash technology marketplace is moving fast; the bottom line is that initial solutions have targeted decreasing IT cost, while the long-term advantage is the increase in the amount of data that can be accessed.
Initial enterprise deployments of flash in traditional architecture storage arrays and legacy applications to help alleviate hot spots. Modern applications are emerging that can fully use the low latency and high I/O of flash architectures. Some examples that David Floyer shared:
- Google search found that it was quicker to get a piece of data from cache in London than getting it from a group of disks [locally] in Mountain View, CA.
- Facebook has very large databases; the performance of traditional storage arrays could not support the requirements. The databases were put in memory with Fusion-io technology.
- Apple Siri must deliver data in less than 3 seconds, and the data is pre-staged into memory.
- IBM Watson (as shown on Jeopardy) is a harbinger of new decision-making capabilities based on leveraging massive amounts of data
Real returns on new applications can take 20 years to roll-out; this will start quickly by first delivering improvements to existing architectures and applications.
If response times can be reduced by a factor of 10, productivity can be improved by a factor of 2.
Disk is not going away; raw and archive data will continue to live on disk (since disk remains 10x cheaper for storage), but production data on disk is at a significant disadvantage to flash alternatives. A forecast of storage spending on flash and traditional storage arrays in 2015 can be seen here.
Flash technology will change the way systems are designed. The requirement for new systems is to pre-fetch the data from disk to have it available on flash. Big data applications will deliver this functionality. Production systems will have 99+% access via data-in-memory instead of disk in the next decade. The technology required to make this transition is the improvement to flash. While NAND flash has potential competitors, none of them have achieved high volume through the consumer market, which helps to lower cost and improve reliability. Flash will be spread throughout many parts of system architectures, with multiple use cases driving deployments. Data should be moved around as little as possible. Flash is improving in durability, performance and price along a fast curve and should continue along this path through 2015. Other technologies that are part of the flash ecosystem are data-in-memory architectures, in-memory databases, I/O tiering, and mixed types of memory: more expensive SLC (one-bit per memory cell) and cheaper MLC (three-bit per memory cell). Flash is much further down the maturity curve than unproven technologies like Racetrack, Memristor, or Phase-change memory.
While the raw cost of flash is higher than disk, flash vendors use compression, deduplication and other techniques to reduce the cost per GB. The cost of active data in a flash-only array is cost competitive with high-speed (SAS/FC) disk. SSD is not an optimal design; better form factors will emerge over the next four years.
Thomas Isakovich of Nimbus Data commented that the common perception that flash will remain 10 times more expensive than disk for a long period of time is not true. The price of flash is dropping faster than disk; and some disk prices are increasing due to shortages. He projects that total acquisition price per GB of flash and 10k/15k SAS or FC disk will reach parity in 2012 even without compression or deduplication. While Nimbus and many others have deduplication support, Thomas stated that it is not a market requirement since some data can not be deduped and there are many places in the stack where deduplication can be done.
The traditional SCSI interface is inefficient for flash architectures and new standards such as NVM Express, SCSI Express and SATA Express are being developed to solve the current interface bottlenecks. Other initiatives, such as atomic writes directly to the flash that don’t use SCSI, can simplify the requirements of the SCSI stack. These building blocks will allow the driving of much greater amounts of data to new architectures.
IT organizations utilizing flash technologies can increase the amount of data that they can process and in turn drive additional revenue without adding new staff. Revere Electric Supply drove 20% additional revenue by putting its databases in flash with flat headcount. IT will want to redesign the applications and reign in server sprawl to bring solutions into smaller, more tightly managed environments.
Flash technology allows not only a power savings of flash over spinning disk; in overall cost, power is a 10% savings. The greater savings will be that the increased power density where using flash will allow for more compute per rack.
Action Item: Flash storage is not a point technology but a disruptive wave to storage architectures that will fit in various architectures for many use cases. The long term question that CIOs should be asking is what technologies can be implemented that will allow employees and customers to have access to an order of magnitude more information that can transform the interactions between companies and customers. Companies can adopt flash storage solutions immediately to take advantage of cost savings while reevaluating long term modernization of applications.
Footnotes: Flash market segments, vendor roundup and pricing forecast in David Floyer's 2011 Flash Memory Summit Enterprise Roundup