Contents |
Introduction
In early 2012, Wikibon outlined the necessity of a shared very-low latency product and was very supportive of EMC's project Thunder, with beta scheduled for 2Q 2012. EMC then cancelled the project, citing no demand, to very vocal dissension from Wikibon. It is good for IT practitioners and investors that EMC has seen the error of its ways, even at the cost of an unknown number of billion dollars. EMC has agreed terms to acquire DSSD, a storage startup building advanced flash devices. DSSD is privately held and the acquisition is expected to close in 2Q 2014.
Jeff Bonwick and Bill Moore, engineers with vast storage operating system experience, founded DSSD in 2010. The problem they set out to solve was how to extract all the performance and capacity from flash chips, and how to connect them natively to applications.
The original funding for DSSD has come from Bechtolsheim, a co-founder of Sun Microsystems, with additional funding from EMC, SAP and others. The DSSD patent assets filed include multi-dimensional RAID within a node and across nodes and self-describing data (aka object storage), and potential patents in native attachment protocols.
DSSD Technology
DSSD uses PCIe NAND flash technology and x86 servers to manage the capacity and bandwidth of flash chips, flash endurance and data management. DSSD will also need a high-speed interconnect between servers to manage metadata (RDMA over InfiniBand??) across PCIe cards and nodes. DSSD is also rewriting Linux file systems to achieve separation of capacity and bandwidth, and to integrate 'native block and object attachment' to clustered compute nodes. The key value proposition of this native attachment is putting very large amounts of non-volatile storage close to the external compute nodes, being addressed as memory. Eliminating the overheads of the SCSI file system build decades ago for managing spinning rust reduces the latency from about 1 millisecond with traditional all flash arrays (AFA) to 50 microseconds or less, a 20 x improvement in latency against AFA arrays, and 300+ times compared with hard disk drives. Even more important for advanced database work is low variance for IO. EMC are claiming that DSSD will deliver terabytes of rack-scale flash storage with an order of magnitude improvement in latency, bandwidth and IOPS. SAP has committed support for the native attachment protocol with their HANA data in-memory database.
DSSD are expecting to bring their product to market in 2015. Expect another directed marketing approach similar to the XtremeIO introduction to protect traditional revenues in 2015/2016.
DSSD & Fusion-io
The most important early market for this technology is the database market. There are two major components of database - transactional & analytic. Although EMC have talked the marketing talk about using Hadoop on this type of technology, this is unlikely to make much financial sense - Hadoop does best by moving processing to a large number of the data nodes in parallel; DSSD is designed as a shared storage device and is not a great fit for Hadoop. The two most important markets are new transactional systems with very high data demands and new large-scale analytics on very large data cubes. The best market is when the two are combined. These types of system will require persistant data close to the processors. These are applications that even with data-in-memory databases, just cannot be run on todays systems. The value of these next-generation applications will be very very high, and will be the foundation of the next wave of business disruption using techniques as yet undreamt of.
The DSSD approach is similar to the approach demonstrated by Fusion-io with its January 2012 demonstration of a billion IOPS using 8 standard HP servers and 64 PCIe flash cards. The key to Fusion-io’s success is its software development of openSource APIs and advanced file systems. Atomic write and NVM compression have been integrated into all the MySQL distributions from MarieDB, Oracle and Percona. API support for KP is also available for non-SQL databases.
Fusion-io have taken a server centric approach to improving performance, although the Fusion-io ION line offers a shared storage solution. DSSD has chosen a shared storage model. Wikibon believes in the long run that Server SANs will become the predominant model, and both DSSD and Fusion-io can adapt to play in both Server SAN and very low latency SAN markets.
Both the EMC and Fusion-io approaches will mean that realization of the potential described in Wikibon's manifesto on next generation design in ”Designing Systems and Infrastructure in the Big Data IO Centric Era”. The potential value of new applications is designing and combining real-time analytics with transaction systems to introduce automated data-driven processes as a normal way to automate business processes. This approach has the potential to revolutionize business models in the later part of this decade with high overall productivity gains for enterprises. A case study on Revere gives an indication of the potential value of a single version of the truth across the whole of a small company using today's technology; very low latency SANs and Server SANs will extend this capability to all enterprises.
EMCs purchase of DSSD will accelerate interest in this very low latency segment, in the same way as EMC generated momentum for SSDs with its introduction in 2008. The key to success will be the adoption of native protocols by independent software vendors (ISVs). The big difference is that at the moment the DSSD protocols are proprietary. In order to grow the market for very low latency and to be successful within it, Wikibon believes that EMC should and will contribute the protocols as open standards.
Other NVM Offerings
HP’s Memristor, IBM’s PCM and Intel NVM are potential next generation NVM offerings, but at the moment look unlikely to make it in volume into either the consumer or general enterprise IT marketplaces. There may be some military applications for these new NVM offerings. Although the next generation NVMs offer great potential if volumes can be achieved, the success of flash in the consumer market and the continued investment to solve problems is resulting in the next generation NVMs continuing to be next generation for some time to come.
This would leave Fusion-io as the only other very low latency flash solution in town at the moment, as flash is the only non-volatile storage supported by the consumer market. That could attract a lot of attention from other vendors looking to compete with EMC's DSSD.
Conclusions
EMC's purchase of DSSD has legitimized the very low latency storage marketplace away from the trading applications and into main stream software. DSSD is an exciting development in the flash NVM marketplace. DSSD will accelerate the change of flash from a storage device simulating a traditional magnetic disk to a true extension of memory. EMC’s entry into this space will galvanize further interest from other server and storage vendors, and SAP's adoption of the native protocol will drive significant interest from other ISVs.
Action Item: CEOs and CIOs should understand fully the potential of this new technology to change software functionality, and ensure that the best and brightest are looking at the potential to disrupt business models in their own industries and adjacent ones.
Footnotes:
Action Item:
Footnotes:
