Data-in-memory databases have become popular where rapid response times are necessary, either to help process a lot of data and iterate rapidly (such as SAP HANA), or in transactional systems requiring sub-second response time (such as advertisement placement systems). For applications that need predicable low transaction latency, the bottleneck is never the processor: modern x86 processors almost always have enough GHz and cores to provide the processing power for transactions. The performance issue always comes from the storage. Until recently, the disk drive has been the only persistent storage device. However, unless the whole database can be held in DRAM, this makes low consistent latency (e.g., sub-second response times) almost impossible to achieve because of the very high variance of disk sub-systems.

Data-in-memory systems, where the whole database is held in memory, have become more popular, with the use of use of topologies such as Membase within NoSQL database systems. However, the problems of dealing with large amounts of data in a memory-based system are extremely difficult to manage:

• The cost of large amount of DRAM on servers (up to 512GB of DRAM and higher is possible) requires extremely expensive servers;
• The larger the DRAM on a server, the slower the restart time of a node from disk-based persistent storage;
• As the amount of DRAM is increased, the probability of the L1 & L2 caches being trashed and potentially decreasing performance becomes higher, and applications become more sensitive and more costly to maintain;
• The cost of protecting the data-in-memory with battery protection of memory and/or multiple distributed copies of data goes up sharply as the number of nodes are increased.

In general, it is cheaper to use a larger number of small servers with (say) 96GB of storage with a relatively small number of nodes, and the resulting database size is limited. However, the limitations of scaleability of database size are significant.

Use of NAND Flash in Databases

NAND flash has superb characteristics for reading random data, and although flash is slower than DRAM, it is possible to create a layer below DRAM and dramatically increase the size of the database. The forecasts that improvements in NAND flash performance and reliability could not be maintained have proven wrong.

However for write, flash is not so optimal. Only 0’s can be written, so a block of data has to be set to 1’s before data can be written. Writing is much slower than reading, especially random writes.

Including NAND Flash in the Data-in-Memory Topology

Figure 1 shows the architecture that Aerospike uses to incorporate NAND flsah productively in its database architecture. It utilized the strength to DRAM, the strength of flash as a read persistent layer, and the strength of blocking up writes to persistent flash storage. The duplication in DRAM within the data center and outside allows protection of data against loss.

Figure 1 – Topology for Data-in-Memory with DRAM & NAND Flash
Source: Wikibon 2012

The key characteristics of this architecture are:

• Provision of low latency consistent performance for IO reads and writes (Wikibon published research showing a 200 times reduction in variance with flash arrays;
• Much greater scaling of Data-in-Flash than Data-in-DRAM systems;
• Much lower costs of Data-in-Flash compared to Data-in-DRAM, particularly as the amount of data increases (Wikibon research shows that a comparison of two-year server and active data costs for a transactional analytic workload with a 2 Terabyte Replicated Database (4 Terabytes in Total), supporting 800,000 operations/second (5,000 Bytes/operation) costs seven times less for Data-in-Flash compared with Data-in-DRAM.

For very applications with very small amounts of data (1TB or less), and ultra-low latency, Data-in-DRAM will remain the preferred topology.

Future Directions for NAND Flash Access

Figure 2 shows that range of new access techniques from companies such as Fusion-IO and Virident that are being tested in beta at the moment. As these techniques are implemented into traditional and new file systems and databases, the potential of Data-in-Flash topologies will become greater and will become mainstream. These file-systems and databases will increasingly utilize very large distributed flash-only storage and obtain storage services from a Software-led Storage architectures, as discussed in recent Wikibon research.

Figure 2 – Directions in Accessing Data Directly to and from NAND Flash
Source: Wikibon 2012

Action Item: In a recent Peer Incite, Dag Liodden, Co-Founder and CTO of Tapad, discussed why he used the Aerospike Data-in-Flash architecture for the deployment of his ad service. Without the development of modern Intel processors and modern databases such as Aerospike that can exploit the processor, DRAM, and flash, the ability to provide real-time advertisement serving and the real-time analytics that optimize the placement would not be possible cost-effectively. CTOs together with the lines-of-business should be looking for business opportunities where processing of big data can provide new revenue

Footnotes: