Introduction to Wikibon Green Validation Reports
Green Validation Reports are distilled from the more technically detailed Wikibon Energy Lab Validation Reports and are intended to articulate the green personality of a vendor’s technology. The lab reports are highly detailed documents prepared by Wikibon in support of clients who are seeking to have their technology declared eligible by utility companies such as Pacific Gas & Electric Company for energy efficiency incentives and other such conservation rebates. Lab reports are the materials submitted by Wikibon on behalf of vendors to the utility in support of the energy incentive qualification process.
Wikibon performs this service under its program called ConserveIT.
This Wikibon Energy Lab Report was submitted in August 2008. All data pertaining to the cost, capacity and performance of solid state storage is of that date. The primary objectives of the Wikibon Energy Lab Report is to:
- Validate the hardware energy efficiency of a particular technology as compared to an established baseline.
- Assess the potential contribution of software technologies to power savings, and validate the actual contribution in actual installations. (green software).
- Quantify the contribution of the hardware and software technologies to a green data center.
- Educate business, technology and utility industry professionals on the impact of technologies on reducing energy consumption.
Wikibon Energy Lab defines and validates the hardware testing procedures, and in addition follows up with reviews of the actual results obtained. The Wikibon Energy Lab works with vendors and utility companies to define how software technologies can be measured in customer environments and helps vendors and their customers measure the impact. When determining the overall effectiveness of a technology actual field results are used. These are the proof points that are mandatory for the utility company to qualify a specific vendor's technology for energy incentives.
Wikibon Energy Lab Reports are not sponsored. Rather they are deliverables required by PG&E and other utilities as part of an incentive qualification process. As part of its Conserve IT Program, Wikibon is paid by the vendor to perform services associated with securing incentive rebates from utilities for end customers that acquire the vendor's technologies. To ensure this process is completely independent, Wikibon lab and field results are sometimes vetted by a third party engineering firm hired by PG&E or other utilities.
Wikibon only produces Lab Validation Reports for technologies that have been qualified for rebate incentives by PG&E or other utilities and have passed strict utility company guidelines. By adhering to this criterion, Wikibon assures its community of the independence of these results.
Management Summary
The use of flash drives to deliver a very high performance storage tier as a replacement for spinning disk is an exciting new technology development in storage. So much so that Wikibon selected the EMC flash implementation for the Wikibon CTO 2009 award1. The concept of flash drives as a very high performance tier, Tier 0, was introduced by EMC in 2008 into the company’s high-end and mid-range storage arrays.
There are two main reasons why flash drives are gaining popularity:
- They process about 30 times more I/O than a single rotating disk drive. One flash drive can perform the same IOPS as up to 30 rotating disk drives, and reduce the power consumption by more than 98%.
- Each I/O is at least six times faster; replacing 4% of traditional disk drives with flash drives typically increases the I/O speed by 25%, which improves the throughput and lowers the power consumption of a system (server and storage) as a whole by 4%.
However, the technology is currently about 10-to-15 times more expensive. Although the price of flash drives is projected to decline faster than the price of traditional drives, flash drives will not replace traditional fibre channel drives in enterprise storage arrays in any reasonable planning horizon without strong financial and cost savings incentives.
Flash drives will be used increasingly to replace high-performance disk drives such as fibre channel drives in common use today for their performance characteristics. These drives represent an important percentage of the terabytes deployed (as much as 10%), and they represent a much higher percentage of the power used in storage arrays (25-35%). Heavy I/O traffic will be directed at these flash drives, sometimes referred to as tier-0 in a tiered-storage architecture. Light I/O traffic will be directed at high-capacity SATA drives. The combination will significantly reduce the overall power and cooling requirements of enterprise storage.
There are severe constraints to adoption of the flash drive technology to replace high-performance disk drives, including software investment, positive user experience, and confidence in the technology. Wikibon believes that incentives that promote early adoption will speed general adoption of flash drives significantly.
This report gives an example of 12 flash drives added to an original configuration of 100 servers and 360 drives enabling a combined direct and HVAC energy savings of 85,529kWh per year, and $66,585 or $66,247 in energy costs over the life of the flash drives. The benefits come from reducing the number of spinning disk drives and from reducing the number of servers required. As a rule of thumb, each additional flash drive replacing 1% of traditional drives with flash drives saves 1% in total system power.
Technology Overview - Flash Drive Technology
On January 14th 2008, EMC announced Enterprise Flash drives on the high-end DMX storage array, and later in 2008 added support on the midrange CLARiiON array. The flash drives are manufactured by STEC, Inc. and are the same size and have the same interface as traditional spinning disk drives. For all intents and purposes, they are interchangeable, albeit with very different characteristics.
Flash drives use NAND memory technology that was made into a mass market with the introduction of Apple iPods. The first iPods, introduced in 2001, used 1 inch disk drives. The iPod mini was introduced in 2005 with flash memory from Samsung. It revolutionized and re-launched the iPod, making it lighter, drop resistant, cooler (in every sense) and increasing battery life significantly. It should be noted that the highest capacity iPods still continue to use disk drives.
One of the biggest technology changes that STEC made to the technology was the use of single layer cell NAND technology as opposed to the multi-layer technology used in iPod and other consumer devices. This enabled the higher reliability that is required in data processing as opposed to media processing.
As a result of consumer use of NAND technology, its cost, quality, and performance has improved dramatically. One of the early drawbacks of NAND technology was the limited number of times that a cell could be written to before becoming “exhausted” and failing to function normally. With the use of buffers and write management software, this problem has been solved for all practical usage. Table 1 above summarizes the key differences between a single enterprise SSD drive and a single enterprise traditional fibre channel disk drive at the drive level.
The major differences highlighted in Table 1 are clear. The cost of the same amount of flash storage is more than 20-times higher. However, the number of I/Os per second (IOPS) that can be driven through a flash drive is more than 30-times higher, and the average response time is 6 (or more) times better.
Flash Drives in Enterprise Storage Arrays
The performance limitation of traditional spinning disk drives relates to their electro-mechanical characteristics. Specifically, the speed of rotation of a disk drive’s platters and the physical movement of the drive’s actuator arm across the platters substantially impede performance.
For example, the maximum rotational speed of a disk drive’s platters is 15,000 rpm, and this has not changed for over a decade. As a result, the IOPS (Input/Output operations Per Second) per drive has hardly increased at all, whereas the data capacity per drive has dramatically increased. Non-spinning flash drives can sustain much higher IOPS, and much lower response times, than spinning disk drives because they use semiconductor technologies to store and retrieve data, thereby eliminating mechanical movement.
The use of flash drives in these applications results in a significant reduction in the number of spinning disk drives required. The reduction can vary from a factor of 3 to 30 times, depending on the application. Wikibon conducted a rigorous analysis of the benefits that EMC customers had achieved with the use of flash drives, the results of which are shown in Table 2 below.
There are two main benefit categories for the use of flash drives, shown in Table 2:
- The workload is IOPS constrained, i.e. the system requires a high number of IOPS to a relatively small data set. Before flash, the only solution for this type of workload was either large amounts of cache (increasing the cache hit rate of the controller), or a technique called “short-stroking”. When the cache hit rates are low (for example, when the transactions access data randomly), short-stroking is the only effective technique. In this method, only the outer part of the disk platter is used for storing data, and the disk actuator arms only have to travel a short distance across the outer edges of the disk platter to retrieve the data. In TPF benchmarks, for example, short-stroking is used extensively to improve performance and throughput. The result is very inefficient use of low-capacity, high-speed disk drives, and the use of additional power and cooling. This technique is well tried and well understood, and a significant number of customers use it for high-performance applications to mask the low number of IOs that hard disk technology can sustain. In Table 2, the first three cases show that the average reduction in traditional drives when flash drives are introduced is about 15:1; the major factor here is the reduction in traditional high-speed low capacity spinning drives.
- The system requires very fast I/O response time in order to meet business service level objectives (SLAs) for either batch or online systems.
Cases 4 and 5 are also very interesting; here the performance constraints are user response time (case 4), and batch processing time (case 5). More flash drives were added to significantly improve throughput. In both cases the alternative was adding additional computing power, and with it significant additional energy requirements. In the batch processing case, the financial company was not meeting its SEC compliance obligations to process the batch by the start of the next daily cycle. The solution before flash drives were introduced was to add an addition 1,000 Mips of mainframe processing power, which would have increased the total system power by over 25%. This case study supports the rule-of-thumb quantifying the reduction in overall systems power achieved by adding flash drives that is developed below. The IOPS that were achieved in case 5 are 4,000, which is nearly 40 times the performance that can be achieved with traditional drives.
For some IO intensive applications response time is the most important factors. For example, if a transaction results in 5,000 data base lookups, and the cache hit rate in the array is 90%, 500 I/Os have to be completed before the transaction can be completed. The response time for the users is going to be well over 5 seconds, and I/O to disk is going to be by far the biggest contributor. If sub-second response time is required (e.g., for user productivity or customer satisfaction) then I/Os have to be accelerated.
The effects of the two types of power savings illustrated above are additive. The first effect is relatively easy to measure, quantify, and verify. The second is much harder in a specific case, but easy to quantify in general. The calculations of the impact of these effects is made in the next section.
The driving force for flash in enterprise storage is I/O performance and moving the I/O of the active data volumes from hard drives to flash drives. This is very different from the driving forces in the consumer world, where the major considerations are a smaller form factor, robustness, and battery life. The most important constraint on the adoption of flash technology is the ecosystem of software and hardware that will need to adapt to be able to direct I/O-intensive data to the flash drives automatically. PG&E financial incentives will play a significant role in accelerating the introduction and adoption of these improvements.
Energy Efficiency of Flash Technologies
The power characteristics of traditional disk drives and flash drives is shown in table 3, using the 15:1 replacement ratio derived from the first part of table 2. The assumptions are based on flash drives within a DMX4 array frame. The source of the information was the EMC power calculator. The methodology was reviewed in depth by Wikibon (see Appendix 2), and the measurements were spot checked by Wikibon by redoing the measurements on whole arrays, and each component of the array. The results were always within 3% of the original measurements.
The energy savings from using 10 flash drives vs. 150 traditional disk drives is shown in Table 3 below. This table was derived from the Wikibon Energy Lab Power Calculator (EMC Flash V4), which was created using data from the EMC power calculator V2.4.2, Wikibon analysis, and PG&E’s methodology for calculating the power distribution and cooling overheads.
Calculation of Energy Savings with Flash Technology
The EMC Power Calculator is an accurate tool which allows the power consumption of different configurations to be compared with each other. Details of the calculator are given in Appendix I, together with an on-site review that Wikibon made of its accuracy. Wikibon concludes that the accuracy was within 5%.
The availability of this tool makes the calculation of the rebates easier. The power consumption for the storage array configuration as will be installed is calculated, as shown in Figure 1 below. The compute environment is a data center with 100 1U blade servers, supported by 360 disk drives. The storage configuration is for CLARiiON CX4 with 360 drives and 6 flash drives. This is an US industry average configuration, based on IDC numbers. The storage power consumption is 6.48 kVA.
The Power Calculator output in Figure 2 below shows the power consumed by the CX storage array without flash, but with the additional 85 drives required. The power consumed is 8.46 kVA, an additional 2 kVA.
In addition to the direct savings from a smaller storage configuration, there are also system wide benefits that derive from the system as a whole. The main benefit comes from faster I/O times. The average I/O time for traditional disk is 6.5ms, and for flash, 1ms.
The operating system tries to keep processors busy. But processors operate at nanosecond speed (10-9). Hard disks operate at millisecond speed (10-3), a million times slower. It is difficult to keep the processor busy except in technical workloads. A component of the time the processor is not busy is called “Wait for I/O”. Table 4 shows typical “Wait for I/O” times for different workloads, with an average of 17%. Flash I/O has no mechanical component and brings down the I/O time to about 1ms or less. By directing some of the I/O to flash drives, the I/O times can be significantly reduced. Using conservative figures in Table 4, the average I/O time can be reduced from 6.5ms to 5ms. This allows an average reduction in I/O wait time of about 4% which translates into a 5% decrease in processor power requirements. This equates to a reduction of five servers in a 100 blade server configuration.
Table 5 shows the combined effect on lower storage power and lower processor power required if 12 (~3%) flash drives are added to a 100 1U server configuration. The overall reduction is 5%, or about 11kVA. As a rule of thumb, adding 1% flash drives will reduced the power consumption of an average configuration by
Table 6 shows a summary of the results that were derived from the Wikibon Energy Lab Power Calculator (V3) which computed the energy saving based on the inputs (see figures 1 & 2) which contrasted the proposed system that contained flash drives and the alternative option that did not. The Wikibon Energy Lab Power Calculator computed an energy saving when the flash drives was implemented of $66,585 over the five year life of this particular project.
Confirmation of Energy Savings with Flash Technology
EMC has tools that can measure the IOPS that are going to the disk drives. The key output is shown in Figure 6 below, which shows the output from the EMC SPA tool. The output is flexible and can show the data over any time period. The key ability of the flash drives is to handle very high peak I/O rates, often referred to as I/O storms. One flash drives can handle the theoretical throughput of 30 conventional 15K spinning disk devices. In practice it can handle even more than that, as usually the data cannot be spread smoothly over 30 disk drives.
The SPA tool can be run on request of PG&E to confirm that the 15:1 ratio of disk-to-flash I/O has been achieved. This data will allow confirmation that the flash drives are working as expected in the first one or two flash installations. After that, these reports would only need to be run to meet audit requirements.
Future projections for Enterprise Flash Drives Costs
Flash drives are the most exciting new technology in storage for many years. This report has shown that on average a single flash drive used appropriately can eliminate 10 traditional rotating disk drives, and reduce the power consumption by 94%. The main reason for their use is that they can process about 30 times more I/O on a single disk drive in 1/6th of the elapsed time.
The technology is currently about 20 times more expensive. The price of flash drives is projected to come down faster than the price of traditional drives over the next five years and more, as shown in Figure 7. However, flash drives will not replace high-density SATA drives in enterprise storage arrays in any reasonable planning horizon (e.g., 10 years). Flash will remain much more expensive than high-capacity SATA devices, and the advantages that are driving the consumer flash markets (size, battery life, and robustness) are not significant factors in enterprise storage.
Wikibon projects that flash drives will be used increasingly to replace high-performance disk drives such as fibre channel drives with lower density and higher rotational speeds. Although these drives represent a small percentage of the terabytes deployed (5-10%), they represent a much higher percentage of the power used in storage arrays (25-35%). Heavy I/O traffic will be directed at these flash drives, as part of a tiered-storage architecture, and will significantly reduce the overall power and cooling requirements of enterprise storage. Figure 4 shows that over the next four to six years, the prices will come down and become very competitive with traditional high-performance disk drives. An update to Figure 4 is available at Enterprise_Flash_Drive_Cost_and_Technology_Projections Enterprise Flash Drive Cost and Technology Projections.
Constraints limiting the adoption of the flash drive technology as a replacement for high-performance disk drives include price, software investment, positive user experience, and confidence in the technology. Reducing these barriers to adoption early will support stprage power consumption reduction, and probably lead to earlier adoption of new storage array architectures designed to take maximum advantage of high I/O bandwidth and low latency of flash. Wikibon believes that incentives that promote early adoption will bring forward general adoption of flash technology within storage arrays significantly.
As the cost of flash drives decreases, so the justification for additional flash drives to improve I/O times and reduce the number of storage devices will increase. Wikibon established the rule of thumb that each additional flash drive replacing 15 hard drives will lead to a 1% reduction in total power of the system. This rule of thumb should hold for many years.
Footnotes:
References: 1 Wikibon CTO Award 2009, Downloaded 12/31/2008
Appendix I – Measurement Methodology
Measurement Methodology
The measurement methodology used was based on EMC’s Power Calculator. This calculator was designed and built by EMC, and first released in December 2006. The current version is V2.4.2 (as of November 2008). The power calculator has the following components:
- A database of measured results for each component of the following storage array systems:
- EMC Celera
- EMC Centera
- EMC EDL
- EMC Connetrix
- EMC CLARiiON
- EMC Symmetrix
- A configuration engine with the rules for valid configurations for each of the arrays systems
- A report system that allows the power consumption to be assessed for three situations
- Typical power consumption
- Maximum Power Consumption
- Recharge Power Consumption
- A database of measured results for each component of the following storage array systems:
For the purposes of assessing the power consumption for PG&E incentive payments, the typical power consumption is the best fit. The maximum or recharge power consumptions are maximums that are used for determining power provisioning, and would overestimate the power savings.
Flash drives are currently only available on the EMC CLARiiON and Symmetrix arrays. Only these parts of the calculator were assessed in this report, though the power calculator will be updated to include flash drives in other arrays types if and when they are announced and measured.
Confirmation of accuracy of the EMC Power Calculator
Wikibon Energy Labs reviewed the EMC power testing facilities and processes for accuracy and completeness. The testing facilities were at the EMC site in Hopkinton, Massachusetts.
The tables 7 and 8 below show the results of different configurations using two different test suites.
Table 9 below show the comparison between the measured results and the results found in the EMC Power Calculator. As is shown, the results were less than 3% different from each other.
Table 10 below shows the results of measuring the actual power consumption of a configuration before and after pulling boards. This test compared the power consumption of components of an array as predicted using the EMC Power Calculator, compared with the values tested in the Lab. The variance was again within 3%.
The conclusion of the Wikibon on-site review and tests was that the EMC Power Calculator was within a 3% error range, and sufficiently accurate to be used as a basis for calculating PG&E rebates. Using the EMC Power Calculator makes the application for rebates significantly simpler for both customers and EMC, and will increas'e the take-up of PG&E incentives and the delivery of flash drives.
Equipment used to measure Power during Wikibon Review
Power Meters were installed in 5 locations. A Voltech PM3000 was used to monitor the System or Logic Bay power. A Fluke 434 Power Meter was used to monitor the Drive Bay power. An Extech hand held power meter was used to monitor power for the DAE with 11-15K drives and 4 Flash Drives. A Voltech PM1000 was used to monitor power for the DAE with 15-10K drives. A Yokogawa WT210 meter was used to monitor power for the DAE with 15-15K drives. Both AC inputs to the redundant power supplies were monitored so that the total power consumption was measured by each of the 5 meters.
Appendix II – Specification of EMC Flash Drives
Specifications
Interface: Fibre Channel 4Gb
Dimensions: 147(L) x 101.6(W) x 20.0(H) (standard 3½ drive size)
R/W Transactional 45K / 16K Random Sustained IOPS
Current capacities: 73GB, 146GB (future capacities up to 512GB)
Read and write performance: 220MB/s; Write: 115MB/s
IOPS Read: 46,000; Write: 16,000
Power consumption: Sleep/Idle mode 5.4W; Operating mode 8.4W
Action Item:
Footnotes: Disclaimer
This report was prepared by Wikibon. Neither Wikibon, PG&E nor any of its employees make any warranty or representations, expressed or implied, or assume any legal liability or responsibility for the accuracy, completeness, or usefulness of any data, information, method product or process disclosed in this document, or represents that its use will not infringe any privately-owned rights, including, but not limited to, patents, trademarks, or copyrights.
This report uses preliminary information from vendor data and technical references. The report, by itself, is not intended as a basis for the engineering required to adopt any of the recommendations. Its intent is to inform the customer of the potential cost savings. The purpose of the recommendations and calculations is to determine whether measures warrant further investment of time and/or resources.
Acknowledgments:
Wikibon is grateful to Pacific Gas and Electric Company and the California Public Utilities Commission for the funding that underlies and makes this study possible. We would also like to thank EMC technical managers for their support and cooperation.