Contents |
Crisis
John Charles, the CIO of California State University, East Bay (CSUEB) and Rich Avila, the Director, Server & Network Operations were looking down the barrel of a gun in late 2007. The total amount of power being used in the data center was 67kVA and the maximum available from the plant was 75kVA. Pacific Gas & Electric Company (PG&E) had informed them that no more power could be delivered. They would be out of power in less than six months. A new data center was planned, but would not be available for two years.
John Charles had another constraint. In February 2007, as a member of the CSUEB executive committee, he had been charged with meeting one of the “foremost concerns to the campus community,” namely to “restore and maintain a physical plant that is consistently attractive, clean, safe, easily accessible, and green1.”
Problem
Rich Avila did a survey of the equipment in the data center, and concluded that the utilization of the servers was less than 6%. They had been installed using an ad hoc process to meet the projected needs of computer projects all over the university. The storage management was also adhoc, with most of the storage being attached directly to servers, and some centralized storage on two small Storage Area Networks (SANs). The utilization of the storage was also very low (10-15%), and the spare capacity could not be used by other applications that were running out of storage. The bottom line of Rich’s findings was that the IT equipment was vastly underutilized, and the aging equipment was power inefficient (shown in Figure 1).2
Solution
Jonathon Taylor is an environmental studies student at CSUEB and was writing a research paper on making data centers more efficient and green. He was also working as a systems engineer in the CSUEB data center, and used the data center at CUSEB as a reference model. He recommended virtualization as a key component of a strategy to reduce power consumption, and presented this paper to his CIO, John Charles. John’s response: “Do it; now.”
To solve the problem, CSUEB operations group did five things:
- Installed three Sun Fire 4600 servers to run VMware and provide the server backbone of a virtualized server environment
- Installed a 3PAR S400 with 92 disk drives and 43 terabytes of virtualized storage to be the backbone of the virtualized storage environment
- Implemented VMware to provide virtualized servers in place of physical devices
- Began migrating servers and storage to VMware and 3PAR, starting with the oldest and least used servers
- Mandated that the default platform for all future applications would be the 3PAR/VMware platform
The users were informed that over the spring and summer of 2008 their equipment would be migrated to a virtual server on a specific date. If the operations group did not hear back (this was the case for the majority), the equipment was migrated to VMware on that date. If there was an objection to the date, operations negotiated a new target date with strong pressure on the user groups to help solve the power and environmental problems of CSUEB.
Results
As of October 2008, 25 servers have been shut down, 17 NAS servers have been migrated to the 3PAR equipment and VMware and turned off, and 21 new applications that would have required additional servers have been installed on 3PAR/VMware with another 15 new applications on track to also be installed. The old SAN equipment has been migrated to the 3PAR storage, and will be removed. So far, power consumption has been reduced by 26% -- from 67kVA to 50kVA — even after adding 21 new applications, with more to come. “Our power crisis has been averted” said Rich Avila.
The S400 storage equipment from 3PAR was a key component of the virtualization strategy. The 3PAR equipment virtualizes the storage, which allows all the disk to be used, and data to be moved dynamically as required. “What prompted the decision to go with 3PAR was the fact that they do thin provisioning ," Jonathon Taylor explains, adding that the University expects to virtualize 100 of its servers by 2009. “That allows us to save a lot of energy.” In addition, a feature called thin provisioning was used, which allows the servers to think that they have a large amount of disk space, but the 3PAR storage array only provides real storage when data is actually written. The combination of 3PAR’s storage virtualization and thin provisioning allow the amount of disk required on the 3PAR equipment to be only 20% of the storage required on the original equipment, even after the addition of several new applications. Wikibon did a detailed analysis on the savings from virtualization and thin provisioning, and found that the average ratio between virtual storage and real storage was 4:1, exactly in line with the CSUEB results3.
Pacific Gas and Electric Company (PG&E) Rebates
The result of this improved storage capacity utilization has been to reduce the number of spinning disks required for an array and, ultimately, the number of arrays as well. This is critical since spinning disks typically account for 80 percent of the power requirement of a storage array. 3PAR’s virtualization and thin provisioning technology are recognized by PG&E as energy efficient, and users of the technology qualify for energy rebates. Prior to installing the 3PAR array, CSU applied for and received a rebate incentive from PG&E for using the technology. The business case showed that the overall power savings from using the 3PAR equipment would be more than $30,000.
PG&E and its regulators are serious about power efficiency and have added data centers as a focus item for helping customers achieve power reduction. “Storage virtualization and thin provisioning technologies help our customers realize significant capacity and cost savings while also addressing critical data center energy issues,” explained Mark Bramfitt, principal program manager for PG&E. “By providing financial incentives, we hope to increase industry adoption of these smart and efficient new storage technologies.”
New Data Center
Jonathon Taylor’s report also influenced the design of the new data center due to come online in 2009. Virtualization of servers and storage will continue in 2009 and beyond. CSUEB is implementing a hot and cold aisle strategy, with an AC power bar across the top of the cabinets. All the backbone equipment will be isolated from the servers, with direct patching to the network. Cabinets that are part of vendor equipment are placed at the ends of the aisles and earthquake-proofed. “We are planning that the new design will improve our data center infrastructure efficiency, or DCiE), index4 to above 70 percent,” said Rich Avila.
Conclusions
Wikibon believes that there are seven major steps for implementing an effective data center green strategy:
- Give specific responsibility for data center efficiency and a data center green strategy to a single executive, and agree upon specific metrics and targets.
- Make data center power a separate budget line item, and ensure that the data center facilities costs (space, power, and cooling) are included in all business cases.
- Project the data center power and power density requirement forward, using as the projections’ bases installed equipment and future equipment, projected power density of future equipment, and local power availability—and communicate these figures to facilities management regularly.
- Implement aggressive virtualization of servers and storage, including thin provisioning and other storage efficiency technologies.
- Eliminate and unplug unused equipment and applications from the data center.
- Implement hot and cold aisle technology in the data center, and raise temperatures significantly (if necessary, eliminate the need for IT personnel to work in the machine room for extended periods). Implement monitoring equipment to monitor air temperatures and airflow, and use outside air for cooling where possible.
- Evaluate selectively outsourcing data center facilities that cannot be improved to meet the targets set in item 1 above.
CSUEB has been effective in the majority of these steps. Notably, CSUEB’s synchronized storage and server virtualization effort has been cited by Wikibon users as a best practice to achieving optimal efficiency. CSUEB has not set specific metrics and targets; instead, it has adopted a general green awareness response, which it believes is more appropriate for a university institution. CSUEB has not been able to utilize outside air in its future data center because of the new data center’s location in the middle of a building.
Overall, a power crisis sparked the evolution of an effective green IT strategy at CSUEB, which is reducing power and improving the cost-effectiveness of IT. Many customers are facing a similar crisis or soon will be. The time to act is now. In general, U.S. companies lag in implementing energy efficient data centers and without swift action will face their own do-or-die scenarios.
References
2 CSU East Bay IT Annual Report 2006/7, downloaded 9/28/2008
4 Green Grid PUE and DCiE Metrics, downloaded 9/28/2008
Footnotes:Legal: © Wikibon 2008. This document is copyright protected by Wikibon and does not fall under the GNU general license terms for Wikibon.org. Links to this article from external sources are allowed, however any other re-distribution of this content for commercial purposes is strictly prohibited. Please contact Wikibon for more information.
The cases cited herein are real. Wikibon case studies are developed independently and their development is not initiated for or funded by any single company. Wikibon reports actual customer experiences and results with no attempt to emphasize any one vendor’s strengths or weaknesses. Read the full disclaimer.
