On St. Patrick’s Day, March 17, 2009, the Wikibon community gathered to discuss the story of California State University, East Bay (CSUEB). John Charles, the CIO of CSUEB and Rich Avila, the director of Server & Network Operations, were looking down the barrel of a gun in late 2007. The total amount of power being used in the data center was 67kVA and the maximum available from the plant was 75kVA. Pacific Gas & Electric Company (PG&E) had informed them that no more power could be delivered. They would be out of power in less than six months. A new data center was planned, but would not be available for two years.
Data Center Scan
The team realized they had to take action quickly. Rich Avila conducted a survey and found the following:
- CSUEB’s 250+ server infrastructure was running at less than 6% utilization;
- DAS and SAN storage was utilized at between 10-15%;
- There was very little sharing across servers. When a server ran out of storage the answer was to buy more storage or sometimes more servers;
- One half of CSUB’s server equipment was more than five years old.
At the time, Jonathan Taylor, an environmental studies graduate student, and a systems engineer in the data center, was doing some research for a paper. Jonathan presented the paper to Charles and recommended implementing storage and server virtualization. Charles, the CIO said ‘do it.’
Action Plan
Rich Avila put forth a strategy to consolidate more than 50 servers down to less than five physical servers and implemented a five-point strategy:
- Install three Sun Fire 4600 servers to run VMware and provide the server backbone of a virtualized server environment;
- Install a 3PAR S400 with 92 disk drives and 43 terabytes of virtualized storage as the backbone of the virtualized storage environment;
- Implement VMware to provide virtualized servers in place of physical devices;
- Began migrating servers and storage to VMware and 3PAR, starting with the oldest and least used servers;
- Mandate 3PAR/VMware as the default platform for all future applications.
CSUEB chose 3PAR for two main reasons: The 3PAR solution was developed from inception with thin provisioning as a fundamental feature, and the product was simple to install.
Avila, who had been a consultant at the university prior to his employment at CSUEB had developed enough credibility with the university’s departments to mandate a ‘forced migration’ to the virtualized environment at a specific point in time. He put forth a schedule that called for installing equipment in the spring of 2008 and beginning the migration in the summer.
Results
CSUEB removed:
- Fifteen pallets of miscellaneous gear, ten cabinets, 27 CRT’s and disconnected 150 cables.
- More than 60 servers, consolidated on to about four physical servers running VMware.
This has helped CSUEB avoid approximately $230,000 in server costs; and avoid power, rack space and maintenance costs of around $45,000 per annum.
CSUEB also received an incentive payment (aka ‘rebate’) from PG&E for installing the 3PAR virtualization and thin provisioning technology. Notably, CSUEB was the first storage customer in PG&E’s territory to receive a rebate check for installing energy efficient equipment.
In addition, power consumption has been reduced from 67 kVA to 51kVA at a cost savings of $2,000 per month. CSUEB now runs a lights out data center, and employees are much happier not having to deal with cool temps and loud humming sounds.
Action Item: CSU East Bay’s crisis was averted by decisive action. Users should learn from this lesson and avoid a potential disaster by getting their asset management house in order and begin to build the case to aggressively consolidate server and storage assets.
Footnotes: