Storage Peer Incite: Notes from Wikibon’s March 17, 2009 Research Meeting
Some Pictures and notes from CSU East Bay's data center.
On Tuesday, the Wikibon community Peer Incite Meeting continued the exploration of effective green IT measures with a stunning case history. Faced with an energy and space crisis, California State University, East Bay (CSUEB), virtualized its servers and storage. The result was a tremendous leap in efficiency, expense reduction, and carbon footprint reduction. The impact is visible in the before and after pictures taken by Wikibon member David FLoyer and CSUEB's Rich Avila. This huge improvement had no negative impacts on service levels and improved IT's flexibility to respond to new computing needs. The full story is presented in the articles below.
The message is clear: Server and storage virtualization, particularly when done as coordinated projects, can result in dramatic improvements in efficiency of operations of many medium-to-large sized data centers. Often, the larger the data center the greater the potential benefits. Today IT faces two major pressures. First, every IT operation needs to cut costs and often, unfortunately, staff, to respond to the new financial realities. Second, organizations need to cut their carbon footprint as soon and as much as possible to try to minimize the impacts of global warming. In most data centers those twin needs clearly begin with virtualization. G. Berton Latamore
Cal State U East Bay's power playbook
On St. Patrick’s Day, March 17, 2009, the Wikibon community gathered to discuss the story of California State University, East Bay (CSUEB). John Charles, the CIO of CSUEB and Rich Avila, the director of Server & Network Operations, were looking down the barrel of a gun in late 2007. The total amount of power being used in the data center was 67kVA and the maximum available from the plant was 75kVA. Pacific Gas & Electric Company (PG&E) had informed them that no more power could be delivered. They would be out of power in less than six months. A new data center was planned, but would not be available for two years.
Data Center Scan
The team realized they had to take action quickly. Rich Avila conducted a survey and found the following:
- CSUEB’s 250+ server infrastructure was running at less than 6% utilization;
- DAS and SAN storage was utilized at between 10-15%;
- There was very little sharing across servers. When a server ran out of storage the answer was to buy more storage or sometimes more servers;
- One half of CSUB’s server equipment was more than five years old.
At the time, Jonathan Taylor, an environmental studies graduate student, and a systems engineer in the data center, was doing some research for a paper. Jonathan presented the paper to Charles and recommended implementing storage and server virtualization. Charles, the CIO said ‘do it.’
Action Plan
Rich Avila put forth a strategy to consolidate more than 50 servers down to less than five physical servers and implemented a five-point strategy:
- Install three Sun Fire 4600 servers to run VMware and provide the server backbone of a virtualized server environment;
- Install a 3PAR S400 with 92 disk drives and 43 terabytes of virtualized storage as the backbone of the virtualized storage environment;
- Implement VMware to provide virtualized servers in place of physical devices;
- Began migrating servers and storage to VMware and 3PAR, starting with the oldest and least used servers;
- Mandate 3PAR/VMware as the default platform for all future applications.
CSUEB chose 3PAR for two main reasons: The 3PAR solution was developed from inception with thin provisioning as a fundamental feature, and the product was simple to install.
Avila, who had been a consultant at the university prior to his employment at CSUEB had developed enough credibility with the university’s departments to mandate a ‘forced migration’ to the virtualized environment at a specific point in time. He put forth a schedule that called for installing equipment in the spring of 2008 and beginning the migration in the summer.
Results
CSUEB removed:
- Fifteen pallets of miscellaneous gear, ten cabinets, 27 CRT’s and disconnected 150 cables.
- More than 60 servers, consolidated on to about four physical servers running VMware.
This has helped CSUEB avoid approximately $230,000 in server costs; and avoid power, rack space and maintenance costs of around $45,000 per annum.
CSUEB also received an incentive payment (aka ‘rebate’) from PG&E for installing the 3PAR virtualization and thin provisioning technology. Notably, CSUEB was the first storage customer in PG&E’s territory to receive a rebate check for installing energy efficient equipment.
In addition, power consumption has been reduced from 67 kVA to 51kVA at a cost savings of $2,000 per month. CSUEB now runs a lights out data center, and employees are much happier not having to deal with cool temps and loud humming sounds.
Action item: CSU East Bay’s crisis was averted by decisive action. Users should learn from this lesson and avoid a potential disaster by getting their asset management house in order and begin to build the case to aggressively consolidate server and storage assets.
Cal State University East Bay's Virtualization Success ... Lessons Learned
The virtualization success at California State University, East Bay (CSUEB), is a good example of cleaning up a mess with good practices. But plan ahead. Do research prior to and during the project and include energy savings in the justification by projecting cost savings. Look to utility companies for guidance and incentives/rebates as they are constantly tweaking their incentive models. Indeed, some require applications to be filed before the project starts.
Also, don't forget to look for server incentives – PG&E, for instance, pays $200 for every server reduced (net). Other equipment may qualify as well. Consider using outside experts such as those at Wikibon’s Conserve IT service.
Action item: Adopt both a short-term and a long-term view. In the case of CSUEB, it must move its data center after the virtualization project finished in order to strengthen the old building to withstand earthquakes, and the remote replication technologies that 3PAR provides could be used to avoid a long outage. It is possible that 3PAR or other vendors will provide “loaner” equipment to assist in this relocation and generate another success story.
IT street cred speeds virtualization adoption
When California State University East Bay had a power crisis in its data center, it called upon someone with 'street credibility' to help implement the fix. Rich Avila had close relationships with the university's departments because he'd worked as a consultant to CSUEB prior to joining the university as an employee.
University department heads, like lines of business leaders, often have the latitude to make capital acquisitions to support new applications. This was the case at CSUEB, and 'server creep' was in full swing at the university. The result: server utilization was less than 10% and storage utilization on direct attached and SAN storage ranged from 10-15%. These are clearly unacceptable metrics, even in good times.
The Wikibon community heard a similar story from BT's Michael Crader last year. Crader, like Avila, used both the powers of persuasion and 'tough love' to combat the friction of migrating physical servers to virtual environments. The formula basically came down to four components:
- Make a strong business case to management that the benefits of virtualization can attack waste and outweigh the drawbacks-- both perceived and real;
- Include storage virtualization in the mix;
- Set hard dates for the cut-over from physical to virtual;
- Communicate, communicate, communicate with the lines of business.
Both organizations we've cited here, BT and CSUEB, have successfully virtulized severs and storage and navigated through political, performance, and other perceived landmines. While this approach did not negate the need to manage the challenges of virtualization (e.g. performance, backup, etc.), the organizations are clearly running more efficient IT with minimal business disruption.
Action item: Organizations must anticipate the friction that will come about from initiating virtualization projects. Empower a project manager with established line-of-business relationships and a track record of success. Set goals and regularly communicate efficiency benefits (including greenness) which can turn resistance into support.
Opportunity Knocks - The Synchronization of Server and Storage Virtualization
In late 2007 California State University, East Bay (CSUEB), faced a computing crisis. The existing data center was using 67kVA, and the maximum power available from the utility plant was 75kVA. The university was designing a new data center, but it would not be available for two years, and at the then present rate of growth the existing data center would be out of power in six months. CSUEB averted the crisis by implementing virtualization for both its server and storage environments. By the end of 2008, 25 servers had been shut down, 17 NAS devices had been migrated and turned off and 21 new applications that would have required additional servers had been installed. Power consumption had been reduced by 26%, saving tens of thousands of dollars in electricity costs, resulting in a much more efficient data center.
CSUEB teaches us a powerful lesson of data center optimization through virtualization with quantifiable results. As power consumption continues to push more data centers towards the limit, virtualization offers a potentially effective solution to:
- Reduce power,
- Remove aging technologies that have become costly to operate,
- Get rid of unnecessary things in the data center and attack underutilized devices,
- Shrink footprint (physical and carbon).
The CSUEB study identifies tangible benefits of synchronizing virtualization of servers and storage. Storage virtualization is the pooling of physical storage from multiple network storage devices into what appears to be fewer storage devices managed from a central console. It is commonly used in a storage area network (SAN). The management of storage devices can be tedious and time-consuming, and the bigger the data center, the bigger the challenge. Storage virtualization helps the storage administrator perform tasks such as backup, archiving, and recovery more easily, and in less time, by disguising the complexity of the SAN. But working on a virtualization project without understanding the broader issues can cause problems - consider the following issues:
- Before server virtualization occurs, a few people depend on each physical resource and hardware changes and disruptions are confined to a small group of users.
- Concentrating workloads increases dependency on storage subsystem availability and performance levels.
- Virtual servers allocate disk capacity per virtual machine and create “templates” that reserve disk capacity that may or may not ever be consumed.
- Thin provisioning and thin replication technologies can significantly improve underutilized disk storage on non-mainframe systems, which averages about 40% at best. Server utilization can average as low as 10%.
- The advent of the new thin technologies can reduce necessary disk capacity and the associated carbon footprint by as much as 60% compared to traditional fat storage.
- A thin provisioning strategy ensures that any unused storage resources are available to basically any application or operating system in the network - "it's time to go on a diet and get thin".
- Increased levels of virtualization can make performance tuning and capacity planning more difficult as it becomes more complicated to attribute the actual measurements to physical hardware lengthening tuning efforts.
Action item: It’s common to go after server virtualization but ignore storage. Synchronizing storage and server virtualization creates advantages including improved storage utilization on both the server and SAN side. Having both server and storage virtualization synchronized presents the data center with an excellent opportunity to re-architect IT.
Designing to reward the buyer with hero reports
Sales and marketing presentations are awash with value propositions before a sale. But after the sale, how does the buyer show his organization quickly and easily that the claimed value propositions were achieved? And if they were not achieved, how does the seller know and correct the product?
A number of features lower power consumption on storage arrays, including virtualization, thin provisioning, thin copying, and various types of MAID. How do you show that thin provisioning is improving utilization of the disks, that the disks have been slowed or stopped, and how much power has been saved?
After reviewing dozens of products in detail at the Wikibon Energy Lab, I have found very few that provide effective hero reports the buyer can use to show his organization what has been achieved. These have almost always been an afterthought, a special script that has to be developed and run, a special program that has to be installed or that requires a services engagement to analyze and display a complex set of data.
Take thin provisioning as an example. The full information on a storage volume comes from data that can be obtained from the storage controller and data that is known by the server operating or file system. To provide an effective thin provisioning hero report needs a combination of both sets of data. This is easy to do early in the product cycle but very difficult to achieve after the product has been installed and run.
If you then collect and analyze that meta-data from numbers of customers, powerful information can be developed. An example discussed on the March 17 Peer Incite was the case study on the virtualization and thin provisioning benefits achieved by customers. The figure in the case study shows the probability of an organization achieving improved levels of efficiency. For example, it shows that the expected increase in storage capacity from virtualization and thin provisioning is 150%.
Action item: Early in the design cycle reviewers should ask two key questions:
- “What are the value propositions of the product?”
- “How can the buyer know (and show others) those values were achieved?”
Early in the sales cycle the buyer should ask one key question:
- “How can I (on my own) show my organization quickly and easily that we have achieved your claims, without special programming or a services contract”
Is Virtualization An Opportunity to Clean House?
Rich Avila, Director of Server & Network Operations at California State University, East Bay (CSUEB), faced the problem of hitting a power ceiling for his datacenter. There was simply no additional electric power available. As we discovered in the recent peer incite session, Rich and his staff to steps to consolidate the majority of various departmental servers onto a virtualized server and storage infrastructure, allowing them to remove a number of application servers and NAS servers which resulted in reduced electric power consumption while adding new applications onto the new infrastructure.
By consolidating servers and storage, Rich now has fewer physical assets to track and maintain. Consider the maintenance issues for the servers that were removed. When it comes time for regular software updates and patches, he and his staff now spend much less time applying updates and patches to all those physical servers. In addition, the administrative overhead of tracking those physical assets has been reduced considerably. Especially in a de-centralized setting, sometimes simply finding all the servers and storage can be a serious undertaking, let alone insuring that they are properly maintained.
Another benefit of reducing the number of physical servers includes a simplified backup and recovery process. The backup environment is simplified because the backup target devices can be centralized rather than having several backup devices spread out in several areas, reducing hardware acquisition and maintenance costs, media costs, and administrative costs in verifying that backups were completed properly and tracking the backup media at various locations.
Action item: Knowing how and when to eliminate underused assets reduces costs, saves administrative and management time, and is a valuable skill in these lean economic times. Look for underused assets with an eye to consolidating them.