Executive Summary

In September 2010, Wikibon reviewed how Terremark is addressing the power efficiency of its data centers. In this paper Wikibon is looking in depth at another service provider wishing to remain anonymous, and called SVII for the purposes of this paper. SVII is a fast-growing managed hosting provider, with 85,000 sq. ft. in operation and 45,000 sq. ft. in the pipeline. By focusing on providing a virtualized server, firewall, and storage infrastructure stack, SVII can offer its customers full security, improved resource utilization, and reduced space and energy costs. SVII estimates that it has avoided spending $67 million dollars per year on energy costs in 2010 from energy saving programs and HP BladeServers and HP 3PAR storage virtualization projects. The project will be completed when the thin provisioning and thin data services provided by the HP 3PAR storage arrays are fully implemented.

SVII has also been very aggressive in improving the efficiency of its data centers. It has used hot and cold aisles very effectively, raising the temperature of the air that is vented directly into the cooling (CRAC) units to 95ºF. In addition SVII uses perforated floor panels and has placed barriers in empty rack slots. SVII has also invested in hot air containment to ensure that the hot air does not mix with cold air. SVII created a hot air plenum above the rows to allow the hot air to flow directly from the racks into the plenum and back to the CRACs. The combination of improvements has reduced the PUE ratio (a measure of data center power and cooling efficiency, see below for definitions) from 2.2 to 1.7. The impact of this is to reduce the power and cooling budget by $5 million per year, and the four-year Net Present Value (NPV) using a 5% interest rate for these savings is $17.5 million.

The overall power and cooling cost avoidance from virtualization and PUE improvements shown in Figure 1 is $72 million per year. The NPV (5%) for the virtualization and PUE projects is $255 million dollars.

Figure 1 – Total SVII Power & Space Cost Avoidance from Virtualization and PUE Improvements
Source: Wikibon 2011

Both SVII and Terremark have realized impressive savings commensurate with their business models. SVII and Terremark together have saved nearly 100MW of power consumption. This is equivalent to avoiding releasing 1.5 million tonnes of CO2 in the atmosphere and is equivalent to reducing the number of cars in the United States by nearly 100,000. Most of that saving (90%+) has come from virtualization of servers and storage.

Wikibon believes that service providers can be more efficient in power and cooling technologies than all but the largest data centers. The most important foundation technologies available to enable power and space improvements are server and storage virtualization. These technologies are key building blocks in the development of Infrastructure 2.0, which will enable IT organizations to be competitive to service providers.

SVII Data Center Thumbnail Sketch

SVII focuses on providing custom managed hosting solutions for complex Internet-facing infrastructures. It manages the total infrastructure, including security, monitoring, storage, data center operations, servers, and applications (administration of the database and the full software stack). SVII operates in four locations and has 130,000 sq. ft. of raised floor in its data centers, as shown in Table 1.

Eighty percent of the SVII space is dedicated to managed hosting solutions, and 20% to traditional collocation services. The power characteristics of the SVII datacenters are shown in Table 2. SVII uses about 16.3 MegaWatts of power.

Table 2 – SVII Power Usage
Source: Wikibon 2011

Within the collocation business, SVII has fewer options for improving power efficiency, as its customers choose the technology and equipment that is deployed. This case study focuses on the custom managed hosting solutions (80% of datacenter space), where SVII has focused on specific virtualization strategy to reduce costs and power.

SVII Virtualization Strategy for Managed Hosting

In order to be able to efficiently manage its custom hosting solutions, SVII focused on providing a virtualized stack of services. The virtualized stack includes:

• Hypervisor – VMware is the leading hypervisor deployed; Microsoft Hyper-V and Zen are also used. The hypervisor provides the ability to share resources among many operating systems operating as virtual machines (VMs).
• Virtual Servers – the hypervisor allows the virtualization of the servers, which can be used to provide both direct server resources to run client applications and as resources to run virtual services such as virtual firewalls. This allows much higher levels of efficiency running on standard hardware, rather than providing multiple types of servers running at low levels of utilization. Fewer servers results in lower energy requirements.
• Virtual Firewalls – By using specialized virtual firewalls running on the virtual servers, SVII is able to provide very high levels of security to its customers, while being able to offer the price competitiveness of a virtual cloud of shared resources.
• Virtual Storage – SVII has implemented storage virtualization using HP 3PAR storage arrays. This has enabled the implementation of secure storage partitions for its customers from a shared storage area network (SAN). The virtualization allows space to be allocated from a pool as required. The HP 3PAR storage virtualization technology also allows the aggressive “thinning” of data by the elimination of zeros and virtual (thin) provisioning when the data is actually used.

Facilities Improvements from Virtualization of Servers and Storage

SVII uses the HP BladeSystem c7000 with Intel Xeon 5400 processors. This enables a very space- and power-efficient server infrastructure. The Xeon 5400 processors consume about 50 Watts per processor, compared to the 80-120 Watts used by earlier processors. This significantly reduces the power consumption per unit of processing power.

The virtualization of the servers using (mainly) VMware allows the efficient allocation of server resources. Applications can be migrated to other server racks if necessary (for example using VMware vMotion), and the virtualization of many virtual servers running on a single physical server dramatically reduces the number of server types and physical servers. The focus on one server type reduces the cost of acquisition and server management, and power requirements.

SVII uses HP 3PAR virtualized storage arrays. The virtualization of storage arrays allows SVII to implement secure storage partitions for its customers from a shared storage area network (SAN) pool. From a management and performance view, each SVII client sees a dedicated storage array populated by the correct balance of drive types. No client is aware of any storage other than their own. The virtualization allows space to be allocated from a pool as required and avoid pre-allocating large amounts of unused contiguous storage. The HP 3PAR storage virtualization technology also allows the aggressive “thinning” of data by the elimination of zeros and using virtual (thin) provisioning by allocating storage only when it the data is written. SVII also uses virtual copying. This means that only the changed data is written to a virtual copy volume. Many additional copies can be taken to improve application availability, recoverability, and testing with very little storage cost or overhead. The net effect of the HP 3PAR storage technologies is that far less spinning disk is required. The major power consumption for storage is the physical spinning of disks.

SVII measures the power and space efficiency of its racks in the data center very carefully. As was shown in Table 2, the power requirement for a managed hosting rack is on average 5,000 Watts. A rack requires 25 sq. ft. of space. SVII estimates that the number of racks required to supply the computing has been reduced by 80% through the application of the server and storage virtualization techniques described above. This enables both significant capital cost reductions and reductions in power and space requirements. Table 3 shows the power and space benefits of the virtualization projects. Currently 2,437 racks (about 34 thousand servers and 500 terabytes of storage) are being used for managed hosting. Without virtualization, SVII projects that it would need five times the number of racks (housing servers and storage). The total cost avoidance in power is projected to be 45MW, and more than 200,000 additional square feet of raised floor would have been required without virtualization. SVII can pass these savings on to its clients as reduced cost of IT infrastructure. With average power costs of $0.1 per kWh and space costs of $100 per sq ft, the power and space cost avoidance is calculated at $67 million per year.

Table 3 – Impact of Virtualization on Managed Hosting Power & Space Requirements
Source: Wikibon 2011

SVII Power Efficiencies in the Data Center

The Green Grid defines data center power efficiency with a metric called the Power Usage Efficiency (PUE).

SVII, like most service providers, has focused on improving its PUE figure. SVII has aggressively implemented hot and cold aisles in its data centers. This avoids mixing of cool air entering the data center with the hot air that is being extracted from the room and increases efficiency. SVII has been very aggressive with the temperature of the hot aisle, which is 95ºF; the higher the delta between the temperatures of the air going into the room and the air exiting, the greater the efficiency of the computer room air conditioner (CRAC) units. SVII also vent the air in the hot aisle directly into the CRAC units and puts barriers in all the empty slots in the racks, which increases efficiency further. SVII has also invested in hot air containment to ensure that the hot air does not mix with the cold air. SVII created a hot air plenum above the rows to allow the hot air to flow directly from the racks into the plenum and back to the CRACs.

As a result of these investments, SVII has reduced its PUE from about 2.2 (meaning that 120% of the energy is used to power the equipment is required to provide power and cool it) to 1.7 (only 70% of the energy is required). The financial impact is shown in Table 4. The yearly savings are about $5 million, and the 4-year NPV is $17.5 million.

Table 4 – SVII PUE Savings
Source: Wikibon 2011

Summary and Conclusions

By focusing on providing a virtualized server, firewall and storage infrastructure stack, SVII can offer its customers full security, much improved utilization of IT infrastructure resources, and much lower energy and space costs. The overall power and space cost avoidance is shown in Table 5.

Table 5 – SVII Green Impact
Source: Wikibon 2011

SVII estimates that it has avoided spending $67 million dollars per year on energy costs from HP BladeServers and HP 3PAR storage virtualization in 2010; the project will be completed when the thin provisioning and thin data services provided by the HP 3PAR storage arrays are turned on fully. SVII has also been very aggressive in improving the efficiency of their data centers, reducing the PUE ratio from 2.2 to 1.7. The benefits of this PUE reduction are $5 million per year.

The overall power and cooling cost avoidance shown in Table 5 is $72 million per year. The NPV of the virtualization and PUE projects using a 5% interest rate is $255 million dollars.

Wikibon is looking at the Infrastructure 2.0 and other key technologies that will enable a more agile and energy efficient IT infrastructure built primarily from volume components. Virtualization of both servers and storage are key components of evolving to Infrastructure 2.0. The business case for avoiding additional expenditure on power & space is overwhelming.

Comparison with Other Service Providers

In September 2010, Wikibon reviewed how Wikibon Terremark is addressing the power efficiency of their data centers. There are many similarities in the way the two service providers have addressed this issue, and the differences reflect the different structure of their business. Table 6 summarizes the differences.

Table 6 – Comparison of Energy Savings from Different Service Providers
Source: Wikibon 2011

Both SVII and Terremark have realized impressive savings commensurate with their business models. Terremark’s business is 90% collocation services, and the savings reflect the fact that the virtualization projects could tackle only 10% of managed services equipment. Terremark is being more aggressive with its data center efficiency (PUE) targets, aiming at 1.5. This is sensible because it improves both its managed service and collocation power efficiency. SVII has focused on managed services (80% of equipment), and has managed to reduce the power requirements of its offerings by 80% by virtualizing servers and storage. The large savings is MW and dollar savings made by SVII come from the fact that 80% of its equipment is focused on managed services.

Together, SVII and Terremark are saving nearly 100MW of power consumption. This is equivalent to avoiding 1.5 million tonnes of CO2 in the atmosphere, and is equivalent to reducing the number of cars in the US by nearly 100,000.

Action Item: Many organizations will take advantage of managed hosting facilities such as SVII instead of building or augmenting their own facilities. Because of the economies of scale, service providers such as SVII will provide a standard, both in cost efficiency as well as power efficiency, against which all IT infrastructures will need to measure themselves.