Snapshot on a page
While CS1 was satisfied with the performance and availability of its tier-one platform of EMC’s Symmetrix storage arrays, rapid growth of tier-two (mainly Windows-based) applications and increased demand for storage to support unstructured content was creating constraints on its business. The cost and time to migrate to new arrays to support CS1’s three-year leasing strategy was onerous (5+ months and up to $50,000 per array).
The company decided to adopt an aggressive virtualization strategy for both servers and storage to improve the flexibility and cost effectiveness of its infrastructure. It is six months into a three-year plan.
Original Storage Snapshot
Some 300TB of EMC Symmetrix were split over two data centers providing excellent availability and performance. Two SANs connecting storage to 600 Unix and Windows servers.
- Time to provision new arrays averaged 5 months, and array migration costs exceeded $50,000 per migration. For a typical high-end array this represents additional costs above original hardware and software expenditures of:
- Loss on buying earlier than required....... ~13%
- Loss on lease payments....................... ~13%
- Cost of array migration.......................... ~17%
- Total cost............................................ ~43% of acquisition cost
- Migrations required shutting down the application for a period of time that was disruptive to business users
- The current Symmetrix tier-one storage was too expensive to be a default for all data, implying a need for tiered storage.
- CS1 plans to virtualize nearly all storage using the IBM SAN Volume Controller (SVC) to allow commissioning, migrations and decommissioning without impacting applications.
- CS1 has commissioned IBM DS4000 modular arrays behind the SVC to provide tier 2 storage and has made it the default storage going forward.
- CS1 has retained the EMC Symmetrix arrays for tier 1 storage, and is moving most of these behind the SVC. CS1 is moving the array-based storage management software to Tivoli.
- Performance and availability considerations for applications behind SVC,
- Ensuring that users were happy with performance of tier-two storage (testing on an application-by-application basis)
- Creating a central group to assign data sets to storage tiers based on performance requirements.
For the storage it has virtualized, CS1 has reduced its storage provisioning time from five months to five days. In addition, payback period is less than one year. With virtualization of both servers and storage, CS1 has improved and continues to expect improvements in availability and time to provision a new virtual server with storage from weeks to minutes.
|Vendor Proposal||Advantages for CS1||Drawbacks for CS1||Overall CS1 Assessment|
|EMC Invista||Performance and availability of Symmetrix, VMware||No references or experience with Invista, unclear software management strategy||**|
|Hitachi USP||Great performance, some references||No support for Symmetrix as tier 1, attached storage; poor services||**|
|IBM SVC||2,500 references, services, redbooks, one software set||Doubts about availability & performance of Symmetrix behind SVC, scaling of SVC||****|
Six months ago, CS1 was very happy with the performance and availability of EMC Symmetrix storage arrays that provided 300 terabytes of highly available high-function single-tier storage for open systems across two data centers 500 miles apart. However, the storage that was growing fastest was that for mainly Windows-based tier 2 applications. On the horizon management could see an increased demand for archive and compliance data storage.
CS1 wanted to implement a strategy that would provide:
- Lower storage hardware and software cost.
- Reduced cost (over $50,000 each time) and time (average five months) for commissioning and decommissioning storage arrays; together with opportunity cost of having to buy storage early, this was estimated to be about 43% of the acquisition cost of a high-end array.
- Higher storage utilizations by seamlessly moving data between tiers on different arrays.
- Support for a server virtualization strategy that had the objective of improving the time-to-deploy for servers and improving availability.
Three vendors -- EMC, Hitachi, and IBM -- were "short listed" for detailed proposal analysis. EMC proposed Invista and CLARiiON storage as the second tier. IBM proposed the SAN Volume Controller (SVC) together with keeping the installed Symmetrix Tier 1 and using IBM DS4000 series modular storage. IBM also included Tivoli software, in particular the Tivoli Provisioning Manager. Hitachi proposed replacing the Symmetrix storage with USP storage for tier 1 and externally connecting AMS modular storage arrays for tier 2.
All the solutions met CS1 requirements. However, the Hitachi solution required the costly early termination of the Symmetrix leases, as Hitachi did not want to support Symmetrix as tier 1 attached storage. The EMC Invista solution required significant changes to the installed SAN switches, and CS1 found that Invista was very early in the adoption cycle. CS1 management was not confident that EMC had the experience necessary for its installation. IBM cited 2,500 successsful SVC installations, provided detailed technical and implementation information such as the IBM Redbooks, and proposed using the IBM Provisioning Manager to reduce commissioning of storage from five months to five days.
CS1 chose and implemented:
- 8-node IBM SVC’s,
- DS4800 modular storage, used as the default storage unless applications met specific service level requirements that warranted tier-one placement,
- Symmetrix retained as tier 1 storage when necessary for performance moved behind the SVC and virtualized,
- Array-based storage management software migrated from Symmetrix to SVC and Tivoli-based storage management software to control software expenses.
CS1 is six months into the plan. CS1’s analysis showed a break even on the project within one year. It plans to implement tier 3 archives using ATA disk drives and complete the virtualization / tiered storage implementation over the next two-year period.
Storage Equipment Installed before
Six months ago, CS1 was very happy with EMC Symmetrix storage arrays that provided 300 terabytes of highly available high function single tier storage for open systems across two data centers 500 miles apart. Most of the disks were 145GB, revolving at 15K RPM. They provided excellent performance and very high availability. EMC provided good storage management tools (in particular TimeFinder™), and support was excellent.
This storage then and now supports 600 Unix and Windows servers, combination of Sun, IBM and HP.
There is a strong move to consolidate and virtualize the servers. VMWare is being used for Windows and native partitioning for the Unix servers. Two SAN fabrics, one in each data center, connect the servers, virtual servers and storage.
The applications running on the servers and storage support about 6,000 users across the United States. SAP is one of the largest systems. The users are very happy with storage performance and availability, although scheduled downtime to migrate arrays is a major source of pain for the customer.
Business Problems related to Storage
Before deciding its strategy, CS1 analyzed the storage trends and found that the fastest growth was in storage attached to Windows-based applications, which would be best addressed with tier 2 systems. On the horizon CS1 could see an increased demand for archive and compliance data storage. IT management concluded that Symmetrix storage arrays were overkill for most tier 2, archive and compliance applications.
CS1 also found that applications that needed tier 2 storage did not need most of the sophisticated storage management software in the Symmetrix arrays. EMC’s licensing agreement resulted in license fees for all the storage attached to a controller, with no way to partition the storage on an array to create one storage pool that could use the software service and another that could not. The software costs paid to EMC were eating up the storage budget.
CS1 did not have a storage charge-back system in place, and IT management was very conscious that senior management in the organization would not support the current rate of storage budget growth and that a lower cost storage strategy was required. A deeper analysis of the storage costs identified some major inefficiencies in the way that storage was managed. CS1 leased all its storage on three year leases, and the time to commission storage and decommission the old storage was long (about five months) and expensive (they outsourced it at costs that varied from $50,000 to $75,000 per array). It also meant that they had to purchase the array five months before it was actually needed. The impact is shown in table 2 below.
|Array Cost Element||Relative Cost|
|Tier 1 Array Cost||100%|
|Cost of migration||17%|
|Cost of buying early (32%/year x 5/12 = 13.3%)||13%|
|Loss on 3 year lease costs||13%|
|Total cost of array||143%|
One of the major factors that contributed to the cost and length of time for commissioning and decommissioning was that applications had to be closed down before data could be migrated, which meant that precious maintenance slots had to be used or that storage managers had to work with users to schedule downtime.
In addition, analysis revealed that the difficult data migration between arrays meant that data was almost never moved from its original allocation. The result that storage allocation was suboptimal and storage utilization low.
As indicated, CS1 was initiating server virtualization. One of the main drivers for the server virtualization initiative was to provision servers quickly in order to respond to the business requirements. Prior to the project, storage was a significant bottleneck to the rapid deployment of servers.
CS1 put a team together to investigate tiered storage solutions from EMC, Hitachi and IBM.
- EMC offered a number of different approaches to the problem:
- EMC offered an in-array tiered storage approach, using different disk drive sizes such as 300GB or 500GB within the array. It offered reduced cost-of-software (~50%) for the high capacity drives. This solution provided lower cost tier 2 storage and good abilities to migrate data within an array but did not provide cost-effective migration between the 12 arrays that CS1 had on the floor. The solution did not adequately address the commissioning/decommissioning problem.
- EMC offered a modular storage second tier using CLARiiON storage. It provided a different and lower cost set of management software based on the Navisphere Management Suite. This solution provided lower cost tier 2 storage and lower cost management software. The solution did not provide abilities to migrate data between the 12 arrays that CS1 had on the floor or address the commissioning/decommissioning problem.
- EMC finally offered the Invista out-of-band virtualization approach together with CLARiiON storage, after IBM had proposed the SAN Volume Controller (SVC) appliance. Invista takes advantage of new intelligence being built into SAN switches from Brocade Communication Systems, Cisco Systems and McData to allow dynamic mapping of applications to their required data. Invista is an out-of band virtualization controller that processes the control and metadata paths, while relying on port-level processing of the intelligent switch to transfer the data path without overhead. As data volumes are migrated to other locations, such as during a technology refresh, that migration does not disrupt the applications.
- This Invista solution met the major requirements of CS1, including the ability to migrate data seamlessly between arrays. However, it required additional changes in the SAN switches as well as different software approaches for the Symmetrix and CLARiiON arrays. EMC did not have a significant installed base of Invista, with installations only starting in the last few months. When compared with IBM’s SVC (see below), CS1 felt more comfortable with IBM’s experience, did not want to change the SAN switches at the same time, and found the storage management software for the SVC more mature than the Invista software.
- Hitachi offered a virtualization solution using the USP controller.
- It offered to place the tier 1 storage on the USP and connect AMS modular storage (or any storage arrays that CS1 preferred) as the tier 2 external storage. This external storage was under the management control of the USP, and provided the ability to virtualize all the volumes of both the internal and external storage. Hitachi storage management software could be used if required on both external and internal storage. The USP controller could be partitioned and software costs managed by partition, which avoided the problem of paying for storage management software not required on tier 2.
- Hitachi did not recommend the use of EMC Symmetrix attached to the USP as tier 1 storage but insisted that the tier 1 storage should be on the USP.
- This solution met the major requirements of CS1, including the ability to migrate data seamlessly between arrays. The approach was architecturally sound, and Hitachi provided a large number of reference installations. The storage management approach was also sound. However, CS1 concluded that the inability (or lack of desire) of Hitachi to support its installed Symmetrix arrays as tier 1 storage would require CS1 to get rid of all of the installed EMC Symmetrix leased equipment to implement the Hitachi solution. This effectively ruled out the Hitachi proposal because of the cost and disruption of breaking the EMC Symmetrix leases.
- IBM offered the SAN Volume Controller (SVC) appliance.
- This is an in-band virtualization solution for heterogeneous storage. IBM offered its DS4000 series as the modular tier 2 storage. The SVC is deployed as a cluster of nodes. For CS1, each of the two 8-node clusters consisted of 1U high rack mounted appliances based on IBM System x servers. Each node has at least four fibre channel ports and was protected by an uninterruptible power supply. The nodes are clustered together so that if a node fails, surviving nodes can take over. The nodes run a Linux kernel and a specialized Virtualization Storage Software environment. Servers access SVC as if it were a storage controller. The SCSI LUNs they see represent virtual disks which are allocated in SVC from a pool of storage made up from one or more managed disks (mdisks). A managed disk is a storage LUN provided by one of the storage controllers that the SVC is virtualizing. The SVC can migrate data seamlessly from mdisk to mdisk, whilst maintaining IO access to the data. Asynchronous remote copy Global Mirror would allow CS1 a remote disaster recovery site at the other data center. The fast-write cache feature was important to CS1 because it allowed Symmetrix to be used as tier 1 storage attached to the SVC (it signals that writes from hosts are completed once they have been committed into the SVC cache but prior to being de-staged to the underlying storage controllers. Data is protected by being replicated to the other node in the IO group). The SVC supports point-in-time copying of data (FlashCopy).
- IBM provided extensive technical information on how the SVC should be installed, including an IBM Redbook. IBM also emphasized that it had more than 2,500 installations of SVC and provided several customer references. IBM also proposed the Tivoli Provisioning Manager, which would use the SVC to completely automate the commissioning and decommissioning of storage. This was very attractive to CS1, as it would significantly reduce the cost of IBM services, and (more importantly) reduce the time taken to commission/decommission from five months to five days.
The team recommended the IBM SVC and DS4000 modular storage as the best strategic option for CS1.
Implementing the Solution Selected
Overall Storage Strategy
Figure 1 shows the CS1 implementation strategy diagrammatically. CS1 wanted to continue to use the Symmetrix arrays as its tier 1 storage, but to move the majority of them under the control of the SVC. This would allow it to discontinue many of the Symmetrix storage software licenses. The implementation plan virtualizes as much of the storage as possible over three years, tying in with the three-year leases on the Symmetrix hardware. If some of the high-IO storage impacts the SAN, CS1 have a plan for a small pool of non-virtualized storage.
CS1’s strategy is to make the default storage the IBM modular storage. A special case has to be made if tier 1 storage is to be used. As there is no chargeback, CS1 had to ensure there were sufficient management controls in place to govern this policy. CS1 is very concerned that the users have a good performance experience as the storage is moved to tier 2. To effect this strategy, CS1 has put in place a storage group within storage administration responsible for ensuring that tier 2 storage is used as the default, placing data optimally on the right tiers and across the storage arrays, and for ensuring that performance requirements are met by comprehensive testing. CS1 is in the process of implementing a better performance and space monitoring and infrastructure monitoring mechanism, based on IBM’s Tivoli software. The storage group monitors I/O performance and does storage capacity planning and performance planning. The objective is to implement tools that monitor the performance, and if something is abnormal, automatically open an incident report. The goal is to operate by exception and automation. For example the Tivoli provisioning manager is being implemented and scripted to automate commissioning and decommissioning of arrays. CS1’s long term objective is to automate storage requests and storage migrations using a policy-based engine. The storage request would include the business requirements and prerequisites, and the policy would dictate where it goes, together with backup and archive criteria. That is, as CS1 points out, a long road!
Wikibon draws the following conclusions from this case study:
- Organizations like CS1 have concluded that virtualization is a prerequisite for any effective implementation of tiered storage.
- Virtualization is now ready for general adoption, with both IBM and Hitachi having significant installed bases of customers, and EMC putting significant muscle behind Invista.
- One of the largest benefits of virtualization is the ability to commission and decommission storage in days rather than months. This reduces the costs of overlapping storage, delays purchase of new storage, and significantly reduces costs.
- A tiered storage strategy means that responsibility for the initial and ongoing placement of storage should be centralized, probably as part of the storage administration.
- Storage and server virtualizations are independent projects, but that they will gain significant benefits from each other if undertaken together.
Wikibon concludes CS1 has implemented a sound strategy and that the pressures for creating additional storage tiers will require an accelerated rate of adoption of virtualization and tiered storage. However a potential weak point of the strategy remains the degree to which CS1 can virtualize tier 1 storage behind IBM’s SVC given performance and availability requirements for tier-one storage.
Legal: © Wikibon 2007. This document is copyright protected by Wikibon and does not fall under the GNU general license terms for Wikibon.org. Links to this article from external sources are allowed, however any other re-distribution of this content for commercial purposes is strictly prohibited. Please contact Wikibon for more information.
The cases cited herein are real however the name of the customer is fictitious. Wikibon case studies are developed independently and their development is not initiated for or funded by any single company. Wikibon reports actual customer experiences and results with no attempt to emphasize any one vendor’s strengths or weaknesses. Read the full disclaimer.