Contents |
Highlights
CS7 is a small financial organization with high transaction rates. Large amounts of data are captured every day, and have to be secured to meet audit and compliance mandates. The amount of data created each day is very variable and impossible to predict. The transaction data is moved each day to a data warehouse system.
CS7 has recently rolled out a major rewrite of the main transaction system. The system architecture of the old system was “vertical” implemented on a few larger servers. The architecture of the new system is “horizontal” on a large number of smaller servers.
As a result of the change in system architecture, the storage infrastructure requirements changed. The systems group was rolling out a lot of servers very quickly, and they all need to be attached to storage. In addition to production systems, storage for test and development systems that replicated the production systems was required.
CS7 small storage team decided that an alternative approach was required to the EMC DMX arrays then currently installed. CS7 liked the reliability and performance, but wanted more flexibility and ease of change.
The solution was the introduction of a Hitachi manufactured high-end array with virtualization. There were two aspects of virtualization that were important
- Port virtualization that allowed many more server ports to be connected to the array with far fewer fabric ports
- Storage virtualization with thin provisioning, which allowed dynamic allocation and reallocation of storage to the new servers with the minimum of storage administrative effort.
CS7 had initially planned to put in two 40 TB EMC arrays for the development and test environment. Instead they planned to put in two 40TB Hitachi arrays. They implemented virtualization initially in the test environment because of the very high change rate in the test environment, and a need to over-allocate storage to simulate the production environment.
After implementing the first Hitachi Array, CS7 found that only one array was needed for the job. The improvement in storage utilization as a result of the virtualization and thin provisioning had significantly exceeded their expectations. CS7 installed the second and two other similar arrays for other storage projects.
CS7 do not use high-level tools to manage storage, but as a policy used the native tools on each array to manage storage. The philosophy is that when things go really wrong, higher level storage management tools get in the way and made it harder to recover.
CS7 found that they saved one array and approximately ½ a head-count in just the initial implementation of virtual ports, virtualization and thin provisioning.
Current Storage Snapshot
CS7 have over a petabyte of storage installed across all the systems in two locations. About one third is on mainframe systems, and two-thrids on open systems. The main tier-1 storage arrays were EMC Symmetrix at the start of the project. The primary servers being deployed for the new transaction system are supplied by Sun with the Solaris operating system.
The main database system for open systems is Oracle 10G (supporting dual core processors). There are some systems using Oracle 9iRack, the previous standard.
The open systems storage is managed by two very experienced storage administrators.
Pain Points
- The new transaction system was running on a very large number of smaller servers, and CS7 were always running out of storage ports and switch ports
- The test environment had to replication the logical set-up of the production environment. Doing this on the current arrays would result in having to have a far greater number of actual disks
- CS7 had been badly burned during a virtual tape project that had resulted in the loss of a significant amount of data, and virtualization (and new technology in general) was still a sensitive subject to senior IT management
- CS7 were unhappy with the lack of flexibility with EMC business agreements
Solution Strategy
The solution strategy was to implement a different array technology for the development and test environment. The Hitachi manufactured high-end array was chosen with 40TB of storage. Virtual ports were used to allow multiple servers to connect to a single port, which reduced the SAN fabric and port requirements. Virtualization and thin provisioning were implemented to allow overallocation of storage in the test and development environments.
Testing of thin provisioning was done with the the main Oracle databases. Appropriate parameters were agreed with the database administrators and communicated as a standard.
Originally two array systems were earmarked for the test and development storage. However, because of the success of the virtualization project, only one was required.
Adoption Issues
The initial step for the small implementation team was to become familiar with the new products and the software running them. CS7 kept the environment simple by using just the native tools. The implementation team devised and implemented new procedures for storage allocation in the new environment.
The implementation agreed new standards for Oracle database systems with the database administrators to ensure compatibility between Oracle and thin provisioning.
The adoption strategy for thin provisioning was again to keep it simple. CS7 chose to have very few virtualized storage pools (one major storage pool) and implement it across a large number of spindles.
The test environments needed to reflect the production environment as closely as possible. The storage administration used thin provisioning to over-allocate the storage. The storage appeared to be the same size as a production system to the test systems. The actual test data was (in most cases) significantly smaller than the real environment, and this lead to significant savings in storage utilization.
Senior management was concerned to reduce the risk of new technology, because of previous virtualized tape problems. The test system was an ideal initial deployment of the technology.
The project achieved significant improvements in storage utilization. Overall it was twice what was previously achieved. One unexpected byproduct of this success was that there was a reduced amount of time for the storage requisition process to authorize and implement new storage. Previously there was room to fine-tune the arrays if necessary if there was a delay in the availability of new storage. With the storage virtualization and thin provisioning, that wiggle room disappeared. In this instance, CS7 took a box 'on loan' from the reseller as the approval worked through purchasing. As a result, CS7 modified the storage procurement process to have a pre-authorized array always in the pipeline.
Benefits
A major benefit of the project for CS7 was to reduce the number of planned 40 TB arrays for the test environment from two to one. In addition, the additional people benefits derived from a better native array storage management, lower storage administrative work because of virtualization and thin provisioning by both server and storage groups, and having to manage only one array was estimated as about equivalent to headcount decrease of 1/2 person .
Comparative Table of CS7 Array Options
Vendor Proposal | Advantages for CS7 | Drawbacks for CS7 | Overall CS7 Assessment |
---|---|---|---|
EMC High-end Array | Good EMC support history, excellent performance and reliability, well established processes and procedures | Limited number of ports (64), no virtual ports, harder to use ECC native storage management tools, no virtualization, no thin provisioning (at the time of decision and purchase), inflexibility in business agreements | ** |
Hitachi High-end Array | Large number of physical ports, very large number of virtual ports, solid virtualization within the storage controller (no additional box), thin provisioning, good references, better flexibility in business agreements, better GUI interface with Hitachi Storage Manager software | Additional storage vendor at a critical time for a new system roll-out, risks of new technology, history of failed virtual tape project meant that management were skeptical of virtualization, use of channel partners for support rather than direct support | **** |
Conclusions
Wikibon draws the following conclusions from this case study:
CS7 was happy with the performance and reliability of tier-one Symmetrix storage for its old production systems and data warehouses. With the new production systems built on a far greater number of smaller servers, CS7 needed greater attachment flexibility and ability to change storage allocations.
The new system test environment was a perfect fit for Hitachi’s high-end array. The virtual ports and virtualization allowed far greater flexibility to set up test environments, and to migrate from array to array dynamically. Not all users will need the large number of ports or this degree of flexibility, especially for more stable production environments. EMC has now announced thin provisioning, but still lacks an effective high-performance in-controller virtualization option.
The CS7 systems administrators used the native storage management tools for each array. They found the Hitachi Resource Manager about three times faster to use that the EMC ECC software. The skill level of the CS7systems administrators is very high, and it would be difficult for many installations to adopt a similar strategy of only using the native tools. Many IT shops may be forced to use high level software for less experienced administrators and have one or two experts (or buy-in as required) to use the native tools when needed.
The history of a failed virtual tape library project were significant data was lost brings home the point that there is always risk when new technologies are introduced. CS7 was wise to initially evaluate storage virtualization technology in a test environment. IT organizations (and especially vendors) are too quick to come to the conclusion that a technology is “bullet-proof” based on the fact that nothing has happened so far. IT organizations should have a recovery plan for catastrophic failure of all technologies.
For CS7, the very high skill level of the storage administrators made adding an additional vendor and new technology a lower risk alternative, especially the initial environment was a test system.
Wikibon would liked to have seen better integration and planning between the storage virtualization project and the upcoming VMware project for server virtualization. Wikibon believes there are significant synergies.
Overall, Wikibon believe that CS7 made an excellent decision to implement the virtual ports, virtualization of storage and thin provisioning. The implementation plan was executed flawlessly, and the expected benefits were clearly achieved and measurable.
Wikibon concludes that all organizations should have developed a clear storage virtualization strategy and thin provisioning strategy, and should be aggressively implementing it. Were possible, it should be integrated into a server virtualization plan.
Legal: © Wikibon 2008. This document is copyright protected by Wikibon and does not fall under the GNU general license terms for Wikibon.org. Links to this article from external sources are allowed, however any other re-distribution of this content for commercial purposes is strictly prohibited. Please contact Wikibon for more information.
Wikibon case studies are developed independently and their development is not initiated for or funded by any single company. Wikibon reports actual customer experiences and results with no attempt to emphasize any one vendor’s strengths or weaknesses. Read the full disclaimer.