Originating Author: Fred Moore

Technology accelerates both integration and fragmentation. It allows geographically separate communities to connect with each other and prosper. Communities that do not connect develop deficiencies in their ability to grow economically and will slow or stall. The worldwide impact of information technology is unprecedented.

While everyone tries to predict the future based on the past, this is no longer possible for the storage industry.

Having peaked in the early 2000's, spending worldwide on storage hardware, software and services is slowly creeping back to dot.com levels and is hovering around $50B[1]. This figure includes the end market value of devices, subsystems, libraries, software, storage networking and associated services. While the numbers are similar to the peak days of 2000, there are many differences.

A growing spectrum of storage choices is emerging

Steady growth marked the storage industry from the early 1970s until late in the year 2000 and was evident in just about every statistic and measurement. Often called the Infinite Disruption, the worldwide events of 2001 brought significant changes to the IT and storage industry landscape. Growth rates slowed, worldwide storage revenue declined, thousands of IT jobs were eliminated while many others moved offshore, and the global economy entered a well chronicled and lasting slump.

In 2000, 91 new storage companies were launched with a first round of funding. In 2001, the number of storage firms reporting net losses hit 69 percent. Vendor roadmaps were pushed out as storage vendors derived lower profitability and, therefore, had less money to invest in their futures. Venture capital money for new storage ideas remains difficult to secure. Only business plans that solve real problems are being funded. As a result, just four new storage companies were funded with first-round money in 2004 and just two in 2005. This has had the effect of slowing down the flow of new ideas and invention, suggesting to some that we are now in a period of re-invention rather than invention. Consolidation is another major storage industry factor. There were over 60 disk drive manufacturers in 1980 and today there are just six. IT executives now list security, data classification, e-mail management, tiered storage, application and database archiving and virtual tape libraries as their immediate storage priorities.

Current perspectives

Shifting focus from Capex to Opex

The IT industry has repositioned itself with several new areas of emphasis. These include a new value system for IT where, for the first time, the real value proposition is becoming more important than the acquisition costs. For years, users perceived the hardware purchase price as their primary acquisition criterion. This has become increasingly misleading and reflects the now old and out-of-date viewpoint that the value of the IT infrastructure exists in hardware. This is like measuring the value of the television industry by the number of sets sold (the old rules) rather than the value of the content being transmitted by television (the new rules). The impact on the storage industry of this shift in focus from Capex (capital expense) to Opex (operating expense) will have lasting effects on the way the value of any storage solution is viewed.

Data protection and security

When it comes to disaster recovery planning, the challenges are mounting for CIOs. They cite the traditional problems, including human-induced errors that can crash a system to an increasing number of natural disasters like fire, hurricanes, floods and earthquakes. But they've also added new concerns ranging from widespread loss of electrical power to the growing intrusion threats from hackers to employee sabotage and terrorist attacks.

Data security is a primary requirement across the board for IT environments. Two aspects of security, confidentiality and protection from data theft, are provided by encryption technology. Recent news coverage has shown how confidential customer information can be seriously compromised when unencrypted data is lost or stolen, either by physical media loss or from malicious spyware.

In the past five years, both disk and tape storage devices have become increasingly more reliable, making intrusion an even greater threat to attaining the required high levels of IT availability. Terrorists and hackers, scam artists and predators, intrusion, a growing number of data security issues and compliance requirements have mandated that companies implement dramatic improvements to their weak and porous Internet security systems. In excess of 80 percent of all Internet traffic is now unwanted material and the number is growing. To date, over 54 million identities have been stolen and an estimated 19,000 more identities are stolen each day. Companies on average are spending over 1,500 hours per incident at a cost of $40,000 to $90,000 per victim. Expect the visionary storage vendors to deliver a wide range of storage security functionality including features like WORM and encryption for stored data. These new systems will use hyper-firewalls, advanced encryption and eventually biometrics, making the data security business much larger than the disk and tape industry combined by 2008. Intrusion prevention systems must soon reach bullet-proof status and be built with a mainframe mentality. Getting all the way there, however, remains a difficult and yet over-arching goal for the storage solution providers and all are adding significantly to their data protection offerings. Scalability

Historically, disk vendors have described scalability in terms of capacity increases and maximum limits. Users have learned in the past year that if performance and capacity don’t scale at the same rate, throughput bottlenecks result and the overall performance capability of the subsystem decreases. In addition, device utilization levels are being reduced in order to sustain acceptable performance levels. A few savvy storage vendors are slowly beginning to address scalability in terms of capacity and performance. In addition, availability will soon join capacity and performance as the third piece of the scalability equation as all are needed to enable the storage infrastructure to maintain equilibrium.

Grid storage

The concept of grid storage is moving closer to reality. Grid storage is a new model for managing storage distributed across multiple systems and networks without requiring a large centralized switching system. Grids offer the promise of ultra-high scalability and availability. Vendor interoperability and standards are still in the definition stage making grid storage a promising architecture of the future.

Open systems

Open systems and interoperability remain distant goals. How long have we been hoping for and talking about these concepts, proving that hope is not a strategy? We acknowledge that these concepts still don’t extend much beyond vendors providing a few specific APIs (Application Programming Interfaces) to close business partners. It’s now time to strike these terms from our vocabulary and wish list, and realize that existing infrastructure components from a variety of vendors can, at a minimum, co-exist in a non-disruptive manner. Like it or not, proprietary systems continue to win the market-share battle.

Open source

However, open source software is becoming increasingly important to many corporate infrastructures. IT staffs are beginning to contribute to existing code bases often resulting in benefits that come back in six months versus 18 months. Once a generic but consistent set of user interfaces and business rules arrive that link to the open source infrastructure, widespread adoption for all types of open source software is expected.

Storage management

The requirement for a centralized, end-to-end storage management solution using a “single pane of glass” for mainframe and non-mainframe systems that really works across multiple operating systems could not be greater! Many of the tried and proven mainframe storage capabilities are just finding their way into non-mainframe systems. Concepts such as HSM, CDP (Continuous Data Protection), snapshot copy, data classification, tiered storage and virtual tape have been well-established for 20 years or more. What’s taking so long?

Beyond mainframes, Unix operating systems host more critical, data-intensive applications than any other operating system. The storage services provided by Unix that are present today are basically the same ones that existed over 20 years ago when Sun Microsystems first released NFS (Network File System) and have seen little improvement since. No one knew then that these distributed computing systems would one day be asked to do the work of a mainframe and to access as much data as their mainframe counterparts, if not more. There have been few significant enhancements to non-mainframe operating systems in terms of storage services for several years.

The Unix, Linux and Windows (formerly called open systems) storage environments have become increasingly larger and now account for roughly 90 percent of all disk capacity shipped annually. As a result of this growth, a non-mainframe equivalent of DFSMS (Data Facility Storage Management Subsystem) has become highly desirable and is now long overdue. For years, the policy-based DFSMS has been widely used on mainframe computers, providing powerful improvements in storage management capability. It is now time for non-mainframe computers to provide comparable and powerful mainframe-like storage management functions as they control major storage environments and the vast majority of digital data. Surprisingly, any specific effort in this direction is not readily apparent.

Switch and fabric-based intelligence

Moving the major storage management functions off the server and into the storage fabric to simplify the storage management task is one of the major but not new architectural directions for the storage industry. The major switch, director and storage subsystem companies have now launched intelligent switch initiatives. Storage subsystem suppliers are also adding storage management intelligence into their offerings. Server-less or outboard storage management functions will gradually evolve beyond backup and recovery techniques to include mirroring, journaling, snapshot copy, point-in-time copies, deduplication and a variety of storage virtualization functions. Though the net value is high, deployment of intelligent switches is slower than expected as existing operational processes must be redesigned.

Higher availability

Fault-tolerant systems exist throughout the IT infrastructure today and can repair themselves after hardware or software failure occurs. Future applications will demand improved self-healing systems that can fix themselves before a permanent failure occurs. With rapid progress in all aspects of embedded microprocessor and dual-core chip development, expect self-healing systems ultimately to become a reality. The value of self-healing systems to the high-availability equation will be unprecedented and these architectures should eventually spread to any piece of computer technology. The new race to achieve beyond five 9s availability is under way.

Conclusion

The storage industry is headed on a new course, leaving behind many of the principles that defined the way business has been conducted for the last 30 years. Nonetheless, expect to be looking directly at device-to-device data movement, libraries of disks and disks in tape cartridges, non-volatile RAM chips and object-based storage, along with a variety of miniaturized nanotechnologies as potential breakthroughs in the years ahead. Companies and individuals in most of the world now rely on information technology to the point that their survival is threatened if it is not available.

Finally, simply reacting to an unfamiliar landscape when change comes is not an option; advanced planning, then setting a course for the future is your best bet to assess the wide spectrum of choices that lies ahead and to anticipate its many exciting challenges. The events of the past have signaled that the coming IT industry will be different than the one we knew so well and will require us to change in order to survive. New storage strategies are emerging that identify the pathway to success and these will often have little or no resemblance to their predecessors.

Notes

[1] Source: Derived market estimates using secondary market research from IDC, Gartner, Horison Information Strategies and ITCentrix.