Co-author Gary MacFadden
Today, businesses of all sizes are challenged to keep up with the growth of digital information, data, and content. The growth in digital records, forecast to top 2.5 zetabytes worldwide by 2012, and the increasing percentage of digital records being designated of critical business value, places the onus on end-users to rethink information management strategies to create and deploy the most effective policies, processes, and technology solutions for digital content. See Functionality Set for Managing Unstructured Data Across the Enterprise
By all accounts, the vast percentage of the business records being created and managed today are digital. With the overwhelming growth of digital content – most notably e-mail with attachments, file system content, instant messages, wiki and web content,images and otherwise referred to as unstructured data – the management of electronic records has become an increasingly critical issue for corporations of all sizes. A few end-user statistics underscore the current state:
- As much as 50% of unstructured data is stored longer than it needs to be for business, regulatory, or legal purposes, with much of this content being retained indefinitely.
- Those firms that keep track of duplicate content, which most find difficult if not impossible to do, find duplication rates for unstructured data run as high as 20:1.
- Firms are increasing their storage capacities by 50% or more per year to accommodate the growth and retention of unstructured data.
- Unstructured data now accounts for up to 50% of all enterprise electronic records, including database records and transaction files.
- 63% of end-users rate the quality of digital records on shared drives to be poor or fair (source Wikibon).
- 42% rate the quality of email management to be poor or fair (source Wikibon).
Businesses can point to several trends and management practices that have led to this current state:
- The proliferation of desktop computing and storage, giving the end-user the ability to create, copy, and store content with little or no enforceable restrictions;
- The advent of email as a fundamental business tool and the basis for business transactions in many organizations;
- Lack of end-to-end information management automation across all content types, that, if available, would allow the enterprise to intelligently declare, classify, store, secure, retain, discover, and ultimately dispose of content;
- Regulatory mandates that encourage organizations to “retain all”, or be subject potentially to scrutiny and fines;
- Poor management practices with regard to retention and disposition policies, and the ability to control and enforce policies across businesses;
- In some cases, the inability to distinguish between digital content required to support the business and business transactions, and courtesy copies, artifacts, or transitory records maintained for operational expediency;
- The immaturity of the records management function when it comes to applying information management practices designed for the physical world to the digital environment;
- The inability to easily include users, policies and rights, and digital content in a clear and real time picture to allow the enterprise to enforce effective security and privacy requirements.
Over recent years, many organizations have implemented expensive infrastructure to manage unstructured data partly in response to new federal regulations. Most are resigned to the fact that these investments have turned out to be reactive, stopgap measures designed to address a specific problem, such as electronic discovery. For example, most commercially available archive and electronic discovery solutions are not truly unified or end-to-end. And since enterprise business practices and IT requirements vary greatly; no one architectural approach is clearly superior to another. In addition, many existing systems and processes don’t scale well. Performance is too often less than optimal, taxing knowledge and legal discovery efforts with additional operational and litigation costs. See EDiscovery woes: Not all the Vendors Fault
For those companies struggling with existing implementations – particularly larger installations of more than 5,000 end-users – many new point solutions have come to market to alleviate known bottlenecks in legacy systems’ authentication, auto-declaration, classification, culling, de-duplication, indexing, mailbox management, and search shortcomings. At the same time, there is a fair amount of M&A activity with some high profile examples, including purchases of Stellent by Oracle and Interwoven by Autonomy following the examples of IBM with the FileNet acquisition and Documentum by EMC. Clearly these vendors and others, such as Microsoft with major improvements coming for Exchange 2010 and SharePoint, believe that more end-to-end integration of digital content management and archiving solutions along with offering buyers key functions such as enterprise search, indexing and deduplication will improve enterprise control over unstructured data.
However, how successful these major vendors, along with Symantec and OpenText, will be in terms of delivering the ease of implementation, functional integration, scalability and speed-to-discovery they promise remains to be seen. Certainly the sector’s history for assimilation and fusing of major components and/or point solutions has been less than stellar.
Complicating the picture for users is the plethora of hybrid solutions incorporating cloud computing and the number of outsourced and SaaS services, including technology titans Google and Dell and an ever-increasing number of Microsoft partners, that offer to manage pieces of the solution or the entire process. It is generally accepted that this outsourced, cloud enabled, SaaS solutions business represents 50% or more of the entire market, with more service providers and solutions entering the market every quarter.
Nonetheless, information management professionals responsible for driving the firms' IM strategy need to consider what business processes need to be re-engineered, what new policies and practices are needed to ensure the effective use of some of these new, integrated, enabling technologies and, of course, identify and resolve technology gaps.
Action Item: The impact and costs associated with the growth of unstructured data requires focus on business process, information governance policy, enforcement practices, and enabling technology. Traditional practices of the records manager must be assessed and reshaped in light of the digital content revolution. CIOs or CIMOs (Chief Information MANAGEMENT Officers) will be well served by taking a holistic view of information management (IM); what has worked, what hasn’t, what’s new in unstructured data management, and what drives business investments in unstructured data management. Drivers to consider are security, privacy, compliance, discovery, server and storage sprawl, and other operational risks and how compliance requirements change the role of the CIO/CTO/Information Management professional.
Footnotes: