Storage Peer Incite: Notes from Wikibon’s March 6, 2007 Research Meeting
David Floyer presented Email archiving: Moving from server room to court room to board room. Email archiving is becoming a mission critical application and is no longer the sole domain of IT.
The discovery by the legal community that email archives present a new vein for business has led to a large number of law suits in which old emails become the key piece of evidence used to convict corporations of wrongdoing. As a consequence of this simple change, email archiving capabilities should no longer be focused on narrow IT interests (e.g. keeping the costs of storing emails low) and are now much more in the domain of the business, specifically the legal function within the business.
Lawyers view email archives as absolutely essential to proper business operations. Email archiving has become a mission critical application. As a result, the technologies that are being utilized for successful email archiving systems are no longer focused narrowly on reducing the cost of writing email to an archive structure but rather at extracting and querying email that is necessary to support a low cost and hopefully (from the business perspective) successful discovery process.
The drivers behind this are both the need to better support the legal activities of defending the corporation as well as the requirements associated with staying in compliance with increasingly information-orientated regulations. However, this process of moving the perspective of email archiving from the IT server room to the court room will further evolve. The capabilities associated with being able to query, extract and rapidly make sense of large amounts of unstructured data will find themselves increasingly in the board room as companies attempt to consider issues proactively like verifying business practices or testing whether or not employee management activities are conducive to the cultural edicts of the corporation.
Over the next few years, we’ll see significant new case law test the limits and requirements of email archiving and that will continue to drive significant new investment in these technologies. We will also start seeing businesses further experiment with the capabilities created by email archiving in other business domains.
Action item: IT must facilitate the transition to a legal business driven email archiving approach, recognizing that approach itself will evolve to increased focus on access to unstructured data for business query. However IT organizations must also help the business fully factor, in a more comprehensive way, how messaging technology affects business risk and opportunity by addressing new email technologies (e.g. hosted email) as well as other messaging technologies (e.g. instant messaging and voice mail) that employees utilize to perform both business and personal communications today.
The center of gravity for email archiving is shifting from the IT server room to the board room, via the court room. This means that the economics of email archiving is shifting as well, from an emphasis on keeping the cost of storing historic emails low (i.e. a focus on TCO) to one of compliance, reducing corporate risk exposure and proactively ensuring that corporate and employee behavior is consistent with the culture and objectives of the organization as a whole.
Moreover, in addition to legal discovery activities, organizations will increasingly integrate email archiving and messaging technologies with other strategic initiatives including document management, information lifecycle management and rapid user access to unstructured data. Efficient infrastructure is essential but insufficient as the real value proposition is shifting to reducing corporate exposure and increasing end user productivity.
Action item: IT organizations need to recognize that as the economics and value proposition of email archiving and other messaging technologies shift, their centers of expertise will also shift to initially serve corporate counsel to support electronic discovery activities and over time to radically re-architect the unstructured information infrastructure of the organization.
Email managers are under pressure to ensure that the cost, performance, and responsiveness of email are maintained. Senior managers notice when things go wrong. To IT, email archiving systems should be implemented in ways that have the minimum impact on operations. For example, the simplest method of populating an email archive system is to use the daily backup as a source. The alternative method of using a journaling feed introduces overhead and complexity that may impact the production email system.
However, using the email backup will not ensure that all the emails will be captured. Emails immediately deleted will not be there. Journaling would ensure that every email is captured. An email manager might well argue “To capture all the emails means installing journaling. That is expensive and just isn’t necessary just to capture a few emails”. The truth of the matter is that counsel must be able to say that they have included all emails in a discovery request. If users can hide emails by deleting them immediately, they will. If that email is discovered in (say) another organization, counsel has lost all credibility, and the organization will loose the trust of any court. The cost of that can be enormously expensive.
Action item: It is a cost of business to ensure that all emails are captured, and that the legal department knows the full story, warts and all. It must be assumed that opposing legal counsels will know everything about email systems and will exploit any weakness mercilessly. The business must ensure that they clearly specify the business requirements for email archiving, including the requirements for provenance, permanence, and comprehensiveness. Otherwise any supposed legal umbrella will be illusionary.
Email archiving shows up as the tip of the iceberg when it comes to managing, securing, and exploiting the unstructured data within an organization. While some are trying to integrate the pieces into a single solution, no vendor has articulated a credible and scalable technology road map to achieve this. Today’s email archiving products can largely be viewed as point solutions that either focus on infrastructure data problems (e.g. EMC's Centera, HP’s RISS), or attempt to solve end-user problems (e.g. Symantec for archiving, Attenex for e-discovery or Merrill for e-discovery services). Each solution does either the infrastructure well, but fails to provide adequate function for the user, or provides the function without adequate performance, scalability, or ability to integrate with other unstructured data in the organization. The companion piece on the vendor implications of Email archiving discusses the reason why there will be a bifurcation of products that will concentrate either on data infrastructure or on end-user function. New highly parallel architectures such as those implemented in Google’s innovative file system point the way, but the future is unclear.
Action item: In designing strategies for unstructured data, IT organizations should avoid strategic dependence on products that try and integrate the infrastructure and end-user function within the same solution. The likelihood of success in the long run remains very limited. Short term tactical adoption of these products is necessary, but the business case should assume a cost of migration to other products within five years.
Email archiving vendors are capitalizing on compliance and legal mandates of corporations. However, the real long-term opportunities in this marketplace lie in creating better ways to handle meta data, scale infrastructure, integrate a variety of messaging technologies and provide facile and speedy access to unstructured data.
Broadly, there are two main opportunities for vendors.
- Infrastructure that provides automatic provenance and single copies of data with high degrees of parallelism and services that can be flexibly invoked by multiple applications.
- Applications that provide value to users by understanding and supporting specific business processes (e.g. eDiscovery, HR, Marketing, R&D).
Importantly, each of these initiatives requires substantially different solutions and strategies.
Action item: Vendors need to clearly identify their place in the value chain, stake their claims and articulate a vision. Infrastructure vendors need to understand the shifting economics of the business that increasingly emphasize risk reduction and over time, increased user productivity. Google-like architectures that write by appending to an existing ‘thread,’ preserve provenance and are highly parallel will gain momentum (many as hosted services).
Applications vendors need to provide the flexibility to choose infrastructure services and let markets determine what’s best from an infrastructure standpoint. Vendors should not lock into specific databases and architectures but rather broaden solutions to run on emerging architectures.
The availability of free, web-based, consumer email services forces firms to complement efforts to implement the right mix of email archiving technology with the perfect mix of end-user education. Use of internal email systems can be monitored to verify that guidelines are being followed. However, users increasingly can circumvent an organization’s email dictates by using 3rd-party email services like gmail, yahoo, etc, to conduct company business. It’s clear that the courts expect firms to be able to access and provide copies of any digital communication conducted in support of business. It’s not clear if the courts will relax this expectation simply because a business document was created and distributed over a 3rd-party mail system.
Action Item: Email governance policies must include use of 3rd-party email systems, with special efforts made to determine the risk to the firm posed by 3rd-party email provider archiving practices.