Storage Peer Incite: Notes from Wikibon’s September 11, 2007 Research Meeting
Moderator: Peter Burris & Analyst: David Floyer
This week Wikibon presents Email archiving: necessity or pleasure. Email archiving is becoming a major thrust in IT in the face of new legal requirements and the increasing use of email in civil and, potentially, criminal actions involving businesses. Once created, these archives can have many other uses. However, warns the Wikibon community, those charged with planning and creating the archive must maintain a tight focus on the basic needs of compliance and legal risk abatement. This focus will be hard to maintain in the face of vendor promotion of a variety of other, often desirable, features and functionality. Software vendors compete on providing the greatest mix of these features, but an archive project that becomes distracted with evaluating these will become bogged down and may well collapse of its own weight. Focusing on the compliance and legal risk needs will in many cases lead organizations to consider simpler solutions that are less expensive and easier to implement but do not provide all the "bells and whistles". However, as long as the solution is based on standards rather than nonstandard data formats that lock the organization into a single vendor solution, such choices will provide a better solution to the overriding needs behind the archive, and other functionality can be added later.
Organizations in some industries not directly cited in the new business legislation, such as several US retail chains, certain manufacturers and other less regulated industries, either are not archiving or are only keeping minimal amounts of their email on the philosophy that the less they keep the less likely they are to have to defend themselves against problems in email. We view this as a dangerous strategy. A civil court action against any organization today will include a demand to produce relevant emails as part of discovery. Companies unable to meet that demand may face a very annoyed judge who can and possibly will impose very harsh punitive damages, as has already been seen in several court actions. Thus we believe that all organizations, regardless of their industry or of whether they are covered by specific legal requirements to archive email or not, should be installing archives as the best insurance and risk abatement policy they can buy against large punitive damages in what otherwise would be routine civil suits. Bert Latamore
Contents |
Email archiving: Necessity or pleasure
With knowledge workers spending 10% or more of their work days on email and the focus on email in both business criminal and civil legal actions, the implementation of solid, defensible, reasonable email compliance practices has become incredibly important. This effort must start with the development of policies and practices that users need to follow in their email handling, along with the creation of email archiving practices that comply with present legal mandates and, to the degree possible, anticipate future court and legislative actions. Otherwise, the enterprise may well find itself exposed to a “witch hunt” through its back email by an aggressive team of attorneys looking to liabilities to attack. While archiving is being pioneered largely in answer to specific legislative requirements to preserve email, even organizations operating in jurisdictions or industries not facing these explicit requirements need to anticipate future civil actions that may focus on their email archives or, worse, lack of them.
Given the clear dictates of new laws in Europe and America, and the sometimes punitive actions of US civil court judges impatient with the inability of corporate defendants in several famous cases to produce a full record of relevant emails in a reasonable time, businesses must focus their archive creation efforts first on compliance and risk abatement, ignoring other potential uses or advantages until after the basic system is built. To do otherwise is to invite design bloat that will increase cost and time, may create archiving systems that do not in fact meet the basic legal needs driving archiving and, in the worst case, may create a project that collapses of its own weight. Maintaining this tight focus in the face of software vendor competition based on providing the widest possible collection of functionalities will be a key challenge for the email archiving team, including storage administrators, and in many cases may lead to selection of a much simpler solution that otherwise might be implemented. Consequently email archive design teams must address three critical issues as they define the architecture:
- They must assess the actual scope of the risk, as measured by potential losses from three main areas – fines from noncompliance, punitive damages, and lack of accountability leading to potential extensive criminal witch hunts. The scope of these risk elements will vary by industry, geography and the need to sustain a history of email activity.
- They must recognize that over the 7-10 year period that emails need to be archived, they will experience at least one media migration regardless of the application or media chosen, driven by the 30% per year decrease in cost of new media acquisition versus steadily rising maintenance costs. We expect that virtual tape libraries will emerge as a favored mechanism for managing these migrations.
- They must anticipate the certainty that the business will request ways to access the archive to generate derivative business value by running future applications against it as an information store. In this context the archive takes on characteristics of a content management system. Therefore, designers need to consider factors such as open access methods that developers can use, and therefore need to avoid solutions that use proprietary data formats that lock the business into a particular vendor.
Ultimately the group running the archive may look like a classic applications team if only because of the potential future use of the archive as a content management system. But near term it is critical that storage administrators work closely with legal counsel and risk management personnel in the business to keep the focus of the archive project on risk, with a recognition of the derivative application benefits in the design process. They must avoid being taken over by email application management teams who will tend to bloat the project with a functionality wish list that may hinder rather than advance the business’s overall risk mitigation objectives.
Action item: Storage administrators are being asked to put forward clear plans to manage email archive infrastructure. When responding, they must focus narrowly on the requirements put forward by the business’s legal and risk management teams and choose appropriate email archive solutions, infrastructure and operating policies to ensure that the business receives the right spectrum of risk mitigation capabilities and not just a new cool set of email application functions.
The irony of email archiving adoption
While risk can come in many flavors, as discussed narrowly in today's summary Email archiving: Necessity or pleasure three types stand out with regard to email archives:
- Fines from government or industry regulation and oversight;
- Punitive damages from litigation;
- Accountability of who directed whom to do what, when.
Dependent on the industry (or often geography) it's perceived risk or lack thereof that drives the pace of email archive investments, at least initially. For example, in the late 1990's SEC rule 17A-4 was put into place and required exchange members, brokers and dealers to keep track of and archive electronic communications like instant messages and emails. It's not surprising that the financial services industry responded (although some firms were slow) and implemented risk reduction measures in the form of straightforward email archiving systems.
Other less regulated industries --like pharmaceutical-- or way at the other end of the perceived risk spectrum --such as retail-- were much slower to respond, largely because oversight agencies weren't pushing them and the perceived risk was lower. However, as firms are learning, the risk of damages from litigation can be enormous.
Wikibon case studies clearly show that while organizations are learning risks are often more a function of corporate asset value (i.e. how much can lawyers extract) versus industry regulation, those industries that are less regulated seem to be more willing to trade risk abatement for email function. Specifically, we've seen organizations add in substantially enhanced email function as part of an archiving project but sacrifice (for budget and complexity reasons) rolling out the archive to international divisions, subsidiaries and/or lower level employees-- leaving themselves exposed.
Here's the irony. Organizations that implement a simple, effective archiving system broadly across the organization, that may in fact sacrifice incremental email enhancements, are probably less at risk than those companies who have chosen not to roll out email archives for major parts of their user population yet will still fund email enhancements to placate business users.
Action Item: Generally, email archives are being justified on the basis of risk mitigation and they must deliver on that promise first and foremost. Keep it simple. To succeed, focus initial email archiving functionality on risk abatement broadly across the organization and avoid unnecessary complexity.
Email archive administration: When is an archive something else?
The business value of an email archive will flow from two sources: (1) reduced exposure to legal fines and damages and (2) derivative mining of knowledge embedded in electronic correspondences. As a system for reducing risk, an email archive will behave like any other archive system: write activity will be nearly certain, records will be managed using highly mature practices, and reads will be rare and limited in scope. Consequently, the practices of administering an email archive will be easily aligned with traditional archive administration, and likely handled by infrastructure personnel. However, as a critical source of data to a dynamic set of extremely valuable search, analysis, and prediction systems, an email archive will behave more like a content management system: writes still will be nearly certain, but access to records will be extremely fluid. Traditional archive administration practices (and technologies) cannot be applied to an email archive under these more dynamic conditions. Indeed, administrative practices are likely to be more aligned with database management administration than archive administration. While storage professionals will be intimately involved in the processes of architecting and implementing an email archive system, their role in administration likely will be reduced to writing and achieving specific – and potentially very high profile – infrastructure service-level agreements for email archive administrators that are more aligned with application management groups.
Action Item: The hybrid nature of an email archive will strain traditional storage administration approaches to archive management. To appropriately manage an email archive, storage professionals should work to establish an administrative regime similar to that found in the database world, making sure to forge administrative rules that do not compromise the fundamental requirement that email archive data be legally verifiable.
Separating email archiving technologies
Two different technology sets are required to mitigate risk:
- Compliance risk (leading usually to fines, but for some regulated industries potentially leading to shutdown) requires ensuring that all relevant emails are collected in an archive. Archive management must be able to prove to stakeholders and a court of law that the archive is complete and immutable, and that they have implemented reasonable fully documented procedures for managing the archive that have been scrupulously followed.
- Litigation and accountability risk which require technologies to analyze the archive, identify, and mitigate risk.
The two technologies are completely different. Establishing the basic archive infrastructure requires more attention to process than technology, and will not be refreshed very often (maybe every 15 to 20 years). It must be able to allow retrieval of emails within a reasonable time (usually 48 hours), and must allow simple and complete export of archive data to technologies that will analyze the data. The technologies will be similar across most industries. Outsourcing will common.
The technologies that analyze, identify and mitigate risk will change often and radically as the arms race between aggressive councils and defendants plays out. The technologies will be different for different industries and departments. Word filters, communication maps, contextual searches, inference machines and other technologies will come and go, and use the archive and other structured and unstructured data sources.
Action item: Separate the technologies and the teams that manage them. Like oil and water, avoid mixing.
Email archiving realignment
As email archiving projects mature, risk mitigation will emerge as the primary objective. As discussed in the storage alert Separating email archiving technologies, the infrastructure email archiving and the exploitation of that archive to identify and mitigate risk have completely different characteristics, and will refresh on completely different cycles.
Email archiving infrastructure technologies and solutions will be common across most industries, and will respond to traditional and broad economies of scale. They will compete to be the platform of choice for the suppliers of exploiting technologies, and should be careful not to compete with these suppliers. New entrants such as Google and Microsoft are likely to exploit their distributed file services to offer web-based solutions. Outsourcing in general is likely to be a significant component of the industry.
Software based vendors that create solutions that identify and mitigate risk will compete based on the ability to integrate different sources of data and on specific industry knowledge. They will require and expect unfettered access to data sources such as an email archive, and will not support suppliers that compete with or impede them. The rewards for success are likely to be high and short term.
Action item: As the email archiving market realigns, vendors should choose where they compete carefully. Strategies that integrate infrastructure and risk analysis and mitigation technologies are unlikely to retain long-term economies of scale. Strategies that lock in customers by restricting their ability to exploit the data in the archive are very unlikely to retain traction.
Source the email archive
If you're already using a service like Iron Mountain for offsite archiving (and even if you're not) it might make sense to consider outsourcing the archive. Organizations have many concerns from SOX to Bliley to Patriot Act, HIPPA and new Federal Rules of Civil Procedure. Keeping documents for a multi-year period --or emails forever in many cases for legal reasons-- is a big change that's occured over the past three to five years because emails are now considered legal documents.
Regulators often require companies to be able to bring up mission-critical applications and that often determines tiering requirements. For example, tier 1 mission-critical within six-to-twelve hours, tier 2 within twenty-four hours or tier 3 within seventy-two hours. Email often falls into tier 1 but after some number of days, they will often migrate to tier 2 or tier 3. This underscores the audit concerns organizations have and the need to make sure adequate processes are in place. Indeed, the lawyer community will often attack an organization's proceesses and procedures as a weak point in a case, sometimes flipping the burden of proof.
Outourcing the archive can, for somewhere between $3 - $7 per user per month get the SLA guarantees required to support these applications that internal IT may not be able to (or want to) worry about. Service level agreements here might include factors like the data hasn't been tampered with, all emails have been captured, procedures are in place and followed correctly, etc. The outsourcing vendor may not have all the bells and whistles functionality but process-wiese they're probably pretty up-to-date.
Action Item: Because risk reduction is the primary driver for email archiving, third party services should be considered as viable options to source the archive. This will allow organizations to more quickly deliver on the promises of the archiving initiative.
