Originating Author: Kaushik Das

Email storage plays a vital role in providing email services in most organizations. For the last ten years or so, it has shaped itself up to be the most important component of email services by providing scalability along with robustness to transform email as a strategic business tool. However, it is not sufficient in view of the business needs of mid-sized organizations that have to accommodate the explosive growth of email in the next ten years to come.

The email traffic growth, along with the size of individual emails, is redefining the business needs of email storage for all mid-sized organizations. For example, a mid-size organization, with 4,000 email users, needs an additional email storage capacity of 1.5 terabytes every month. Since businesses now send as much as 60% of their business-critical data via email, the backup need and backup convenience for email storage has also become important for IT managers.

Email data, as another form of electronic communication, is now an integral part of business records. Mid-sized businesses now have a requirement to satisfy federal compliance regulations such as Sarbanes-Oxley, SEC Rule 17a for maintaining accessible and secure business records. They also have to respond to judicial discovery request if any. Legal retention periods are based on content and increasingly content resides in emails. As such, the storage of email content necessitates sufficient storage as well as a technology that is up-to-date and can read and process the current email data after a potentially long period of time.

Legal retention is based upon content not container or format. Retention for an email contract or clinical trial data is same regardless of it is in a Word doc, in an email or on paper. There is no standard legal retention for email.

Some specific challenges for IT managers, administrators and users are:

• PST archival (for MS outlook) has its own drawbacks. Since this solution has no centralized location, IT managers find it difficult to manage, search and retrieve archived (or backed up) mesasages. Users are forced to create PST archives or suffer undesirable system behaviors and/or manage mailbox space constraints.
• According to CA, significant time is spent on back up (8-12 hours per week) and recovering (5-6 hours per week) archived messages or attachments.

Capability of email storage architecture

Email storage architecture delivers a complete overhaul in order to accommodate an explosive growth in email, backup and legal compliance for judicial discovery in addition to 24x7x365 availability. The goal of a mid-sized organization is to eliminate the current storage drawbacks by using a centralized, easily searchable repository with customized automatic archiving capabilities, and real-time retrieval of messages. Capabilities include:

• 24x7x365 availability and resilience to interruptions – The email facility should be available to all users on an uninterrupted basis. In order to reduce downtime, the storage architecture must be resilient to overcome various interruptions. Also, faster recovery is as important as uninterrupted availability of email systems.

• Backup (or archiving): Introduce automated email archive and retrieval process. The automated technologies can significantly reduce the amount of time spent by both end-users and system administrators on archiving activities.

• Record management and regulatory compliance: Email audit necessitates an extensive search and retrieval facility, including message, links and attachment content. The storage architecture will also comply with legal requirements to manage and reproduce email business records appropriately upon request.

A relevant article on Email archiving discusses how to save and protect the data contained in email messages.

Specific operational goals of architecting email storage

The investment that is likely to be required to implement an email storage infrastructure is between $2 million to $3 million, with an elapsed time of between 8 to 10 months. Annual costs, thereafter, will be approximately $0.5 million which will cover IT costs for updates and labour. [Note: These figures assume a Standard wikibon business model organization with $1 billion in revenue, with 4,000 employees and an IT budget of $40 million per year. The scenario assumes that 180 terabytes are installed, with a mixture of high performance SAN, mid-range storage solutions, some direct attached storage, and some NAS]. The cost assumes that older technology is phased out with latest hardware and software to reflect the latest technological developments. The email data has a legal retention period of 12 years, which assumes that data created with the current technology will not be processed or read by newer technologies after 12 years. All the benefits will come from the business and virtually none from IT due to the large increase in infrastructure spend. Apart from investing on improving the performance, the email storage infrastructure requires an “Insurance policy" dimension to the investment in order to protect the organization from litigation costs and fines.

A successful email storage implementation will:

• Deal with an ever-growing e-mail volume with increased size for individual email messages containing larger attachments, i.e. successfully manage the explosive email growth
• Provide 24x7x 365 availability with higher reliability (minimal or no downtime) and higher performance (i.e., performance doesn't decrease when actual storage approaches the fixed-size email server capacity)
• Provide backup convenience without shutting down the e-mail operation and data consistency during such live backup
• Allow the users to conduct key business transactions like contract negotiation or selling of goods and services
• Allow the IT managers, responsible for email storage and infrastructure, to retain email messages and attachments in order to support regulatory compliance, avoid legal fines or litigation costs, and satisfy auditing requirements
• Allow less IT resources to manage the email storage and infrastructure
• Allow up to 50% reduction in storage capacity requirements
• Allow leveraging the “email knowledge asset” through knowledge management tools, sometimes referred to as data mining, to maintain a competitive advantage
• Reduce the time spent for searching emails. Users spend almost two hours every day to manage their mailboxes. See the next point how productivity can be improved based on this.

The major expected business productivity improvements will normally include:

• Less time spent by end-user to search for older messages. With sophisticated email storage infrastructure, administrators allow unlimited mailbox capacity which, in turn, offers two main advantages – prevents the loss of important messaging data and reduces the time needed to manage one’s mailbox and archive. A specific productivity improvement, for example, comes from the fact that the users don't have to create PST files.

Other potential impacts on organization include:

• The ability to produce messages in order to meet demands connected to the freedom of information act (FOIA) or similar guidelines issued by local, state and federal agencies

• The ability to achieve ROI – with “no” significant missed opportunity (e.g. there is no downtime), increased productivity of user population and a higher service level expectation on the part of customers

• The paramount importance of an email retention policy for non-regulated businesses

• The best practices that can reduce storage and discovery costs as well as legal and regulatory risks

Risks of architecting email storage infrastructure

It's imperative that enterprises attempt to mitigate the risks that arise when implementing the email storage architecture. Whereas storage is a physical entity, storing and managing email data requires a solution that involves a combination of people, technology and processes. Again, a single solution is not sufficient to address all the challenges that an IT department will encounter to properly implement email storage (and to deliver a certain service level agreement or SLA). In order to achieve success, the email storage architecture should assess the risks that are associated with the following:

• Storage and backup demand - it's always tricky to know the correct storage requirement for a mid-size business, due to increasing volume and attachments. Also, the backup infrastructure (e.g. no back up window due to 24x7 utilization) is heavily stressed when recovering business-critical data with the needed granularity.

• Performance demand - businesses like to ensure 24/7/365 availability (with no downtime or performance degradation). IT managers need to implement an extremely resilient system that offers uninterrupted email service.

• Legal compliance and judicial discovery requirement – there are emerging guidelines for data management practices. Businesses face serious consequences when email data is not managed properly. For example, they must be able to produce records meeting audit criteria within a stipulated time.

Email storage architecture initiative

Expectations (out-of-scope)

It's reasonable to expect that an overall enterprise storage architecture has been defined and it's clear where the email storage fits in this architecure. However, The following items are necessary and should be in place for a successful email storage architecture implementation:

• An overall IT risk assessment, including threat probabilities and overall business impact
• The technology trends for hardware and software
• The legal compliance requirements for countries other than US
• Defining appropriate email data management metrics
• A key decision maker has been identified and roles various business groups are defined and understood

Analyze Phase

Acceptance test considerations:

A successful email storage analysis is complete when the sponsor (or the stake holder) considers the initial business case and accepts it. In particular, a business benefit analysis has been performed such that the sponsor has agreed to the fact that the benefits will only come from the business and not from the IT (cost savings) due to the large increase in infrastructure spend. Upon acceptance, the sponsor will proceed to the design phase.

Key analysis milestones:

• Existing email systems documented and understood
  • Understand the current email storage by knowing the number of email users, the volume and size of email storage assigned to each individual user, and the growth characteristics of the email service
  • Understand the extent to which the email is currently used for key transactions (contract negotiation/selling of goods and services)
  • Understand the email storage constraints on users (size of mailboxes, amount of storage in mailboxes)
  • Understand the backup requirements for IT and the end-users themselves
• Analysis for future email storage
  • Can an enterprise provide unlimited email storage for every individual user within a certain budget? For example, if an enterprise currently allocates a storage of 500 megabytes, can a properly architectured system can increase the limit by three times or elliminate it completely?
• Key business objectives defined and presented
  • Design a first-pass email storage system and get agreement on the budget considering cost of equipment, services, IT staff, and user staff to implement and maintain the email storage system
  • Get agreement on implementing legal compliance (the legal team must validate the requirements). In the US, the specific requirements are SEC Rule 17a and Sarbanes-Oxley. In other countries, there will be country-specific industry legislation.
  • Get different stake holders (such as IT, legal, record management) to agree to work as a team
  • Get agreement on performance such as uninterrupted service, minimum downtime, easy access to backup records (and within a stipulated cost) and high-speed recovery
  • Get agreement on key user productivity benefits
  • Get agreement on reducing email risks (such as litigation, fines) in the form of judicial discovery
  • Create a business case, present it to stakeholders, and get agreement to go to the design phase

Design Phase

Acceptance test considerations

The design phase will be deemed complete when the design has been accepted by the sponsor and agreed to by key groups such as IT, legal and record management, and agreement has been reached to proceed to the deploy phase or kill the initiative.

Key design milestones:

The design phase should take 3-4 months for most companies.

• Primary vendor decided
  • Decide on vendor hardware and software technologies available and issue RFP/solicit bids
• Email storage procedures designed
  • Design email storage procedures and processes around the hardware and software decided
  • Enforce an email management policy. State that all email content is the property of the organization and employees should limit personal use of their email id
  • Define what is and what is not appropriate email content across the email user base
  • Define retention schedules to help user-base to retain emails on folders for a short period of time for ongoing business and how to identify, classify and archive email
  • Implement correct email management practices to find out what the organization legally and operationally needs to retain, determine the retention period, and implement that policy consistently.
  • Use of metadata to evaluate email messages, if needed.
• Email storage policies are updated to reflect Sarbanes-Oxley regulations or SEC Rule 17a and risks are identified
  • Policies should be documented and followed.
  • Legal and compliance teams should review and approve changes
  • Access to email data must require identification and authentication.
• Pay particular attention to the existing email storage to maintain full email service (until new email storage is deployed) according to the SLA.
• Determine training requirements for operations
• Design test procedures and scripts
• Design risk mitigation systems (for the email storage project)
• Design of legal discovery systems if required
• Design of internal audit systems to reduce risk if required
• Design of external audit systems to reduce risk (e.g., compliance with record retention statutes) if required

Deploy Phase

This phase should take about 4-6 months and cost about $1 to $2 million.

Acceptance test considerations:

The implementation phase is complete when the new email storage infrastructure is installed, tested and brought into service. The system, including transitioning to the new system, should be evaluated for its performance, usefulness and user-friendliness. The (internal or external) auditors must evaluate the legal compliance, including the risk mitigation aspects of the new email storage infrastructure.

Organizations like to deploy email storage architecture by putting into practice certain activity milestones:

Key deployment milestones:

• Email storage infrastructure built
  • Installation of hardware and software functionality
  • Installation of any changes required to current email system
  • Smooth transitioning to the new system
  • Update and creation of new processes and procedures, with full documentation

• Email storage infrastructure tested
  • Testing of equipment, software, and features (functionalities)
  • Risk mitigation applications implemented
  • Risk mitigation applications as designed, implemented and signed off by user departments
  • Migration & Cut-over to new email storage infrastructure is completed
• End-user issues verified
  • User training and documentation completed
  • Update and creation of new processes and procedures, with full documentation
  • Help desk operatives trained and documentation updated
• Business issues tested
  • Testing for back up, availability, downtime and recovery
  • Legal compliance issues, including capability for judicial discovery , revisited and verified
• Email storage infrastructure initiative wrapped up
  • Procedures set up for full operation monitoring (availability, downtime, recovery)
  • Procedures set up for weekly, monthly and annual testing
  • Procedures set up for adding email storage, functionalities, and risk reduction applications
  • Final review of documentation
  • All project staff released and full hand-over to IT operations

Summary for email storage architecture initiative

This initiative prepares a mid-sized business to face the new business reality of email communication and to achieve success by managing a well-architectured email storage infrastructure. It brings forward the justification to evaluate and consider a new email storage infrastructure not only to satisfy the need for storage and performance demand but also for mitigating risks with appropriate legal compliance and judicial discovery. Many organizations, both regulated and non-regulated, will soon determine that a properly architectured email storage is a sound business strategy that is necessary to reap the benefits of high-quality business communication along with protection from reputation-tempering events, lawsuits, and fines.