The Case for Archiving Solutions

The uncontained and rapid growth of electronically stored information (ESI), in particular, unstructured data including all forms of email and messaging, documents, images, video, voice and increasingly social media/networking along with evolving retention and retrieval requirements are driving the demand for affordable, scalable, trusted archiving solutions in hosted, on-premise and hybrid configurations.

First Generation Solutions: Retain, Authenticate, Comply and the SEC

Immature solutions began appearing in the late 1990’s in response to anticipated new SEC rulings affecting the financial industry for retention of email in “a non-rewritable, easily accessible format” such as WORM with retention periods up to 6 or more years. At this point, the first priority was on retention, authentication and compliance.

When the financial industry proved it could successfully implement email archiving solutions, it was only a matter of time before other industries would follow suit, or be obliged to do so, and other regulatory entities would take notice Perhaps naively, most archive buyers and vendors did not anticipate the sheer volumes and diversity of ESI their solutions would be required to manage or the breadth of capabilities that would be needed.

First Gen Solution Problems: Scalability, Interoperability, Compatibility

Most first generation (First Gen) solutions for message archiving were originally designed and built to address other problems and functions such as searching web pages, optimizing storage or managing relational data not for searching information contained in emails and word documents or managing the entire path of an email and other messages from creation and retention to deletion across file shares, backup tapes and other dispersed technology assets.

First Gen solutions may have saved time and resources upfront but they generally lack flexibility or massive scalability and too often need additional point solution functionality to perform ediscovery searches, to cull data or execute legal holds and desktop searches. This forced customers into deploying a set of solutions with little or no integration between them that did not efficiently support business processes or workflows.

Marginal solutions may work well enough for firms with lower volumes of ESI. However, larger more complex enterprises have become dissatisfied with this arrangement, Ironically, many of these First Gen in-house solutions are provided by long time trusted technology partners who may have the right intentions but an inferior solution.

Tsunami of Contributing Factors

In the intervening years, several factors have come into play to irrevocably change the way all enterprises great and small manage ESI in all of its forms:

• Growth of ESI: Industry pundits estimate that data in the enterprise is doubling every 18 months with unstructured data representing more than 80% of that total. (Structured data is primarily contained in relational data bases, CRM, ERP solutions and the like). This growth has strained the scalability limits of most first generation solutions and blown out many a storage capacity plan as well as pushed backup and recovery time objectives to their limits.

• FRCP Rulings: Since the enactment of the Federal Rules of Civil Procedure (FRCP) governing the preservation and presentation of ESI in December 2006, enterprises have seen a dramatic increase in litigation activities and ediscovery requests. With the initial emphasis on retention, most first generation archiving solutions did not meet legal department requirements spawning a plethora of point solutions.

• New Regulations: Pick your poison: GLBA, Basel II Accords, NASD rulings , SOX, the Patriot ACT, ISO Standards, HIPAA, NRC, FTC, the FDA and the soon to be established FINREG.

• Unsustainable Business Practices: Given an increasingly aggressive regulatory and litigable business atmosphere paired with most individual’s penchant for not getting rid of electronic “stuff” or consistently applying company policies related to management of ESI, enterprises are forced to adopt technology that supports their evolving business needs or face stiff penalties and fines along with lower worker productivity and a much higher cost or doing business.

The Second Generation: The Rise of Point Solutions

As a result of these factors, customers with first generation solutions that did not scale or meet their ediscovery, regulatory or worker productivity requirements turned to point solutions.

Over the last decade, a phalanx of vendors and system integrators looking to capitalize on the market opportunity have stepped into the breach. Search vendors, storage and storage management vendors, archiving and database vendors, email and email management vendors, back up and recovery vendors, enterprise content management (ECM) and workflow vendors, messaging and security vendors, a host of ediscovery and compliance solutions and vendors and system integrators that offer all the above - not to mention hardware appliance solutions and services that provide completely outsourced or hosted solutions.

Second Generation solutions offer vastly improved functionality in the following categories:

• Archiving hundreds of object types: Mail, Images, Instant Message, Video, Voice, etc.,
• Centralized Policy Management: Control to maintain, support company policies,
• Deduplication: Compression, File Deduplication, Single Instancing, Stubbing,
• Ediscovery modules: To address the Collection and Culling of large data stores,
• Enterprise Search: Across multiple Unstructured Data Repositories and File Shares,
• Security: Authentication, Data protection, Encryption,
• Storage: Optimized and Inexpensive, Object based, Content Addressable, NAS, SATA, ILM

Unfortunately, point solutions offer tactical relief for archiving solution shortcomings but not a long term strategy for dealing with evolving organizational requirements.

The expression, “In the land of the blind the one eyed man is king” is appropriate when referring to the maturity of archiving solutions today. Yet, many archiving and information management vendors set expectations too low. Six months to do a POC and trial for less than 100 mail boxes is far too long. Don’t laugh, it’s happening today. Vendors that have problems with stubbing and compression, quickly rebuilding corrupted indexes or restoring folders and are reluctant to offer reasonable performance SLAs and support should be avoided.

Raising the Bar with Next Generation Archiving Solutions

Depending on whose numbers you believe, the combined market for in-house, hosted and hybrid Enterprise Information Archiving Solutions (EAS) will grow roughly 4X in the next five years; to somewhere upwards of $6 billion; ostensibly from replacement of FirstGen systems, migration to or from hosted or hybrid systems and from first time implementations. Some analyst reports list more than thirty viable vendors in the space with additional vendors large and small entering the market every few months. Given the inherent problems with FirstGen and point solutions in the long run, the growth in both number and average size of ESI objects and the potential growth of the market it should be no surprise that a next generation of solutions will enter the market forcing older, inferior systems vendors to upgrade their offerings or provide clients an entire “forklift” solution to keep their business.

Characteristics of NextGen information archiving solutions include better integration of modules, faster and more flexible searching, better server and storage utilization through use of grid architectures, single instancing of stored data (SIS), use of tiered storage or storage pools, automatic tagging for compliance, dual-writing capability, improved policy management engines, automatic folder and index restoration, user interfaces, plug-ins, access to loose files and a limited number of repositories. The better NextGen systems will cost a lot less to implement and maintain because of how the above mentioned characteristics dramatically improve the over TCO and ROI of a solution. Attributes to look for in Next Gen solutions include,

• Superior integration of modules: Archiving, calendaring, collaboration, compliance, ediscovery, mailbox and message management, social media, web applications
• Faster and more flexible searching: Access to loose files, contextual, enterprise, proximity, web, access to both structured and unstructured data
• Integrated Analytics: BI, web, data mining, search in a single integrated package
• Auto-Classification and Indexing: Automatically and/or manually designate categories, easy development and maintenance of taxonomies
• Improved server and storage utilization: Deduplication, grid architectures, single instancing data (SIS), limited number of repositories, cloud services, virtualization
• Scalability: Manage 5 or 50k mailboxes, 1 billion-plus objects with no performance hit
• Flexible policy management engines: By administrator, custodian, matter, user
• Data protection and security: Backup, cloud, automatic folder and index restoration
• Superior user interfaces: Web interface, plug-ins, easy to learn and use
• True partnering: Superior customer service and adaptability to unique requirements
• Lower Total Cost of Ownership (TCO): Affordability
• Defensibility: Secure, reliable, justifiable, scalable

Defensibility Defined

The most prized Information Management (IM) solution attribute today and in the foreseeable future is Defensibility. Certainly ediscovery and archiving vendors aggressively tout this concept, especially as it relates to an entity’s court-ordered or regulatory activities and its ability to produce, in a timely fashion, any written documents or ESI. However, in practice defensibility is much more far reaching. In the parlance of the IM space, defensibility can apply to:

• The ability to demonstrate that appropriate, achievable, consistent policies governing the management of physical and electronic records have been developed and implemented and that employees have been informed and educated on those policies as well as offered ongoing training and updates.
• The ability to demonstrate repeatable processes that support a firm’s need to comply with legal or regulatory requirements.

• The ability to respond to legal or regulatory ediscovery requests in a timely fashion to thwart questionable litigations or potential fines that could be levied due to inability to produce ESI.

• The implementation of solutions that offer predictability whether it be to support compliance with retention policies, the ability to capture all appropriate ESI or the ability to scale as more content is managed electronically and be assured that all needed ESI is captured and preserved.

• The creation of an information security strategy that limits both external and internal risks and breaches when they occur.

• A risk management strategy that identifies potential liabilities, improves disaster preparedness and protects corporate and personal assets.

Strategic Planning for Success

Before making a decision to implement an EAS solution for the first time or upgrading an older system, organizations need to develop a Strategic Information Management (SIM) plan that includes the creation of a cross-functional team from key departments and lines of businesses and establishing policies for disposition of ESI. (See 10 Critical Steps to follow BEFORE Selecting an Enterprise Wide Information Archiving Solution) A SIM team is ready to move forward when the organization has buy-in from the CEO and upper management, it understands who the stakeholders are and what the business drivers are for implementation, how policies and workflows will be updated and the team has done a preliminary TCO analysis along with a rudimentary ROI calculation. In short, when you have a strategic IM plan in place.

Vendors can charge by the mailbox or by the server. In the case of hosted services they can charge by the month. Industry average costs for a fairly robust solution including email and storage management, archiving, compliance, ediscovery and mailbox management can run from $5 to $10 dollars per mailbox per month including hardware, software and support amortized over a 4 or 5 year timeframe.

The Mid-Size Entity Dilemma: Too Many Choices

Generally speaking, when it comes to selecting an archiving solution large and small organizations have it easier than mid-size firms. The assumption is smaller firms have less ESI volume, fewer litigations or information access requests, are less likely to need highly scalable solutions and can do with marginal in-house or cloud based solutions that provide functionality suites for archiving, ediscovery, mailbox management, records management and search. Large firms with over 10k mailboxes have a limited number of choices as there are realistically only a handful or so of solutions able to manage the volumes and magnitude of their archiving challenges.

On the other hand, mid-size firms from 500 to 5,000 mailboxes are a target for the vast majority of archiving solutions that work both ends of the spectrum including those solutions that would produce marginal results. Mid-size firms may soon be approaching the ESI volumes that many large firms experience today but they typically don’t have the resources that large organizations have to dedicate to a records management position, additional support from a cross-functional team or the expertise to develop a SIM plan.

Meanwhile, hosted or hybrid solutions that do not suit large firms will perform much better for mid-size organizations and even some basic hosted or in-house solutions suitable for smaller firms may perform well enough to plug IM requirement holes in the near term. But with more than 100 archiving vendors and growing to choose from in multiple flavors and configurations the choices are not simple. And larger vendors don’t necessarily have the best performing solutions.

Bottom Line

• Small firms are more likely to benefit from in-house archiving suites that offer a range of complementary applications at relatively low price points. Even if vendors do not offer a complete functionality set, many small companies will opt for using basic tools from the likes of Barracuda Networks and Microsoft, or perhaps contract for Cloud based solutions from someone like Google, Mimecast, or Sonian.

• Mid-size firms have a plethora of choices including archiving heavyweights Autonomy, EMC, HP, IBM, Iron Mountain, OpenText and Symantec which they can compare with smaller vendor firm solutions. Based on the functionality, needed an offering from a smaller vendor with extremely competitive pricing might be just as effective. Like smaller entities, mid-size firms with between 500 and 5,000 mailboxes may not have nearly as much in-house records management and technology expertise as large firms. However, their need to develop a Strategic Information Management (SIM) Plan is becoming increasingly more necessary as content and data stores continue to grow and the business and regulatory climate continues to increase the risk of litigation, fines or lost business opportunities if archiving is not safely in place.

• The largest firms with more than 10,000 mailboxes need to consider scalability and robustness of solution on a level that sets them apart from smaller firms. All of the archiving heavyweights will vie for this business along with large archive specialists such as Daegis/Unify and ZL Technologies. Large firms can usually tolerate more point solutions as they have in-house expertise to manage technology solutions and providers.

• Regardless of an organizations size, there is an archiving solution available that will meet the vast number of requirements. However, buyers should be wary that one solution does not fit all sizes regardless of what the vendor says and having a SIM plan puts the user in control of vendor selection and avoids reacting to vendor hype and plugging holes as problems arise.

Action Item: Especially Mid-size and Large organizations need a Strategic Information Management (SIM) plan and archiving roadmap in place before selecting an archive solution whether it be in-house, hosted or hybrid. Be wary of vendor claims, check references thoroughly, and guided by the SIM plan, develop a checklist for selecting an archiving vendor that is a true partner who shares your vision.

Footnotes: