Making Hadoop Safe for Mission Critical Applications

We all know there’s lots of excitement and buzz surrounding Hadoop, but talk to some CIOs in “non-web” industries about moving mission critical apps to the open source Big Data framework and you’re bound to hear a little fear in their voices.

They’re worried that Hadoop is not ready for primetime because it has a single point of failure. That is, if the NameNode in a cluster goes down, the entire cluster goes down. Spinning clusters back up into working order following a NameNode failure takes time and, by definition, mission critical applications can’t go down … ever. Until the SPOF is solved, more than a handful of Fortune 500 companies will continue paying Oracle through the nose rather than risk a disruption to critical apps.

The Hadoop community has taken various stabs at the SPOF issue over the last year, however. MapR, for example, eliminates the concept of a single NameNode by distributing metadata across nodes. Hortonworks is working with VMware to allow VSphere customers to deploy High Availability NameNode and JobTracker nodes on virtual machines, making automated failover possible. And Cloudera includes HA in Hadoop 2.0-based CDH4 (though there are questions about Hadoop 2.0 being ready for primetime.)

WANdisco takes a different approach all-together. The company makes its living selling peer-to-peer, active-active data replication technology based on its patented algorithm. WANdisco customers use the technology to keep data and files in synch across data centers and geogoraphies. With its recent acquisition of AltoStor, WANdisco plans to release its own Hadoop management software appliance and Hadoop distribution that applies its bread-and-butter WAN replication technology to the open source Big Data framework.

“We can take our secret sauce, which is this patented active-active replication algorithm, and apply it to Hadoop to make it bullet-proof for enterprise deployments,” WANdisco CEO and Chairman David Richards told me yesterday on theCUBE. “We have something coming out called the Non-Stop NameNode … that will ensure that Hadoop stays up 100% of the time, guaranteed.”

Click to watch the entire interview with WANdisco’s Richards and Sundar.

Jagare Sundar, co-founder of AltoStor and now CTO & VP Engineering of Big Data at WANdisco, told me the key to the Non-Stop NameNode is that in fact there is no single NameNode as such. Rather, WANdisco utilizes a peer-to-peer rather than a master-slave architecture, as most Hadoop deployments use. Each node in a cluster can service reads and writes, meaning true load balancing is possible.

Sundar, who previously served as Director of Hadoop Engineering at Yahoo!, said this means administrators can bring down a node that is serving as the NameNode for maintenance and a peer node will pick up right where it left off. This has profound implications for running mission critical applications on HBase, the most popular database for Hadoop at the moment:

We also guarantee that applications that run on top of our Non-Stop NameNode such as HBase will continue to run uninterrupted.  Applications that you run on top of HBase will not even know of the existence of multiple NameNodes or that one went down or came back. This is truly the only active-active solution out there. All the other ones are passive standby, hot standby, cold standby, all of them take time to come back up and start providing service. There’s no load balancing with any of these solutions. HBase is very fragile when it’s run on top of some of these solutions so that’s a clear advantage.

This is an important issue because it relates directly to the somewhat lackluster Big Data applications space. For the business, the promise of Big Data is in mission critical applications that allow marketers, finance and other LOB workers to monetize data to achieve competitive advantage. But CIOs aren’t going to move new or existing mission critical applications on to a platform that is at risk for major downtime. There are other issues of course, including the need to make application developer tools for Big Data easier to use. But hardening Hadoop is at the top of the list for CIOs.

*DISCLAIMER: For a list of Wikibon clients, click here.


, ,