My 5 Key Lessons Learned at Hadoop World 2011

Hadoop World 2011 was bursting at the seams last week. As Cloudera CEO Mike Olson put it, the Sheraton in New York City was “fire marshal full.” The official count was 1,400 attendees, but I suspect that number was even higher. Word is Cloudera had to turn away hundreds who just showed up at the door for the conference.

That’s a good sign for Cloudera as a company and Hadoop as a whole, which leads me to the first of my five key takeaways from Hadoop World.

1. The Hadoop Ecosystem is Not Just Growing, But Thriving. Attendance at Hadoop World 2011 doubled from the previous year, and most of the attendees were practitioners and many were contributors to the Hadoop project. Cloudera now has competition in the form of Hortonworks and MapR. The ranks of vendors – including exciting start-ups like Digital Reasoning, innovative independent players like Attivio, and time-tested stalwarts like Informatica and Teradata — building Hadoop-related tools and apps is large and diverse, and their numbers will only grow thanks to the availability of yet more seed money via the Big Data Fund and increasing VC interest in all things Big Data. With all the activity in the Hadoop ecosystem, there’s little doubt we are on the verge of a wave of Big Data innovation.

Cloudera CEO Mike Olson on Thriving Hadoop Ecosystem

Watch live video from on

2. Hadoop is Enterprise-Ready (And Not Just Web-centric Enterprises Ready.) Hadoop early adopters were mostly large Web companies, and for good reason. They were the firs organizations to run up against Big Data. But when non-Web companies like JPMorganchase, whose roots stretch back to the 18th century and whose enterprise-level requirements are second to nobody, start embracing Hadoop, I think its safe to declare that Hadoop is Enterprise Ready. The Web companies validated Hadoop, a thriving ecosystem of practitioners, contributors and vendors added crucial management and integration features, and now non-Web companies are ready to take the Hadoop plunge.

Cloudera’s Amr Awadallah on Hadoop’s Enterprise Readiness

Watch live video from on

3. 2012 will be The Year of Killer Hadoop Apps. Now that the Hadoop infrastructure layer is enterprise-ready, the next step is building killer applications that can be easily deployed and consumed that deliver significant business value. And that will be the dominant meme of the Hadoop world in 2012. Start-ups like Tresata, Revolution Analytics and Datameer are already making great strides on Hadoop applications and you can be sure a slew of newcomers will hit the scene over the next 12 months with the same goal.

Tresata’s Abhi Mehta on Big Data Revolution

4. Data Scientist is A Multi-Disciplinary Role. It’s well known there’s a dearth of Data Scientists on the job market, but why? The reason is Data Scientists require a blend of computer science, math and social science skills that you don’t often find on a single individual. Cloudera’s Jeff Hammerbacher, who basically invented the Data Scientist role while at Facebook, gave a great explanation of what it takes to be a Data Scientist on theCUBE. The bottom line is that there needs to be more training and educational resources directed towards developing a strong workforce of skilled Data Scientists in order for Hadoop to reach its full potential.

Cloudera’s Jeff Hammerbacher on Requirements for a Data Scientist

5. Professional Services Will Play Key Role in Hadoop Adoption. Until there are enough Data Scientists to go around, and it’s going to be awhile, professional services organizations will be crucial for most enterprises to effectively apply Hadoop to achieve real business problems. Even when the ranks of Data Scientists fills out, services providers like Think Big Analytics are needed to help executives and other business types understand the potential of this newfangled technology called Hadoop and to help enterprises develop a data-driven culture.

Think Big Analytics’ Rick Farnell on The Role of Professional Services in Big Data Environments

We have lots (and I mean lots) more coverage of Hadoop World here and our Big Data Manifesto here, so if you missed the show (or you just want to relive it!) check out all our live video from #theCUBE and written coverage on Wikibon and SiliconANGLE.

So what were your key takeaways from Hadoop World? Join the Wikibon community (we’re 16,000 strong now and growing) and get in on the Hadoop/Big Data discussion. One thing I’ve learned over the years is that the more smart people engaged in a technology, the faster it matures and the more valuable it becomes for end-users. I look forward to continuing the conversation with you! 


, ,