Big data is a big deal. Everybody has it, and it is changing the entire computing stack from infrastructure to applications, says Ping Li, partner at Accel Partners and a member of the board at Hadoop database technology developer Cloudera.
“One thing we've seen at Cloudera is that everyone's got big data,” he told SiliconAngle Founder John Furrier in a panel interview webcast over SiliconAngle.tv recently. The full transcription is available on Wikibon.org. “They start off with a terabyte, and by the end of the conversation they find they have petabytes.”
The major reason for this is that big data is changing the definition of data itself. “Cloudera and Hadoop are trying to change the boundaries of what database technologies and database management can do,” he said. “The reason people didn't capture all the data is because there was no cheap, efficient way to store and derive value from semi-structured and unstructured data, so they were throwing it away.” Hadoop changes the paradigm, allowing companies to analyze all this data to derive meaning and business insights.
That changes the game at every level. But, says El Dorado Ventures partner Charles Beeler, “Clearly the place where you're going to see the most momentum is the closer you get to the end-user of a product.” Virtual desktop, for instance, is an area that his VC company is interested in and that he believes is ready to enter a high growth phase in the enterprise.
Another indication of the impact of big data and related technologies is that companies are becoming interested in how Facebook, Google, and Yahoo! designed their data centers to handle the huge volume of data they are capturing off the Internet. Mr. Li says. “It's very interesting to see how these Internet data centers are becoming thought leaders in terms of where some of the new ground-breaking technologies are, whether it's in the storage layer or network or compute layer.” And while he warns that companies need to be careful about transplanting infrastructure from a consumer Web company to an enterprise environment, the private and hybrid cloud environments large enterprises are building borrow a lot of ideas from the big Web companies.
Another huge trend in business that relates to big data is the consumerization of IT and strong interest in harnessing social media in the enterprise. Several major companies, including Cisco Systems, have their own internal versions of Facebook that they use in large part to promote communications across the normal internal boundaries of large organizations.
“IT executives are running around with their hair on fire trying to figure out how to deal with all these iPads and so forth entering the world,” says Pete Sonsini, partner at New Enterprise Associates (NEA). “No one knows exactly what the right answer is. A lot of startups are trying to bring social media and consumerization into the enterprise, but a lot of the problems are still unsolved.”
This is a fundamental change of mindset for corporate IT. For instance, says Mr. Beeler, an IT executive at a major financial company told him, “You know our job used to be to block this stuff. Now we have guys going home at night, and they're more efficient getting things done at home than they are in the office because of the limits we put on them. We've got to change that paradigm.”
Action Item: Organizations need to look at how they can harness social media and other Web technologies to improve the organization and in particular to foster creativity and the development of ideas that may stretch across the boundaries of different departments and operating units within the organization. They also need to be more open to incorporating outside resources. The security threats are out there certainly, but so are the possibilities for increasing productivity and creativity, as well as improving employee morale.