Big Data

From Wikibon

Revision as of 18:47, 25 February 2014

Big Data Research and Analysis from The Wikibon Project

The best of Wikibon's Big Data research and analysis. Follow Wikibon Big Data Analyst Jeff Kelly on Twitter and LinkedIn. For a list of Wikibon clients, click here. Also see Wikibon's research on disruptions to infrastructure on the Software-led Infrastructure page.

Big Data Market Forecasts

NEW REPORT FEBRUARY 10, 2014
The Big Data market is on the verge of a rapid growth spurt that will see it over the $50 billion mark worldwide within the next five years. The analysis in this in-depth research report highlights Wikibon’s five-year forecast for the Big Data market as a whole as well as vendor revenue estimates. It includes full analysis of the leading market growth catalysts and advice for enterprise CIOs and business leaders looking to capitalize on Big Data and business analytics.

The market for Hadoop/NoSQL software and services topped $540 million in 2012 as measured by vendor revenue. Over the next five years, Wikibon forecasts this market to grow to nearly $3.5 billion, a compound annual growth rate (CAGR) of 45%. Demand for new approaches for processing, storing, analyzing and delivering value from Big Data is the main catalyst driving the Hadoop/NoSQL software and services market.

The research behind this article drills down into the database components, Big Data SQL database revenue, and Big Data NoSQL database revenue, highlighted in Figure 1. The total represents 10.4% of total Big Data in 2012, and remains at about 10% of total through 2017. Both SQL and NoSQL databases have strong roles to play in Big Data solutions.

Big Data Manifestos and Definitions

Big Data is the new definitive source of competitive advantage across all industries. Enterprises and technology vendors that dismiss Big Data as a passing fad do so at their peril and, in our opinion, will soon find themselves struggling to keep up with more foreword-thinking rivals. For those organizations that understand and embrace the new reality of Big Data, the possibilities for new innovation, improved agility, and increased profitability are nearly endless.

Wikibon defines the Data Economy as the emerging economy in which organizations succeed or fail based in large part on their ability to leverage data and analytics to improve operational efficiencies, to make better tactical and strategic decisions, and to create innovative products, services and business models to meet & exceed customer expectations.

Big Data is emerging from the realms of data science projects to help companies better understand the masses of data being created both inside and outside the enterprise, make better decisions that result in real business value, and act in real-time to better serve customers and target markets. The IT techniques and tools to execute big data processing are new, very important and exciting. In this report, Wikibon provides its definition of enterprise Big Data and what it means for practitioners.

Industrial Internet and Internet of Things

The main difference between the consumer/social Internets and the Industrial Internet is in how and how much value is created. For consumer/social Internets, the majority of value is created from advertisements. The value created from the Industrial Internet is much greater from the same amount of data, and has three components.

While Wikibon believes that Big Data is the new definitive source of competitive advantage across all vertical markets, the stakes are highest – financial and societal -- in those industry sectors that make up the Industrial Internet. The high value and consequences of harnessing this data, therefore, demand the development of specialized platforms, data models, and analytic capabilities to meet these unique, stringent, and critical requirements.

Big Data in Vertical Markets

Early adopters of Big Data technologies and methodologies in the pharmaceutical industry are mining vast troves of clinical, market, and legal data to identify which drugs under consideration for development have the best chance of navigating the grueling path from discovery to pharmacy shelves.

Retailers such as Walmart and Amazon are among the innovative Big Data early adopters, using data to provide value-add services that both benefit customers and improve the bottom-line. Today, retailers continue to blaze an innovative path in Big Data.

The commercial aviation industry is ripe for innovation in the form of Big Data analytics. Jet engines create significant data exhaust during the course of operations, data that can and should be analyzed to increase operational efficiencies and to facilitate preventive maintenance of faulty and soon-to-fail parts.

Advanced metering infrastructure (AMI), commonly known as smart meters, that “phone home” energy usage data every 15 minutes are increasingly replacing aging, traditional meters that require manual readings. The result is a deluge of machine-generated data that offers both opportunities and challenges for utilities companies.

A handful of innovative media companies – both “old school” news operations such as the Associated Press & the New York Times and new media companies including Prismatic and Zite – are leveraging Big Data to deliver timely, relevant content to users and derive new lines of revenue.

Big Data would for the first time provide an efficient and practical way for banks to make loan decisions based on highly personalized analytics at scale. The result would be a more accurate view of an applicant’s true credit risk and a resulting increase in loan activity, particularly to those applicants whose credit scores alone don’t tell the whole story.

Other Big Data Research

While a majority a Fortune 500 companies have Big Data deployments in production, and a significant percentage of mid-sized enterprises have proof-of-concept and pilot projects underway, Wikibon estimates that close to half have not realized the level of value anticipated at their onset.

The Hadoop distribution race is important because no technology can achieve mass adoption in the enterprise unless risk-averse IT departments believe it will stand-up under pressure, and Hadoop is no exception. Consider the apocryphal saying, “Nobody ever got fired for buying IBM,” but replace IBM with Hadoop. Sounds almost fanciful, no? But making that statement a truism is now the mission of every Hadoop distribution vendor.

Low-latency and hyperscale computing are and will continue to radically change the fundamental design of databases and systems. The shape of the IT industry is being radically altered by the use of Big Data and related technologies, including flash storage. CIOs must prepare themselves for this new paradigm, which gives IT shops a golden opportunity to add significant value to the enterprise.

CIOs that believe they can survive and thrive in the Era of Big Data by making marginal improvements to existing data management technologies and infrastructures are mistaken. Rather, the enterprises that come to dominate the 21st century will be those that invest in new technologies and processes that allow them to ingest, analyze, and act on streaming data from all manner of sources in real-time and develop closed-loop feedback systems to continually optimize business processes.

Big data is a topic of significant interest to users and vendors at the moment. Wikibon has completed significant research in this area to define big data, to differentiate big data projects from traditional data warehousing projects and to look at the technical requirements. In this paper Wikibon looks at the business case for big data projects and compares them with traditional data warehouse approaches.

• Cloud and Big Data Upending IT Stalwarts
• IBM Aims to Bring Watson to the Enterprise
• Hadoop Pure-Play Business Models Explained
• Qubole Tries to Up-level the Hadoop Conversation with Managed Cloud Service
• Attunity Adapts Change Data Capture for Big Data Analytics
• Treasure Data Looks to the Cloud to Unlock Big Data Value
• Can Pivotal Compete with AWS?
• Big Data Adds Complexity, Nuance to the Data Quality Equation
• GE Unveils Industrial-Grade Big Data Platform and Analytics Applications
• HP Unifies Big Data Assets in HAVEn Platform
• IBM Accelerates Its Big Data Portfolio
• Intel Validates Hadoop Market
• EMC Integrates Greenplum DB and Hadoop with Pivotal HD
• Data in DRAM is a Flash in the Pan
• Don’t Let Misconceptions Limit Your Hadoop Options
• Hardware Efficiency Important to Overall Hadoop ROI
• Accumulo: Why the World Needs Another NoSQL Database
• Google and VMware Provide Virtualization of Hadoop and Big Data
• Clearing Up Some Confusion on the Hadoop Wars
• Google Playing the Big Data Long Game
• Ask More of Your Data, More Often
• Google Playing the Big Data Long Game

Big Data Videos from theCUBE

Collection of live Big Data event coverage and on-location reports. Watch theCUBE live here.

theCUBE 2014

SiliconANGLE and Wikibon are bringing theCUBE for three days of live coverage and interviews at #BigDataSV 2014 in Santa Clara, February 11-13, 2014. Also see all videos from this show on SiliconANGLE's YouTube playlist. Full list of videos and articles from the event here.

SiliconANGLE and Wikibon are bringing theCUBE for one day of live coverage and interviews at MIT ECIR Cyber Security Workshop 2014, January 7th, 2014 in Cambridge, Mass. Also see all videos from this show on SiliconANGLE's YouTube playlist. Full list of videos and articles from the event here.

theCUBE 2013

SiliconANGLE and Wikibon are bringing theCUBE for three days of live coverage and interviews at HP Discover Barcelona, Spain, December 10th – 12th, 2013. Also see all videos from this show on SiliconANGLE's YouTube playlist. Full list of videos and articles from the event here.

SiliconANGLE and Wikibon are bringing theCUBE for three days of live coverage and interviews at Amazon AWS re:Invent in Las Vegas, November 12th – 14th, 2013. Also see all videos from this show on SiliconANGLE's YouTube playlist. Full list of videos and articles from the event here.

SiliconANGLE and Wikibon brought theCUBE for two days of live coverage and interviews to IBM Information on Demand in Las Vegas, November 4th – 5th, 2013. Also see all videos from this show on SiliconANGLE's YouTube playlist. Full list of videos and articles from the event here.

SiliconANGLE and Wikibon brought theCUBE for three days of live coverage and interviews at #BigDataNYC with coverage of #HadoopWorld in New York City, October 28-30, 2013. Also see all videos from this show on the SiliconANGLE Big Data NYC 2013 YouTube playlist. Full list of videos and articles from the event here.

SiliconANGLE and Wikibon is filming two days of live video at Oracle OpenWorld 2013 in San Francisco, September 23rd – 25th, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed two days of live video at Splunk .conf2013 in Las Vegas, October 1st – 2nd, 2013. Also see all videos from this show on SiliconANGLE's Splunk 2013 YouTube playlist.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon is filming two days of live video at the Tableau Customer Conference in Washington, D.C., September 9th and 10th, 2013. Also see videos on the playlist on SiliconAngle's YouTube channel.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon is filming two days of live video at the HP Vertica Big Data Conference in Boston, August 6th and 7th, 2013. Also see videos on the playlist on SiliconAngle's YouTube channel.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon is filming two days of live video at the MIT CDOIQ Symposium 2013 in Cambridge, July 17th and 18th, 2013. Also see videos from this show on SiliconAngle's YouTube playlist.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon will film two days of live coveragefilmed live video at SAPPHIRE 2013 in Orlando from May 1415, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed two days of live video at Hadoop Summit 2013 in San Jose, Calif. on June 26 and 27, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed one day of live video at MongoDB Days 2013 in New York City on June 21, 2013.
Full list of videos and articles from the event here.

SSiliconANGLE and Wikibon is filming three days of live video at HP Discover 2013 in Las Vegas, June 11th through 13th, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed one day of live video at GE's Industrial Internet event in San Francisco on June 18, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed one day of live video at Accel Partners Symposium 2013 at Stanford University on April 30, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed three days of live video at Strata Conference Santa Clara 2013 on February 26-28, 2013.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed video with numerous Big Data start-ups at the annual fundraiser for TUGG, or Technology Underwriting theGreater Good, in Boston on January 31, 2013.
Full list of videos and articles from the event here.

theCUBE 2012

SiliconANGLE and Wikibon filmed video at the opening of hack/reduce, a Big Data accelerator in Cambridge, Mass., on November 8, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed two days of live video at Strata Conference + Hadoop World 2012 on October 24 & 25, 2012 in New York City.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed two days of live video at IBM Information On Demand in Las Vegas, October 22 and 23, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed two days of live video at Splunk .conf 2012 in Las Vegas, September 11 and 12, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed three days of live video at Oracle OpenWorld 2012 in San Franciso, California, October 2-4, 2012.
Full list of videos and articles from the event here.

Original Content: SiliconANGLE and Wikibon filmed live video at Cassandra Summit in Santa Clara, Calif. on August 8, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed two days of live video at Hadoop Summit in San Jose, Calif. June 13 and 14, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed three days of live video at HP Discover in Las Vegas, June 5 through 7, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed video at the Cambridge, Mass. Headquarters of Atlas Venture on May 27, 2012. 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed live video at HBaseCon 2012 in San Francisco on May 22, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed three days of live video at SAPPHIRE 2012 in Orlando, Fla., May 14 through 16, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed three days of live video at EMC World and Data Science Summit in Las Vegas, May 21 through 23, 2012.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed three days of live video at O'Reilly Strata Conference: Making Data Work 2012 in Santa Clara, Calif., Feb. 28 through Mar. 1, 2012.
Full list of videos and articles from the event here.

theCUBE 2011

SiliconANGLE and Wikibon filmed two days of live video at Hadoop World 2011 in New York City, Nov. 8 & 9, 2011.
Full list of videos and articles from the event here.

SiliconAngle and Wikibon filmed 3 days of videos at EMC World in Las Vegas, May 9-11, 2011.
Full list of videos and articles from the event here.

SiliconANGLE and Wikibon filmed live video at Hadoop World 2010 in New York City, Oct. 12, 2010. 2011.
Full list of videos and articles from the event here.

Additional Coverage from theCUBE

• IBM Edge 2013
• EMC World 2013
• OpenStack Summit 2013
• Knowledge13
• VMworld 2012
• IBM Edge 2012
• Oracle OpenWorld 2011
• VMworld 2011
• HP Discover 2011
• SAP SAPPHIRE 2011

Big Data Peer Incites from The Wikibon Project

Live discussions between Big Data practitioners and the Wikibon community.

Where exactly does the value in Big Data lie? In a recent Wikibon Peer Incite call, Relay Technology Management’s Dr. Brigham Hyde zeroed in on the answer. Deriving value from Big Data, said Hyde, requires unifying unstructured content & data with structured data in a way that allows end-users to gain deeper insights than possible with analysis of structured data alone.

The Wikibon Community convened to consider whether the advent of Big Data with its focus on multiple kinds of data including unstructured data-types has turned the data warehouse into a dinosaur. The center piece of the discussion was a recent Wikibon study of the advantages and disadvantages of the Oracle Exadata IT appliance and similar offerings expected from IBM and HP, and a survey of early big data users.

Providing targeted advertisements on the web is a timely business. And by timely, that means milliseconds. That's how long Tapad has to help its clients analyze a web user's behavior profile, make a determination as to how much they are willing to bid to place an ad in front of that use, execute the auction and serve up the ad to the user. In this Peer Incite, Tapad discusses how it makes this real-time process hum using the Aerospike NoSQL database.

Big Data Infographics from The Wikibon Project

Infographics and visualizations focused on Big Data and analytics.

The reality is that Big Data is today – here and now – delivering on its many promises. This infographic chronicles actual use cases and broad deployments of today along with the growth areas such as the Industrial Internet that will continue to gain traction over the next couple of years. There is also a visualization of Wikibon's exclusive market forecast including a break-down by product and solution categories.

Consumers generate over 3/4 of all data and therefore are a major contributor to the information explosion. This infographic compares the data footprint created through digital usage by Boomer, Gen-X and Millennial generations. The accompanying blog post discusses the underlying infrastructure architectural changes that are required to cope and take advantage of this massive volume of data.

Big Data can be a beast. Data volumes are growing exponentially. The types of data being created are likewise proliferating. And the speed at which data is being created – and the need to analyze it in near real-time to derive value from it – is increasing with each passing hour. But it is possible to tame the Big Data Beast, as evident in this infographic from The Wikibon Project. It includes real-world use cases and practitioner success stories.

While the concept of data science has been around for decades, the notion of a data scientist has become an in-demand career leading to a rise of a new generation of data scientists. Social media platforms such as Facebook depend on data science to create innovative, interactive features that encourage users to get interested and stay that way. Increasingly, more traditional firms in a variety of industries are realizing the need to find, hire and retain the new rock stars of the business world. Here is a visual walk-through of this concept and the role of the data scientist.

What’s critical to realize is that 35% more digital information is created today than the capacity exists to store it; and this number will jump to over 60% over the next several years. Most of these gigabytes of data will pass through the servers, network, or routers of an enterprise, which becomes responsible at that moment for managing that content, protecting user privacy, watching over account information, and protecting copyright.

Additional Big Data Links and Resources

• Big Data News from SiliconANGLE
• Video from theCUBE and SiliconANGLE on YouTube
• The Top 20 Influencers in Big Data
• Wikibon's Complete Archive of Big Data Professional Alerts