Premise

The effective management use of Big Data is a competitive and survival imperative for all mid-size and large organizations over the next decade.

Big Data Stage 1: Internal Data Ontology

• IT sells and funds a Big Data vision to the board and senior line of business managers, and updates and resells constantly;
• Big Data is not Hadoop, or indeed any software package, set of software packages or piece of hardware;
• Big Data includes structured data, unstructured data, images and video: particular attention should be paid to creating video metadata;
• Big Data becomes more valuable when combined with more Big Data;
• Organizations should start with the Big Data they already create and collect in stage 1;
• Big Data stage 1 is creating a Big Data organization and data ontology;
• The key Big Data resource is data analysts;
• Good Big Data analysts are a scarce and valuable commodity and should work together;
• Four major skills are required for a Big Data analyst team:
  1. Advanced statistical & numerical skills;
  2. Deep knowledge of the industry & part of the enterprise they work for;
  3. Large scale data analysis skills;
  4. Deep knowledge of industry and government data sources, their reliability, extractability and cost;
• Data analysts should be few, well equipped, and paid well for success;
• The key output metric for data analyst teams is actioned insights;
• Every Big Data insight should result in:
  • Changes(s)in business process;
  • Business processes monitored by real-time capture of and use of the data found in the original data analysis;
• The big data ontology will extend into the production systems as the systems are modified to help create, index and manage big data metadata in real time.

Big Data Stage 2: Adding External Data

• Stage 2 of Big Data is extending the ontology to external data sources, and linking updates to existing and new applications to enhance the ontology;
• Moving and ingesting large amounts of Big Data should be avoided like the plague:
  • The cost of moving Big Data is too high and it takes too long;
  • Process remote Big Data remotely and combine extracts remotely or locally;
• Learn how to evaluate and use external sources, government sources, and external data providers;
• As much as possible of external Big Data processing should be outsourced.

Big Data Stage 3: Participating in Data Markets

• Stage 3 of Big Data is creating and selling big data constructs on an ongoing basis. The constructs become ever more complex and valuable, each version building on the last;
• The major revenue and profit from big data will be made by data providers who sell access to (via APIs) and extracts of their data;
• Very large (and potentially profitable) data providers will be companies such as Google, Microsoft, Yahoo, telecommunication companies and Government institutions;
• The value of a specific Big Data models and construct will decline over time as new constructs and connections are created;
• All organizations should be data providers and data consumers, will buy and sell access to data as the paradigm changes from data marts to data markets;
• Creation of Big Data will need to be built-in to applications, products and process design: particular attention should be paid to building in mechanisms to return information about deployment and value of products and services;
• New system designs using flash and other persistent storage technologies will drive a top-down process flow where production application and big data applications run in parallel against the same data input streams, and where the analytics feed back into production applications in real-time.

The winners in big data will be organizations that combine data from many sources, have the skills and DNA to use the results to improve their own endeavors and learn how to trade in the big data market.

Action Item: Creating and implementing a Big Data strategy is a vital project that should be strongly coordinated by IT. Big Data works best when connected to other big data, both inside and outside of the organization.

Footnotes: