Traditional Enterprise Data Flow
The traditional model of value extraction from enterprise data has been a sequential process. Operational systems update operational databases. Batch processes extract data multiple operational databases into data warehouse(s) and adds a historic timeline. Then ETL routines load data marts into the appropriate BI tools.
This sequential process was forced by the limited number of database calls that could be executed, because of the IO and bandwidth constraints of disk-based IO technology.
Big Data Potential
The potential value of Big Data Comes from:
- Extending the ability to derive value from all the data in the organization:
- data originating from operational systems;
- data from the analytic systems and from personal productivity systems;
- the “exhaust” data from systems such as HR and call centers;
- external data from social media and many other sources;
- Keeping all the original data without transformation, enabling:
- Direct access to the data without having to ingest it into an intermediate database
- Combining operational data and other data sources in a common big data repository
- The ability to create real-time analytic systems that will feed back directly and in real-time into operational systems
The current set of traditional data warehouses deal with Hadoop as just another data source to be ingested via connectors to Hadoop databases.
The true Big Data potential is realized when data-led applications can be developed that run operational and analytical from the same set of data.
The Hadapt approach
In a recent announcement, Hadapt provides a framework which integrates Hadoop structured and unstructured data, BI tools and SQL without the requirement to ingest the Hadoop data. In theory this construct allows both operational and analytical systems to be operating at the same time on the same data sources.
The main business benefit from this approach is improved speed, efficiency and effectiveness in analyzing big data. The process is simplified with fewer steps, the original data is available to the BI analyst, and time to analysis results is improved.
For large-scale enterprise systems other asset management data services would require integration, such as metadata services, master data management and data governance (ideally in real-time). In memory techniques and high-speed high-bandwidth IO would be required ensure performance in many environments.
The Hadapt approach is a significant improvement over the current approach of connectors ingesting data into current data warehouse and BI solutions from Oracle, Teradata and many other vendors. The Hadapt demonstration and integration with Tableau is neat and illustrative of the power of integrating text and sentiment analysis on unstructured data linked with structured data.
This is a first step however, and will need the ability to link or provide many other data services to provide an integrated software-defined infrastructure for big data.
Action Item: CIOs, CTOs and Business analysts should be aware of the Hadapt approach, and include it in investigations for BI solutions. More time and development will be required before Hadapt can be considered a strategic framework for operational and analytic systems integration.