Growing data volumes have a direct impact on data integration performance. Simply put, traditional extract, transform, and load (ETL) tools developed in the 1990’s were designed to integrate relatively small volumes of data from source systems like CRM and ERP applications into data warehouses on a daily, sometimes weekly basis.
Since then, data volumes have grown exponentially, and multiple terabyte data warehouses are common. This inevitably leads to data integration performance constraints, as integration jobs have likewise grown in complexity, size, and frequency. Syncsort, a Woodcliff Lake, N.J.-based firm best known for its data protection software, claims its DMExpress platform overcomes data integration performance bottlenecks by looking up values within reference sources more efficiently and allowing users to automatically “push” lookup operations into the database.
Syncsort aims to compliment existing data integration software, like that from Informatica or IBM, rather than replace it. DMExpress includes a metadata exchange feature that allows users to export existing data integration jobs and “snap” them in to the Syncsort platform, according to the company.
The company claims the current version of its platform, DMExpress 6.5 (Figure 1), increases 5x data integration performance with 75% less CPU.
Source: Syncsort
Syncsort got its start in 1967, supporting data management jobs in mainframe environments. Currently, the company mostly sells its data integration platform directly to customers but is in the process of establishing a more robust channel pipeline, according to Syncsort CEO Flavio Santoni.
Action Item: Companies experiencing data integration performance issues due to increased data volumes should seek improved, new and/or complimentary technologies remedy bottlenecks. Those with legacy data integration deployments should ask incumbent vendors how they are helping customers overcome data integration challenges in the era of big data. If incumbent vendors do not have a viable approach or require a significant increase in fees to take advantage of big data integration capabilities, users should consider third-party vendors such as Syncsort to address integration bottlenecks.
Footnotes:
