Metadata is the data about your data, and management of it can be critical, particularly in conjunction with reporting systems and data marts. “When you look at metadata, it may tell you that, ‘This is the account balance as of last night at 5 p.m.,’” notes Bill Fox, a senior consultant with TreeTop Technologies. “Data lineage metadata tells me that it is a combination of three different fields from three different systems, one of which has another upstream system. So I know that it’s not just the account balance, but the balance upon which the sales rep gets his commission, which is modified or adjusted according to other business rules, and so on.”
But while understanding the context of existing data may be crucial to the business goals of a company, too few organizations pay attention to data lineage metadata, warns Fox. Yet data lineage provides execs and managers the crucial information to assess the quality of reported business data. Equally important, it helps application developers make better decisions.
Too often, metadata doesn’t tell companies enough, because different departments are using different tools and techniques to enter and report the metadata, and few are investing sufficient time to reference it in ways that it can be leveraged. The reason many companies fail to manage data lineage metadata is that they don’t have strong documentation rules and don’t handle data lineage metadata in a consistent fashion. It’s not easy.
Fox advises that companies stay ahead of the problem before they have a mountainous, unmanageable pile of disparate metadata. Tools to deal with data lineage—some of which use a wiki approach and some of which use automated code analysis—are pricey, so they cannot go on everyone’s desktop. But key people should have these tools, and policies should be established for how it is organized and published so downstream users can use it to the organization’s advantage.
Action Item:
Footnotes: Reprinted with permission © 2008 TreeTop Technologies
From "View from the TreeTop" Volume 2 Issue 2 February 2008