For several years people have been using the terms metadata Registry and Repository inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.

First lets take the definition of a Repository. Webster defines a repository as …a place, room, or container where something is deposited or stored.. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.

Collecting, administering and leveraging meta data presents challenges and risks that must be surfaced and managed to avoid unpleasant surprises in the areas of data warehousing, data administration and the system development life cycle at large. Absent careful planning, the surprises can overwhelm the benefits of any meta data initiative. Without accurate, current, high-quality meta data, development teams in both data warehousing and transactional systems are on a slope of diminishing returns, working harder and harder to maintain many-to-many interfaces. Meta data affects system analysis, version and change control, system interoperability, intersystem visibility and transparency, and related factors at an enterprise level.

The number one risk to meta data projects is that the team will end up with documentation, not actionable insight into diverse IT systems interoperations. Of course, system documentation is generally a useful and, at times, an essential IT artifact. However, it is subject to a number of well-known shortcomings. When produced in the form of electronic documents, it is an idle wheel. Documentation does not automate or move any part of the development or maintenance process. Documents are often obsolete the very day they are published. A document does not know whether it is obsolete or not; and a labor-intensive, manual effort is required to keep documents current. Inaccurate (outdated) information is often worse than no information at all because it is misleading. In contrast, a data modeling tool from which database data definition language (DDL) can be produced or which can be imported into an ETL (extract, transform and load) or query-and-reporting tool is a mechanism that enables meta data-driven design or maintenance. Because the meta data interchange between tools is incomplete, some manual labor will be required to manage the risk of a meta data idle wheel. Teamwork and discipline remain essential to managing this situation.

A few weeks ago, I was asked to help oversee a client's business intelligence project that was running behind schedule. By the time I joined the project, the requirements and design had been completed, and the delivery team had finished most of its development work. The ETL process was working properly, and several reports had been developed. However, the project had been stalled for several weeks, having made little or no progress. The reason for the delay was that each of the reports that had been developed needed to be "certified."

Approximately 15 reports had been requested for the first release of the system. The requirements team had been told that most of these would replace reports that users were currently receiving. The team was provided with a set of the existing reports, which were mainly Microsoft Excel spreadsheets. The reports looked simple enough. Each column had a heading that identified the data it contained, and the team was told where the data was located. An estimate was made for how long the reports would take to develop. So what was the problem? In a word, metadata.

Over the past twenty years, enterprises have created many diverse systems to manage their information and data. Individual systems combine a myriad of hardware configurations, operating systems, databases, and applications. Often, individual enterprises have found themselves with several disparate information systems among their divisions and departments, especially after mergers or acquisitions have broadened the scope and depth of the enterprise.

As the world, not to mention the enterprise, networks more completely, the enterprise needs to integrate its diverse systems to operate and analyze its resources more effectively. Numerous external sources, from partner information resources to real-time data feeds, have become available. The enterprise needs to marshal and integrate these disparate systems. At the heart of the systems integration challenge lies an information integration challenge.

Model-driven integration differs from the programmed integration. Programmed integration relies upon hard-coding a finite, and inextensible, solution to a particular challenge. Model-driven integration focuses on abstracting the information content into a model that describes the enterprise’s information resources. This model captures the nature of the information the enterprise has within its systems and the way the enterprise uses data in its daily operations.

How are you addressing the single most difficult problem facing data warehouses today? Data Quality. When the quality of data is compromised, incorrect interpretation and use of information from your data warehouse can destroy the confidence level of its customers, YOUR users. Once the user's confidence in your warehouse is eroded it is a question of time before your system will no longer exist.
This data quality quandary often results from system architectures that fail to identify "bad" data before it is loaded into the data warehouse. This missed opportunity leads to a dramatic increase in the time and costs that companies expend to reconcile and audit information in the warehouse. Insertion of technical meta data "tags" directly into the data warehouse's dimensional data model design and the extraction, transformation and loading (ETL) processes corrects this situation by providing a practical means to measure data quality precisely at a table row level of granularity.

This article is the first portion of a two-part series on implementing data quality through meta data. This installment examines the role meta data can have in the data warehouse model and data acquisition designs for information content and quality. Part two of the series will examine the beneficial technical meta data tags that can be incorporated into an architecture to measure data quality and provide flexibility to the system design.