For several years people have been using the terms metadata Registry and Repository inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.

First lets take the definition of a Repository. Webster defines a repository as …a place, room, or container where something is deposited or stored.. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.

Collecting, administering and leveraging meta data presents challenges and risks that must be surfaced and managed to avoid unpleasant surprises in the areas of data warehousing, data administration and the system development life cycle at large. Absent careful planning, the surprises can overwhelm the benefits of any meta data initiative. Without accurate, current, high-quality meta data, development teams in both data warehousing and transactional systems are on a slope of diminishing returns, working harder and harder to maintain many-to-many interfaces. Meta data affects system analysis, version and change control, system interoperability, intersystem visibility and transparency, and related factors at an enterprise level.

The number one risk to meta data projects is that the team will end up with documentation, not actionable insight into diverse IT systems interoperations. Of course, system documentation is generally a useful and, at times, an essential IT artifact. However, it is subject to a number of well-known shortcomings. When produced in the form of electronic documents, it is an idle wheel. Documentation does not automate or move any part of the development or maintenance process. Documents are often obsolete the very day they are published. A document does not know whether it is obsolete or not; and a labor-intensive, manual effort is required to keep documents current. Inaccurate (outdated) information is often worse than no information at all because it is misleading. In contrast, a data modeling tool from which database data definition language (DDL) can be produced or which can be imported into an ETL (extract, transform and load) or query-and-reporting tool is a mechanism that enables meta data-driven design or maintenance. Because the meta data interchange between tools is incomplete, some manual labor will be required to manage the risk of a meta data idle wheel. Teamwork and discipline remain essential to managing this situation.

Over the past twenty years, enterprises have created many diverse systems to manage their information and data. Individual systems combine a myriad of hardware configurations, operating systems, databases, and applications. Often, individual enterprises have found themselves with several disparate information systems among their divisions and departments, especially after mergers or acquisitions have broadened the scope and depth of the enterprise.

As the world, not to mention the enterprise, networks more completely, the enterprise needs to integrate its diverse systems to operate and analyze its resources more effectively. Numerous external sources, from partner information resources to real-time data feeds, have become available. The enterprise needs to marshal and integrate these disparate systems. At the heart of the systems integration challenge lies an information integration challenge.

Model-driven integration differs from the programmed integration. Programmed integration relies upon hard-coding a finite, and inextensible, solution to a particular challenge. Model-driven integration focuses on abstracting the information content into a model that describes the enterprise’s information resources. This model captures the nature of the information the enterprise has within its systems and the way the enterprise uses data in its daily operations.

How are you addressing the single most difficult problem facing data warehouses today? Data Quality. When the quality of data is compromised, incorrect interpretation and use of information from your data warehouse can destroy the confidence level of its customers, YOUR users. Once the user's confidence in your warehouse is eroded it is a question of time before your system will no longer exist.
This data quality quandary often results from system architectures that fail to identify "bad" data before it is loaded into the data warehouse. This missed opportunity leads to a dramatic increase in the time and costs that companies expend to reconcile and audit information in the warehouse. Insertion of technical meta data "tags" directly into the data warehouse's dimensional data model design and the extraction, transformation and loading (ETL) processes corrects this situation by providing a practical means to measure data quality precisely at a table row level of granularity.

This article is the first portion of a two-part series on implementing data quality through meta data. This installment examines the role meta data can have in the data warehouse model and data acquisition designs for information content and quality. Part two of the series will examine the beneficial technical meta data tags that can be incorporated into an architecture to measure data quality and provide flexibility to the system design.

In Part 1 of this article, we explored the concepts of detail and diversity in meta data. We noted that meta data projects tend to cluster into three distinct groups based on their mix of meta data detail and diversity. These three use cases can be described as "control," "inform" and "plan."

The "control" use case typically involves an individual IT technical service provider group (e.g., DBA, security, network, systems management) working within its domain. Existing system management tools are leveraged as collectors of meta data for better control, reporting and analysis of the environment being managed. Data is captured at the most detail level possible; however, including only that meta data directly controlled by the technical domain (i.e., low diversity). Virtually every technical service department in your organization has a homegrown database to track and manage IT assets, yet none of them are integrated or even known outside their department. These efforts are typically run as system management improvement projects and not necessarily recognized as meta data projects at all. Unfortunately, these projects also represent the source system or the system of record for all other meta data related efforts.

They say what goes around comes around and this certainly applies to the meta data information repository. Over the last 25 years, each great evolution of information systems technology and best practices has bred the same set of issues and the invariable vendor response - the meta data repository (or some derivation). What is this problem that keeps recurring? Simply put, as newer technologies are used to develop systems, the increase in complexity breaks the system management status quo. The answer always involves storing information about the system components in a central place for analysis - i.e., the meta data repository. It hasn't always been called this, but nonetheless, the basic approach to solving IT management complexity is the same. We are now witnessing this event playing out once more, but I'm getting ahead of the story.

This article seeks to examine the phenomena of meta data repositories, identifying common patterns of process and technology. Despite what appears to be great similarities, differing IT constituencies and problem domains suggest that a one-size-fits-all solution approach will not work. Finally, in part 2, I will define a strategy for successfully implementing an IT knowledge management program based on meta data repository concepts.

Let’s take a quick meta data management quiz. What is the most common form of meta data architecture? It is likely that most of you will answer, “centralized”; but the real answer is “bad architecture”. Most meta data repository architectures are built the same way data warehouse architectures were built: badly. The data warehouse architecture issue resulted in many global 2000 companies rebuilding their data warehousing applications, sometimes from the ground up. Many of the meta data repositories being built or already in use need to be completely rebuilt.

Meta-data in most simple terms means "structured data about data" or "data that describes something (that may or may not itself be data)". Examples of meta data include: definition of the data element, business names of the element, systems abbreviations for that element, the data type and size of the element, source location, data steward, alternate alias, alternate spelling etc. In other terms, meta-data is any information used to aid identification, description, characteristics, location of, access to, data elements and information.