For several years people have been using the terms metadata Registry and Repository inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.

First lets take the definition of a Repository. Webster defines a repository as …a place, room, or container where something is deposited or stored.. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.

Collecting, administering and leveraging meta data presents challenges and risks that must be surfaced and managed to avoid unpleasant surprises in the areas of data warehousing, data administration and the system development life cycle at large. Absent careful planning, the surprises can overwhelm the benefits of any meta data initiative. Without accurate, current, high-quality meta data, development teams in both data warehousing and transactional systems are on a slope of diminishing returns, working harder and harder to maintain many-to-many interfaces. Meta data affects system analysis, version and change control, system interoperability, intersystem visibility and transparency, and related factors at an enterprise level.

The number one risk to meta data projects is that the team will end up with documentation, not actionable insight into diverse IT systems interoperations. Of course, system documentation is generally a useful and, at times, an essential IT artifact. However, it is subject to a number of well-known shortcomings. When produced in the form of electronic documents, it is an idle wheel. Documentation does not automate or move any part of the development or maintenance process. Documents are often obsolete the very day they are published. A document does not know whether it is obsolete or not; and a labor-intensive, manual effort is required to keep documents current. Inaccurate (outdated) information is often worse than no information at all because it is misleading. In contrast, a data modeling tool from which database data definition language (DDL) can be produced or which can be imported into an ETL (extract, transform and load) or query-and-reporting tool is a mechanism that enables meta data-driven design or maintenance. Because the meta data interchange between tools is incomplete, some manual labor will be required to manage the risk of a meta data idle wheel. Teamwork and discipline remain essential to managing this situation.

A few weeks ago, I was asked to help oversee a client's business intelligence project that was running behind schedule. By the time I joined the project, the requirements and design had been completed, and the delivery team had finished most of its development work. The ETL process was working properly, and several reports had been developed. However, the project had been stalled for several weeks, having made little or no progress. The reason for the delay was that each of the reports that had been developed needed to be "certified."

Approximately 15 reports had been requested for the first release of the system. The requirements team had been told that most of these would replace reports that users were currently receiving. The team was provided with a set of the existing reports, which were mainly Microsoft Excel spreadsheets. The reports looked simple enough. Each column had a heading that identified the data it contained, and the team was told where the data was located. An estimate was made for how long the reports would take to develop. So what was the problem? In a word, metadata.

You've probably heard the word "metadata," but what exactly is this mysterious information? Equally important how should it be handled? Failure to adequately capture and preserve metadata associated with electronic discovery materials has been considered spoliation of evidence and grounds for significant discovery sanctions. Conversely, a failure to erase metadata from outgoing law firm correspondence can inadvertently communicate a wealth of inappropriate information to recipients. This article reveals how metadata is created, its purpose, and situations where metadata can assume significant importance.

What is metadata?

People normally define an electronic document, spreadsheet, e-mail message or other digital data as the content they can see, the information intentionally added by the document's author. However, almost all computer programs automatically add additional information to a document, usually storing it at the beginning or end of the electronic file where the program can easily find it. This information is generically known as "metadata."

When a file is attached to an e-mail message or copied from one location to another, much of its associated metadata is also transferred with the file. Some metadata, however, is automatically updated to reflect the new location or other actions involving the file. As a result, computer files that appear identical when viewed or printed may have different metadata.

The world of information technology has "grown-up" dramatically in the last fifteen years -- the term of my comparably short career. From the days of punching cards and feeding deck readers at midnight at the university computer lab to the world of dot-coms, electronic business, and business intelligence, one might believe that they have seen it all.

But not even close … One can only imagine what the next fifteen years have in store for us. Post-Y2k and for the foreseeable future, the need and speed to manage data, information, and knowledge will (if it has not already) become THE business driver.

Data within your database is just that, data. It becomes information only when you can effectively extract and distribute in an understandable form. Using meta-data has come a long way; it allows for free-flowing information and is putting the ability to extract information into the hands of end-users.

When I first started my career in data processing, I began as a COBOL programmer. Back then, meta-data was strictly “data about data” and, in those days, we had to rely on s FDs, copy books, and a scarce set of documentation that comprised our meta-data. Such resources told us where data was located and what particular objects (tables) and columns meant.

Today, meta-data has grown into a complete subject area encompassing such terms as Knowledge Management, Corporate Data Dictionaries, and Enterprise Meta-Data Repositories. All of these classifications have the single goal to categorize the underlying information buried within databases so that end users can get to information faster. They do this by employing tools that allow end users to unlock the mysteries of what information is available and where it is located.

Imagine that you are cornered in the break room by the CEO or called into a sales meeting and asked to produce a report that describes demographic information for current customers. This task seems to be quick and easy, at least at face value. But, you quickly remember that you have distributed databases that control different product lines, and the way objects have been defined within those databases are quite different. To compound the issue, your CEO is starting to describe “customers” as anyone who has purchased products or services from you including potential customers that are not stored in a database yet but are in multiple spreadsheets maintained by your sales reps. No doubt, it is becoming obvious that these requirements will be changing and you begin to feel like you will be trying to hit a moving target.