Metadata Management and the Enterprise Data Catalog

Executive summary

Every organization already has metadata. It lives in database schemas, pipeline code, spreadsheet headers, wiki pages and the memories of long-serving analysts. What most organizations lack is managed metadata: context that is captured systematically, curated by accountable people, connected across systems and used in daily work. The enterprise data catalog is the platform through which that management becomes visible, but the catalog is an outcome of the capability, not a substitute for it.

This paper distinguishes four categories of metadata (business, technical, operational and social) and shows why each matters for findability, understanding, trust, impact analysis, automated governance and regulatory evidence, such as the records of processing required by the General Data Protection Regulation (GDPR). It compares three architecture options, describes the catalog's capabilities as a connected set rather than a feature list, shows how active metadata turns the catalog into a point of control, and identifies the open standards that keep metadata portable between tools.

• Metadata management is a capability with owners, standards, workflows and metrics; the catalog is its primary platform.
• Automated harvesting provides breadth; human curation provides meaning. Neither is sufficient alone.
• Active metadata turns documentation into control: the same metadata that describes sensitivity can enforce masking, route approvals and prioritize pipelines.
• Adoption follows usefulness. A catalog that answers the questions people ask, inside the tools they already use, becomes the default starting point for data work.

The implementation guidance favors scoping by domain and critical data elements, a minimum metadata standard at onboarding, curation workflows with named stewards, and metadata quality measures such as description completeness, owner assignment and freshness. Governance of metadata itself, including ownership of definitions and change control, is treated as a first-class concern. The paper ends with measures of value, the pitfalls that turn catalogs into graveyards, and a maturity path from documented to active metadata.

Why metadata matters

Metadata is the context that makes data usable: without it, a table is a set of columns rather than a statement about the business.

Metadata is commonly defined as data that describes other data. A more useful enterprise definition is the information needed to find, understand, trust, manage and use data assets. DAMA-DMBOK treats metadata management as a knowledge area in its own right and as a dependency of the others, because governance, quality, security, integration and architecture each depend on knowing what data exists, what it means, where it came from and who is accountable for it. Metadata management is the discipline of capturing, curating, integrating and delivering that context so that it stays accurate as systems change. The business case rests on six outcomes.

The regulatory point deserves emphasis. Article 30 of the GDPR requires controllers to maintain records of processing activities that cover, among other things, the purposes of processing, the categories of data subjects and personal data, the categories of recipients, transfers to third countries and, where possible, the envisaged time limits for erasure; processors keep a narrower record of the processing they carry out on behalf of each controller. An organization that knows which systems hold personal data, why, for whom and for how long already has a metadata capability, and the same holds for risk data aggregation in banking under the Basel Committee's principles (BCBS 239). Where metadata is managed, regulatory evidence is a report; where it is not, it is a project.

This paper covers metadata for structured and semi-structured enterprise data and the catalog that manages it. The business glossary and data classification are disciplines in their own right; here they appear as catalog capabilities and as inputs to active metadata, the paper's central theme: the shift from metadata that documents data to metadata that operates it.

Categories of metadata

Practitioners group metadata into categories so that producers, capture methods and responsibilities can be assigned sensibly. DAMA-DMBOK distinguishes business, technical and operational metadata; many practitioners add a fourth category, social or collaborative metadata, generated by the user community rather than by systems or stewards. A single catalog entry typically combines all four, but each has a different producer, rate of change and failure mode when neglected.

Business metadata describes what data means and how it may be used: term definitions, owning domain and steward, critical data element (CDE) designation, business rules and calculation logic, quality expectations, sensitivity classification, permitted uses and retention requirements. Its producers are stewards and subject matter experts, and it changes slowly but consequentially. Neglected, it yields three departments reporting three different customer counts, each technically correct.

Technical metadata describes structure and physical implementation: schemas, tables, columns, data types, constraints, keys, file formats, partitions, application programming interfaces (APIs), pipeline code, transformation logic and the lineage between them. Systems produce it and catalogs can harvest it automatically, which makes it the most complete category and, on its own, the least meaningful. Because it changes constantly, technical metadata maintained by hand is wrong within weeks.

Operational metadata describes what happens when systems run: job execution times and status, row counts, freshness, error and access logs, query frequency, storage volumes, cost and service-level attainment. It is the raw material for observability and for the usage statistics that separate important datasets from abandoned ones. It arrives continuously and in volume, and it is valuable mainly in aggregate or when linked to the asset it describes.

Social metadata, also called collaborative metadata, comes from the user community: ratings, endorsements, questions and answers, tags, bookmarks and the implicit signal of who uses what. It captures tacit knowledge that formal curation misses, such as the fact that a certified table is correct but a particular derived view is what analysts actually rely on. Its weaknesses are noise and decay, so the catalog must moderate it and present it alongside authoritative metadata.

The categories determine both architecture and operating model. Technical and operational metadata must be harvested, because no team can maintain them by hand at enterprise scale. Business metadata must be curated, because no harvester can infer intent, accountability or permitted use. Social metadata must be invited, because it appears only where people already work. Applying one method to all four either drowns stewards in schema documentation or leaves a harvested inventory without meaning.

Metadata architecture

Metadata architecture decides where metadata is stored, how it gets there and how people and tools reach it. Three patterns recur. A centralized repository holds all metadata in one store with one model, fed by connectors. A distributed pattern leaves metadata in the tools that create it, such as the warehouse information schema, the orchestrator's run history and the business intelligence (BI) tool's semantic model. A hybrid or federated pattern keeps detail near its source but synchronizes a curated subset into an enterprise catalog that acts as a catalog of catalogs, linking entries across platforms through shared identifiers and a shared metadata model.

Most large organizations end up federated whether they planned to or not, because platforms arrive with their own catalogs and no single repository can represent every tool's metadata at full fidelity. The design question is therefore what to federate. A workable rule is that the enterprise catalog holds the identity, ownership, classification, business definition, quality status and dataset-level lineage of every asset in scope, while platform catalogs hold fine-grained technical detail and link back through stable identifiers. The figure below shows the resulting layers.

Harvesting versus push

Metadata enters the catalog in two ways. Harvesting, or pull, means the catalog connects to a source on a schedule, reads its system tables, logs or APIs, and reconciles the result with what it already holds. It is simple to operate and covers legacy systems that cannot be modified, but it introduces latency, depends on connector quality, and misses transient objects and runtime behavior. Push means the source, pipeline or orchestrator emits metadata events as work happens. Push is near real time and captures facts that no scan can see, but it requires instrumentation and a standard event format, and it fails silently when a team forgets to emit.

A mature architecture uses both and reconciles them. Harvesting establishes the baseline inventory and catches what instrumentation misses; push keeps lineage and operational metadata current between harvests. Reconciliation rules must be explicit: which source is authoritative for which attribute, how conflicts are flagged, and how an asset that disappears from a harvest is retired rather than deleted. Without them, a catalog accumulates ghost entries and contradictory facts.

ISO/IEC 11179 as a conceptual anchor

ISO/IEC 11179, the multipart international standard for metadata registries published jointly by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), supplies a vocabulary that is useful even where no formal registry exists. Its core distinction is between a data element concept, which associates a property with an object class, such as the birth date of a person; a value domain, the set of permissible values and their representation, such as a date format or a code list; and a data element, which binds a data element concept to a value domain and is what appears as a column or field in a schema. The standard also treats registry entries as administered items, each with a registration status and responsible organizations, an arrangement that anticipates the stewardship workflows of catalog tools.

The practical lesson is to model meaning separately from representation. One concept, customer birth date, may have many data elements across systems with different formats; the catalog should link all of them to one glossary term rather than defining each column independently. This separation is what makes vertical lineage (from business term to physical column), cross-system quality rules and consistent classification possible.

Catalog capabilities

An enterprise data catalog is the operational face of metadata management: a connected set of capabilities rather than an inventory with features attached.

The capabilities below are described one at a time, but their value lies in their connections: a classification tag drives an access policy, a quality score appears next to the search result, and a lineage graph carries a glossary term from source column to report. When evaluating or building a catalog, test the connections rather than the checkboxes.

Harvesting and crawling

Automated harvesting builds and maintains the technical inventory: databases, schemas, tables, views and columns, files in object storage, streaming topics, BI reports and semantic models, machine learning features, and APIs. Capable harvesters record schema changes over time, sample values to infer types and detect sensitive patterns, read statistics such as row counts and last-modified times, and scan incrementally to spare source systems.

Business glossary linkage

The glossary holds approved business terms with definitions, synonyms, relationships, owners and status. Linkage associates physical assets, usually columns and datasets, with those terms, so that a search for gross written premium returns the tables and reports where the concept is implemented. Links can be proposed by name matching, lineage propagation and classification models, but a steward should confirm them for critical data elements.

Lineage display

Lineage shows how data moves and changes between assets. In the catalog it appears as a navigable graph, expandable from dataset to column level where that detail exists. The catalog rarely captures lineage itself; it consumes lineage from parsers, orchestrators and runtime events, ideally through an open event standard, and presents it beside ownership, quality and classification so that an impact analysis can be completed in one place.

Classification and sensitivity tags

Classification assigns each asset, and where needed each column, to a sensitivity level and to regulatory categories such as personal data, special category data, payment card data or material non-public information. Tags may be applied manually, inherited through lineage, or proposed by pattern-based and model-based detection, with steward confirmation for high-stakes categories. Classification is the most direct bridge to active metadata, because access, masking, retention and transfer rules can be expressed against tags rather than against individual tables.

Quality and certification signals

Consumers need to know whether a dataset can be relied on. The catalog therefore shows quality rule results, trends and open issues, together with a certification status assigned through a governed workflow, for example draft, under review, certified and deprecated. Certification needs explicit criteria (owner assigned, definition linked, quality thresholds met, lineage captured, access policy defined) and a review date, so that the badge remains a current signal.

Ownership and stewardship workflows

Every asset in scope needs an accountable data owner and an operational steward, and the catalog is where that assignment is recorded, requested and reviewed. Workflow capabilities include assigning owners at onboarding, routing definitions and classification changes for approval, escalating unanswered questions, notifying owners of schema changes to their assets, and periodic attestation, in which owners confirm that their metadata is still correct.

Access requests and entitlements

When consumers find a dataset they cannot yet use, the catalog should let them request access in place, show the policy that applies, route the request to the owner or an automated approver, and record the grant with its purpose and duration. This creates one record of who may see what and why. The platform's policy engine enforces the entitlement; the catalog orchestrates and records it.

Search, ranking and APIs

Effective search accepts business language as well as technical names, filters by domain, owner, classification, certification and freshness, and ranks results by certification, usage, recency, metadata completeness and endorsements. Ranking is a governance decision in technical clothing: promoting certified assets above popular but uncertified ones is how the catalog steers behavior. APIs expose the same metadata to pipelines, policy engines, notebooks and BI tools, so that the catalog can serve as a system of record for automation.

Collaboration

Collaboration features gather social metadata: ratings and endorsements, questions and answers attached to assets, usage statistics such as query counts and distinct users, and the ability to follow an asset and receive change notifications. They are inexpensive to provide and are often the most honest indicator of which assets matter, so usage should feed back into ranking, curation priority and cost control.

Active metadata

Active metadata is metadata that is continuously collected, analyzed and used to trigger actions in other systems, as distinct from passive metadata, which is collected and displayed for people to read. The term was popularized by the analyst firm Gartner, whose research describes active metadata management as continuous access to and analysis of metadata across the data estate. The distinction is one of use, not of type: the same classification tag is passive on a catalog page and active when a policy engine reads it to apply column masking. Active metadata is also the working content of the data fabric pattern, in which a metadata layer spanning distributed platforms drives integration, discovery and policy enforcement rather than merely documenting them. Four families of automation are common.

The loop has a prerequisite that is easy to underestimate: identity resolution across systems. A policy can act on a column only if the catalog knows that the column in the warehouse, the field in the source application and the measure in the BI tool are the same thing. The connect stage, highlighted in the figure above, in which assets are matched to stable identifiers and linked to glossary terms, is the hinge between collection and automation; skipping it produces automation that acts confidently on the wrong objects.

Designing for active metadata changes several catalog decisions. Metadata must be machine-readable and governed as data, with schemas, versioning, quality rules and service levels of its own. Latency matters: a classification that arrives a day after a table is created leaves a day of exposure, which argues for push-based capture of new assets. Actions need safeguards: dry-run modes, confidence thresholds, human confirmation for irreversible actions such as deletion, and an audit trail recording which metadata drove which action. Each automated rule needs an owner.

Metadata that no system reads is documentation; metadata that systems act on is control.

Open standards and interoperability

Metadata outlives the tools that hold it: catalog platforms are replaced, estates are consolidated after mergers, and lineage collected by one orchestrator must be readable by the next. Interoperability therefore belongs in the architecture from the start, as a metadata model that can be exported in full, identifiers that do not depend on one vendor, and collection formats that any tool can emit and consume. The open standards in the table below cover most needs.

Lineage shows why a run-time event standard matters. Parsing SQL and pipeline code reconstructs lineage but misses what happened when a job ran: which version executed, against which partitions, producing which outputs. With a standard such as OpenLineage, every orchestrator and engine emits the same event shape, and the catalog, the observability tool and the policy engine read one stream instead of maintaining separate parsers. Where provenance must be kept as evidence, for example for the data used to train a machine learning model, PROV offers a record that is independent of any pipeline tool. The FAIR principles, which ask that data be findable, accessible, interoperable and reusable, were written for scientific data but are a useful test for enterprise metadata as well.

Open-source catalogs are an established category; Apache Atlas, DataHub and OpenMetadata are examples. Project health varies (Amundsen, once a widely cited example, has been archived for inactivity), which is itself an argument for portability. Open-source projects also define exchange mechanisms: Egeria, a project of the Linux Foundation's LF AI & Data, provides an open metadata type system, APIs and event payloads through which tools can exchange metadata, and at least one major commercial catalog exposes APIs compatible with Apache Atlas. Open-source catalogs remove license cost but need a team to run, upgrade and extend them, while commercial catalogs package connectors, workflow and support in exchange for fees and some dependence on the vendor's model. The protective criteria are the same either way: a documented metadata model, bulk export, open APIs and support for the standards above.

Implementation approach

Catalog programs fail more often on scope than on technology. Inventorying every table produces an uncurated index that nobody trusts, and curation spread evenly across it is wasted. Scope should follow two filters: the domains with the strongest demand and sponsorship, and within them the critical data elements on which regulatory reporting, key decisions and customer processes depend. The CDE list is shared with the data quality program; the catalog is where it becomes visible and linked to each physical implementation.

A minimum metadata standard

Before the first source is onboarded, the program should publish a minimum metadata standard: the attributes every in-scope asset must carry before it counts as onboarded. A workable standard is short, because each attribute adds curation effort.

• A business-readable name and description
• The owning domain, with a named owner and steward
• The sensitivity classification
• The source system and refresh cadence
• Linked glossary terms for any critical data element
• The certification status

Everything else is optional until use justifies it. A standard that asks for twenty attributes on day one produces boilerplate, not metadata.

Onboarding sources

Onboarding is a repeatable procedure rather than a project per source; the first and last steps below are the ones most often skipped.

1. Register. Record the source with its owner, platform contact, environments and the connector or push mechanism.
2. Harvest and resolve. Harvest the technical inventory and resolve identities against existing entries, so that no table is registered twice.
3. Classify and propagate. Run automated classification, and propagate tags and glossary links through lineage where confidence is high.
4. Assign and complete. Assign the owner and steward, who complete the minimum standard for assets in scope and mark the rest as not curated.
5. Publish and schedule. Publish the entries, announce them to consuming teams, and schedule the first attestation.

Curation workflows

Curation turns the harvested inventory into meaning, and it must run as a workflow with a queue, priorities and service levels. Priority follows usage, CDE status and classification, so that a widely queried table holding personal data is curated before an empty staging schema. Definitions follow the propose, review, approve and publish cycle of the business glossary; schema changes create steward tasks instead of silent drift; and periodic attestation closes the loop, with owners confirming on a fixed cadence that their metadata remains correct.

Adoption and measurement

A catalog is adopted when it is the fastest route to an answer, and adoption has to be designed rather than requested. The first lever is search success: the share of searches that end in an asset opened, an access request raised or a question asked, rather than abandoned. Search fails when uncurated staging tables dominate results, when business terms return nothing for lack of glossary links, or when ranking buries certified assets. Reviewing failed searches each week and curating what people were looking for is the most direct adoption tactic available.

The second lever is embedding. Few people open a separate application to check a definition, but most will read a tooltip in the BI tool, a column description in the SQL editor or a certification badge in a notebook. The catalog should expose its metadata through APIs and plug-ins to the tools where data work happens, and accept ratings and questions from them in return. Mandates produce compliance visits, not habits.

Metadata quality metrics

Metadata is data, and its quality is measured with the same dimensions the data quality program applies elsewhere. The metrics below are reported per domain and per owner, so that every gap has an address. Complete coverage is expected for CDEs and certified assets, while the long tail is measured but not forced. Completeness is easy to game with boilerplate, so a sample of descriptions should be read each cycle, and machine-generated descriptions should be marked as such until a steward confirms them.

Metrics of value

Quality metrics say whether the catalog is fit to be useful, not whether it is used. Value metrics answer that question and belong in the same report, so that an immaculate but unused catalog is visible as a problem. Most come from catalog logs, access request systems and ticket queues; time to find needs a short recurring survey question.

Governing metadata

Metadata needs governance of its own, because it records decisions that people will dispute: what a term means, who owns a dataset, how sensitive a column is and whether an asset deserves certification. The enterprise data governance framework supplies the bodies and escalation paths. Within it, three questions recur: who owns a definition, how a change is controlled and how a conflict is resolved.

Ownership of definitions

Ownership splits by metadata category. A business definition belongs to a term owner, a business leader accountable for its meaning across the enterprise, with a glossary steward curating the entry. Asset-level metadata such as ownership, classification and certification belongs to the data owner of the asset. Technical and operational metadata belongs to the platform team that runs the harvesters and event pipelines. The classification scheme itself is owned by the classification program together with security and privacy; the catalog applies the scheme rather than defining it.

Change control and conflict resolution

Definitions, classifications and ownership assignments are versioned with effective dates, so that an earlier report can be read against the definition that applied when it was produced. Changing a CDE definition or raising a classification is a change with impact: lineage identifies the affected assets, consumers are notified, the change takes effect after a notice period, retired items are deprecated rather than deleted, and dependent policies are re-evaluated.

Conflicts over meaning are normal and should be settled at the lowest competent level: first the stewards of the domains involved, then the data owners, and only enterprise-wide disputes reach the data governance council, which records its decision as precedent. Many apparent conflicts dissolve when the glossary holds two explicitly related terms, each with its own context and owner, instead of one ambiguous definition. Where real divergence remains, the council decides which definition governs enterprise reporting.

Roles that carry metadata governance

Five roles carry this governance; titles vary, but responsibilities should not.

Metadata is a record of decisions; governing metadata means deciding who may make them and how they may change.

Metadata also needs security and lifecycle rules of its own. Sample values, the lineage of restricted datasets and access request histories can all be sensitive, so catalog entries need their own access controls, and metadata history needs a retention rule like any other record.

Pitfalls that turn catalogs into graveyards

The failure modes of catalog programs are consistent across industries, and most are visible within the first year to anyone who looks at usage rather than at the number of assets registered. Each has a structural remedy described earlier in this paper.

Maturity path and roadmap

Maturity is best assessed per domain, because one domain is often active while another is undocumented. Each stage in the figure below produces the trust and plumbing the next one needs; skipping a stage is the usual reason automation acts on wrong or missing metadata.

An indicative roadmap follows the stages. The first two quarters establish the foundation for one or two domains: the minimum standard, named owners and stewards, glossary linkage, harvesting of the priority platforms and a first attestation. The following four quarters expand by domain, add run-time lineage from priority pipelines, connect classification to the policy engine and put the catalog inside the BI and query tools. Active metadata begins with one closed loop and widens as each loop proves reliable. The plan belongs inside the enterprise data roadmap rather than beside it, so that catalog milestones are funded and sequenced with the programs that depend on them.

Conclusion

Metadata management is the discipline that makes every other data discipline workable at scale, and the enterprise data catalog is where it becomes visible and measurable. The path from documentation to active metadata runs through unglamorous work: a short standard, named owners, curation run as a workflow, identity resolution, and open standards that keep metadata portable. Six first steps put that work in motion.

1. Scope deliberately. Choose one or two domains with strong demand and sponsorship, and agree the critical data elements within them.
2. Publish the minimum standard. Fix the short list of attributes every in-scope asset must carry before the first source is onboarded.
3. Name the people. Assign owners and stewards, and run curation as a workflow with a queue, priorities and service levels.
4. Connect before automating. Harvest the priority platforms, resolve identities and link critical data elements to glossary terms.
5. Close one loop. Pick a single active metadata loop, such as masking newly detected personal data, and make it work end to end.
6. Measure use as well as quality. Embed the catalog in BI and query tools, and report metadata quality and value together.

Organizations that do this work gain a catalog people start from, metadata that machines can act on, and regulatory evidence that is produced rather than assembled; those that skip it acquire one more repository to maintain.

Glossary

Active metadata

Metadata that is continuously collected and analyzed and that triggers actions in other systems, such as policy enforcement or pipeline orchestration.

Attestation

A periodic confirmation by a data owner that the ownership, classification and description of their assets are still correct.

Business glossary

A governed collection of business terms with approved definitions, owners, relationships and links to physical data.

Catalog of catalogs

A federated enterprise catalog that holds a curated subset of metadata from platform-native catalogs and links to them through shared identifiers.

Catalog product owner

The role accountable for the catalog as an internal product: its roadmap, minimum standard, integrations and adoption.

Certification

A governed status indicating that a dataset meets defined criteria for ownership, definition, quality, lineage and access policy.

Critical data element (CDE)

A data element whose quality or misinterpretation would materially affect a decision, an obligation or a customer outcome.

Curation

The human work of adding meaning, ownership, classification and context to harvested metadata.

Data element

In ISO/IEC 11179, the binding of a data element concept to a value domain; what appears as a column or field in a schema.

Data element concept

In ISO/IEC 11179, a property of an object class, such as the birth date of a person, independent of how it is represented.

Data steward

A person accountable for curating and maintaining the business metadata of a domain and for first-line resolution of conflicts over meaning.

Harvesting

Automated, scheduled collection of metadata from source systems by a catalog connector.

Identity resolution

Matching the records of the same asset across systems to one stable identifier, so that metadata and policies apply to the right object.

Lineage

A description of how data moves and is transformed between systems, datasets and columns, from origin to consumption.

Metadata

Information needed to find, understand, trust, manage and use data assets; commonly grouped as business, technical, operational and social metadata.

Minimum metadata standard

The set of attributes every asset in scope must carry before it is considered onboarded to the catalog.

Push

Metadata capture in which sources, pipelines or orchestrators emit metadata events as work happens, rather than waiting for a scheduled harvest.

Value domain

In ISO/IEC 11179, the set of permissible values for a data element and their representation, such as a format or a code list.