Data Fabric

Executive summary

Enterprise data is permanently distributed across on-premises databases, several public clouds and a growing number of SaaS applications. Consolidating all of it into one platform is neither feasible nor desirable, yet every new source adds manual work to map, document, classify and secure it, and that work grows faster than the teams doing it. A data fabric addresses the problem by turning the metadata that systems, pipelines, catalogs and users already generate into an engine that does part of the work. It does not remove the work; it shifts a growing share of it from people to automation guided by metadata.

This paper argues that a fabric is assembled, not bought. It connects catalog, integration, quality, policy and orchestration capabilities, many of them already present, through a shared metadata graph, and it automates only what governance has explicitly authorized.

• A fabric has six logical layers, from sources and integration up to consumption, with the catalog and metadata graph as the foundation that every other layer reads from and writes to.
• Activation depends on four kinds of metadata (technical, business, operational and social) and on a continuous cycle of collecting, analyzing, recommending and automating.
• The boundary between recommending and automating is the most important governance decision in a fabric, and it should move only by recorded decision.
• Four use cases carry most of the business case: governed self-service access, integration acceleration, impact analysis and evidence, and cost and performance optimization.
• Fabric and mesh are complements: mesh decides who is accountable for data, and fabric capabilities can become the self-serve platform that makes that accountability affordable.

The later sections set out a metadata-first implementation path that expands by domain, a worked example of automating one policy end to end, the prerequisites and skills involved, the pitfalls that turn a fabric into one more tool, the measures that show it is working, a readiness checklist and four first steps.

An architecture, not a product

The term data fabric has been popularized by industry analysts, notably Gartner and Forrester, and vendors have since attached it to many products. Gartner describes a data fabric as a data management design that uses knowledge graphs, semantics and active metadata-based automation to make data integration pipelines flexible, reusable and augmented, regardless of deployment option, use case (operational or analytical) or architectural approach. It also stresses that a fabric is not one single tool or technology. This paper uses the term in that architectural sense.

A data fabric uses active metadata, semantics and automation to integrate, govern and deliver data wherever it lives.

The mechanism is a connected graph: metadata generated by systems, pipelines, catalogs and users is collected into it, analyzed continuously and used to recommend and automate integration, governance and delivery tasks. Each new source still has to be mapped, documented, classified and secured, but more of that work can be done from what the graph already knows about similar sources, owners and policies.

No single product delivers this. A fabric is assembled from catalog, integration, quality, policy and orchestration capabilities that share metadata, and most enterprises already own several of them. The architectural work lies in connecting them through the graph and deciding, as a matter of governance, what they may do automatically.

Definition and components

A working definition has to separate the architectural idea from the products marketed under its name, and then say what the architecture is made of. This section covers what a fabric is and is not, its six logical layers, the metadata graph that connects them and the integration styles it orchestrates.

What a data fabric is, and what it is not

A fabric is a set of capabilities, some already present in most enterprises, connected through a shared metadata graph so that integration, governance and delivery can be partly automated. Because its premise is that data will remain distributed, it is multi-environment by design, and it grows incrementally rather than arriving in one release.

The misreadings matter because each leads to a predictable disappointment. Many products deliver important parts of a fabric, but none delivers all of it. A fabric does not replace the warehouse, lakehouse or integration engines; it coordinates them. A catalog bolted onto a lake is inventory, not activation, and virtualization alone addresses access rather than governance. Most importantly, a fabric does not decide who owns data or what the rules are: it executes decisions that people and governance bodies have made. Treating it as a one-time project guarantees that the metadata decays and the automation becomes untrustworthy.

Six layers of a data fabric

The layers below are a logical view. In practice several may be served by one product, and one layer by several.

At the base are the sources and the integration styles that move or expose their data: batch, change data capture, streaming and virtualization. Above them sits the catalog and metadata graph, which records assets, business terms, owners, lineage, quality results and policies as a connected knowledge graph and exposes the semantic layer that consumers understand. It is the foundation of the stack.

The active metadata engine analyzes the graph and produces recommendations, and it is what distinguishes a fabric from a well-documented estate. Policy automation turns governance decisions into enforced rules, and orchestration and DataOps run the pipelines, tests and deployments that result, with observability feeding operational metadata back into the graph. Both act on recommendations only within agreed guardrails. At the top, the consumption layer presents one governed surface to consumers, regardless of where the data physically resides.

The metadata graph connects every component

The defining structural feature of a fabric is that components exchange metadata through a shared graph rather than through point-to-point configuration. Each component publishes what it knows to the graph and consumes what the others publish.

The exchange runs in both directions. A policy engine reads classifications and owners from the graph, an integration engine reads schemas and mappings, and a consumer reads business terms and trust signals. Recommendations flow outward to the engines, and their results flow back as new metadata, which is what makes the loop active rather than a one-time inventory.

Two design decisions matter early. Open metadata standards and APIs, such as OpenLineage for lineage events, keep the graph independent of any single tool. And the graph needs an owner with a governance mandate, because whoever controls it controls what the fabric can automate: ownership of the graph is a governance decision, not an IT default.

Integration styles the fabric orchestrates

A fabric does not introduce new ways of moving data. It chooses among the established integration styles, which Gartner's description of a fabric expects to be combined, and reduces the manual effort of each.

In every case the engines that execute the style stay in place. The fabric contributes the metadata-driven part of the work, from generated mappings and drift detection to policy enforcement at query time, and the ability to choose the right style for each new requirement instead of defaulting to whichever tool a team knows best. It selects and orchestrates engines; it does not replace them.

Metadata activation

Activation is what turns a catalog into a fabric. Gartner's account of a fabric centers on converting passive metadata, recorded for a known purpose such as a schema or a glossary entry, into active metadata that is analyzed continuously and used to drive recommendations and actions. This section covers the kinds of metadata a fabric collects, the cycle that activates them, the analysis techniques involved and the boundary between recommending an action and executing it.

Four kinds of metadata

Activation depends on breadth, because recommendations are only as good as the evidence behind them. Gartner names four kinds of metadata that a fabric should collect and analyze: technical, business, operational and social.

The four kinds differ sharply in how hard they are to obtain. Technical metadata is the easiest, because systems and tools expose it. Business metadata cannot be harvested at all: it depends on stewards and owners, and it is where most fabrics are weakest. Operational metadata records what actually happens and feeds anomaly detection and optimization. Social metadata reveals which assets are trusted in practice, which is often different from which assets are officially endorsed.

How metadata activation works

Activation runs as a cycle of four steps. It is simple to state and demanding to run.

Collection uses connectors, agents and APIs to harvest metadata and, critically, to link it: this column belongs to this asset, which this pipeline produces, which this owner is accountable for, which this policy governs and which these users query. Without the links there is a catalog but no graph. Analysis runs continuously over the graph, using the techniques described below. Recommendation converts analysis into proposed actions with the evidence attached, such as a mapping between two schemas or a quality rule derived from an observed distribution. Automation executes the subset of actions the organization has authorized, from provisioning pipelines to enforcing masking and optimizing placement, and routes everything else to people. Each executed action generates new operational metadata, which closes the loop.

Analysis techniques that make metadata active

Each analysis technique answers a different question about the estate and feeds a different kind of recommendation.

Inferring lineage from code and logs is far more reliable than asking teams to document flows by hand, and usage mining shows which assets analysts actually rely on rather than which ones are documented as important. Every technique depends on its inputs: schemas and data samples, pipeline and query logs, access and audit logs, quality rule results, catalog and glossary edits, and tickets and requests. The more of these the graph receives, the better the engine performs.

Recommend or automate: where a person stays in the loop

The boundary between recommending and automating is the most important governance decision in a fabric. The figure below shows a typical split.

The boundary is not fixed. Recommendation acceptance rates show when the engine is reliable enough for a category of action to be promoted to automation, and every promotion should be a recorded governance decision. Moving the boundary silently is how fabrics lose the trust of the people they are meant to support.

Use cases, and fabric versus mesh

A fabric earns its investment through a small number of use cases that manual processes cannot deliver at scale. It also has to be positioned against data mesh, with which it is often, and wrongly, presented as a rival.

Four use cases that carry the business case

Governed self-service access is the most visible: time to data falls without loosening control, which is the combination manual processes cannot deliver. Integration acceleration matters most in hybrid and multi-cloud estates, where onboarding a source becomes routine work rather than a project. Impact analysis and regulatory evidence rely on end-to-end lineage combined with policy metadata, so that questions such as which reports a change will break, where personal data flows and who accessed it are answered on demand rather than assembled under pressure. Cost and performance optimization turns usage and operational metadata into concrete savings that can be measured against the platform bill.

Fabric and mesh compared

Fabric and mesh are often presented as competing choices, which misreads both. Gartner's own analysts have argued that the two approaches complement each other, and the comparison below shows why: they differ on almost every dimension because they answer different questions.

Because ownership of data in a fabric frequently remains with a central platform group, the approach suits heterogeneous estates where business domains lack the capability to own data products. A mesh, introduced by Zhamak Dehghani, moves that ownership to the domains and suits organizations with many autonomous domains and a high rate of change. Neither is a prerequisite for the other, and each covers the other's weakness: fabrics without clear ownership accumulate uncurated metadata and automate on top of noise, while meshes without automation ask every domain to build the same plumbing.

Fabric capabilities as the self-serve platform of a mesh

Combined, the fabric's catalog, graph, recommendation engine and policy automation become the self-serve platform that a mesh requires. Mapping them to the three planes of a mesh platform makes the complementarity concrete.

The mesh experience plane, where consumers discover products and the federation supervises conformance, is served directly by the fabric's catalog, metadata graph, lineage and conformance dashboards. The data product experience plane, where domain teams build and operate products, draws on the fabric's recommendation engine, templates, inferred quality rules and policy automation: exactly the undifferentiated work a domain should not build itself. The infrastructure plane maps to the fabric's integration engines, virtualization, orchestration and storage. Domains own the products while the fabric automates their plumbing, and the global policies the federation agrees are the rules the fabric engine enforces.

The combination is useful in both directions. Where domains are not yet ready to own data, the fabric delivers governed access and integration value without waiting for the operating model to mature. Where domains are strong, the fabric keeps autonomy from degrading into fragmentation, because shared metadata, identifiers and policies are enforced consistently across every product.

A fabric automates what the metadata knows; a mesh decides who is accountable for what the metadata describes.

A fabric is bounded by what the metadata knows: it can automate mappings, policies and placements only for data that has been described, classified and connected in the graph. A mesh is bounded by what the organization decides: it assigns accountability for describing, classifying and serving data to the domains that create it. Leadership teams can therefore stop asking which to choose and ask two separate questions instead: what platform capabilities are needed, and who should be accountable for which data.

Implementation, pitfalls and measures

The practical questions are where to start, what has to be in place first, what typically goes wrong and how to know the fabric is working. The path below is deliberately metadata-first, because every later capability depends on a graph that can be trusted.

An implementation path that expands by domain

Each phase has an exit criterion that matters more than its timing. The foundation is complete when the catalog is the agreed system of record for metadata and lineage for priority pipelines flows without manual documentation. The engines are connected when quality and policy tools read classifications, owners and lineage from the graph rather than from their own configuration, and stewards review recommendations as routine work. First automation is complete when one policy runs end to end with an evidence trail and a measured change in time to data, which becomes the core of the business case. Expansion then proceeds domain by domain in order of value and readiness, retiring the manual integration and access steps that automation replaces. The periods are indicative; the sequence and the exit criteria are what matter.

Worked example: automating one policy end to end

The example below shows what end to end means in practice. The chosen policy has clear value and a measurable scope: personal data is masked for every role that lacks a documented purpose, across all analytical stores.

Detection and classification come first, because the policy can apply only to columns the graph knows are personal: sensitive data detection proposes classifications, and stewards confirm them and record the result as metadata on each column. The rule is then expressed once in the policy engine, as a function of classification, role and purpose, versioned and tested like code, rather than configured separately in each platform. Enforcement uses the access layer of each platform, so new columns inherit the rule as soon as they are classified, with no per-data-set setup. Finally the fabric produces evidence automatically (coverage of classified columns, enforcement logs and the exception register) for review by privacy and internal audit. The same pattern applies to retention, residency and quality policies.

Prerequisites and skills

Most fabric programs that stall do so on prerequisites rather than on technology.

Without a designated catalog of record, the graph has no authority. A policy engine cannot reason about entitlements that exist only in spreadsheets, so the identity foundation has to express roles, groups and purposes. Metadata standards make metadata from different tools comparable, and tools that cannot expose metadata or accept policy through APIs become islands. Sponsorship has to fund the fabric as a product with a backlog, not as an installation that ends at go-live. The skills follow from the same list, and steward time in particular has to be allocated rather than assumed.

Pitfalls that turn a fabric into another tool

The pitfalls mirror the prerequisites. Buying a fabric is the most expensive; automation without guardrails is the most dangerous, and its remedy is an explicit list of what may be automated, a log of every action and visible exceptions. An engine trained on uncurated metadata recommends confidently and wrongly, and stewards soon stop trusting any of its recommendations. And over-reliance on virtualization is a performance trap: virtualization should provide reach, while physical integration carries load.

Measures of a working fabric

Measures should show whether the fabric changes outcomes, not merely whether it is installed.

Time to provision data is the headline measure because it is what consumers feel. Recommendation acceptance does double duty: it is a quality signal for the engine and for the metadata it learns from, and it guides which categories of action can be promoted to automation. All six should be reported as trends per domain; absolute values depend on scope and are not comparable across organizations.

Readiness checklist

The checklist translates the prerequisites into conditions that can be verified before investment scales. The first two items establish authority over meaning, the next three prove that the graph can be populated and acted on without manual effort, and the last three give the first automation a purpose, the people to review it and a baseline against which to show improvement.

Where to start

The first steps establish the minimum foundation without a large program, and each produces something usable on its own.

1. Designate the catalog of record. Name it, and name owners for critical data assets, which creates the authority the graph needs.
2. Switch on automated lineage. Start with the pipelines behind priority reports. This populates the graph with movement and transformation metadata at low cost and immediately supports impact analysis.
3. Automate one high-value policy end to end. Select it, automate it with evidence, and use it to prove the approach and produce the first measurable outcome.
4. Baseline time to provision data. Measure it now and review it as each domain joins, which keeps the program accountable for consumer outcomes.

A data fabric is not a product to install but a discipline of turning metadata into action: collecting it broadly, curating what only people can supply, and automating only what governance has agreed to automate. Organizations that start with the catalog, lineage and one policy, and then expand domain by domain, build a fabric that earns trust as it grows. Metadata that only describes data is documentation; metadata that acts on it is a fabric.

Glossary

Data fabric

An architectural approach that uses active metadata, semantics and automation to integrate, govern and deliver data across distributed environments; not a single product.

Active metadata

Metadata that is analyzed continuously and used to drive recommendations and automated actions, as opposed to passive metadata recorded for a known purpose and consulted by people.

Technical metadata

Metadata about structure and mechanics, such as schemas, data types, keys, formats, pipeline definitions and query plans.

Business metadata

Metadata that supplies meaning and accountability, such as glossary terms, definitions, ownership, classification and retention rules.

Operational metadata

Metadata about what actually happens, such as pipeline runs, freshness, volumes, failures, quality results, query history, access logs and cost.

Social metadata

Metadata about how people relate to data, such as usage, ratings, comments, questions and certifications.

Metadata graph

A knowledge graph that links assets, terms, owners, pipelines, policies and users, and through which fabric components exchange metadata.

Catalog of record

The catalog designated as the authoritative system of record for metadata, with named owners for critical assets.

Semantic layer

A business-friendly representation of data, built on the glossary and graph, that consumers query in familiar terms.

Lineage

The record of where data comes from and how it moves and changes on its way to reports, models and other consumers.

Change data capture

An integration style that detects changes in a source system and propagates them in near real time.

Data virtualization

An integration style that queries data in place across sources without copying it.

Policy engine

A component that evaluates rules, such as access or masking rules, against metadata like classification, role and purpose, and enforces the result.

DataOps

The application of automated testing, deployment and observability practices to data pipelines and data products.

Recommendation acceptance

The share of the engine's recommendations that stewards accept; a quality signal used to decide what may be promoted to automation.