Data Architecture & Integration

Enterprise Data Architecture

Principles and Reference Patterns

On this page

Enterprise data architecture identifies the data an organization needs and designs the blueprints that describe how that data is captured, stored, moved, protected and served. Done well, it lets a delivery team find the authoritative source for the data it needs, choose a sanctioned pattern and inherit the right controls. Done badly, or not at all, it leaves a landscape that grew one project at a time and that every change makes more expensive.

This paper positions the discipline within enterprise architecture, drawing on DAMA-DMBOK, TOGAF and the Zachman Framework, and sets out nine architecture principles with their rationale and implications. It describes the artifacts an architecture function must maintain, compares ten reference patterns from the enterprise data warehouse to data mesh and data fabric, and explains how to select and combine them. Architecture governance, cloud and security considerations, metrics, common pitfalls and a maturity path complete the picture.

Written for Enterprise and data architects, chief data officers, data platform leads, and solution architects who set or apply the blueprints for how an organization stores, moves, protects and serves its data.

Executive summary

Most large organizations did not design their data landscape; they accumulated it. Each application brought its own database, each analytics initiative its own copy of customer and product data, and each integration project its own point-to-point interface. Nobody can say with confidence where the authoritative version of a critical data element lives or what will break when a source changes. Enterprise data architecture is the discipline that replaces accumulation with design.

Following DAMA-DMBOK, this paper treats data architecture as identifying the enterprise's data needs and designing and maintaining the blueprints that meet them. Those blueprints are a connected set of artifacts rather than one diagram: an enterprise conceptual data model organized into subject areas, a map of systems of record and data flows, a small number of reference architectures, a standards catalog and a log of architecture decisions.

  • Principles come before patterns: nine principles, each with a rationale and implications, give architects and review boards a stable basis for decisions as technology changes.
  • No single pattern serves every need: the paper compares ten patterns, from the warehouse and lakehouse to the streaming backbone, data mesh and data fabric, on purpose, strengths, weaknesses and fit.
  • Targets are reached through transition architectures: a roadmap that sequences change, retires technical debt deliberately and records decisions matters as much as the target state.
  • Architecture governance and data governance are distinct but inseparable: the design authority decides how structures and flows are built, while the data governance council decides who owns data, what it means and which source is authoritative.

The paper closes with the metrics that show whether an architecture is working, such as reuse, redundancy, time to provision data and standards adherence; the pitfalls that recur in practice, including ivory-tower architecture, pattern-of-the-year adoption and the absence of a transition plan; and a maturity path from an undocumented landscape to largely automated architectural compliance.

Why architecture must be intentional

An enterprise does not have a data architecture because somebody drew one; it has one because thousands of decisions were made about where data lives and how it moves, whether or not anyone wrote them down.

The question is therefore not whether an organization has a data architecture but whether that architecture is intentional. An unintentional architecture is the by-product of project decisions taken one at a time, each reasonable in isolation and collectively expensive. Its symptoms are familiar, and because they are structural, tactical fixes do not remove them.

An intentional architecture does not eliminate complexity; a large organization's data landscape will remain plural. What it provides is a shared frame for that plurality: agreed principles, a vocabulary of subject areas, a designated authoritative source for each kind of data, a small number of sanctioned patterns and a visible route from the current state to a better one. This paper describes how to build and operate that frame.

The treatment stays at the enterprise level. Schema design is covered only as far as architecture needs it, and integration techniques, data mesh and data fabric appear here as patterns within the overall architecture rather than in full operational depth.

An architecture that is not written down still exists; it is simply unmanaged.

Definitions and scope

DAMA-DMBOK describes data architecture as identifying the data needs of the enterprise, regardless of structure, and designing and maintaining the master blueprints to meet those needs; the blueprints then guide data integration, control data assets and align data investment with business strategy. The definition has two halves that should not be separated. Identifying needs is a business activity: which subject areas matter, which decisions depend on them, and what quality, timeliness and protection they require. Designing blueprints is a technical activity whose value comes entirely from answering those needs.

Within DAMA-DMBOK, data architecture is one of the knowledge areas arranged around data governance, alongside data modeling and design, data storage and operations, and data integration and interoperability, among others. It is neither the sum of the platforms an organization owns nor a synonym for data modeling. It is the layer of decisions above both: which data the enterprise needs, where each kind is authoritatively held, how it flows, and which structures and standards every solution must respect.

Place in enterprise architecture

In TOGAF, data architecture is developed in Phase C of the Architecture Development Method (ADM), Information Systems Architectures, together with application architecture, between the business architecture of Phase B and the technology architecture of Phase D. The data and application work within Phase C may run in either order or in parallel, but both follow the business architecture. The sequence encodes a discipline: data architecture is derived from business capabilities and information needs, and it constrains rather than follows the choice of technology. TOGAF also supplies apparatus that data architects borrow: a template that states each principle with its rationale and implications, an architecture repository that records the standards new architectures must comply with, architecture compliance reviews and an architecture board.

The Zachman Framework plays a different role. John Zachman describes it as an ontology rather than a methodology: a classification scheme that arranges descriptive representations of an enterprise in a grid of six interrogatives (what, how, where, who, when and why) against six rows, which correspond to the perspectives of successive audiences, from the executive's view of scope to the operating enterprise itself. For data architects it is a completeness check: the what column reminds the team that an inventory of business things, a conceptual model, a logical model, a physical design and the deployed data are different artifacts for different audiences.

Table 1Three reference points for enterprise data architecture
FrameworkWhat it isWhat data architects take from it
DAMA-DMBOKA body of knowledge for data management, with governance at the center of its knowledge areasThe definition and scope of data architecture and its links to modeling, integration, storage and governance
TOGAFAn enterprise architecture standard built around the Architecture Development MethodThe position of data in the ADM, the principle template, transition architectures, compliance reviews and the architecture board
Zachman FrameworkAn ontology that classifies the descriptive representations of an enterpriseA completeness check across audiences and levels of abstraction, especially in the what column

Levels and boundaries

Enterprise data architecture sits at the top of a hierarchy of increasingly concrete descriptions, shown in the figure below. Solution data architectures apply the enterprise blueprints to a business area or program, data models define the structures of specific stores and interfaces, and the platform of storage, compute, network and security services carries everything above it. Each level constrains the one beneath it and feeds lessons upward.

  1. Enterprise data architecture

    Needs, principles, subject areas, systems of record

  2. Solution data architecture

    Patterns applied to a business area or program

  3. Data models

    Conceptual, logical and physical structures

  4. Platform and infrastructure

    Storage, compute, network and security services

Figure 1Levels of data architecture, from enterprise blueprints to platform services

Two boundaries deserve explicit statement. Data architecture covers both the operational plane, where transactions are captured and processes run, and the analytical plane, where data is integrated, historized and analyzed; organizations that architect only the analytical plane inherit every inconsistency of the operational one. It also covers unstructured content, streams and externally sourced data, not only relational tables.

Architecture principles

Principles are the stable core of an architecture. Technologies and patterns change every few years; the reasons for preferring one design over another change much more slowly. Following the TOGAF convention, each principle is stated with a rationale and implications, because a principle without implications is a slogan. Nine principles can be remembered, debated and applied in a review; thirty cannot.

Table 2Nine enterprise data architecture principles
PrincipleStatementPrimary implication
Data is a shared assetData belongs to the enterprise, not to the capturing system or teamAccess by default within policy; no unjustified private copies
Authoritative sourcesEach subject area has designated systems of record and distribution pointsOther copies are derived, traceable and refreshed
Separate storage and computePersist data independently of the engines that process itOpen storage formats; elastic, workload-specific compute
Decouple through interfacesSystems exchange data only through published, versioned interfacesNo cross-boundary database access; contracts and change notice
Metadata-driven designMetadata describes structures, flows and controls and drives automationCatalog as a build-time dependency; lineage captured, not drawn
Security and privacy by designProtection follows classification and is built into every patternClassification before provisioning; controls inherited from the platform
Fit-for-purpose storageMatch the store to the workload rather than using one store for allPolyglot persistence within a sanctioned list; integration budgeted
Open formats and standardsPrefer open, documented formats and interfaces over proprietary onesPortability tested; proprietary features used knowingly and recorded
Design for changeAssume sources, volumes, regulations and tools will changeSchema evolution, versioning and transition plans are mandatory

Shared asset and authoritative sources

The first principle addresses the most common failure of accumulated landscapes: data trapped in the system that captured it. Treating data as a shared asset means that the default answer to a legitimate request within policy is yes, that the platform bears the cost of sharing rather than each project negotiating it, and that producing teams are accountable for the usability of what they publish. Ownership is assigned to business domains through data governance; architecture provides the channels through which owned data is shared.

The second principle corrects a persistent misunderstanding. The goal is not a single physical copy of each piece of data, because replication is inherent in distributed systems, caching, analytics and resilience. The goal is that each subject area, or each segment of it, has a designated system of record where data is created and corrected, and designated distribution points from which consumers obtain it. Every other copy is derived, carries lineage to its source and is refreshed rather than independently maintained.

Decoupling, metadata and protection

Separating storage from compute is the common shape of cloud data platforms: data persisted in open formats on durable object storage can be read by several engines, scaled independently of them and retained when an engine is replaced. Decoupling through interfaces applies the same logic between systems. Data crosses a boundary only through a published, versioned interface, whether a service, an event stream or a governed data product, and never through direct access to another system's tables, so interface contracts and change notice periods become architectural standards.

Metadata-driven design recognizes that structures, flows and controls cannot be managed by hand at enterprise scale. Schemas, lineage, classifications, ownership and quality rules are recorded in a catalog that pipelines and platforms read at build and run time, so that lineage is captured by tooling rather than drawn afterward. Security and privacy by design follows directly: when classification is metadata the platform can read, masking, encryption and access policy are inherited by every pattern. That is the architectural form of the obligation in Article 25 of the General Data Protection Regulation (GDPR) to implement data protection by design and by default.

Fit, openness and change

Fit-for-purpose storage accepts polyglot persistence: a relational store for transactions, a document store for flexible product data, a graph store for relationships, a columnar store for analytics and a log for events. The discipline lies in keeping the list of sanctioned stores short and in budgeting the integration that each additional store creates.

Open formats and standards protect the organization's options. Data held in open file and table formats and exchanged through standard protocols can move between engines and vendors, so the cost of leaving is known rather than discovered. Design for change means schema evolution rules, versioned interfaces, replayable pipelines and a transition plan for every significant component, together with a record of why each decision was made, so that successors can tell which constraints still apply.

Architecture artifacts

Blueprints are useful only if they exist, are current and are consulted. An architecture function therefore maintains a small, named set of artifacts, each with an owner, a refresh cadence and a place in the architecture repository. Organizations that attempt to maintain dozens of artifact types find that none of them is trusted.

Table 3Core data architecture artifacts and their maintainers
ArtifactWhat it containsMaintained byTypical cadence
Enterprise conceptual data modelSubject areas, core entities and relationships in business termsEnterprise data architect with domain stewardsQuarterly review; change by decision
System-of-record mapAuthoritative sources and distribution points per subject areaData architecture with governance sign-offOn every system change
Data flow mapMovement of data between systems, planes and partiesDomain architects; generated where possibleContinuous from the catalog
Reference architecturesSanctioned patterns, components, interfaces and controlsDesign authorityAnnual review
Standards catalogApproved formats, interfaces and products with lifecycle statusDesign authoritySemiannual review
Architecture decision recordsContext, options, decision and consequences of each significant choiceDeciding architect; indexed centrallyAt decision time

Conceptual model and subject areas

The enterprise conceptual data model names the things the business cares about, such as customer, product, contract, location and event, and the relationships among them, in business language and independent of any system. Its first product is the set of subject areas, which become the organizing scheme for ownership, the system-of-record map, catalog navigation and domain boundaries. A workable enterprise model fits on one page; detail belongs in the subject-area and logical models that sit beneath it.

System-of-record and data flow maps

The system-of-record map records, for each subject area and where necessary for each attribute group within it, the system in which data is created and corrected and the points from which it is authoritatively distributed. It is among the artifacts most often requested by auditors, regulators and integration teams, and among those most often missing. The data flow map complements it by showing how data actually moves between systems, planes and external parties; where a catalog captures lineage, the map should be generated rather than drawn.

Reference architectures, standards and decisions

Reference architectures describe the sanctioned way to assemble components for a recurring need, such as an analytical platform, an integration backbone or an external data exchange. Each names the components, their interfaces, the controls that must be present and the situations in which the pattern should and should not be used. The standards catalog, which corresponds to the standards held in a TOGAF architecture repository, lists approved formats, protocols, products and conventions, each with a lifecycle status such as emerging, approved, contained or retiring.

Architecture decision records (ADRs), a lightweight practice popularized by Michael Nygard, are short, numbered documents that record the context of a significant decision, the decision itself and its consequences; many templates also record the options considered. They are the memory of the architecture: principles say what the organization values, and ADRs show how those values were traded off in specific cases. A decision that is not recorded will be relitigated, usually by people who lack the context in which it was made.

Reference patterns

A reference pattern is a proven arrangement of components that solves a recurring data problem. Each pattern below is described by its purpose, strengths, weaknesses and fit. None is universally superior and most enterprises run several; a table at the end of this section summarizes them, and the figure that follows it shows how they combine into one layered architecture.

Operational and analytical planes

The first distinction is between the operational plane and the analytical plane. The operational plane runs the business: transactional systems, customer-facing applications and the integration between them, optimized for low-latency reads and writes of current data with strong consistency within each system. The analytical plane helps the organization understand the business: integrated, historized data organized for query, reporting and machine learning, optimized for throughput and for change over time. Much architectural harm comes from ignoring the boundary, for instance by running analytics against transactional databases or by using the warehouse as an operational integration hub.

Warehouse, lake and lakehouse

The enterprise data warehouse integrates data from many sources into a consistent, historized store for reporting and analysis. Two lineages shaped it. Bill Inmon's approach, extended in the corporate information factory, builds a normalized warehouse that is subject-oriented, integrated, time-variant and non-volatile, and derives data marts from it. Ralph Kimball's bus architecture builds dimensional marts around business processes and integrates them through conformed dimensions. The warehouse's strengths are consistency, mature tooling and well-understood governance; its weaknesses are the lead time of modeling before use and a poor fit for semi-structured content. It fits regulated reporting and any domain where agreed definitions matter more than speed of onboarding.

The data lake stores data of any structure in raw form on low-cost storage and applies structure on read. The term is generally credited to James Dixon, then chief technology officer of Pentaho, who contrasted the lake's water in its natural state with the cleansed, packaged bottled water of a data mart. The lake's strength is the speed and cost at which new sources can be landed and explored; its weakness is governance, because without catalog, ownership and quality controls raw storage becomes a swamp in which nobody can find or trust anything. It fits exploration, feature engineering and the retention of raw history, provided it is operated with warehouse discipline.

The lakehouse adds warehouse capabilities to lake storage by placing an open table format, such as Apache Iceberg, Delta Lake or Apache Hudi, over files in object storage, which brings ACID transactions, schema enforcement and time travel. The term was popularized by Databricks, and the architecture is set out in a research paper by Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia. Several engines can read and write the same tables, which realizes the separation of storage and compute, and the medallion convention of bronze, silver and gold layers organizes progressive refinement. The lakehouse's strengths are openness and one copy serving engineering and business intelligence alike; its weaknesses are the operational maturity it demands and the temptation to treat the layer convention as a substitute for modeling. It fits organizations consolidating lake and warehouse estates.

Operational data store and master data hub

The operational data store (ODS) is an integrated, subject-oriented store of current or near-current data from several operational systems, used for operational reporting and for serving integrated views back to operations. Unlike the warehouse it is volatile and holds little history. Its strength is a consistent, current view across systems without burdening them; its weaknesses are the cost of another integrated store and the risk that it quietly becomes the warehouse or the integration hub. It fits customer service views and operational dashboards.

The master data hub manages golden records for core entities such as customer, product, supplier and location, matching and merging records from contributing systems and distributing the result. Its implementation styles, from a registry that only cross-references identifiers, through consolidation and coexistence, to a centralized hub in which master data is created, trade intrusiveness for control. Its strength is a governed, de-duplicated view of the entities every other pattern depends on; its weaknesses are organizational effort, matching complexity and synchronization latency. It fits wherever the same entity is created in several systems.

Streaming, virtualization and data services

The event streaming backbone moves data as a durable, ordered log of events to which producers publish and from which consumers subscribe; platforms such as Apache Kafka are examples, and change data capture (CDC) tools feed database changes into it. Its strengths are decoupling of producers from consumers, low latency, replayability and a single path of change that both planes can consume. Its weaknesses are the skills it requires, the difficulty of expressing complex joins and history as streams, and schema governance across many topics. It fits event-driven integration and real-time analytics.

Data virtualization presents a logical, queryable layer over distributed sources without moving the data, translating a federated query into source queries. Its strengths are speed of delivery, avoidance of copies and usefulness where data must not be moved for legal or contractual reasons. Its weaknesses are performance that depends on the weakest source, load pushed onto operational systems, limited history and difficult transformation. It fits cross-source access to moderate volumes and prototyping before physical integration.

The data services layer exposes data through managed application programming interfaces (APIs), typically behind a gateway that handles authentication, rate limiting and versioning. It is the operational counterpart of the analytical data product: a contract-bound, discoverable interface through which applications and partners obtain data without touching the source. Its strengths are decoupling, reuse and policy enforcement at one point; its weaknesses are unsuitability for bulk movement and the governance of hundreds of endpoints. It fits application integration, partner exchange and synchronous retrieval of master and reference data.

Data mesh and data fabric

Data mesh, introduced by Zhamak Dehghani, is an organizational and architectural approach rather than a platform. Its four principles are domain ownership of analytical data, data as a product, a self-serve data platform and federated computational governance. Its strength is scaling data delivery with the number of domains by removing the central team as a bottleneck. Its weaknesses are the organizational maturity it assumes, duplicated skills across domains and the difficulty of interoperability without strong federated standards. It fits large organizations with many autonomous, engineering-capable domains.

Data fabric, which Gartner frames as a design concept, uses active metadata, a knowledge graph of the data landscape and automation to integrate, govern and deliver data across heterogeneous platforms with less hand-built pipeline work. Where mesh distributes ownership, fabric centralizes intelligence about data while leaving the data where it is. Its strength is leverage on metadata the organization should be capturing anyway; its weaknesses are dependence on metadata quality and coverage, the immaturity of fully automated integration, and vendor definitions that vary widely. It fits heterogeneous estates with a strong catalog foundation. Mesh and fabric are complementary: a mesh needs a metadata backbone, and a fabric needs owners for the data it connects.

Table 4Reference patterns compared
PatternPrimary purposeStrengthsWeaknessesBest fit
Enterprise data warehouseIntegrated, historized data for reportingConsistency, mature tooling, governanceLead time; weak with unstructured dataRegulated and financial reporting
Data lakeRaw data of any structure at low costFast onboarding, flexibility, scaleSwamp risk; weak governance by defaultExploration, data science, raw history
LakehouseWarehouse capabilities on open lake storageOpenness; one copy for many enginesOperational maturity; evolving toolingConsolidating lake and warehouse estates
Operational data storeCurrent integrated view for operationsNear-current data; relieves sourcesAnother store; scope creepService views, operational reporting
Master data hubGolden records for core entitiesGoverned, de-duplicated entitiesOrganizational effort; synchronization latencyEntities created in several systems
Event streaming backboneDurable, ordered movement of change eventsDecoupling, low latency, replaySkills, schema governance, complex joinsEvent-driven integration, real time
Data virtualizationLogical access without moving dataSpeed of delivery; no copiesSource load, performance, limited historyCross-source access; movement constraints
Data meshDomain-owned analytical data productsScales with domains; clear ownershipAssumes maturity; interoperability effortMany autonomous, capable domains
Data fabricMetadata-driven integration across platformsLeverages metadata; less hand-built workDepends on metadata quality; varied definitionsHeterogeneous estates with a strong catalog
Data services layerManaged APIs for application accessReuse, contracts, policy at one pointUnsuited to bulk; endpoint sprawlApplication and partner integration
Consumption
  • BI
  • Applications
  • Data science
  • Partners
  • AI
Serving and access
  • Semantic layer
  • APIs
  • Data products
  • Virtualization
Integrated storage
  • Lakehouse or warehouse
  • Operational data store
  • Master data hub
Ingestion and movement
  • Streaming backbone
  • CDC
  • Batch pipelines
Sources
  • Transactional systems
  • SaaS
  • Devices
  • External feeds
Catalog and metadata
Security, governance, observability
Figure 2A layered enterprise reference architecture that combines the patterns

In the figure above the patterns are layers of one architecture rather than alternatives. Streaming and batch coexist in ingestion; integrated storage holds the lakehouse or warehouse, the operational data store and the master data hub; and a serving layer exposes data through a semantic layer, data products, APIs and virtualization. The catalog, security, governance and observability run across every layer. Mesh and fabric are not layers: mesh organizes ownership across the layers, and fabric automates the cross-cutting foundations.

Selecting and combining patterns

Pattern selection is a matter of fit rather than fashion. Five criteria dominate: the latency consumers need, the consistency they can tolerate, the governance obligations attached to the data, the total cost of the pattern over its life, and the skills the organization has or can acquire. The table below sets out the question to ask under each criterion and the direction in which the answer pushes the choice.

Table 5Selection criteria and the patterns they favor
CriterionQuestion to askWhat the answer changes
LatencyHow fresh must data be when consumed: days, hours or seconds?Seconds push toward streaming, an ODS and APIs; days allow batch loading
ConsistencyMust every consumer see the same value at the same moment?Strict consistency points to the system of record or a transactional hub
GovernanceHow regulated, sensitive or definition-critical is the data?Regulated data needs modeled, controlled stores and inherited policy
CostWhat are storage, compute, movement and staffing costs over several years?Open formats and shared platforms lower lifetime cost; count movement and staffing
SkillsCan the organization build and run the pattern over time?Mesh, streaming and polyglot estates demand skills; otherwise prefer fewer, managed patterns

The matrix below crosses the first criterion, freshness, with a structural factor the table does not capture: how many autonomous domains produce the data. Together they indicate where an architecture's emphasis should lie.

Freshness and latency requirement

Central real-time platform

Streaming backbone feeding a shared lakehouse

Event-driven mesh

Domain data products published over a shared backbone

Central warehouse

Batch integration, conformed dimensions, one team

Federated platform

Domain-owned products on a shared batch platform

Autonomy and number of producing domains

Figure 3Pattern emphasis by freshness requirement and domain autonomy

Most organizations need more than one quadrant, because freshness needs and domain autonomy vary across subject areas, so the matrix guides emphasis rather than exclusive choice. Combining patterns is normal and should be explicit: a streaming backbone commonly feeds the lakehouse, the operational data store and the master data hub from one stream of change; a lakehouse is the storage beneath a mesh of domain-owned products; and a fabric's metadata layer spans warehouse, lake and operational systems alike. The combinations to avoid are those that duplicate a function, such as two integration hubs for the same data, or that hide a missing decision, such as virtualization used instead of designating a system of record.

Transition architectures and roadmaps

A target architecture that cannot be reached is a poster. TOGAF's later phases, Opportunities and Solutions (Phase E) and Migration Planning (Phase F), exist because the route matters as much as the destination, and their device is the transition architecture: a described, stable intermediate state that delivers value on its own and reduces risk for the next step. A data architecture roadmap should be expressed as two to four transition architectures, each naming the systems it changes, the data it migrates, the interfaces it retires and the capability it enables.

  1. Document the baseline honestly. Inventories, the system-of-record map as it is, data flows as they run, and known defects.
  2. Describe the target in the same artifacts. Use the same models and maps for the target, so that gaps are visible item by item.
  3. Group gaps into transition architectures. Each should leave the landscape consistent and deliver a measurable outcome.
  4. Sequence by dependency and value. Attach funding and owners, and record the rationale in an architecture decision record.
  5. Re-plan at each transition. The target will move, and the roadmap should move with it rather than be defended.

Managing technical debt

Technical debt in data landscapes takes recognizable forms: undocumented pipelines, duplicated copies of critical data, point-to-point interfaces that bypass the backbone, hand-maintained reporting stores and platform versions out of support. Managing it starts with a debt register that names each item, its owner, the interest it charges in effort and risk, and the transition architecture in which it will be repaid. Not all debt should be repaid, because some systems will be retired before their debt matters. The strangler fig pattern, named by Martin Fowler, in which a new component incrementally takes over an old one's responsibilities behind a stable interface, repays large items without a risky cutover.

Architecture governance

Architecture governance is the set of structures and processes through which an organization ensures that solutions conform to its architecture, that principles are applied consistently and that exceptions are deliberate. It should be as light as the organization's risk tolerance allows, because heavy governance is bypassed rather than obeyed. The elements below are common to most workable models.

Design authority

The accountable architecture leadership, typically the enterprise data architect with domain architects, which owns principles, reference architectures and standards and decides contested designs.

Architecture review board

A standing forum that reviews significant solutions against principles and standards at defined gates, records decisions and tracks exceptions.

Principles compliance

Assessment of a design against each principle with a stated finding: compliant, compliant with conditions, exception requested or non-compliant.

Exceptions

Time-limited waivers with an owner, a rationale, a compensating control and an expiry date, after which they are renewed, remediated or escalated.

Fitness functions

Automated checks, a concept from evolutionary architecture, that continuously test an architectural property, such as no direct cross-domain database access or every table registered in the catalog.

Review should be concentrated where decisions are expensive to reverse: a new system of record, a new integration pattern for a subject area, a new store type, an external data exchange, or any design that moves regulated data across a boundary. Routine solutions should pass through automated checks and self-assessment against a checklist, with the board sampling rather than inspecting everything. Reviews that arrive after a design is built are theater; the board's calendar should align with program gates so that it sees designs while options are still open.

Relationship with data governance

Architecture governance and data governance are distinct and interdependent. Data governance decides who owns data, what it means, which policies apply and which source is authoritative; architecture governance decides how structures, flows and platforms are built to honor those decisions. The system-of-record map is the clearest shared artifact: data governance designates authoritative sources, and architecture encodes and enforces the designation. Where the two are not connected, architecture designs channels for data nobody owns, and governance issues policies no platform can enforce.

Data governance decides whose data it is and what it means; data architecture decides how it is kept, moved and served. Neither works alone.

Cloud, security zones and environments

Cloud platforms have changed the economics of data architecture without changing its principles. Storage is cheap and elastic, compute is rented by the minute, managed services replace much platform engineering, and separation of storage and compute is the default. Cloud has also introduced constraints that architects must design around explicitly: data gravity, egress charges, residency obligations and the differences between providers.

Data gravity, egress and residency

Data gravity, a term coined by Dave McCrory, is the observation that large bodies of data attract the applications and services that use them, because moving data is slow and expensive while moving processing is not. The location of the primary integrated store therefore determines where most processing will end up, and that choice should be made deliberately rather than inherited from the first migration. Egress charges reinforce gravity: providers typically charge to move data out of a platform or region, so architectures that shuttle large volumes between clouds or back to on-premises systems incur recurring costs that should be modeled before the pattern is chosen.

Residency and sovereignty requirements determine where data may be stored and processed and under whose jurisdiction. The GDPR's rules on international transfers in Articles 44 to 49, sector regulation in financial services and health, and national data localization laws can each constrain region choice, replication and the location of support staff. The EU-US Data Privacy Framework provides a lawful basis for transfers to certified US organizations, but its two predecessors were invalidated by the Court of Justice of the European Union and the framework itself has been challenged in the EU courts. Residency should therefore be treated as a metadata attribute of each data set that drives placement and replication rules, so that a change in the legal position becomes a change of configuration rather than a redesign.

Many large organizations operate more than one cloud and retain on-premises systems through acquisition and history rather than design. The sensible posture is to accept a hybrid estate while refusing gratuitous duplication: choose a primary location for each subject area's integrated data, use open formats so that engines can change without moving data, place workloads with the data they consume, and reserve cross-cloud movement for services genuinely unavailable elsewhere. Portability is a property to be tested, for instance by proving that a critical data set can be read by a second engine.

Security zoning and environments

Security zoning applies the classification scheme to the architecture. A common arrangement separates a landing zone, where raw data arrives under restricted access; curated zones, where governed data is served under role- or attribute-based access; and restricted zones for data whose sensitivity requires tokenization, column-level encryption or approved-purpose access. Zones are implemented through network segmentation, identity and access management, key management and policy enforcement in the platform, and they should be defined once in the reference architecture so that every solution inherits them. ISO/IEC 27001 and the organization's control framework supply the controls, and GDPR Article 32 requires security appropriate to the risk for personal data; the architecture decides where each control is enforced.

Environments complete the picture. Development, test and production environments must be isolated at the identity and network level, and production data must not flow into non-production environments without masking, pseudonymization or substitution by synthetic data. Pipelines, schemas and policies should be promoted between environments as versioned code through the same deployment mechanisms as application software. An architecture in which production data is copied to developer workstations has a zoning design only on paper.

Metrics and pitfalls

Metrics

An architecture function that cannot show its effect will be judged by the volume of its documents. The metrics below are chosen because each reflects a principle and can be measured from artifacts the function should already maintain: the catalog, the system-of-record map, the standards catalog and the review records. Targets should be set per domain and trended over time rather than compared with external benchmarks, which rarely share definitions.

Table 6Metrics for an enterprise data architecture function
MetricHow it is measuredWhat it indicates
ReuseConsumers per governed data set, API or data productWhether shared assets are shared in practice
RedundancyCopies of each critical data element outside designated distribution pointsProgress toward authoritative sourcing
Time to provision dataElapsed time from approved request to usable accessFriction in the platform and governance path
Standards adherenceShare of reviewed solutions compliant without exceptionFitness of standards and quality of guidance
Exception ageOpen exceptions by age and by expired statusWhether waivers are temporary or permanent
Interface conformanceShare of cross-boundary interfaces using sanctioned patternsDecoupling versus point-to-point growth
Decision coverageSignificant changes with an architecture decision recordWhether the architecture is remembered

Common pitfalls

The failures of architecture functions are as consistent as the symptoms of accumulated landscapes, and most have a structural remedy that this paper has already described. The table below pairs each pitfall with its cure.

Table 7Common pitfalls and their remedies
PitfallWhat happensRemedy
Ivory-tower architectureBlueprints are produced away from delivery teams and reviewed only at the endEmbedded domain architects, early clinics and decisions recorded in the open
Pattern-of-the-year adoptionEach new pattern is adopted as a replacement rather than a tool, so the estate gains a layer with every cycleExplicit selection criteria and retirements that accompany every addition
No transition planA target state without transition architectures, funding and retirements leaves teams to improvise the route, usually by building a parallel landscapeTwo to four funded transition architectures, each with named retirements
Single-copy fundamentalismPursuing one physical copy of enterprise data discredits the architecture when copies inevitably appearDesignated systems of record and distribution points, with every other copy derived and traceable
Platform before principlesA platform is selected and the architecture derived from its feature list, which makes the vendor's roadmap the enterprise's architecturePrinciples and reference architectures agreed first; platforms chosen against them
Analytics-only scopeThe analytical plane is architected carefully while operational systems exchange data through undocumented point-to-point interfaces, so every inconsistency is inherited downstreamArchitecture that covers both planes, with interface standards for operational integration
Governance by inspectionA review board is relied on to catch everything, which produces queues and bypassesStandards encoded in platform defaults and fitness functions, with review reserved for costly decisions

A staged path forward

Architecture maturity is reached in stages, and attempting to skip stages produces documents without adoption. The path below is indicative; domains within one organization will sit at different stages, which is acceptable provided the enterprise-level artifacts exist and the direction is shared.

  1. Stage 1
    Undocumented

    Landscape known only to individuals; copies uncounted

  2. Stage 2
    Documented

    Inventories, system-of-record map, baseline data flows

  3. Stage 3
    Standardized

    Principles, reference architectures, standards, reviews

  4. Stage 4
    Managed

    Roadmap, debt register, metrics, exceptions with expiry

  5. Stage 5
    Optimizing

    Automated compliance, fitness functions, live roadmap

Figure 4A maturity path for enterprise data architecture

Each stage creates the conditions for the next. A roadmap, a debt register and metrics become meaningful only once principles and reference architectures exist, because until then they have nothing to measure against. The final stage, in which compliance is largely automated and the roadmap is revised continuously, is reached by organizations that treat the architecture as a product with users rather than as a standard to be enforced.

Where to start

  1. Document the baseline. Inventory stores and interfaces, record the system-of-record map as it actually is, and trace the flows that carry critical data.
  2. Agree principles and reference architectures. Adopt a short set of principles with rationale and implications, publish a few reference architectures, and start reviewing significant decisions against them.
  3. Build a credible roadmap. Express the target as two to four transition architectures, open a technical debt register, and record each significant choice in a decision record.
  4. Measure and adjust. Report reuse, redundancy, provisioning time and standards adherence by domain, and revise any standard that attracts repeated exceptions.
  5. Automate compliance. Encode the most important standards as fitness functions and platform defaults, so that review effort shifts to the decisions that are expensive to reverse.

Conclusion

Enterprise data architecture succeeds when it changes decisions. The artifacts, patterns and governance described here exist to make the right design the easy design: to let a solution team find the authoritative source for the data it needs, choose a sanctioned pattern that fits, inherit the controls that classification requires and record what it decided for those who follow.

The route is unglamorous: principles specific enough to exclude something, designated authoritative sources in place of the myth of a single copy, patterns chosen for fit rather than fashion, transition architectures that retire as much as they add, and governance concentrated where decisions are expensive to reverse. Organizations that reach that state spend less on rediscovering their own data and more on using it.

Glossary

Architecture decision record (ADR)
A short document recording the context, decision and consequences of a significant architecture choice, often with the options considered.
Authoritative source
A designated system or distribution point from which consumers are expected to obtain a kind of data.
Data fabric
A design concept that uses active metadata, knowledge graphs and automation to integrate, govern and deliver data across heterogeneous platforms.
Data gravity
The tendency of applications and services to locate near large bodies of data because moving data is costly and slow.
Data lake
A store that holds data of any structure in raw form on low-cost storage and applies structure when the data is read.
Data mesh
An organizational and architectural approach in which business domains own analytical data and publish it as products on a self-serve platform under federated governance.
Enterprise conceptual data model
A business-level model of the enterprise's core entities, relationships and subject areas, independent of systems.
Fitness function
An automated test that continuously checks an architectural property of a system or platform.
Lakehouse
An analytical architecture that adds warehouse capabilities, such as transactions and schema enforcement, to lake storage through an open table format.
Master data hub
A system that matches, merges and distributes golden records for core entities such as customer, product and supplier.
Operational and analytical planes
The two broad regions of a data landscape: systems that run the business and systems that analyze it.
Operational data store (ODS)
An integrated, subject-oriented store of current or near-current operational data, volatile and holding little history.
Polyglot persistence
The deliberate use of several storage technologies, each chosen for the workload it serves best.
Reference architecture
A sanctioned, documented arrangement of components, interfaces and controls for a recurring class of solution.
Subject area
A high-level grouping of related business entities, such as customer or product, used to organize models, ownership and maps.
System of record
The system in which a particular kind of data is created and corrected and which holds its authoritative version.
Transition architecture
A stable, described intermediate state between the baseline and the target architecture that delivers value on its own.

References and further reading

  1. DAMA International, DAMA-DMBOK: Data Management Body of Knowledge, second edition.
  2. The Open Group, The TOGAF Standard, 10th Edition.
  3. John A. Zachman, A Framework for Information Systems Architecture, IBM Systems Journal, vol. 26, no. 3.
  4. W. H. Inmon, Building the Data Warehouse, fourth edition, Wiley.
  5. Ralph Kimball and Margy Ross, The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, third edition, Wiley.
  6. Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia, Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics, Conference on Innovative Data Systems Research (CIDR).
  7. Zhamak Dehghani, Data Mesh: Delivering Data-Driven Value at Scale, O'Reilly Media.
  8. Neal Ford, Rebecca Parsons, Patrick Kua and Pramod Sadalage, Building Evolutionary Architectures, second edition, O'Reilly Media.
  9. Martin Fowler, Strangler Fig Application, martinfowler.com.
  10. Michael Nygard, Documenting Architecture Decisions.
  11. EDM Association (formerly the EDM Council), Data Management Capability Assessment Model (DCAM).
  12. ISO/IEC 38505-1, Information technology: Governance of IT, Governance of data, Part 1: Application of ISO/IEC 38500 to the governance of data.
  13. ISO/IEC 27001, Information security management systems: Requirements.
  14. General Data Protection Regulation (GDPR), in particular Articles 25, 32 and 44 to 49.

This paper offers general guidance and is not legal, regulatory or professional advice. Summaries of laws, regulations and standards are for orientation only; confirm the obligations that apply to your organization with qualified advisors. Product and vendor names are used for illustration and imply no endorsement.

© ByteCI, LLC. All rights reserved.

Put this into practice

Want help applying this?

ByteCI helps federal and private-sector teams turn guidance like this into working architecture, roadmaps and delivery. Tell us about your situation.