Data Integration and Interoperability

Executive summary

Most enterprise data is used far from where it is created. An order captured in a commerce platform feeds fulfillment, finance, analytics, fraud detection and regulatory reporting, each with its own format, timing and definition of an order. Data integration is the discipline of moving and consolidating that data; interoperability is the ability of the systems involved to exchange data that each can correctly interpret and use. Together they form a DAMA-DMBOK knowledge area and, in most organizations, one of the largest consumers of data engineering effort.

The effort is structural. Each interface must bridge differences in structure, meaning, timing and ownership, and must be kept working as both ends change. Organizations that let interfaces multiply point to point accumulate undocumented dependencies that slow every change; organizations that standardize on a few patterns, publish interfaces under contracts and capture lineage automatically turn integration from a recurring cost into a reusable asset.

• No single pattern fits all needs: batch, change data capture, event streaming, virtualization, APIs, file transfer, replication and reverse ETL each suit a distinct combination of latency, volume, source capability and consumer breadth.
• Interoperability must hold at four levels (technical, syntactic, semantic and organizational): protocols, schemas and registries address the first two, while glossaries, reference data, contracts and agreed responsibilities address the rest.
• Reliability comes from idempotent processing, explicit delivery semantics, handling of late and out-of-order data, and replayable pipelines built on retained raw data.
• Every interface needs an owner, a contract, automated lineage, a change and deprecation policy, and security controls proportionate to the data it carries.

The paper defines the field, catalogs the patterns and their trade-offs, explains the interoperability foundations and a layered reference architecture, works through the engineering concerns, sets out governance and testing practices, and closes with a selection framework, metrics and a maturity path. Streaming is treated as one option among several: valuable where decisions depend on fresh data, and an expensive distraction where a nightly load would serve.

Why integration absorbs so much effort

Data creates value when it moves from the system that captures it to the people, processes and models that need it; integration is the work of making that movement reliable.

Every enterprise runs on data that crosses boundaries. A customer record created in a sales application is enriched by a service desk, billed by finance, scored by a risk model and reported to a regulator. Each hop is an integration between two systems designed independently, with their own structures, update cycles and definitions. DAMA-DMBOK treats the design and management of these flows as a distinct knowledge area, Data Integration and Interoperability, alongside data architecture, data quality and metadata management.

Integration consumes a large share of data effort for structural reasons. The number of possible point-to-point interfaces grows with the square of the number of systems, and a large enterprise operates hundreds of applications. Each interface must bridge differences of format, structure, meaning and timing, and must be maintained as both endpoints change; a change to a widely consumed source ripples through dozens of pipelines. Fragile integrations produce stale dashboards, duplicated records, reconciliation breaks and incidents that surface first with the business user, and slow integration delays every initiative that depends on new data.

Definitions and scope

Data integration is the set of processes that move and consolidate data within and between data stores, applications and organizations, transforming it where necessary for use in a new context. Interoperability is the ability of two or more systems to exchange data and to use the data that has been exchanged. Integration is something an organization does; interoperability is a property its systems have. A nightly table copy is integration; two systems exchanging payment messages in a shared standard, each interpreting the fields identically, are interoperable.

Interoperability is usefully described at four levels, summarized in the table below, and each must hold for an exchange to work. The failures that reach the business are more often semantic or organizational than technical.

Frameworks divide the levels differently. The European Interoperability Framework distinguishes legal, organizational, semantic and technical layers and treats syntax as part of semantic interoperability, while HIMSS describes foundational, structural, semantic and organizational levels for healthcare. The four-level split used here is a working model for enterprise data exchange; what matters is that structure, meaning and responsibility are each agreed explicitly.

Flows also differ in latency, the time between a change at the source and its availability at the target. Scheduled batch moves accumulated data at intervals of hours or days; micro-batch shortens the interval to minutes; near-real-time flows deliver changes within seconds, typically through change data capture (CDC) or messaging; and streaming flows process each event as it arrives. Latency is a requirement derived from the decisions the data supports, not a feature to be maximized.

Two further terms recur throughout. An interface is an agreed boundary across which data is exchanged, defined by a transport, a schema, semantics and a change policy; a pipeline is the managed, repeatable process that moves and transforms data across one or more interfaces. The knowledge area spans operational integration between applications, analytical integration into warehouses and lakehouses, exchange with partners and regulators, and the return of curated data to operational tools. It excludes the design of target structures and the operation of platforms, which belong to data modeling and to data observability practice respectively.

Interoperability is agreement made executable: shared transport, shared structure, shared meaning and shared responsibility.

The pattern catalog

Integration patterns are recurring solutions to the problem of moving data between systems, each trading off latency, volume, transformation capability, coupling and operating cost. Mature organizations maintain a small, deliberate portfolio of patterns with a standard implementation for each, rather than letting every team solve the problem afresh. The ten patterns below are compared side by side in a table at the end of this section.

ETL and ELT

Extract, transform, load (ETL) transforms data in a dedicated engine or staging area before loading it into the target; extract, load, transform (ELT) loads raw data into the target platform first and transforms it there with the platform's compute. ETL suits targets with limited compute, requirements to mask sensitive data before it reaches a shared platform, and transformations that need specialized engines. ELT suits cloud warehouses and lakehouses, where retained raw data enables reprocessing and transformations are expressed in SQL and versioned as code, provided that classification and masking are applied at load.

Scheduled batch and micro-batch

Scheduled batch moves accumulated data at fixed intervals, as full snapshots or as incremental extracts keyed on a timestamp or sequence column. It is simple, cheap and adequate for reporting and regulatory uses whose cadence is daily or slower. Its weaknesses are latency, load spikes, and incremental logic that depends on modified timestamps, which many sources do not maintain for deletes or bulk updates. Micro-batch runs the same logic every few minutes, narrowing latency without event semantics; if the consumer acts on each record individually rather than on the set, a streaming pattern is more appropriate.

Change data capture

Change data capture (CDC) delivers the inserts, updates and deletes applied to a source instead of re-reading the whole dataset. Three approaches are in use, and they differ sharply in reliability and in the load they place on the source.

Log-based CDC is the usual way to feed operational changes into analytical platforms and event streams. Its design questions are how the initial snapshot is reconciled with the change stream, how source schema changes propagate, and whether consumers receive raw change events or a materialized current-state table. Open-source connectors such as Debezium are one example of the category. The figure below traces the path of a single change.

Event streaming and publish/subscribe

In event streaming, producers append events to a durable, ordered log organized into topics, and consumers read at their own pace, independently of one another; publish/subscribe is the delivery model in which a message reaches every subscriber of a topic. Distributed logs such as Apache Kafka are the common example. The pattern decouples producers from consumers in time, lets many consumers reuse one feed, retains history for replay and supports stream processing for enrichment and alerting. Its costs are operational skill, duplicate and late-event handling by consumers, ordering that is guaranteed only within a partition, and mandatory schema discipline, since one producer change can break every consumer. It fits event-driven processes and operational analytics, not set-based periodic work such as month-end close.

Messaging and queues

Message queues deliver each message to one consumer from a group, with acknowledgment and redelivery on failure. They remain the right choice for commands and work distribution, where a task such as issuing a payment instruction must be performed once. Queues provide buffering, back-pressure and guaranteed handoff, but not broad reuse or replay, because consumed messages are removed. An enterprise service bus adds routing and transformation around queues and topics; integration platform as a service (iPaaS) products offer similar capabilities as a managed cloud service.

Data virtualization and federation

Data virtualization presents data from several sources through one logical layer that translates queries and combines results at runtime without copying data; query federation is the underlying technique. It suits low-volume, ad hoc or exploratory access, prototyping before a physical pipeline is justified, and cases where residency rules prohibit copying. Its weaknesses are performance bounded by the slowest source, load on operational systems, no retained history and heavy transformations that cannot be pushed down to the sources. It complements physical integration rather than replacing it.

API-based integration

Application programming interfaces (APIs) expose data or operations through request and response calls, usually over HTTPS with representational state transfer (REST) conventions or GraphQL, secured by tokens or certificates. They suit on-demand access to current state, transactional operations, exchange with external parties and self-service access by many consumers. They are poor for bulk movement: paginated extraction of millions of records is slow, rate limited and hard to make consistent. Software as a service (SaaS) vendors often offer APIs as the only access path, which pushes organizations toward connector frameworks and incremental extraction.

Managed file transfer

Managed file transfer (MFT) moves files over secure protocols such as SFTP or AS2, adding scheduling, encryption, integrity checks, non-repudiation, checkpoint restart and audit trails. Files remain the common currency of exchange with partners, banks, regulators and legacy systems, and MFT makes that exchange controlled rather than ad hoc. Its weaknesses are implicit schemas carried in file layouts, batch latency, and the burden of partner onboarding and key management, so file interfaces need written specifications and arrival monitoring.

Replication

Replication maintains a copy of a database or storage volume elsewhere, physically or logically, with little or no transformation, for availability and disaster recovery, read offloading, geographic distribution and reporting replicas. It is fast and faithful but reproduces the source model unchanged, codes and quirks included, so it is a transport mechanism rather than an integration solution. Reporting from replicas of operational systems works until several systems must be combined.

Reverse ETL

Reverse ETL moves curated data from the analytical platform back into operational tools such as customer relationship management, marketing automation or finance applications, so that segments, scores and consolidated attributes are available where people act. It reuses governed definitions but writes into systems of engagement from a platform designed for reads, which raises questions of ownership, conflict resolution and privacy. Each reverse flow should be treated as an interface with a contract, not as a convenience export.

Interoperability foundations

Moving bytes between systems is the easy part. Interoperability requires that the receiver can parse what arrives, interpret it correctly and keep doing so as the sender evolves. Four foundations make this possible: shared models where they are justified, explicit schemas in well-chosen formats, registries that enforce compatibility, and semantic agreement through glossaries, reference data and standards.

Canonical models and their limits

A canonical data model is a system-neutral representation of shared entities such as customer, product or order, into which every producer translates and from which every consumer translates. It replaces the quadratic growth of pairwise translations with linear growth and concentrates semantic decisions in one place. Its limits are well known: an enterprise-wide canonical model tends toward the union of every system's attributes, becomes too large to understand and too slow to change, and forces domains with legitimately different definitions into one compromise. The practical response is to scope canonical models to a domain or a specific exchange, keep them small and versioned, and accept explicit translation at domain boundaries, which domain-driven design describes as a bounded context with a published language.

Schemas and formats

A schema is the formal description of the structure and types of exchanged data, and the choice of encoding format determines how schemas are carried and enforced. The four formats in the table below cover most structured exchange in data platforms. Delimited and fixed-width text files remain common in partner exchange and carry no schema at all, which is why they need written specifications.

Row-oriented binary formats such as Avro and Protobuf suit record-at-a-time movement because each record is compact and can be written and read individually. Columnar Parquet suits analytical storage because queries read only the columns they need and compression is more effective. JSON offers readability and web interoperability at the price of size and type ambiguity, so dates, decimals and large integers need explicit conventions. A common architecture carries events in Avro or Protobuf, lands them in Parquet-based open table formats and exposes JSON through APIs.

Schema registries and compatibility

A schema registry stores versioned schemas for each topic, table or interface, lets producers and consumers reference a schema by identifier, and enforces a compatibility mode when a new version is registered. The modes in common use are set out in the table below.

These names follow the widely used Confluent Schema Registry, in which backward compatibility is the default; other registries offer equivalent modes under similar names. The registry turns a social problem, producers breaking consumers, into a mechanical check that fails the deployment before damage is done. It should be the system of record for interface schemas, integrated with the data catalog, with the mode chosen per interface according to who can upgrade first.

Breaking changes, such as most field renames and incompatible type changes, fail every compatibility check and require a new major version, usually a new topic or interface, published alongside the old one. Formats differ in the detail: Avro can map a renamed field through an alias and promote an int to a long, while Protobuf's binary encoding identifies fields by number rather than by name, which is why its field numbers must never be reused.

Semantic agreement

Schemas guarantee that a field called amount can be parsed; they say nothing about whether it is gross or net, in which currency, or whether refunds are negative. Semantic interoperability requires agreed definitions, units, code lists and identifiers: a business glossary for definitions, managed reference data for code lists such as ISO 3166 country codes and ISO 4217 currency codes, master data for shared identifiers, interface documentation that binds each field to a glossary term, and ISO 8601 timestamps that carry an explicit offset from Coordinated Universal Time (UTC). Where these are weak, every consumer re-derives meaning independently, and reports disagree.

Industry standards

Industry standards show semantic agreement at scale. ISO 20022, the universal financial industry message scheme, provides a common modeling method, a central repository of business processes and message definitions, and XML as its principal syntax; it is used for payments, securities, trade services and cash management, and payment systems in many markets have adopted it or are migrating to it. HL7 Fast Healthcare Interoperability Resources (FHIR) defines healthcare data as modular resources, such as Patient, Encounter and Observation, exchanged through web APIs in JSON, XML or RDF; Release 4 is the most widely implemented. Electronic data interchange (EDI), in the ANSI ASC X12 and UN/EDIFACT families, standardizes supply chain documents such as purchase orders, invoices and advance ship notices exchanged with trading partners over AS2 or managed file transfer.

Adopting any of them is a project rather than a format choice. Each allows optional elements and local extensions, so communities publish implementation guides or profiles that narrow the options, and two compliant parties can still fail to interoperate without agreeing a profile. Mapping internal models to the standard and validating messages against the profile are integration work in their own right.

Pipeline reference architecture

Whatever patterns an organization chooses, the pipelines that implement them share a common anatomy. Naming the components explicitly lets teams standardize each one, reuse platform services and locate failures quickly. The figure below shows the layers from ingestion at the bottom to serving at the top, with monitoring and metadata running across all of them.

Ingestion connects to sources and brings data across the boundary with the least possible interpretation. A standard connector library for databases, files, APIs, SaaS applications and event sources, with consistent handling of credentials, retries and checkpoints, is the most reusable platform investment. Every batch or event should carry provenance: source, extraction time, connector version and checkpoint position.

The landing zone stores data as received, immutable and partitioned by arrival time, with retention long enough for replay and audit; lakehouse conventions call this the bronze layer. Unchanged raw data is what makes reprocessing possible when a transformation bug or a late rule change is discovered. A staging area holds data in transit; it is transient and not for consumers.

Orchestration manages dependencies, schedules and triggers, retries with back-off, alerting and the backfill of historical partitions. Code-first orchestrators express pipelines as directed acyclic graphs, which brings version control and review to scheduling logic. Use time triggers where sources are periodic and event triggers where data arrives irregularly; mixing the two without clear rules produces runs on empty inputs and missed late files.

Transformation converts raw data into curated structures: standardizing types and codes, applying business rules, conforming dimensions and building the models that consumers query. Transformations should be declarative where possible, versioned, tested and separated from movement logic, so that the same rule is not re-implemented in every pipeline; frameworks that compile SQL models with dependency tracking, such as dbt, are one example of the category.

Quality gates check data against expectations before release, and quarantine holds records or batches that fail. Gates test schema conformance, mandatory fields, referential integrity, ranges, volume against expected bounds and freshness. The failure policy must be explicit: reject the batch, quarantine the failing records and release the rest, or release with a warning flag. A circuit breaker that halts propagation when failure rates exceed a threshold stops one bad load from contaminating many products.

Serving exposes curated data in the forms consumers need, including tables and views, APIs, topics, files and semantic models, each with access control. Monitoring and metadata run across every layer, capturing lineage, run history, freshness, volume and schema changes and publishing them to the catalog and to alerting; a pipeline without monitoring is only presumed to work.

Engineering for reliability

The difference between an integration that runs quietly for years and one that generates weekly incidents lies in a handful of engineering concerns that every pattern must address. Each is cheap to design in and expensive to retrofit.

Idempotency and delivery semantics

An operation is idempotent when applying it more than once produces the same result as applying it once. Idempotent pipelines can be retried and replayed safely, which is the foundation of reliable operations: load with merge or upsert semantics keyed on a stable business or surrogate key, derive deterministic identifiers rather than generating them at load time, and design event consumers to tolerate a repeated event. Append-only loads without a key are a frequent source of duplicates in analytical platforms.

Delivery semantics describe what a transport guarantees. At-most-once delivery never redelivers, so messages can be lost; it suits telemetry where gaps are acceptable. At-least-once delivery retries until acknowledged, so messages can be duplicated; it is the default for durable systems and requires idempotent consumers. Exactly-once processing is achievable within a system that controls both the log and the processing state: Apache Kafka, for example, combines idempotent producers with transactions that commit consumer positions together with the output. Delivery into other systems requires their cooperation, so across system boundaries the achievable goal is effectively-once processing: at-least-once delivery combined with idempotent handling and deduplication, so that the outcome is the same as if each event had been processed once.

Ordering, lateness and replay

Ordering is guaranteed only where it is designed. Partitioned logs preserve order within a partition, so records that share a partition key, such as an account identifier, stay in order while records with different keys do not; CDC delivers events in commit order from one database log. Consumers that need the current state of each entity should rely on a version or sequence number in the payload rather than on arrival order. Global ordering across an enterprise is impractical and rarely needed once the right key is chosen.

Late-arriving and out-of-order data is normal: devices reconnect, partners resend files, upstream batches fail and rerun. Stream processors work in event time rather than processing time, use watermarks to estimate how late data may still arrive, and route events that miss the allowed lateness window to a side output for correction. Batch systems recompute the last few partitions on each run, and dimensional models handle a fact that arrives before its dimension row with an inferred placeholder member that is completed later.

Backfills and replays reload history after a bug fix, a rule change or the arrival of a new consumer. They are safe only when raw data is retained, transformations are deterministic and idempotent, and outputs are partitioned so that a range can be recomputed without touching the rest. Replays must respect log retention and downstream capacity; a replay that floods an operational consumer is an incident in its own right, so every pipeline should document how it is backfilled before it reaches production.

A pipeline that cannot be replayed cannot be repaired; it can only be patched.

Change, duplicates, time and cost

Slowly changing data describes attributes that change occasionally, such as a customer's address. Dimensional modeling distinguishes type 1 handling, which overwrites the value; type 2, which adds a new row with effective dates so that history is preserved; and type 3, which keeps the previous value in an additional column. Integration must capture enough information, including effective dates where available, for the target to implement the chosen type. Bitemporal designs also record when the system learned of each change, which regulatory reporting often requires.

Deduplication removes repeats introduced by retries, resent files and overlapping extraction windows. It needs a reliable key, which may be a business key, a source key with a sequence number or a hash of the payload, and a window over which duplicates are checked. Time zones cause silent errors: store instants in UTC with the originating offset preserved, distinguish business dates from timestamps, and test daylight saving transitions. A figure that moves by one day at a time zone boundary is a classic reconciliation break.

Latency versus cost is the final trade-off. Lower latency raises platform and operating cost and narrows the pool of engineers able to support the pipeline. The right latency is derived from the cadence of the decision the data serves, expressed as a service level objective (SLO) and reviewed when needs change; a latency budget allocated across capture, transport, transformation and serving shows where extra effort buys little.

Governing integrations

Integration governance keeps interfaces owned, understood and safe as they multiply. It is light when embedded in the platform and the delivery process, and burdensome when applied as after-the-fact review. Six elements matter: ownership, contracts, documentation and lineage, change management, security and auditability.

Ownership and contracts

Every interface and pipeline needs a named owner. The producing domain owns the interface it publishes, including schema, semantics, quality and change policy, just as it owns the application behind it. The pipeline owner, usually a data engineering or platform team, is accountable for operation, monitoring and incident response, and the business data owner remains accountable for appropriate use wherever the data flows. Pipelines orphaned by reorganizations are where incidents accumulate first.

An interface contract makes the producer's commitments explicit: schema and types, field definitions tied to the glossary, quality guarantees, freshness and availability objectives, support channel, classification, and the policy for change and deprecation. Contracts live in version control beside the code that fulfills them and are enforced by the schema registry and automated checks; the same discipline underpins the data products of a data mesh.

Lineage and change management

Documentation and lineage should be captured automatically from orchestration, transformation and ingestion tooling, because hand-written documentation decays within months. Open standards such as OpenLineage, together with catalog integrations, register every pipeline, its inputs, outputs, owner and schedule as a by-product of deployment. Lineage turns a schema change at a source into an impact list rather than a surprise, and it is the evidence regulators expect when they ask how a reported figure was derived.

Change management follows a simple discipline. Additive changes are made compatibly and announced; breaking changes create a new version published in parallel with the old one; consumers receive notice and a deprecation window long enough to migrate; and the old version is retired on a published date once usage monitoring confirms that no consumer remains. The window length belongs in the contract as a known term, not a negotiation at each change.

Security and auditability

Security controls travel with the data. Encrypt data in transit with Transport Layer Security (TLS 1.2 or later, preferably TLS 1.3) and at rest in landing zones and queues. Keep credentials and keys in a secrets manager with rotation, never in code, configuration files or orchestration variables. Run each pipeline under its own service identity with least-privilege access to exactly the sources and targets it needs. Propagate classification labels so that masking, row-level restrictions and retention rules apply at the target, and move only what consumers need; copying whole tables for five columns enlarges both the attack surface and the compliance scope. Article 32 of the General Data Protection Regulation (GDPR) requires security measures appropriate to the risk, and the Article 30 record of processing activities depends on knowing where personal data flows.

Auditability requires that every run leaves evidence: what ran, when, with which code version, reading which inputs and producing which outputs, with record counts and checksums that allow source-to-target reconciliation, retained for as long as the data is subject to audit. For the organizations they cover, two regimes make this evidence mandatory rather than merely good practice: Sarbanes-Oxley (SOX) requirements for internal control over financial reporting, and the Basel Committee's principles for risk data aggregation (BCBS 239), which expect risk data to be aggregated on a largely automated basis and reconciled with source and accounting data.

Testing, monitoring and anti-patterns

Integrations deserve the testing discipline applied to application code. Five kinds of test cover most needs, and all of them should run in a continuous integration pipeline on every change.

• Unit tests check transformation logic against small fixtures.
• Contract tests verify that a producer's output still satisfies its published schema and consumer expectations.
• Integration tests run the pipeline end to end against masked or synthetic data.
• Reconciliation tests compare counts, sums and key sets between source and target after each run.
• Regression tests on reprocessing compare new outputs with the previous version of the same partition and require every difference to be explained.

Monitoring in production watches freshness against the contract, volume against expected bounds, schema drift, distribution shifts in key fields, run duration and failure rates, with alerts routed to the pipeline owner and status visible to consumers through the catalog. Every alert needs a runbook. Monitoring is part of a pipeline's definition of done, not an afterthought.

Certain anti-patterns recur in nearly every large organization. They are rarely the result of poor engineering; they emerge from local optimization under delivery pressure and persist because nobody owns the whole picture, and most remedies are organizational before they are technical.

Selecting patterns

Pattern selection should be a repeatable decision rather than a preference. Eight criteria cover most cases.

• Latency. How fresh the data must be for the consuming decision.
• Volume and change rate. How much data moves and how often it changes.
• Source capability. What the source can offer: log access, an API, an event feed or only files.
• Transformation complexity. How much reshaping, joining and rule logic the flow requires.
• Consumer breadth. The number and variety of consumers that will use the data.
• Consistency. Whether consumers need transactional snapshots or can tolerate eventual consistency.
• Operational maturity. Whether the team can run the pattern reliably over time.
• Total cost. The cost of running the flow for its expected life, including staffing.

The matrix below collapses the two most discriminating criteria, latency and breadth of consumption, into a starting recommendation that the other criteria refine.

When streaming pays

Moving from batch to streaming pays where the business acts on individual events within minutes, where sources can emit changes through log-based CDC or native events, and where consumers can handle event semantics: fraud and anomaly detection, inventory and order status, personalization, operational alerting and the synchronization of many downstream systems from one source. It does not pay for periodic, set-based processes such as financial close or regulatory batch submissions, where the nightly load is cheaper and easier to audit. Where it does pay, a staged migration limits risk.

1. Start from a constrained decision. Choose a consumer decision demonstrably constrained by latency, and agree the latency objective with that consumer.
2. Capture changes once. Use log-based CDC or native events at the source, feeding a durable log governed by a schema registry and a contract.
3. Protect existing consumers. Let batch consumers read from tables materialized from the log, so that nothing breaks and the source is read only once.
4. Add stream processing selectively. Build it only for the consumers that need it, with idempotent sinks and explicit handling of late events.
5. Retire the legacy extract. Switch it off when its consumers have moved and reconciliation between the two paths has run clean for an agreed period.

Measuring progress

Metrics

Integration should be measured as a capability, not pipeline by pipeline. The metrics below balance reliability, reuse, governance and cost, and are best reported quarterly to the architecture and governance forums as trends rather than single values.

Maturity path

Progress follows a recognizable sequence, and the sequence matters: organizations that attempt event-driven integration before interfaces have owners and contracts recreate point-to-point spaghetti at higher speed. Each stage has an entry condition that the previous stage establishes.

Standardization requires an agreed pattern portfolio and a platform team that owns the shared tooling. Governance requires the interface register, the schema registry and automated lineage, together with the willingness to refuse new point-to-point interfaces. Product orientation requires producing domains to accept ownership of what they publish, with funding to support it, and event-driven integration requires all of the above plus the operational maturity to run streaming infrastructure. Most organizations are at different stages in different domains, which is acceptable as long as the direction is shared.

Conclusion

Data integration and interoperability will remain a large share of data work because it is where the independent decisions of many system owners meet. Organizations that manage it well do not eliminate that work; they make it repeatable, by choosing from a deliberate portfolio of patterns, agreeing structure and meaning through schemas, registries, glossaries and standards, building pipelines that are idempotent, replayable and monitored, and giving every interface an owner and a contract.

Streaming belongs in that portfolio, not above it. Capturing change once at the source and serving both event-driven and periodic consumers from it is the direction many enterprises are taking, and it rewards those who have first put ownership, contracts and lineage in place. Four practical steps start the work.

1. Inventory existing interfaces. Build an interface register that records producers, consumers, transport and schedule for every flow that crosses a system boundary.
2. Name owners. Assign a producing domain and a pipeline owner to each interface, starting with the most widely consumed.
3. Standardize the supported patterns. Publish the pattern portfolio with one sanctioned implementation of each, and route new integration requests through it.
4. Bring the most widely consumed interfaces under contract. Enforce their schemas through the registry, capture lineage automatically and publish change and deprecation terms.

Glossary

Canonical data model

A system-neutral model of shared business entities used as the common form for exchange among many systems.

Change data capture (CDC)

A technique that identifies and delivers the inserts, updates and deletes applied to a source, most reliably by reading its transaction log.

Compatibility mode

A schema registry rule, such as backward, forward or full, that constrains how a new schema version may differ from earlier ones.

Data virtualization

A logical layer that presents data from several sources through one interface, translating queries and combining results at runtime without copying the data.

Delivery semantics

The guarantee a transport offers about whether a message may be lost or duplicated: at-most-once, at-least-once or exactly-once.

Effectively-once processing

At-least-once delivery combined with idempotent handling and deduplication, so that the outcome matches processing each event exactly once.

ELT

Extract, load, transform: loading raw data into the target platform and transforming it there.

ETL

Extract, transform, load: transforming data in a separate engine or staging area before loading it into the target.

Event streaming

Publishing immutable events to a durable, ordered log from which many consumers read independently and can replay.

Idempotency

The property that applying an operation more than once has the same effect as applying it once.

Interface contract

An explicit, versioned agreement between a producer and its consumers covering schema, semantics, quality, service levels and change policy.

Interoperability

The ability of two or more systems to exchange data and to use correctly the data that has been exchanged.

Managed file transfer (MFT)

Controlled file exchange over secure protocols with scheduling, encryption, integrity checks and audit trails.

Quarantine

A holding area for records or batches that fail quality gates, pending correction or release.

Reverse ETL

Moving curated data from the analytical platform back into operational tools such as customer relationship management or marketing systems.

Schema registry

A service that stores versioned schemas for interfaces and enforces compatibility rules when schemas change.

Watermark

In stream processing, an estimate of how far event time has progressed, used to decide when a window can be closed.