Chains of synchronous calls couple teams and fail together, and nightly batch is too slow for processes that must react as things happen. Event-driven architecture decouples producers from consumers by exchanging facts, commands and streams through a broker: a producer records a fact once, and consumers react at their own pace. The decoupling is real, but it is paid for in failure modes of its own, from duplicates, reordering and lost or poison messages to dependencies that nobody can see.
This paper sets out how to design asynchronous application programming interfaces (APIs) deliberately. It explains the dimensions of coupling and when to use request-response, queues, events or streams; how to design events with a common envelope, stable identity, ordering keys and classified payloads; how to describe and evolve contracts with the AsyncAPI specification and schema registries; which delivery semantics and patterns make each effect happen once; and how to run and govern the broker as a platform.
Written for Enterprise and integration architects, platform and streaming leads, and senior engineers who design integration between services, systems and partners.
Executive summary
Asynchronous integration lets services, systems and partners cooperate without waiting for one another. A producer publishes a fact or a command, a broker holds it, and consumers act on it when they can. That arrangement removes the two couplings that make call chains fragile, the need for both parties to be running at the same moment and the need for the sender to know who receives, but it introduces new obligations. Messages arrive twice, out of order, late or not at all, and consumers that used to be visible as callers become invisible subscribers.
This paper argues that each of these failure modes has a known design answer, and that the answers should be platform defaults rather than discoveries made in production.
- Interaction style is chosen per requirement; most estates use request-response, queues and streams side by side.
- Events are designed with a common envelope (CloudEvents), stable identity, ordering keys, past-tense names and classified payloads.
- Contracts are described in AsyncAPI, registered in a schema registry under a compatibility mode, evolved additively and tested on both sides.
- Delivery is assumed to be at least once. Exactly-once effects come from the transactional outbox, idempotent consumers with an inbox, and idempotency keys on external side effects, not from the broker alone.
- The broker is run as a governed platform, and governance rests on ownership: the producer owns each event's schema and meaning, and consumers register before they subscribe.
The closing sections cover the pitfalls that undo the decoupling, a four-stage maturity path and a 90-day plan that produces working evidence rather than a program plan.
Synchronous chains, slow batch and new failure modes
Most integration estates grew in two directions at once. Online paths became chains of synchronous calls, each service calling the next, so that a slow dependency three hops away stalls the user, every release must be coordinated, and an outage anywhere becomes an outage everywhere. Offline paths stayed on nightly files, so the business learns about orders, payments and exceptions hours after they happened.
Call chains couple teams and fail together; batch is too slow; events decouple, but only when their new failure modes are designed for.
Event-driven architecture answers both problems, and it brings failure modes that call chains and files never had. None of them is exotic, and each has a known design answer, summarized below and developed in the rest of this paper.
| Failure mode | How it appears | Design answer |
|---|---|---|
| Duplicates | The same message delivered twice after a retry, rebalance or replay | Idempotent consumers keyed on event identity; an inbox table |
| Reordering | Events for one entity processed out of sequence | An ordering key per entity; consumers that tolerate or re-sort |
| Lost messages | A state change committed but never published | The transactional outbox and at-least-once delivery |
| Poison messages | One malformed message blocks a queue or partition | Bounded retries and an owned dead-letter queue |
| Invisible dependencies | Nobody knows who consumes a topic or what a change will break | A catalog entry per event type, with registered consumers |
| Schema drift | Payloads change shape and break consumers in production | A schema registry with a compatibility mode checked in the pipeline |
Why asynchronous
Asynchronous integration is a family of styles rather than one technique, and choosing among them is a choice about which kinds of coupling to remove.
Four interaction styles and four dimensions of coupling
Request-response
The caller waits for an answer; both sides must be up at once (HTTP, gRPC)
Commands on queues
A message asks one receiver to act; the queue holds it until a worker takes it
Events on topics
A producer publishes a fact; any number of consumers react, now or later
Streams and logs
An ordered, retained log; consumers read at their own pace and can replay
Tightest couplingLoosest coupling
Request-response is the most coupled style: both sides must be up, the caller knows the callee's address, and the caller waits. Commands on queues remove the waiting and the need for simultaneous availability, though the sender still intends one receiver to act. Events on topics remove knowledge of the receiver entirely, because the producer publishes a fact in the past tense and any number of consumers react. Streams add retention and replay. Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns supplies the underlying vocabulary: a command message asks a receiver to do something, an event message announces that something happened, and a document message simply transfers data.
| Dimension | The question it asks | What removes or manages it |
|---|---|---|
| Temporal | Must both parties be running at the same moment? | Queues and logs remove it |
| Location | Must the sender know who receives, and where? | Topic names replace addresses |
| Contract | How much of the message's shape and meaning does each side depend on? | Schemas, compatibility rules and envelopes manage it |
| Technology | Must both sides share a protocol, language or library? | The broker and standard protocol bindings mediate it |
Messaging removes temporal and location coupling. Contract and technology coupling remain, which is why schemas, envelopes and brokers exist: they cannot remove a consumer's dependence on what a message means, but they can make that dependence explicit, versioned and checkable.
Choosing an interaction style by requirement
No style is best. Most estates use all three mechanisms below, chosen per interaction and recorded in the contract.
| Requirement | Request-response | Queues and topics | Streams and logs |
|---|---|---|---|
| Answer latency | Immediate answer required | Acknowledged now, processed soon | Processed in order, at the consumer's pace |
| Consistency | Strong within one call | Eventual; design for the delay | Eventual; replay rebuilds state |
| Fan-out | One caller, one callee per call | Topics deliver one fact to many | Many consumer groups read the same log |
| Auditability and replay | Only what the server logs | Queues typically delete on acknowledgment | Retained log; replay and backfill |
| Consumer autonomy | Caller depends on callee uptime | Consumers run at their own cadence | Consumers choose position and pace |
| Failure isolation | Failures cascade up the chain | Broker buffers; dead letters isolate | Lag grows; producers keep writing |
| Typical use | Queries, validations, user actions | Work distribution, notifications | Analytics, projections, integration at scale |
The decision usually turns on one question: does the caller need an authoritative answer now? A validation does, so it is a call. Work that must be done but not answered immediately, such as generating a document, is a command on a queue. A fact that matters to several parties, such as an order being placed, is an event on a topic. Where order, retention and replay matter, as for building read models or feeding analytics, a stream keeps the log so that new consumers can rebuild state from history.
The two broker mechanisms are built differently. Queue brokers, such as those implementing the Advanced Message Queuing Protocol (AMQP), and cloud queue services typically delete a message once it is acknowledged, while log-based platforms, such as Apache Kafka and cloud streaming services, retain an ordered log that each consumer reads from its own position. Martin Kleppmann draws the same distinction in Designing Data-Intensive Applications.
Living with eventual consistency
Eventual consistency is the price of decoupling, and it is a product question before it is a technical one.
For users and interfaces
- Show the accepted state honestly: received before confirmed, with a time expectation
- Read your own writes from a source that has the change, or echo it locally
- Make pending states first-class, with a status check and a completion notice
- Never present a derived view as authoritative for money, stock or eligibility
- Deliver final answers as notifications, not by polling a lagging view
For business processes and controls
- Enforce cross-service invariants by detection and correction: reconcile, then compensate
- State the lag each process tolerates and alert when it is exceeded
- Give operational reports a current-as-of timestamp
- Treat compensations as business actions with owners, limits and audit trails
- Where no delay is tolerable, keep the data in one service and one transaction
For users, the interface should tell the truth about state. An order is received before it is confirmed, and the confirmation arrives as a notification, because polling a view that may lag teaches users to distrust it. For business processes, invariants that span services are enforced by detection and correction: reconciliation finds the discrepancy and a compensation corrects it. Because a compensation such as a refund is a business action, it needs an owner, limits and an audit trail, and stating the lag each process can tolerate turns lag into a business metric. Where no delay is tolerable, the service boundary is in the wrong place: the data belongs inside one service and one transaction.
Designing events
An event is a contract between teams that may never meet, so its kind, envelope, name, keys and payload deserve the same care as a public API.
Four kinds of event and their trade-offs
| Kind | What it carries | Strength | Trade-off |
|---|---|---|---|
| Notification | An identifier and the fact; consumers call back for detail | Small, stable, hard to misuse | Callbacks restore coupling and load |
| Event-carried state transfer | The fact plus the state consumers need | No callback; consumers work autonomously | Larger payloads can leak models; copies go stale |
| Domain event | A business fact in the domain's language | Meaningful, reviewable, long-lived contract | Needs modeling discipline and owners |
| Change-data-capture event | A row-level change read from the database log | No producer code; complete and ordered | Exposes tables, not intent; schema drift |
Martin Fowler's article What do you mean by "Event-Driven"? separated four patterns that are often conflated: event notification, event-carried state transfer, event sourcing and Command Query Responsibility Segregation (CQRS). The table uses the first two, adds the domain event of domain-driven design, and adds change data capture (CDC) as a common event source; event sourcing and CQRS are persistence and query patterns, discussed later.
The choice is mostly about coupling. Notifications keep payloads small but invite callbacks, which put load and coupling back on the producer. Event-carried state transfer lets consumers work from local copies even when the producer is down. Domain events make the best contracts between teams, and collaborative modeling techniques such as event storming supply the discipline they need. CDC events, read from the database log by tools such as Debezium, suit replication and analytics better than cross-team contracts. Whatever the kind, start thin and add only the fields consumers demonstrably need.
A common envelope: CloudEvents
A common envelope lets every consumer, router and tool handle event metadata the same way, whatever the payload. CloudEvents, a graduated project of the Cloud Native Computing Foundation (CNCF), defines that envelope; its specification version is 1.0.
CloudEvents envelope
Common metadata around any payload
id
Required; unique within its source
source
Required; the context that produced the event
specversion
Required; 1.0
type
Required; reverse-DNS prefix, past tense
time
Optional; when the occurrence happened
subject
Optional; the entity the event is about
dataschema
Optional; a link to the payload schema
data
The payload, typed by datacontenttype
Optional attributes and extensions add what the four required attributes do not: the partitioning extension, for example, defines a partitionkey attribute, and the Kafka protocol binding describes an opt-in key mapper that copies it into the Kafka record key. Protocol bindings map the envelope onto HTTP, Kafka, AMQP, MQTT, NATS and WebSockets, among others, so the same event can cross transports without losing its metadata.
Identity is the attribute most often misunderstood. The specification requires that the combination of source and id be unique for each distinct event, allows a producer to resend a duplicate with the same id, and lets consumers treat events with identical source and id as duplicates. The deduplication key is therefore the pair, not the id alone, and producers must reuse the id when they retry.
The last rule matters because a fact in a retained log is still personal data under the General Data Protection Regulation (GDPR): storage limitation in Article 5 and the right to erasure in Article 17 apply to topics as much as to tables. Keeping personal data out of long-retained topics, or encrypting it with per-subject keys that can be destroyed, are common ways to honor erasure in an append-only log.
Contracts and schemas
An event contract has three parts: a description of the channels and messages, a schema for each payload, and rules for how both may change. Each part can be checked by a machine, and each should be.
From AsyncAPI document to enforced contract
Describe first
AsyncAPI document before producer code: servers, channels, operations, messages, examples and an owner
Define schemas
Each payload referenced as JSON Schema, Avro or Protocol Buffers, with the envelope alongside
Register
Schema published to the registry under a compatibility mode; breaking changes rejected
Lint and review
Naming, envelope and classification linted on every commit; review only for new types
Publish and test
Channel added to the catalog with owner and consumers; contract tests on both sides
The AsyncAPI specification does for event-driven interfaces what OpenAPI does for HTTP: it describes servers, channels, the operations an application performs on them and the messages exchanged, with headers, payload schemas and examples. In version 3 of the specification, operations are declared separately from channels, each with an action of send or receive, which makes publish-subscribe, request-reply and streaming interactions explicit. AsyncAPI is an open specification under the Linux Foundation, and several of the open source linters used for OpenAPI also check AsyncAPI documents.
Writing the document before producer code is the asynchronous form of API-first design. Payload schemas are referenced rather than embedded, so one definition serves the registry, generated code and documentation. Registering the schema under a compatibility mode makes the pipeline, not a reviewer, reject incompatible changes. Contract tests close the loop: producer tests validate emitted messages against the registered schema and examples, and consumer tests replay recorded sample messages against the consumer's handler.
Schema formats and compatibility modes
| Aspect | JSON Schema | Apache Avro | Protocol Buffers |
|---|---|---|---|
| Encoding | Self-describing JSON text; larger on the wire | Compact binary; the writer's schema is held in a registry | Compact binary; field numbers, not names, define the wire format |
| Evolution | Conventions, unless a registry enforces them | Writer and reader schemas resolved at read time; defaults make additive change explicit | Fields survive renames; removed field numbers are reserved, never reused |
| Validation | Rich: formats, patterns, enumerations, ranges | Types, unions and defaults | proto3 has no required fields; validation lives in code |
| Natural fit | HTTP, webhooks and polyglot JSON consumers | Log-based streaming and data platform ingestion | Estates that already generate gRPC clients and servers |
No schema format is right for every estate. AsyncAPI's default schema format is a superset of JSON Schema, and the specification also accepts Avro, Protocol Buffers and other formats when the schema format is declared. The practical consequence of Avro's design is that a message cannot be decoded without its writer's schema, which makes the registry part of the data path; Protocol Buffers trade some validation for one definition that serves both gRPC services and messages.
A schema registry, whether open source or a managed cloud service, stores every version of every schema and enforces a compatibility mode when a new version is registered.
| Mode | What it guarantees | Who upgrades first |
|---|---|---|
| Backward | Consumers using the new schema can read data written with the previous one | Consumers |
| Forward | Consumers using the previous schema can read data written with the new one | Producers |
| Full | Both backward and forward compatible with the previous schema | Either side, independently |
| Transitive variants | The same check against every earlier version, not only the last | As for the base mode |
Backward compatibility is the default in Confluent Schema Registry, a widely used implementation. The transitive variants matter for streams with long retention, where a consumer may replay data written under any earlier version, not just the previous one.
Evolving an event
Compatible change, same event type
- Add optional fields with defaults; never change a field's type, unit or meaning in place
- Consumers are tolerant readers: they ignore unknown fields and fail only on fields they use
- The registry's compatibility mode checks every schema change in the pipeline
- Each change is recorded in the changelog and catalog, and registered consumers are notified
- Examples and the AsyncAPI document stay in step with the schema, or the build fails
A new type or major version
- A field is removed, renamed or retyped, or its meaning changes, such as gross to net
- The ordering key or identity rules change
- The change is named explicitly: a new type, or a version such as
billing.invoice.issued.v2 - Old and new run side by side, usually by dual-publishing, with consumers tracked per version
- The old type is deprecated with dates and retired when no registered consumer reads it
Evolution rules preserve the decoupling that events promised. Meanings never shift in place, because a field whose meaning changes silently is worse than a field that disappears: consumers keep running and produce wrong results. A change to the ordering key or identity rules needs a new type for a related reason, since consumers would otherwise partition or deduplicate wrongly. Retiring the old type follows the same discipline as retiring a synchronous API version: notice periods, sunset dates in the catalog and retirement only when usage reaches zero.
Delivery semantics and patterns
Delivery guarantees describe what the broker promises; everything beyond that is the application's responsibility. Getting the boundary right is the difference between a system that occasionally sends two emails and one that occasionally sends two payments.
Delivery guarantees and what each demands of the consumer
| Guarantee | What the broker provides | What the consumer must do | Fits |
|---|---|---|---|
| At most once | Fire and forget; no redelivery after a failure | Tolerate loss | Metrics, telemetry, low-value signals |
| At least once | Redelivery until acknowledged; duplicates follow | Deduplicate by event identity; make effects idempotent | Most business events |
| Exactly once, broker scope | Transactional reads and writes within one broker | Stay inside the broker; external effects can still repeat | Stream processing pipelines |
| Effectively once, end to end | At-least-once delivery plus ordering within a key | Outbox, inbox, idempotency keys, deduplication store | Payments, orders, inventory |
At least once is the right default for business events, and it obliges every consumer to deduplicate and to make its effects idempotent, because over time duplicates are certain. Some brokers offer exactly-once semantics within their own scope. Apache Kafka, for example, combines an idempotent producer with transactions that write atomically across partitions and commit consumer offsets in the same transaction, which stream processing pipelines use for read-process-write work. The guarantee stops at the broker's edge, and the feature's designers are explicit that side effects on external systems are not covered: an email, a payment or a call to another service can still repeat.
Effectively-once processing end to end is therefore an application property, built from at-least-once delivery, ordering within a key, the outbox and inbox patterns and an idempotency key on every external side effect. Kleppmann explains why duplicate suppression has to work end to end, with an operation identifier that travels from the original request to the final effect.
Exactly-once effects across systems come from idempotent consumers and deduplication, not from the broker alone.
Transactional outbox and inbox, end to end
- Producer
- Relay
- Broker
- Consumer
- External system
- Producer to its outbox table
State change and event row written in one local transaction
- Relay to broker
A poller or CDC relay publishes the event, retrying until acknowledged
- Broker to consumer
Delivered at least once to every subscribed consumer group
- Consumer to its inbox table
Event identity recorded with the business change; a duplicate is skipped
- Consumer to external system
Irreversible effects carry the event identity as an idempotency key
- Consumer to broker
Acknowledge or advance the position only after the local commit
A state write and an event publish are two writes, and a crash between them leaves a change unpublished or an event without its change. The transactional outbox removes the dual write by inserting the event into an outbox table in the same local transaction as the state change; a relay, either a poller or a change-data-capture process reading the database log, then publishes it. The inbox is the consumer-side mirror, detecting a duplicate before business logic runs.
Two details make the pattern hold. Effects that cannot roll back, such as a payment instruction, carry the event identity as an idempotency key to the external system, so that a retried call is recognized there too. And because the consumer acknowledges only after its local commit, a crash at any point replays safely. The inbox needs a retention window at least as long as the longest plausible redelivery, including deliberate replays.
Six operating patterns
Ordering keys and their limits
Partition by entity key for per-key order; across keys there is none. Retries, dead letters and rebalances break order unless consumers tolerate it.
Retries with backoff
Retry transient failures with exponential backoff and jitter, cap the attempts, and never retry a validation error; retries multiply load downstream.
Dead letters and poison messages
Park a message that fails every attempt with its error and context; alert, triage with an owner, fix or discard, then replay deliberately.
Backpressure and consumer lag
Consumers pull at their own pace and lag is the unprocessed work. Set lag objectives, scale consumers by partition and shed load upstream.
Request-reply over messaging
A correlation identifier and a reply channel turn a command into a conversation; use it only where an answer is needed and the caller can wait.
Webhooks for external consumers
HTTP callbacks for partners: signed with a shared secret and a timestamp, retried with backoff, idempotent by event identity and replayable from a delivery log.
Ordering holds only within a partition or a first-in, first-out queue, so the choice of ordering key decides what stays in sequence. A message that fails every attempt is a poison message, and without a dead-letter queue it blocks everything behind it. Request-reply over messaging is useful but should stay rare, because it reintroduces waiting. Webhooks extend events to partners over HTTP, and signing each delivery over the payload and a timestamp lets receivers verify its origin and reject replays.
Sagas, CQRS and event sourcing
| Pattern | What it is | Use it for | Avoid it when |
|---|---|---|---|
| Saga | A long operation run as local transactions with compensating actions, coordinated by choreography or a process manager | Business processes that span services without distributed transactions | An invariant must hold at once; move the boundary instead |
| CQRS | A write model separated from read models fed by events | Fast, tailored queries without burdening the owning service | The service is simple; a second model doubles the upkeep |
| Event sourcing | State stored as an append-only stream of events and rebuilt by replay | Complete history, audit trails and temporal queries | Record keeping is simple or the team is new to the style |
These three patterns are often adopted together but should be chosen separately. Hector Garcia-Molina and Kenneth Salem introduced the saga for long-lived database transactions; in services, coordination is either choreography, where services react to one another's events, or orchestration by a process manager that issues commands, and either way sagas need timeouts, idempotent steps and visible state. CQRS, a name usually credited to Greg Young, buys fast queries at the price of eventual consistency and projections that must be rebuildable, each with an owner and a lag objective. Event sourcing brings hard problems of its own: evolving old events, snapshots and erasure. It is an internal persistence choice, and its events should not become the integration contract by default; integration events are published on purpose.
Platform and operations
The broker is shared infrastructure and should be run as a product: a platform with a topology chosen on purpose, defaults that teams inherit and procedures rehearsed before they are needed.
Broker topology and tenancy
Isolation required by regulation, tenants or blast radius
Dedicated clusters per workload
Separate brokers for regulated or tenant-specific workloads; simple ownership, duplicated operations
Per-domain clusters under standards
Each domain runs its own cluster under central standards; a relay carries cross-domain events
One shared cluster
Namespaces and quotas suffice; invest in conventions and automation first
Central multi-tenant cluster
A platform team runs namespaces, quotas, access policies and chargeback; common in large estates
Domains and teams sharing the platform: few to many
Two forces decide topology: how many domains and teams share the platform, and how much isolation regulation, tenants or blast radius demand. A small estate is well served by one shared cluster, where conventions and automation pay off before more infrastructure does, and as teams multiply, a central multi-tenant cluster run by a platform team becomes the common shape. A central topology offers one policy point and one view of traffic; per-domain clusters offer autonomy and smaller failure domains at the cost of drift. In every shape, topic configuration is code, applied by the pipeline.
Topic governance, security and operations
| Concern | Practice | Why it matters |
|---|---|---|
| Naming and ownership | Names follow domain.entity.fact; one owning team per topic | Findable, and someone answers when it breaks |
| Classification and retention | Personal data labeled in the schema; retention set per topic | Privacy obligations follow the data into the log |
| Quotas and capacity | Quotas per producer and consumer; partitions planned | One noisy tenant cannot starve the rest |
| Identity and access | Workload identity; read and write rights granted per topic | Least privilege at the topic, not the cluster |
| Encryption and replay access | TLS in transit, encryption at rest; replay rights granted separately | Replay exposes history, not just current state |
| Tracing across hops | W3C Trace Context carried in message headers | One trace from first request to last consumer |
| Lag, dead letters, latency | Alerts on lag, dead-letter depth and end-to-end latency | The health signals of an asynchronous system |
| Replay, backfill, recovery | Replay by position or time; replication and recovery drills | Recovery is a rehearsed procedure, not a hope |
Each practice is a platform default that a team inherits with the topic rather than configuring by hand. Retention is a design decision, not a disk setting. Workloads authenticate with their own identity, through mutual Transport Layer Security (TLS) or OAuth client credentials. Carrying World Wide Web Consortium (W3C) Trace Context in message headers lets a consumer's span link to the producer's, so one trace can follow a request across asynchronous hops. Backfill runs into a separate consumer group, so that rebuilding history does not disturb live consumers.
Governance, pitfalls and maturity
Events decouple teams at run time, which makes the agreements between them more important, not less.
Governing events by ownership
Governance of events starts from one rule: the producer owns the schema and the meaning of the event, because only the producer knows what a fact signifies and when it is emitted. Consumers may request changes; they do not dictate them.
- Every event type has a catalog entry: owner, schema and version, semantics, retention, classification and registered consumers.
- Consumers register before subscribing, so the producer knows whom a change affects and whom to notify.
- One versioning and deprecation policy covers all event types, with notice periods and a sunset date for each retired type.
- Decision rights are written down: who approves a new topic, a breaking change, replay access and a retention change.
- Governance runs as code: naming, envelope, classification and compatibility checks execute in the pipeline.
Without the consumer list, no producer can assess the impact of a change or notify the teams affected, which is why onboarding is explicit.
Find
Find the event in the catalog; read its schema, semantics and retention
Request access
Request access to the topic; receive credentials and a consumer group
Prove
Pass the consumer contract test and register as a known consumer
Subscribe
Set lag and dead-letter alerts, then subscribe in production
Pitfalls that undo event-driven architecture
| Pitfall | How it shows up | Remedy |
|---|---|---|
| Events used as remote calls | A producer waits for a reaction; consumers must respond | Use request-response or a command; events are facts |
| Fat events that leak internal models | Consumers parse tables and flags; each refactoring breaks them | Domain events in business language; curated payloads |
| Missing idempotency | Duplicate payments or emails after a retry or replay | Event identity as key; inbox table; deduplication |
| No schema registry | Payload drift found in production; no compatibility check | Registry with a compatibility mode enforced in the pipeline |
| One topic for everything | Consumers filter noise; retention and access cannot differ | Topics per event type or aggregate; a naming convention |
| Broker treated as a database | Consumers query the log for state; retention grows unbounded | Read models and projections; retention by purpose |
| Hidden coupling by interpretation | Consumers infer meaning from combinations of fields | Document semantics; add explicit fields and types |
Each pitfall is a shortcut that looked reasonable at the time. Events used as remote calls recreate synchronous coupling without its clarity: if the producer needs a reaction, a command or a call is the honest choice. Long retention is not a pitfall in itself; Ben Stopford's Designing Event-Driven Systems argues that a retained log can serve as a shared source of truth. The pitfall is querying the log for current state instead of building views, and keeping data with no purpose-bound retention. Each pitfall also has an early signal the platform can detect, and the remedies belong in the reference implementation and the pipeline.
A staged maturity path
- Stage 1
Ad hoc
Point-to-point calls and nightly files; no schemas, catalog or idempotency; consumers unknown
- Stage 2
Brokered
Shared broker, naming convention and topic ownership; registered schemas; a reference outbox
- Stage 3
Contracted
AsyncAPI and catalog entries for every type; registered consumers; sunset dates; owned alerts
- Stage 4
Platform-led
Tenancy, quotas, chargeback and recovery drills; routine replay; events designed collaboratively
The four stages describe a path most organizations recognize, assessed per domain rather than for the whole enterprise. At the ad hoc stage duplicates are fixed by hand and nobody can list a topic's consumers. The brokered stage establishes the conventions and a reference implementation of the outbox and idempotent consumers that teams copy, and skipping it rarely works, because conventions must exist before tooling can enforce them. At the platform-led stage, new events are designed in event storming sessions and measured per domain.
Where to start: the first 90 days
The first 90 days should produce working evidence rather than a program plan. Each step below closes one of the failure modes described at the start of this paper.
- Days 1 to 30: know the estate. Inventory topics and their consumers, name an owner for each topic, and adopt the CloudEvents envelope for new events.
- Days 1 to 30: stand up the registry. Stand up a schema registry and register existing schemas under a compatibility mode.
- Days 31 to 60: publish the rules. Publish the naming, versioning and deprecation policy, and lint AsyncAPI documents in the pipeline.
- Days 31 to 60: ship a reference. Ship one reference producer and consumer with outbox, inbox and dead-letter handling for teams to copy.
- Days 61 to 90: prove it. Turn on lag and dead-letter alerts, run a replay drill to prove consumers tolerate redelivery, and review the catalog with domain leads to agree ownership.
Together these steps move a domain from ad hoc toward contracted, with evidence at each step that the decoupling is holding. Asynchronous integration succeeds when its failure modes are designed, not discovered.
Glossary
- Event
- A record of a fact that has happened, published by its producer for any number of consumers.
- Command
- A message that asks one receiver to perform an action.
- Event-carried state transfer
- An event style that includes the state consumers need, so they can work from a local copy.
- Domain event
- A business fact expressed in the language of the domain that produced it.
- Change data capture (CDC)
- Reading row-level changes from a database's log and publishing them as events.
- CloudEvents
- A CNCF specification for a common event envelope, with required id, source, specversion and type attributes.
- AsyncAPI
- An open specification for describing event-driven interfaces: servers, channels, operations and messages.
- Schema registry
- A service that stores schema versions and enforces a compatibility mode when new versions are registered.
- At-least-once delivery
- A guarantee that a message is redelivered until acknowledged, so duplicates must be expected.
- Idempotent consumer
- A consumer whose effects are the same whether a message is processed once or many times.
- Transactional outbox
- Writing an outgoing event to an outbox table in the same transaction as the state change, for a relay to publish.
- Inbox
- A consumer-side table of processed event identities, used to skip duplicates within the consumer's own transaction.
- Dead-letter queue
- A holding place for messages that fail every processing attempt, for triage and deliberate replay.
- Consumer lag
- The amount of published but unprocessed work for a consumer, the core health signal of a stream.
- Saga
- A long-running operation split into local transactions, with compensating actions to undo completed steps.
- CQRS
- Command Query Responsibility Segregation: separating a write model from read models fed by events.
- Event sourcing
- Storing state as an append-only stream of events and rebuilding it by replay.
References and further reading
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions.
- Martin Kleppmann, Designing Data-Intensive Applications (the second edition is co-authored with Chris Riccomini).
- Martin Fowler, What do you mean by "Event-Driven"?
- Cloud Native Computing Foundation, CloudEvents specification, version 1.0, with its protocol bindings and extensions.
- AsyncAPI Initiative, AsyncAPI Specification, version 3.x.
- Confluent, Schema Registry documentation: schema evolution and compatibility.
- Apache Software Foundation, Apache Avro specification; Google, Protocol Buffers language guide.
- Neha Narkhede and Guozhang Wang, Exactly-Once Semantics Are Possible: Here's How Kafka Does It (Confluent).
- Chris Richardson, Microservices Patterns, in particular the transactional outbox, saga and idempotent consumer patterns.
- Hector Garcia-Molina and Kenneth Salem, Sagas, Proceedings of the ACM SIGMOD International Conference on Management of Data.
- Ben Stopford, Designing Event-Driven Systems: Concepts and Patterns for Streaming Services with Apache Kafka.
- Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software.
- Alberto Brandolini, Introducing EventStorming.
- W3C, Trace Context (W3C Recommendation).
- General Data Protection Regulation (GDPR), in particular Article 5 on storage limitation and Article 17 on the right to erasure.
This paper offers general guidance and is not legal, regulatory or professional advice. Summaries of laws, regulations and standards are for orientation only; confirm the obligations that apply to your organization with qualified advisors. Product and vendor names are used for illustration and imply no endorsement.
© ByteCI, LLC. All rights reserved.