Continuous Integration and Continuous Delivery

Executive summary

Much of the cost and risk of software delivery accumulates between a commit and that change running in production: branches diverge until merging becomes a crisis, rare releases carry many changes and are feared accordingly, and manual deployments fail under pressure. Continuous integration (CI) and continuous delivery (CD) remove that accumulated risk by making integration, testing and deployment frequent and automatic.

Continuous integration merges every change into a shared mainline frequently and verifies each merge with an automated build. Continuous delivery keeps every passing change releasable, so that release is a decision rather than a project; continuous deployment, releasing every passing change automatically, is a further step taken per service. Common to all three is the deployment pipeline: a staged, automated sequence that builds a change once, promotes that single artifact through progressively broader checks, and is the only path to production.

• Integrate small changes to the trunk continuously, build on every change, and fix a broken build first.
• Build the artifact once and promote the same immutable, versioned artifact through every environment, with provenance recorded as it is built.
• Balance tests as a pyramid, use contract tests at service boundaries, and treat flaky tests as defects.
• Separate deploying from releasing, limit blast radius with blue-green and canary releases, and change databases by expand and contract.
• Define pipelines as code on a shared platform, instrument DORA's delivery metrics, and let the pipeline produce audit evidence.

The practices are the same at every scale; what grows with size is the machinery behind them, from a single pipeline definition to shared templates, elastic build farms and formal change enablement. The paper closes with the recurring pitfalls and a staged path that puts first the commitments that are hardest to retrofit.

From commit to production

The distance between a commit on a developer's machine and that change running in production is where much of the cost, and much of the risk, of software delivery collects.

For many teams, integration is a phase rather than a habit. Developers work for days on separate branches, and combining them becomes an event in itself, with conflicts that surface only at the merge. Release is a second event: a manual push that happens rarely, carries many changes at once, and is feared in proportion to how long it has been deferred. The symptoms are large batches, long-lived branches, late surprises and a production push that needs people awake at night.

Continuous integration is the practice of merging every developer's work into a shared mainline frequently, at least daily, and verifying each merge with an automated build and test run, so that integration problems are found within minutes while the change is small. Kent Beck developed it as one of the practices of Extreme Programming, and Martin Fowler's widely cited article on the subject is the standard description of it. Continuous delivery, set out by Jez Humble and David Farley in their book of that name, extends the discipline to the whole path to production: software is kept in a releasable state at all times, so that deploying is a business decision rather than a technical scramble. Humble describes the goal as the ability to get changes of every kind (new features, configuration changes, fixes and experiments) into production safely, quickly and sustainably. Continuous deployment goes further and releases every passing change with no human gate. Continuous delivery makes release possible at any time; continuous deployment makes it automatic, a choice made per service.

The mechanism that carries a change from commit to production is the deployment pipeline: an automated, staged sequence that builds the change once, subjects it to progressively broader checks, and ends in a release. It is the single path to production. Nothing reaches a production environment except through it, so every change is built, tested and recorded the same way, and the pipeline becomes the place where quality, security and compliance are enforced.

The size lens

This paper reads each practice through three bands of scale, adopted as a convention rather than as a maturity ranking. Small is one or two teams, a handful of deployable units, a single production environment and no dedicated operations or platform function. Medium is several teams, tens of services or applications, shared environments, a platform or enablement team beginning to form, and some regulated or formally controlled change. Large is many teams, hundreds of services, multiple platforms or regions, a dedicated platform organization, and formal change, audit and compliance obligations. The bands describe the shape of delivery, not revenue or headcount, and one organization may contain all three.

Continuous integration

Continuous integration rests on a shared agreement that the mainline, or trunk, is always in a working state. Trunk-based development supports this most directly: developers integrate small changes to the trunk frequently, either directly in very small teams or through short-lived branches that last hours or at most a couple of days, including the time spent waiting for review. Work that takes longer is hidden behind feature flags or built with branch by abstraction, so that it can be merged incomplete without being switched on. Branches that live for weeks defeat the purpose, because the trunk moves beneath them and the eventual merge grows; small, frequent commits keep each integration cheap to review and cheap to reverse.

Every change pushed to the trunk triggers an automated build that compiles the code, runs the fast tests and reports a clear pass or fail. Speed is a feature of this build, not a luxury. Extreme Programming's guideline of a build that completes in about ten minutes remains a sound target for the commit build: feedback in minutes is acted on, while feedback that takes an hour arrives after the developer has moved on to something else.

When the build breaks, fixing it is the team's highest priority, because a broken trunk blocks everyone who wants to integrate; the change is fixed within minutes or reverted. Much of the verification happens before the merge: a pull request runs the same build and test suite against the proposed change, and a reviewer reads it for design and correctness. Automated checks (formatting, linting, tests and security scans) leave people to judge only what people can. One caution: running a build server against long-lived feature branches is not continuous integration, however green the dashboard, because no one's work is being integrated with anyone else's until the branch finally merges.

Branching models compared

The branching model shapes how easily a team can integrate continuously. Trunk-based development is the model continuous integration assumes; the others add merge cost or release coupling that grows with the number of contributors.

GitFlow deserves a specific note because it is so often adopted by default. Its author, Vincent Driessen, has added a reflection to the original description recommending that teams practicing continuous delivery adopt a much simpler workflow instead, and reserving GitFlow for software that is explicitly versioned and must support several releases in the field at once.

The deployment pipeline

The deployment pipeline is the deployment process made explicit, automated and visible. Where continuous integration keeps the trunk healthy, the pipeline carries a specific build through every check that must pass before production.

A pipeline is a sequence of stages, each a gate a change must pass. The commit stage runs first and fastest: it compiles the code, runs unit tests and quick checks such as static analysis, and produces the artifact every later stage uses. Automated acceptance tests exercise the artifact against business scenarios in a production-like environment, non-functional checks probe performance, capacity and resilience where they matter, and manual or exploratory gates are inserted only where human judgment adds something automation cannot. The final stage is release to production.

A principle runs through every pipeline: build the artifact once, and deploy that same artifact everywhere. The commit stage produces a single, versioned binary, package or image, and every later stage up to production uses that identical artifact. Rebuilding per environment reintroduces the risk the pipeline exists to remove, because a binary rebuilt for production is not the one that passed the tests. Promotion, not rebuilding, moves a change forward as it gains confidence.

Because the same artifact runs everywhere, the environments must resemble one another closely enough that a test in one predicts behavior in the next. What legitimately differs is configuration, not code: connection strings, credentials, feature-flag states and scaling parameters are kept outside the artifact and supplied at deployment, so that one artifact is genuinely promotable. Infrastructure as code, which defines environments in version-controlled files and provisions them automatically, is what makes that parity achievable rather than aspirational.

A pipeline definition in practice

The workflow below, written for GitHub Actions as one widely used example, shows the build-once principle in a few lines. The commit stage builds and publishes an image tagged with the commit identifier; the acceptance and production jobs deploy that same image rather than building their own; and the production job runs in a protected environment that can require a named approver.

# Illustrative GitHub Actions workflow: build once, promote the same image
name: deliver
on:
  push:
    branches: [main]

jobs:
  commit-stage:
    runs-on: [self-hosted, linux]
    outputs:
      image: ${{ steps.publish.outputs.image }}
    steps:
      - uses: actions/checkout@v4
      - run: make lint test          # compile, static analysis, unit tests
      - id: publish                  # registry login omitted for brevity
        run: |
          IMAGE="registry.example.com/shop/orders:${GITHUB_SHA}"
          docker build -t "$IMAGE" .
          docker push "$IMAGE"
          echo "image=$IMAGE" >> "$GITHUB_OUTPUT"

  acceptance:
    needs: commit-stage
    runs-on: [self-hosted, linux]
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy.sh staging "${{ needs.commit-stage.outputs.image }}"
      - run: ./scripts/acceptance-tests.sh staging

  production:
    needs: [commit-stage, acceptance]
    runs-on: [self-hosted, linux]
    environment: production        # protection rules can require approval
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy.sh production "${{ needs.commit-stage.outputs.image }}"

Two refinements matter in real pipelines. Images are better promoted by their content digest than by a tag, because a tag can be moved to point at different content while a digest cannot. And the same shape, a commit stage that publishes once and later stages that only deploy, is exactly what shared templates should encode, so that individual teams inherit it rather than reinvent it.

Testing in the pipeline

A pipeline is only as trustworthy as the tests it runs. The test pyramid, described by Mike Cohn in Succeeding with Agile as the test automation pyramid, balances a suite with a broad base of unit tests that check small pieces of logic in isolation and run quickly, a narrower middle of service or integration tests, and a thin top of end-to-end tests that exercise the whole system through its user interface or public interfaces. Fast, cheap, stable tests should greatly outnumber slow, brittle ones. The common failure is the inverted suite, dominated by slow end-to-end tests, which is flaky, costly to maintain and gives feedback too late. Pushing tests down the pyramid wherever a focused test will do is a continual discipline.

Where a system is composed of many services, end-to-end tests that span them are especially fragile, because any service can break the test. Contract tests verify each service against the agreed interface, or application programming interface (API), of its neighbors. In the consumer-driven form, a consumer records what it expects of a provider, and the provider verifies every build against those recorded expectations, catching incompatibilities without a full end-to-end environment. Tools such as Pact implement the pattern, and the contracts double as living documentation of who depends on what.

Test data management is where many suites become slow or unreliable. Tests that share a mutable database interfere with one another and cannot run in parallel, so each test is given its own isolated, known starting state, with synthetic or anonymized data preferred over production copies that carry sensitive information. A flaky test, one that passes and fails without any change to the code, is a defect in its own right: once a red result might be noise, people re-run until green and the pipeline's signal is lost. Flaky tests are quarantined out of the gating path, tracked, and fixed or deleted.

Keeping the suite fast as it grows uses two levers. Parallelization spreads tests across many workers. Test selection runs only the tests a change can affect, with the full suite run on a schedule or before release so that selection errors are caught.

Artifacts, versions and provenance

The artifact the commit stage produces is the unit the pipeline promotes. It is immutable: once built it is never modified, only superseded by a new version. It is addressable, with an identity, ideally a cryptographic digest of its content, that names this exact content and nothing else. And it is self-contained enough that deploying it needs only the artifact and its configuration. These properties let the pipeline promise that the thing tested is the thing released.

Every artifact carries a version. Semantic versioning, published as the Semantic Versioning specification, expresses a version as major, minor and patch numbers: a major increment signals an incompatible change to the public interface, a minor adds functionality in a backward-compatible way, and a patch makes backward-compatible fixes, so that a consumer knows what to expect from an upgrade. Build metadata (the commit identifier, build number and pipeline run) is attached alongside, so that any running artifact traces back to the source it was built from.

Artifacts live in an artifact repository that holds versioned binaries, packages and images and serves them to deployments: the commit stage publishes to it, and every later stage pulls from it rather than rebuilding. The same repository should proxy and cache third-party dependencies, so that builds draw from a controlled, scanned source rather than directly from the internet. A reproducible build goes further, producing a bit-for-bit identical artifact from the same source, build environment and instructions by pinning every input, which turns the artifact into reliable evidence that it represents its source and nothing else.

Provenance and the software supply chain

The build is also where provenance is established: verifiable evidence of how an artifact was produced, from what source, by which pipeline and with what inputs. Signing lets a deployer verify that an artifact came from the expected pipeline unaltered. A software bill of materials (SBOM), a complete inventory of the components an artifact contains including transitive dependencies, usually produced in a standard format such as SPDX or CycloneDX, makes it possible to answer quickly, when a vulnerability is disclosed, whether a given release is affected. These records are cheapest and most trustworthy when the pipeline generates them as it builds.

The software supply chain, the chain of source, dependencies, build systems and artifacts that produce a release, is a target in its own right, because compromising a widely used build step reaches everything it builds. Supply-chain Levels for Software Artifacts (SLSA), a framework maintained under the Open Source Security Foundation (OpenSSF), describes increasing levels of assurance for that chain. Version 1.2 of the specification defines a build track and a source track, summarized below. A pipeline that builds once on a hosted platform, pins its inputs and records signed provenance has most of what the build track asks for; the source track adds controls on the repository itself.

Deployment strategies and safe change

Reaching production is not the same as changing it safely. A deployment strategy governs how a new version replaces the one running, so that a bad change exposes few users and recovery is fast.

Several strategies trade infrastructure cost for safety. A rolling deployment replaces instances of the old version with the new a few at a time, so the service stays up throughout; it is, for example, the default strategy for Kubernetes Deployments. A blue-green deployment runs two production environments that are as identical as possible and switches traffic from one to the other in a single step, with an immediate switch back if needed. A canary release sends a small share of traffic, or a chosen group of users, to the new version first, watches its health signals, and widens the rollout only if they stay good. Each narrows the exposure of a release in a different way.

Deployment and release can also be separated. A feature flag is a runtime switch that lets code be deployed to production while the feature stays off, to be turned on later, for a subset of users, or off again instantly without a redeployment. Pete Hodgson's widely used classification distinguishes release flags, which hide unfinished work, from experiment flags for A/B tests, operational flags that act as kill switches, and permission flags that open features to particular users. A dark launch goes further, exercising new code against real traffic without exposing its results. Flags accumulate as debt if never removed, so each needs an owner and an expiry, and release flags in particular should be short-lived.

When a release goes wrong, rollback returns to the previous known-good version and roll forward fixes the problem in a new one. Rollback is faster when available, which is why immutable artifacts and blue-green or canary mechanisms are worth having, but it is not always safe, because some changes cannot be undone. The hardest case is the database.

Changing the database safely

The pattern that makes database change compatible with frequent deployment is expand and contract, also called parallel change. First expand: add the new structure without removing the old, so that both versions of the code can run. Then migrate: update the application to write both structures and read the new one, and backfill existing data. Finally contract: once nothing uses the old structure, remove it in a later deployment. Each step is individually safe and reversible, which a single large migration never is. Renaming a column, in PostgreSQL syntax, becomes three separate releases:

-- Release 1, expand: add the new column alongside the old one
ALTER TABLE customer ADD COLUMN phone_number varchar(32);

-- Release 2, migrate: code writes both columns and reads the new one;
-- backfill existing rows (in batches on a large table)
UPDATE customer SET phone_number = phone WHERE phone_number IS NULL;

-- Release 3, contract: only after no deployed code reads or writes the old column
ALTER TABLE customer DROP COLUMN phone;

These techniques of evolutionary database design are set out by Scott Ambler and Pramod Sadalage in Refactoring Databases, and schema migration tools that version each change and apply it through the pipeline make them routine. A blue-green or canary release depends on them, because during the release the old and new versions of the application share one database.

Configuration and secrets follow the rule set for artifacts: they live outside the build and are supplied at deployment, so that the same artifact runs everywhere. Secrets (credentials, keys and tokens) are never baked into an artifact or committed to source control, because a secret in an image is a secret leaked. They are held in a managed secret store, injected at run time, rotated on a schedule, and searched for by scanners that run on every change.

Pipelines as code and pipeline architecture

A pipeline is itself software. Its definition (the stages, steps, triggers and environments) lives in version control alongside the code it builds, is reviewed the same way, and is versioned with the application. Defining it as code makes it reproducible and auditable: the history shows who changed the path to production and when, and a pipeline can be recreated from its definition rather than clicked together by hand.

As the number of pipelines grows, duplicating their definitions becomes unmanageable, and every team solving the same problem differently becomes a governance gap. Shared templates and libraries factor the common shape (build, test, scan, sign and deploy) into reusable definitions that individual pipelines extend, so that a change to a template reaches every pipeline that uses it. Templates are where an organization encodes its standards once, and they should themselves be versioned, tested and released like any other shared code.

Pipelines run on runners, the machines or containers that execute each step, and build infrastructure ranges from a single runner to elastic fleets that scale with demand. Caching of dependencies, build outputs and container layers is among the cheapest ways to keep builds fast, and ephemeral, identical runners keep builds clean and reproducible because no state leaks from one build to the next. Repository structure also shapes the architecture: a monorepo simplifies shared and cross-project changes but must build and test only what a change affects, while a polyrepo is simpler per repository but pushes cross-service coordination outward, into versioned dependencies and contracts.

With many services, pipelines compose. Fan-out runs independent work in parallel, and fan-in gathers the results, assembling independently built services into a release and running the tests that only make sense across them. Designing these dependencies explicitly keeps total pipeline time bounded as services multiply, and it makes visible which services can still be released independently.

The paved path

Rather than ask every team to assemble these pieces, a platform team provides a paved path: a golden pipeline, built from shared templates and backed by the artifact store, secret management, policy checks and observability, that a team adopts to get a compliant path to production without building one. The paved path is the easy path and the compliant path at once, and it succeeds by being better than the alternative rather than by being mandated. Platform engineering extends the same idea beyond pipelines to environments, deployment and operations, and GitOps, in which the desired state of each environment is declared in version control and reconciled automatically, is a common way to implement the final deployment step on such a platform.

A pipeline must stay fast, because slowness compounds. When feedback is slow, developers batch changes, failures become harder to diagnose, and the team drifts back toward the large, infrequent integrations continuous integration was meant to remove. Pipeline speed is therefore a primary concern of the platform, maintained through parallelization, caching, test selection and a fast commit stage, and watched as a trend like any other service level.

What changes with scale

The practices in this paper are the same at every band; what changes is how they are implemented, owned and governed. A small team and a large organization both integrate to a trunk, build once and automate their gates, but they differ in who owns the pipeline, how definitions are shared, what infrastructure runs the builds, how change is approved, and how much evidence the result must produce.

Some things do not change with scale, and naming them prevents a large organization from believing that size lets it abandon them. The trunk stays the integration point, because more teams make long-lived branches more dangerous, not less. Build once and promote stays the rule, because more environments make rebuilding more risky. Automated gates stay mandatory, because more change makes manual verification less able to keep up.

What scales is the supporting machinery: one team's single definition becomes a library of shared templates, a handful of runners becomes an elastic build farm, and two environments become many. Governance changes most visibly, as a large organization layers formal change management, separation of duties and audit obligations on top of peer review. Standard changes and automatically generated evidence, described in the next section, are how it does so without giving up frequent delivery.

Governance, measurement and compliance

A pipeline is also an instrument. The software delivery metrics published by DevOps Research and Assessment (DORA), the research program behind the book Accelerate and not the EU regulation of the same acronym, can be read directly from the pipeline and the systems around it. DORA's model has grown from four keys to five metrics: change lead time, deployment frequency and failed deployment recovery time measure throughput, while change fail rate and deployment rework rate measure instability. The pipeline records when changes are committed and deployed, how long they take, which deployments needed immediate intervention, and which were unplanned responses to incidents, so a mature pipeline produces these measures as a by-product rather than as a separate exercise.

The pipeline also has its own health. Pipeline duration, the time from commit to a releasable artifact, governs how fast feedback arrives; queue time reveals capacity pressure on runners; and build failure and flakiness rates measure how far the signal can be trusted. These are operational metrics read as trends, and a worsening trend is the signal to invest before the pipeline becomes the bottleneck.

Formal change management and continuous delivery can coexist, but only if routine change is treated as routine. ITIL 4 describes change as an enablement practice and defines a standard change as a low-risk, pre-authorized change that is well understood, follows a documented procedure and needs no separate approval each time. A well-governed pipeline is exactly such a procedure: if the path to production enforces testing, review, scanning and separation of duties, a deployment through it can be classed as a standard change, approved once as a process rather than repeatedly as an event. This is how organizations keep both frequent delivery and formal change governance, reserving individual approvals and advisory boards for the genuinely non-standard.

Separation of duties, the control that no one person can both make and release a change unchecked, is often assumed to need manual handoffs, but a pipeline enforces it better. Code requires a reviewer other than its author, deployment runs through the pipeline rather than by direct access, and the roles that change the pipeline definition, approve a release and reach production are kept distinct, with protected branches and environments enforcing the rules rather than trust.

Because the pipeline mediates every change, it is the natural place to produce audit evidence, recording who committed, who reviewed, which tests and scans ran, which provenance and SBOM were generated, and who approved the release, all as a by-product of delivery. Regulated financial and payment systems, where these obligations are strongest, depend on such evidence being retained and producible on demand. The Secure Software Development Framework published by the National Institute of Standards and Technology as SP 800-218 sets out the secure development practices that a pipeline of this kind can enforce and evidence directly.

Pitfalls and a staged path

The ways continuous integration and delivery go wrong are consistent across organizations, and most are an earlier principle neglected. Collecting them lets a design review check for each.

A staged path

No team adopts all of this at once, and the order matters: put first the practices that are expensive to retrofit and that everything else depends on.

1. Integrate continuously. Adopt trunk-based development with short-lived branches, build and test on every change, and make a broken build the first thing the team fixes. This is the foundation the rest depends on.
2. Build a pipeline. Automate the path from commit to a tested, versioned, immutable artifact, defined as code and built once, and add the test pyramid, acceptance tests and an artifact repository so that promotion replaces rebuilding.
3. Deliver continuously. Automate deployment with a safe strategy such as rolling, blue-green or canary, separate configuration and secrets from the artifact, handle database change through expand and contract, and keep every change releasable.
4. Scale and govern. Factor shared templates into a paved path, record provenance and SBOMs, instrument DORA's metrics and pipeline health, treat routine deployments as standard changes, and let the pipeline generate audit evidence.

Conclusion

Continuous integration and continuous delivery turn the path from commit to production from a rare, risky event into a repeatable, automated process. Continuous integration keeps the trunk healthy by integrating small changes often and verifying each quickly. The deployment pipeline carries a single, versioned artifact through progressively broader checks to a release, enforcing quality, security and compliance as it goes, and safe deployment strategies and disciplined handling of artifacts, databases, configuration and secrets keep releases reversible.

The commitments are few and best made early: integrate to the trunk, build the artifact once, automate the gates, and keep the mainline releasable. They hold at every band of scale, and what grows with size is the machinery behind them (shared templates, build farms, formal change enablement and automatic evidence), not the principles themselves. Organizations that make these commitments find that the path to production stops being a source of risk and becomes the most reliable process they run, the foundation on which infrastructure automation, security in the pipeline and reliable operations are built.

Glossary

Continuous integration (CI)

The practice of merging every developer's changes into a shared mainline at least daily and verifying each merge with an automated build and test run.

Continuous delivery (CD)

The practice of keeping every change that passes the pipeline in a releasable state, so that deploying to production is a business decision rather than a technical effort.

Continuous deployment

The practice of automatically releasing to production every change that passes the pipeline, with no manual gate.

Deployment pipeline

The automated, staged sequence that builds a change once and carries it through progressively broader checks to a release, serving as the single path to production.

Commit stage

The first, fastest pipeline stage, which compiles the code, runs unit tests and quick checks, and produces the artifact later stages use.

Trunk-based development

A branching model in which developers integrate small changes to a shared trunk frequently, using short-lived branches or feature flags rather than long-lived branches.

Test pyramid

A model for balancing a test suite with many fast unit tests, fewer service or integration tests, and few slow end-to-end tests.

Contract test

A test that verifies a service against the agreed interface of its neighbors rather than against the running services themselves.

Flaky test

A test that passes and fails without any change to the code under test, treated as a defect because it erodes trust in the pipeline's signal.

Immutable artifact

A built unit of software that is never modified after creation, only superseded by a new version, so that the thing tested is the thing released.

Semantic versioning

A versioning scheme of major, minor and patch numbers that signals incompatible changes, compatible additions and compatible fixes.

Reproducible build

A build that produces a bit-for-bit identical artifact from the same source, build environment and instructions.

Provenance

Verifiable evidence of how an artifact was produced: from what source, by which build platform and with what inputs.

Software bill of materials (SBOM)

A complete inventory of the components an artifact contains, including transitive dependencies, used to assess exposure when a vulnerability is disclosed.

SLSA

Supply-chain Levels for Software Artifacts, a framework of build and source track levels that describe increasing assurance for the software supply chain.

Blue-green deployment

A strategy that runs two production environments and switches traffic from the old to the new in one step, allowing an immediate switch back.

Canary release

A strategy that sends a small share of traffic to a new version first and widens the rollout only if its health signals are good.

Feature flag

A runtime switch that lets code be deployed to production while the feature it contains stays off until it is turned on, separating deployment from release.

Dark launch

Running new code against real production traffic without exposing its results to users.

Expand and contract

A pattern for backward-compatible change, also called parallel change, in which new structure is added, data and code migrate, and the old structure is removed in a later step.

Paved path

A supported, ready-made route to production, such as a golden pipeline, that a platform team offers so that the easy path is also the compliant one.

Standard change

A low-risk, pre-authorized change that follows a documented procedure and needs no separate approval each time.