Observability, Reliability and Incident Management

Executive summary

A change that has shipped is not finished; it has to be operated, and operating it well preserves its value. The common failure mode is deploy-and-forget: a change ships, attention moves on, and the system runs unwatched until it breaks, whereupon untrusted alerts, a scramble of experts and diagnosis by guesswork take over. Observability, reliability engineering and incident management replace that pattern with ownership, evidence and learning.

• Instrument services for observability with logs, metrics and traces, correlated by shared context and unified by a vendor-neutral standard.
• Define reliability with service level objectives, and govern the pace of release with error budgets.
• Alert on the symptoms users feel, and keep on-call humane and every page actionable.
• Respond to incidents with a defined, blameless command structure, and mitigate before diagnosing.
• Learn from every incident and near miss, tracking action items to closure.

These practices rest on a simple ownership principle: the teams that build services operate them. The practices do not change with the size of the organization; what changes is their weight and their owners. The paper closes with reliability engineering practices, an operating model, cost management and a staged path to maturity.

Operating what you deliver

A change that reaches production is not the end of the work; it is the start of a second life that the team which built it must be able to see, understand and keep running.

Many teams treat a deployment as the finish line. The change ships, attention moves on, and the system runs unwatched until something breaks. When it does, alerts that no one trusts fire too late or too often, a small group of experts is summoned to diagnose by intuition, and the fix depends on who remembers the system best. This is deploy-and-forget, and it produces alert fatigue, a hero culture resting on a few individuals, and outages diagnosed by guesswork. The practices in this paper move that cost forward, into design and daily running, where it is far smaller.

Owning production

Teams own what they build, and operating it in production is where that ownership is tested. The people who wrote a service understand its failure modes best, so they stay responsible for it in production rather than handing it across a wall. Reliability is the feature on which every other feature depends, because a capability a user cannot reach is worth nothing; it is designed, measured and budgeted, not left to luck.

Site reliability engineering (SRE), described in Google's Site Reliability Engineering book, applies software engineering methods to operations: automating repetitive work, setting measurable objectives, and spending engineering effort to make systems observable and resilient. Benjamin Treynor Sloss, who founded the practice at Google, characterizes it as what happens when software engineers are asked to design an operations function. SRE is a set of practices, not only a job title, so a small team with no site reliability engineer still applies its ideas.

Three size bands

This paper reads every practice through three size bands, adopted as a convention rather than as empirical categories. Small is one or two teams, a handful of deployable units, one production environment and no dedicated operations or platform function. Medium is several teams, tens of services, shared environments, a platform or enablement team forming, and some regulated change. Large is many teams, hundreds of services, multiple platforms or regions, a dedicated platform organization, and formal change, audit and compliance obligations. The bands describe the shape of delivery, not revenue or headcount, and one organization may contain all three.

Observability foundations

Observability, a term borrowed from control theory, is the degree to which a system's internal state can be understood from the data it emits. A system is observable when an operator can answer a new question about its behavior, one not anticipated when the system was built, without shipping new code to gather the data. Monitoring answers known questions, such as whether a server is up or a queue is growing; observability supports the unknown question asked for the first time during an incident.

Three kinds of telemetry carry most of the load. Logs are timestamped records of discrete events, detailed but costly at volume. Metrics are numeric measurements aggregated over time, cheap to store and fast to query. Traces follow a single request across the services it touches, showing where time was spent and where it failed. Two further signals are worth naming: events, which record a notable state change such as a deployment or a configuration change, and continuous profiles, which show where a program spends processor time and memory, down to the line of code.

Structured and correlated telemetry

A log is far more useful when it is structured, emitted as named fields rather than free text, so that it can be searched and aggregated by machine. A request that crosses many services can be followed only if a correlation identifier is attached at the edge and carried through every call, so that the logs, metrics and traces of one request can be joined. Context propagation is the mechanism that passes it across process boundaries, and the W3C Trace Context standard defines the HTTP header that carries it; OpenTelemetry uses that format by default. The example below shows an incoming header and a log record emitted while handling the request, joined by the same trace identifier.

# Incoming request header (W3C Trace Context)
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

# Structured log record emitted while handling it (timestamp omitted)
{
  "severity_text": "ERROR",
  "service.name": "checkout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "e457b5a2e4d86bd1",
  "http.route": "/orders",
  "http.response.status_code": 502,
  "duration_ms": 3012,
  "message": "payment provider timed out"
}

With that identifier in place, an operator who finds the error in the logs can open the full trace of the request, see which downstream call timed out, and compare it with the latency metrics for that dependency, all without guessing. Timeouts, retries and other patterns between services shape what traces show, and tracing is where their effects become visible.

A vendor-neutral standard

OpenTelemetry, a graduated Cloud Native Computing Foundation (CNCF) project, is the vendor-neutral standard for instrumenting code and collecting telemetry. It defines application programming interfaces (APIs) and language software development kits (SDKs), semantic conventions that give common telemetry consistent names, a wire protocol (OTLP), and a Collector that receives, processes and exports data, so that instrumentation is written once and sent to any compatible backend. OpenTelemetry is not itself a backend: storage, querying and visualization are left to the tools an organization chooses. The signals it supports are traces, metrics and logs, plus baggage for propagating context; it treats events as a specialized kind of log record, and profiling is the least mature of its signals, so teams should check the project's documentation for the maturity of each signal in their languages before depending on it.

Telemetry is not free. The cost of metrics is driven largely by cardinality, the number of distinct combinations of label values, because each combination is stored and queried as a separate stream of values. High-cardinality context such as a user identifier or a request identifier belongs in sampled logs and traces rather than in metric labels.

What to measure first

To decide what to measure first, established methods name a few signals. The Site Reliability Engineering book names four golden signals: latency, the time it takes to serve a request; traffic, the demand placed on the system; errors, the rate of failing requests; and saturation, how full the service is. Two complementary methods sharpen the focus. The RED method, introduced by Tom Wilkie, measures rate, errors and duration for each service. The USE method, introduced by Brendan Gregg, checks utilization, saturation and errors for every resource, such as processors, memory, disks and network links. Services are watched with RED, resources with USE, and the golden signals connect the two.

Service level objectives and error budgets

Reliability is meaningful only when it is defined, and three terms from the Site Reliability Engineering book make the definition precise. A service level indicator (SLI) is a precisely defined, quantitative measure of one aspect of the service, such as the proportion of requests served successfully within a time limit. A service level objective (SLO) is a target value, or range of values, for an SLI over a window of time. A service level agreement (SLA) is a contract with users, explicit or implicit, that carries consequences when its objectives are missed; if missing an objective has no consequence, it is an objective, not an agreement. Internal objectives are set tighter than anything promised externally, so the team has a margin in which to act before users or contracts are affected.

Choosing indicators

A good SLI measures what a user experiences, not what a server reports. The way to find one is to follow a user journey and ask what working means at each step. For a request-response service, availability and latency are common; for a data pipeline, freshness and completeness matter more, and they depend on the data quality checks that run inside the pipeline. An indicator expressed as a ratio of good events to valid events stays comparable as traffic changes.

An objective is a judgment, not an aspiration. Set too high, it wastes effort on reliability no user values and leaves no room to ship; set too low, it disappoints the people who depend on the service. The Site Reliability Engineering book's guidance is practical: do not simply adopt current performance as the target, keep indicators simple, avoid absolutes such as perfect availability, have as few objectives as will cover what matters, and start with a looser target that can be tightened rather than a strict one that must be relaxed. History shows what is achievable; users' needs decide what is required. Each additional increment of availability costs more than the last, and the target stops where users stop noticing.

Error budgets

The complement of an objective is an error budget: the unreliability the objective permits over the window, which the team may spend on change, experiments and planned maintenance. The arithmetic is simple and worth making visible, because it turns an abstract percentage into minutes a team can reason about.

For a request-based indicator the budget is a share of requests rather than minutes: at 99.9 percent, one valid request in a thousand may fail. While the budget remains, the team can take the risk of shipping; when it is exhausted, an error budget policy, agreed in advance with product and business stakeholders, changes what is allowed, for example pausing risky releases until the service is back within objective and directing engineering effort to reliability. Writing the policy down beforehand removes the argument from the moment of stress.

Alerting on the budget

An error budget also gives a better basis for alerting. Rather than paging whenever a threshold is crossed, burn-rate alerting, described in The Site Reliability Workbook, pages according to how fast the budget is being consumed relative to the objective. A burn rate of 1 spends exactly the budget over the window; a burn rate of 14.4 sustained for an hour spends 2 percent of a 30-day budget in that hour. The workbook recommends pairing a long window, which shows that a meaningful amount of budget is being spent, with a short window, which shows that the problem is still happening, so that alerts fire quickly and stop soon after recovery.

Expressed as a Prometheus alerting rule, the first of these looks like the example below. The same pattern, written once in a shared rule library, gives every service consistent alerting on its own objective.

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_requests_total{job="checkout", code=~"5.."}[1h]))
              / sum(rate(http_requests_total{job="checkout"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="checkout", code=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="checkout"}[5m]))
          ) > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: Checkout is spending its error budget 14.4 times faster than sustainable
          runbook_url: https://runbooks.example.com/checkout/error-budget-burn

Reviewing objectives

Objectives are reviewed on a schedule, because a target set once drifts out of date as the product and its users change. A review asks whether the indicator still reflects the user journey, whether the objective still matches what users expect, and whether the error budget policy has been honored when the budget ran out. An objective no one acts on is retired rather than left to mislead.

Alerting and on-call

An alert should fire when users are affected, not when an internal cause is present. A full disk matters only if it threatens the service; one failed node in a healthy cluster does not. Alerting on symptoms, the conditions users feel such as elevated errors or latency, keeps alerts few and meaningful and keeps working when the cause changes. The Site Reliability Engineering book frames monitoring as answering two questions, what is broken and why: symptoms answer the first and should drive pages, while causes answer the second and should page only when they are definite and imminent. Symptom-based alerts also map onto service level objectives, because the symptom and the indicator are often the same measurement.

Pages, tickets and dashboards

Not every signal deserves to wake someone. A page interrupts a person immediately and is reserved for conditions that need action now and require human judgment; a ticket records work that can wait for working hours; a dashboard holds information to consult, not act on. Every page must be actionable and must link to a runbook, a short document describing how to assess and mitigate the condition. A page with no action is noise, and a page with no runbook forces each responder to reconstruct what a previous one already learned. If the response to a page is purely mechanical, it should be automated rather than paged.

Sustainable on-call

On-call is sustainable only when it is humane. The rotation needs enough people that any individual carries the pager infrequently and can recover between shifts, handoffs pass on the state of the system, and the work is recognized through compensation or time off. Much of what makes on-call painful is toil, which the Site Reliability Engineering book characterizes as the work of running a production service that is manual, repetitive, automatable and tactical, leaves no enduring value behind, and grows in step with the service. The book treats reducing it as engineering work: Google's SRE teams aim to keep toil below half of each engineer's time, so that it does not crowd out the project work that makes the system better.

On-call changes shape with size. In a small organization, everyone who writes the service is on call for it, and the rotation may be only a few people. In a medium organization, rotations are defined, handoffs formal, and a forming platform team may take first-line response for shared components. In a large organization, multiple rotations follow the sun across regions, with defined severities and a platform organization operating shared services. The principle holds at every size: the people who build a service stay close to its operation.

Incident management

An incident is not the time to invent a process; the teams that recover quickly are the ones that agreed how they would respond before the pressure arrived.

An incident is an unplanned disruption or degradation that needs a coordinated response. Declaring one, explicitly and early, signals that normal work pauses, a response structure takes over and communication begins. The threshold to declare is kept low, because declaring early and standing down is cheaper than declaring late. The Site Reliability Engineering book offers simple triggers: declare if a second team is needed, if the problem is visible to customers, or if it remains unsolved after an hour of concentrated analysis. Severity is set by the impact on users and the business, described in words rather than by a formula, and it drives who is summoned, how often updates are sent and how far the incident is escalated.

Incident command

A coordinated response needs defined roles. The incident management framework in the Site Reliability Engineering book is based on the Incident Command System (ICS), a model from emergency response known for its clarity and scalability, and it names four roles. The incident commander holds the overall picture, assigns roles and keeps the incident document, without making the fix personally. The operations lead makes the diagnostic and mitigating changes, and the operations team is the only group changing the system during the incident. The communications lead keeps responders, stakeholders and users informed on a set cadence. The planning lead handles longer-running work such as filing bugs, arranging handoffs and recording how the system has been changed from normal so that it can be restored afterward. Many organizations add a scribe who records the timeline as it happens, a role PagerDuty's published incident response guide also defines.

In a small team one person holds all of these roles, which is correct rather than a failure, because the value of the model is the checklist of responsibilities, not the headcount. Roles are split off as an incident grows, starting with communications, which is the first duty to be dropped when the person coordinating is also typing the fix.

Running the response

During an incident, communication is as important as the fix. The incident commander keeps a living incident document that several people can edit at once, recording what is known, what has been tried and who is doing what. Updates go out on a regular cadence, even when the only news is that work continues, and a status page lets users see what is known without contacting support. The first goal is to stop the harm, not to understand it: mitigation before diagnosis restores service by the fastest safe means, often by rolling back the most recent change or shifting traffic away from a failing component, before investigating why it failed. Rollback and progressive delivery are the most common mitigations, because a change deployed shortly before the problem began is the most likely cause.

When responders cannot mitigate, or an incident outlasts a shift, the response is escalated along a defined path or handed off with a clear summary of state. Command is handed over explicitly, with the outgoing commander confirming that the incoming one has accepted it, so that a long incident does not depend on one person staying awake. Some incidents are security incidents, where the cause is an attack or a breach. These follow the same structure but add steps such as preserving evidence, containing the attacker's access and coordinating with security, legal and, where required, regulators; the decision to treat an incident as a security incident is made early, because it changes who is involved and what must be retained.

Learning from incidents

After an incident, the team writes a postmortem: a record of what happened, how it was handled and what will change. Blameless culture is practiced here. Blameless means the review assumes everyone acted reasonably given what they knew at the time, and looks for the conditions that made the failure possible rather than a person to blame, as John Allspaw argued in describing blameless postmortems and a just culture. The purpose is a system that fails less, not a name to attach to the failure.

Contributing factors, not a single cause

Complex systems rarely fail for a single reason. Richard Cook's How Complex Systems Fail argues that such systems run in a partly degraded state, held up by many overlapping defenses, so that a failure is the rare alignment of several weaknesses; searching for the root cause stops at the first plausible human action and misses the conditions that let it matter. Sidney Dekker's work describes a new view of human error as a symptom of deeper trouble in the system, not its cause: the question is why the action made sense to a competent person at the time, and the remedy is to change the system that framed the choice.

Acting on what is learned

A postmortem that ends in understanding but not change is incomplete. Each lesson becomes an action item with a single owner and a date, tracked to closure like any other commitment, because items that are never done teach the organization that postmortems are theater. Sharing postmortems across teams, rather than keeping them within one, lets an organization learn from failures it did not suffer directly and makes writing about failure ordinary; a searchable archive becomes a resource the next responder consults.

Measuring incidents with care

Incidents are measured, but their metrics mislead when read carelessly. Time to detect, time to mitigate and time to restore describe the phases of a response. They relate to, but are not the same as, the failed deployment recovery time that DevOps Research and Assessment (DORA) uses as a delivery metric: DORA's measure covers only failures caused by a deployment, while an incident process tracks recovery from every cause. Averaging durations, as in a mean time to restore service, commonly abbreviated MTTR, is treacherous. As Štěpán Davidovič's Incident Metrics in SRE shows, incidents are few and their durations vary widely, so an average over a small, skewed sample is unstable, easily moved by one outlier, and poorly suited to decision-making or trend analysis. These measures are read as distributions and as prompts for investigation, not as a single figure to target. A near miss carries much of the same information at none of the cost, so reviewing near misses surfaces weaknesses before they combine.

Reliability engineering practices

A system's reliability is built, not wished for, through deliberate investments that are tested before the system is asked to survive without them.

A system fails when demand exceeds what it can serve, so capacity is planned rather than discovered in production. Capacity planning estimates future demand and provisions for it with margin; load testing confirms, before release, how the system behaves as demand rises and where it breaks. Together they replace the hope that a system will cope with evidence of how much it can take, and they tell the team which resource will saturate first.

Designing for failure

A resilient system degrades gracefully, shedding or simplifying work under stress so that an overloaded service returns a reduced result instead of nothing. Between services, timeouts stop a caller waiting forever, retries with exponential backoff and jitter recover from transient faults without synchronized retry storms, and a circuit breaker stops calling a failing dependency so that it can recover. Bulkheads isolate resources so that one failing dependency cannot exhaust the threads or connections every other request needs. The safest change is one that can be undone quickly, so progressive delivery and fast rollback limit the blast radius of a bad change and are the first mitigations a responder reaches for.

Rehearsing failure

A system's response to failure is known only if failure is tried. Chaos engineering, set out in the Principles of Chaos Engineering and developed in the book by Casey Rosenthal and Nora Jones, is the discipline of experimenting on a system to build confidence that it can withstand turbulent conditions in production. An experiment starts from a hypothesis about steady-state behavior, introduces events that mirror real failures, runs where the behavior is real, and keeps the blast radius as small as possible; mature programs automate experiments to run continuously. Game days rehearse an incident together, finding gaps in tooling and runbooks before a real event does, and disaster recovery exercises rehearse the loss of a whole environment or region. In regulated sectors, such as financial services, supervisors may require evidence that recovery has been tested against defined objectives, which makes these exercises an obligation as well as a good practice.

Readiness and the reliability hierarchy

Before a service takes production traffic, a production readiness review checks that it has objectives, alerts, runbooks, planned capacity and an owner on call, lighter or heavier by band. These practices fit together in an order the Site Reliability Engineering book presents as a service reliability hierarchy, which it credits to work by a group of Google SREs that included Mikey Dickerson. Each level rests on the ones below, so a weakness low down undermines everything above. Monitoring is the foundation, followed by incident response, postmortem and root-cause analysis, testing and release procedures, capacity planning, and development, with the product at the top, reliable because everything beneath it is sound.

What changes with scale

The practices in this paper do not change between the size bands; their weight, their owners and their formality do. The table below separates the two.

Three things hold at every size: the teams that build services own them in production, alerts fire on the symptoms users feel, and failures are reviewed without blame. Letting any of these lapse as an organization grows is a common and costly mistake, because they are what make the rest work rather than overhead to shed.

Other things change, and should. A dedicated reliability function appears only when there are enough services to justify it. A telemetry platform run by a platform team replaces direct use of a hosted tool once many teams need consistency and cost control. Severity definitions and communication become formal, and compliance evidence becomes an obligation under audit. The error runs both ways: large-organization formality on a small team adds cost without reducing risk, while small-team informality on a large estate invites the failures that structure exists to prevent.

Operating model, tooling and cost

Where reliability engineering sits is a choice with trade-offs. An embedded model places site reliability engineers inside stream-aligned teams, close to the service but thinly spread. A central model gathers them into one team that consults across the organization, which concentrates expertise but risks rebuilding the wall between building and operating. A platform model, common at scale, has them build and run the observability and delivery platform that other teams use. Many organizations combine the three.

As a service approaches production, and again at major launches, a readiness review confirms that it is ready to operate, with objectives, alerts, runbooks, capacity and on-call established. At scale these reviews are standardized and recorded; in a small team they are a checklist the team runs for itself.

Managing telemetry cost

Telemetry grows with the system and can cost more than the system it watches, so its cost is managed deliberately. Sampling keeps a representative fraction of high-volume data such as traces: head-based sampling decides when a request starts, while tail-based sampling, which the OpenTelemetry Collector supports, decides after a trace completes and can keep every slow or failed request. Retention tiers age older data into cheaper storage and summaries, and cardinality limits cap the label combinations that drive metric cost. The choice to build or buy observability tooling is a trade-off to record, not a matter of principle: packaged tools bring capability quickly, while a built or self-hosted stack, often provided through platform engineering, fits unusual needs and can control cost at scale. Instrumenting with an open standard keeps that choice reversible.

Measuring the practice

The practice measures itself, and the trend matters more than the level. A few indicators show whether the discipline is healthy, each read by the team that owns it as a prompt for the next improvement, not as a target set from above, which would invite gaming.

Pitfalls and a maturity path

The failures of observability and reliability work are repetitive, and most invert a principle from the earlier sections. They are collected here so that a review can check for each, with the remedy beside it.

A maturity path

No team arrives at all of this at once. A staged path puts first the practices that are expensive to retrofit and defers the rest until scale justifies them.

1. Foundations. Structured logs, metrics and traces joined by correlation identifiers; a few symptom alerts, each linked to a runbook; the teams that build services on call for them; and lightweight blameless postmortems.
2. Objectives and response. Service level objectives on key user journeys with error budgets, burn-rate alerting, a defined incident command structure, and action items tracked to closure.
3. Reliability engineering. Capacity planning and load testing, progressive delivery and fast rollback, chaos engineering and game days, disaster recovery exercises, and production readiness reviews.
4. Scale and govern. A telemetry platform run as a service, objectives and an error budget policy across services, formal severity and communications, compliance evidence from the operating tools, and the practice measured as trends.

Conclusion

Operating what you deliver is the other half of delivery. A change is done not when it ships but when the team can see how it behaves, has agreed what reliable means for it, is alerted when users are affected, responds to incidents with a structure rather than heroics, and learns from every failure without blame. Observability makes the system explain itself, service level objectives define the reliability that matters, incident management limits the harm, and postmortems turn each failure into durable improvement.

These practices do not change with the size of the organization; their weight, owners and formality do. A small team runs them as a shared habit, with a hosted tool and a handful of alerts; a large organization runs them through a platform, a dedicated reliability function and formal process, each at the weight the system needs. Pipelines, platforms and security controls all meet here, in production, where the value of the work is finally kept or lost.

Glossary

Observability

The degree to which a system's internal state can be understood from the telemetry it emits, including answers to questions not anticipated in advance.

Telemetry

The logs, metrics, traces, events and profiles a system emits about its own behavior.

Cardinality

The number of distinct combinations of label values in a metric, each of which is stored as a separate stream of values.

Context propagation

Passing trace identifiers and related context across process boundaries so that telemetry from one request can be joined.

OpenTelemetry

A Cloud Native Computing Foundation project that provides vendor-neutral APIs, SDKs, conventions, a protocol and a collector for telemetry.

Site reliability engineering (SRE)

A practice that applies software engineering methods to operations, treating the reliable running of a service as an engineering problem.

Service level indicator (SLI)

A quantitative measure of one aspect of a service's behavior, usually expressed as a ratio of good events to valid events.

Service level objective (SLO)

A target value or range for a service level indicator over a window, defining the reliability the team intends to meet.

Service level agreement (SLA)

A contract with users that sets reliability commitments with consequences, typically looser than the internal objective.

Error budget

The amount of unreliability an objective permits over a window, which the team may spend on change.

Error budget policy

An agreement, made in advance, on what changes when the error budget is exhausted, such as pausing risky releases.

Burn rate

How fast a service consumes its error budget relative to its objective; a burn rate of 1 spends exactly the budget over the window.

Toil

Manual, repetitive, automatable operational work that scales with the service and produces no lasting improvement.

Runbook

A short document, linked from an alert, that describes how to assess and mitigate a condition.

Incident commander

The role that coordinates an incident response and holds the overall picture without making the technical fix personally.

Blameless postmortem

A review after an incident that seeks the contributing factors and system improvements rather than individuals to blame.

Mean time to restore (MTTR)

The average duration of recovery across incidents; unstable over small, skewed samples and best read as a distribution.

Chaos engineering

The discipline of experimenting on a system, against a hypothesis, to build confidence that it withstands turbulent conditions.

Production readiness review

A check before launch that a service has objectives, alerts, runbooks, capacity and an owner on call.