Observability, Resilience, and Operations for Microservices

Executive summary

In a system of many services, something is always failing somewhere. Instances restart, dependencies slow down and queues back up, and users notice only if the system was not built to expect it. Reliability therefore depends less on preventing failure than on seeing it quickly, containing it where it starts and responding in a practiced way.

This paper argues that observability, resilience and operations should be delivered as platform defaults, backed by an operating model in which every service has an owning team.

• Observability rests on linked metrics, logs and traces produced through OpenTelemetry, with trace context propagated across every hop and a standard set of signals per service.
• Service levels turn observations into decisions: SLIs measure what consumers experience, SLOs set targets, and error budgets with a written policy govern the pace of change.
• Alerts page on symptoms and fast budget burn; causes stay on dashboards, slow burns become tickets and known failures are handled by automation.
• Resilience patterns are matched to failure modes rather than applied everywhere, and each one is proven by experiment.
• Ownership is explicit: an on-call team per service, a readiness review before go-live, incident roles practiced in advance and blameless postmortems afterward.

The closing sections cover testing in production, the pitfalls that recur, a five-stage maturity path and a 90-day plan sized to produce evidence rather than a program plan.

Failure is partial, continuous and far from its cause

A monolith fails rarely and completely; a system of many services fails constantly and partially. At any moment some instance is restarting, some dependency is slow and some queue is backing up. A single request may cross a gateway, several services, a broker and several data stores, so any one log shows only a fragment, and the service reporting an error is often not the one that caused it.

With many small services, something is always failing somewhere; the symptom appears in one service and the cause in another.

Three consequences follow when nothing is designed in.

The answers are observability that sees across hops, resilience patterns that contain failure, and an operating model that names an owner for every service. A small, well-established vocabulary anchors the first of these.

Observability

Observability is the ability to ask new questions of a running system without shipping new code. In a system of many services it rests on signals that complement one another, a standard way of producing them and a small set of signals every service exposes.

Three complementary signals, and two extensions

Each signal answers a different question. Metrics say whether something is wrong and by how much, cheaply and across many requests. Logs say what happened in one place. Traces say where a request went and where it waited, a question a monolith never had to ask. Among the observability patterns Chris Richardson catalogs for microservices are log aggregation, application metrics, distributed tracing, exception tracking, health check APIs, audit logging and logging of deployments and changes. The practical rule is to link the signals: a metric alert leads to an example trace, the trace leads to the logs of each span, and a deployment marker on the graph explains the timing.

OpenTelemetry: one vendor-neutral path from code to backend

OpenTelemetry, a graduated Cloud Native Computing Foundation (CNCF) project formed when the OpenTracing and OpenCensus efforts merged, is an open, vendor-neutral observability framework used across the industry.

OpenTelemetry provides an API and a software development kit (SDK) per language, semantic conventions that name attributes consistently, a Collector that receives, processes and exports telemetry, and OTLP for transport. Traces, metrics and logs are stable OTLP signals, and a profiling signal is in development. Because instrumentation is separate from the backend, services are instrumented once and the backend can change without touching them, and commercial and open source backends widely accept OTLP.

Propagation is what makes a trace span services. W3C Trace Context, a W3C Recommendation, defines the traceparent header, which carries the trace identifier, the parent span identifier and sampling flags, and the tracestate header for vendor-specific data. The same values travel in message headers on asynchronous hops, where the consumer starts a span linked to the producer's.

Golden signals and the RED method

Google's Site Reliability Engineering (SRE) book names four golden signals for monitoring a user-facing system: latency, traffic, errors and saturation. Tom Wilkie's RED method (rate, errors and duration) is the request-centric subset for any service that answers calls, and Brendan Gregg's USE method (for every resource, check utilization, saturation and errors) covers the resources underneath. The table applies them per service and adds what microservices make necessary.

Two disciplines keep this affordable. Cardinality stays bounded when metrics carry a few labels, such as operation, consumer and status; unbounded values, such as user or request identifiers, belong in traces and logs. Sampling, at the head or the tail of a trace, and retention tiers keep volume proportional to value. Dashboards follow one platform-generated template per service and roll up per tier, so an incident commander sees the user journey at a glance.

Service levels and alerting

Observations become decisions through service levels. The definitions in the SRE book are the ones to standardize on.

Choosing service-level indicators per API

The Site Reliability Workbook recommends expressing an SLI as a ratio of good events to total events, which puts it on a scale of zero to 100 percent that maps directly onto an error budget, measuring it as close to the consumer as practical, and keeping to a small number of SLI types. Stating latency as the share of requests served within a bound, rather than as an average, makes it an error-budget quantity like availability. Correctness matters where a fast, successful, wrong answer is the real failure, and freshness is the indicator asynchronous flows most often lack: a read model hours behind returns success while failing its consumers.

The objective comes from the service tier recorded in the catalog and from what consumers need, not from whatever the service happened to achieve in the previous period. An SLO should be tighter than any SLA that depends on it, so that the contract is breached only after the objective has already raised the alarm; the SRE book makes the same argument for keeping internal objectives tighter than those advertised to users.

From objective to error budget to policy

The error budget connects objectives to behavior. If an API promises 99.9 percent availability over 30 days, its budget is the remaining 0.1 percent, about 43 minutes of complete unavailability or the equivalent in partial failures, spent on incidents or on the risk that comes with change. Burn rate expresses how fast the budget is being consumed relative to the SLO: a burn rate of 1 exhausts it exactly at the end of the window. The Site Reliability Workbook recommends alerting on several windows at once, with a short window that lets an alert reset quickly once the burn stops.

The policy is the part most organizations skip. It is written before the budget is spent, agreed by product and engineering, and applied without renegotiation: for example, feature releases to a service that has exhausted its budget pause until reliability work restores headroom. Reviews then adjust objectives on evidence, because an objective nobody can meet, or whose breach nobody notices, is not an objective. Error budgets turn reliability into a shared, measurable decision between product and engineering rather than a recurring dispute.

Routing alerts: page, ticket, automate or record

Alert routing decides whether on-call is sustainable. The governing rule, from the SRE book's chapter on monitoring distributed systems, is to page on symptoms that users experience and keep causes as diagnostic context: the page says checkout is slow, and the dashboard shows the saturated connection pool behind it. Every page should be urgent, actionable and backed by a runbook, which keeps the top-right quadrant small. Service tiers set the thresholds: a critical tier pages around the clock, a standard tier pages in business hours, and a best-effort tier only raises tickets. A weekly review of every page, asking whether it was actionable and whether it recurred, is the cheapest reliability practice available.

Resilience patterns

Resilience patterns confine a failure to where it started. They are easy to describe and easy to misapply, so this section starts with the two that every call needs, then sets out six containment patterns and matches each to the failure it addresses.

Timeouts and deadlines, retries and budgets

Michael Nygard's Release It! opens its catalog of stability patterns with timeouts, for good reason: a call without one turns a slow dependency into blocked threads, and blocked threads into cascading failure. The SRE book's chapter on addressing cascading failures describes deadline propagation: every call in the tree that grows from one request shares the same absolute deadline, so downstream servers can drop work whose caller has already given up.

Retries make transient failures invisible and sustained failures worse. The same chapter recommends randomized exponential backoff, a limit on retries per request and a server-wide retry budget, and warns that retrying at several layers multiplies load. Amazon's guidance on timeouts and retries makes the point with an example in which three attempts at each of five layers multiply the load on the bottom layer 243 times. A 429 Too Many Requests or 503 Service Unavailable response with a Retry-After header is the server's instruction, and clients should follow it.

Containing failure: six patterns

Most of these patterns are cataloged in Release It!. A circuit breaker moves between closed, open and half-open states: it opens after enough failures so that callers fail fast instead of waiting, lets a trial request through after a pause, and closes again when the dependency recovers. Load shedding rejects work before queues grow without bound, and per-consumer rate limits at the gateway are shedding by policy. Fallbacks need a product decision per operation about what a reduced answer looks like, because engineers alone cannot decide whether a stale price or a missing recommendation is acceptable. Liveness and readiness checks, as container platforms such as Kubernetes expose them, and outlier ejection in load balancers and proxies remove failing instances from traffic before users notice.

Matching the pattern to the failure mode

A decision table stops resilience from becoming a pile of patterns applied everywhere. One row deserves emphasis: a traffic surge is shed by policy before autoscaling absorbs it, because scaling takes minutes and shedding takes milliseconds. Every row remains a hypothesis until it is exercised. Patterns that have never been triggered are assumptions, and cascading failures grow in the gaps between assumptions, which is why the evidence column is filled by the experiments described under testing, pitfalls and maturity.

Traffic, deployment and the platform

Resilience and observability scale only when the platform provides them. Three platform decisions matter most: where traffic policy lives, how changes reach production, and what every new service gets by default.

Where traffic policy lives

Most estates use all three. A service mesh is implemented by proxies in the data path: a sidecar beside each workload or, in sidecarless designs such as Istio's ambient mode, a shared proxy on each node with optional proxies for Layer 7 policy. Istio and Linkerd are common examples. In-process libraries remain necessary for what only the application knows: which operation may degrade, what to cache and which fallback is acceptable. A mesh enforces policy; it does not design resilience.

Progressive delivery with automated rollback

Progressive delivery exposes a change to a growing share of traffic while comparing its indicators with the version it replaces, and reverses automatically when they regress. Blue-green deployment keeps two complete environments and switches traffic between them, which makes rollback a routing change; canary releases shift traffic gradually, with progressive delivery tools automating the analysis and the rollback; and feature flags separate release from deployment. Database changes follow the expand-and-contract pattern so that old and new versions can run at once. The research behind Accelerate, by Nicole Forsgren, Jez Humble and Gene Kim, found that speed and stability are not a trade-off: teams that deploy most often also tend to have lower change failure rates and faster recovery.

The internal platform: golden paths for running services

A golden path is a template and pipeline from which a new service is created already instrumented, deployed progressively, registered in the catalog with an owner, wired to paging and tagged for cost, so that it is observable, deployable and owned on day one. The platform decides what should not be decided per service. Autoscaling works within limits that capacity planning sets from expected peak and headroom. Critical services spread across availability zones by default, since a zone is the common unit of failure, while multi-region deployment adds replication and consistency questions and is reserved for tiers whose objectives require it. In the terms of Team Topologies by Matthew Skelton and Manuel Pais, this is the work of a platform team that offers the platform as an internal product.

The operating model

Technology contains failure; the operating model decides who notices it, who acts and what the organization learns.

Ownership and operational readiness

Ownership is the foundation: the team that builds a service runs it, watches it and is on call for it, with the platform carrying the undifferentiated load. On-call is part of ownership rather than an optional extra, and it must be sustainable, with page volumes reviewed. The operational readiness review, which the SRE book describes as a production readiness review, checks that ownership is real before traffic arrives. A service is ready for production when its owning team can answer yes to every item below, and the review records the evidence.

Two items are often missed. Queues, topics and event flows need owners as much as services do, and cost per service belongs to the same owner, because reliability and cost are traded constantly. The catalog that records owner, tier, dependencies and objectives doubles as the dependency map an incident commander reaches for. When a team is reorganized, ownership transfers explicitly or the service is retired; an orphaned service is never acceptable.

Incident management and learning from incidents

Incident management is a practiced structure, not an improvisation. Severity levels are defined by user impact and tier, so anyone can declare an incident, and the SRE book advises declaring early: closing a minor incident costs little, while starting the process hours into a growing one costs a great deal. Its chapter on managing incidents defines the roles: an incident commander who holds the overall picture and delegates, an operations lead who works the system, a communications lead who posts updates at a set cadence, and, in larger incidents, a planning role that handles longer-running tasks such as handoffs. The first goal is mitigation by rollback, shedding, failover or a flag, with root cause deferred until users are served again.

Afterward, a blameless postmortem reconstructs the timeline, names contributing factors rather than a single root cause or a single person, and produces actions with owners and dates that are tracked like any other work. Time to restore service, one of the four delivery measures popularized by Accelerate, is the natural summary metric; the DevOps Research and Assessment (DORA) program now tracks the narrower failed deployment recovery time, so organizations should be clear about which they report. The postmortem backlog then feeds the resilience work of the following quarter.

Testing, pitfalls and maturity

Every pattern and process in this paper is an assumption until it has been tested, and the tests that matter most cannot all run before production.

Testing before production and testing in production

Testing in production is not a substitute for testing before it; it is an admission that no test environment reproduces production data, traffic and topology, so pre-production is a filter rather than a guarantee. Synthetic monitoring is often the first detector of a broken journey, and shadow traffic is safe only for operations without side effects.

Chaos experiments follow the Principles of Chaos Engineering: build a hypothesis around steady-state behavior, measured as system output such as successful checkouts; vary real-world events, such as a failed dependency or added latency; run experiments in production; automate them to run continuously; and minimize the blast radius. Game days rehearse the incident process itself with the real team. The results of both fill the evidence column of the failure-mode table above.

Pitfalls that recur

The pitfalls recur because each is the path of least resistance under delivery pressure. Missing timeouts and retries everywhere are the two most common contributors to cascading failure, usually library defaults that nobody reviewed. Unbounded fan-out makes tail latency a certainty: as Jeffrey Dean and Luiz André Barroso show in The Tail at Scale, when a request waits on many calls, the slowest of them sets the response time. Batching, caching and concurrency caps help, but the lasting fix is often a change of service boundary. The remedies are cheapest as platform defaults, applied to every new service and backfilled to the critical tier first.

A staged maturity path

A maturity path gives teams a shared description of where they stand and what comes next, assessed per tier and per team. The instrumented stage matters more than it looks, because it produces the data that every later stage consumes: objectives cannot be set without indicators, and resilience cannot be verified without traces. Moving one stage typically takes quarters, the critical tier moves first, and no stage is skipped.

Where to start: the first 90 days

The first 90 days should produce evidence rather than a program plan, starting with the critical tier.

1. Days 1 to 30: name owners and instrument. Name an owning team in the catalog for every service, and instrument the critical tier with OpenTelemetry through the platform's standard path.
2. Days 1 to 30: bound every call. Set an explicit timeout and a retry budget on every call in the critical tier.
3. Days 31 to 60: set objectives. Define SLIs and SLOs per tier, and replace cause-based pages with burn-rate alerts.
4. Days 31 to 60: prepare to respond. Run readiness reviews for the services about to ship, and publish runbooks, incident roles and severity levels.
5. Days 61 to 90: rehearse and review. Run the first game day to exercise the incident process and one resilience hypothesis, baseline cost per service, and review error budgets with product owners.

By the end of the period the critical tier is owned, observable, bounded and rehearsed, and the organization has the data to set its next objectives. Reliability is designed in and then rehearsed; it cannot be monitored in after the fact.

Glossary

Observability

The ability to ask new questions of a running system from its telemetry, without shipping new code.

Span

A timed operation within a trace, such as one service call, linked to its parent in the request's tree.

Trace context

The identifiers that link spans across process boundaries, carried in W3C Trace Context headers.

OpenTelemetry

A CNCF project that provides vendor-neutral APIs, SDKs, a Collector and a protocol (OTLP) for producing and exporting telemetry.

Golden signals

Latency, traffic, errors and saturation: the four signals the Google SRE book recommends monitoring.

RED method

Rate, errors and duration, measured for every request-driven service.

USE method

Utilization, saturation and errors, checked for every resource.

Cardinality

The number of distinct label combinations a metric can take; unbounded cardinality makes metrics expensive.

Service-level indicator (SLI)

A carefully defined quantitative measure of some aspect of the service provided.

Service-level objective (SLO)

A target value or range for an SLI over a window.

Error budget

The shortfall from perfect service that an SLO allows over its window.

Burn rate

How fast a service consumes its error budget relative to the SLO; a burn rate of 1 exhausts the budget at the end of the window.

Circuit breaker

A pattern that stops calls to a failing dependency, fails fast and probes until the dependency recovers.

Bulkhead

A partition of resources, such as a connection pool per dependency, that confines the effect of a failure.

Retry budget

A limit that keeps retries to a small share of traffic, so they cannot multiply load during an incident.

Progressive delivery

Exposing a change to a growing share of traffic while comparing indicators, with automated rollback on regression.

Golden path

A supported template and pipeline that make an organization's practices the default for new services.

Blameless postmortem

A written review of an incident that identifies contributing factors and actions without assigning personal blame.

Chaos engineering

Experimenting on a system to build confidence in its ability to withstand turbulent conditions in production.