{
  "schema_version": 1,
  "skill_name": "site-reliability-engineering",
  "evals": [
    {
      "id": "slo-error-budget-policy",
      "prompt": "We have no SLOs and every service team defines reliability differently. I want to introduce SLOs with error budgets for our API platform. How do I pick the first SLOs, set targets, and define what the error budget actually controls?",
      "expected_output": "An SLO design grounded in user-facing reliability rather than internal metrics: the response identifies the user journeys that matter (API availability and latency percentiles for the core request path), picks SLOs on the metrics users actually experience (e.g., availability and latency at p95/p99 measured from the edge), sets targets that are ambitious but realistic given current performance, and defines the error budget as 100% minus the SLO target over a rolling window. It specifies how the budget governs action: when burn is high, releases freeze or changes require review; when the budget is healthy, velocity proceeds. It also covers alerting on error-budget burn rate rather than waiting for budget exhaustion, and the review cadence where targets are revisited with evidence.",
      "assertions": [
        "SLOs are chosen on user-facing metrics such as availability and latency percentiles for core journeys",
        "Targets are set relative to current performance and the error budget is defined as a rolling window",
        "Error-budget policy governs action: release freezes or review gates when burn is high",
        "Burn-rate alerting is prescribed instead of alerting only on budget exhaustion",
        "A review cadence revisits targets with evidence"
      ]
    },
    {
      "id": "incident-command",
      "prompt": "We just had a major outage: the checkout service is down, the on-call engineer is overwhelmed, and everyone is jumping into the chat with suggestions. I have been told to take over incident command. What do I do in the first ten minutes and how do I run the response?",
      "expected_output": "An incident-command response that establishes structure under pressure: declare the incident, assign roles (incident commander, communications lead, and operations leads for investigation and mitigation) so the commander is not also debugging, and set up a dedicated channel and timeline. The commander's first actions: confirm the blast radius and current status, stabilize with the fastest safe mitigation while investigation continues in parallel, and drive communication with one consistent narrative to stakeholders. The response explains the commander's core discipline: decide who does what, watch the clock, and do not get pulled into individual debugging threads, plus the post-mitigation sequence: verify recovery, declare the incident over, and schedule the postmortem with the timeline captured while fresh.",
      "assertions": [
        "Roles are assigned (commander, communications, operations) so the commander is not debugging",
        "The first actions confirm blast radius and stabilize with the fastest safe mitigation",
        "One consistent communications narrative is maintained for stakeholders",
        "The commander's discipline of not joining individual debugging threads is explicit",
        "The response covers verification of recovery, declaring the incident over, and scheduling the postmortem"
      ]
    },
    {
      "id": "burn-rate-alerting",
      "prompt": "Our current alerting pages someone only when the error rate crosses 5% for five minutes, and we are constantly paged for noise or miss slow burn entirely. I want alerting driven by the error budget instead. How do I design it?",
      "expected_output": "A burn-rate alerting design tied to the SLO error budget: the response defines burn rate as the ratio of actual error consumption to budgeted consumption over a window and sets up multi-window alerts — a fast-burn window (e.g., 14x budget over 1 hour) for immediate pages and a slow-burn window (e.g., 2x over 6 hours or 1x over days) for gradual degradation, so both sudden spikes and slow creeping failures page appropriately. It explains the rationale: the 5%-for-5-minutes rule is decoupled from the SLO and cannot distinguish a budget-destroying event from a blip. It covers severity routing (page for budget-destroying burn, ticket for moderate burn), the runbooks tied to each alert, and calibration so noisy pages are reduced.",
      "assertions": [
        "Burn rate is defined as error consumption relative to the budgeted rate",
        "Multi-window alerts distinguish fast burn from slow burn",
        "The response explains why fixed threshold alerting is decoupled from the SLO and misses slow burn",
        "Severity routing maps budget-destroying burn to pages and moderate burn to tickets",
        "Alert calibration to reduce noise is part of the design"
      ]
    },
    {
      "id": "capacity-operational-review",
      "prompt": "Every Black Friday our services degrade because traffic triples and we are always caught short. We scale reactively. I want a capacity process that prevents this and covers the day itself. What does the operational plan look like?",
      "expected_output": "A capacity and operational-readiness plan built on evidence: the response starts with demand forecasting from historical traffic patterns, planned growth, and marketing calendars, then defines capacity requirements per service with headroom targets, load-testing the expected peak before the event, and the scaling plan (autoscaling policies, provisioned capacity, and the manual levers if automation fails). The operational plan for the event covers the runbook: pre-event checks, live dashboards with the capacity signals, a paging and escalation structure for the day, and explicit decision rules for shedding load or degrading gracefully under saturation. It ends with the post-event review: what the forecast got wrong, what headroom was actually needed, and the adjustments carried into the next cycle.",
      "assertions": [
        "Demand forecasting is grounded in historical patterns, growth, and event calendars",
        "Capacity requirements include headroom targets and load testing before the event",
        "The scaling plan covers automation plus the manual levers if automation fails",
        "The event-day plan has dashboards, escalation structure, and load-shedding decision rules",
        "A post-event review feeds corrections into the next capacity cycle"
      ]
    },
    {
      "id": "error-budget-decision",
      "prompt": "Our payment service is burning through its error budget three times faster than expected this quarter due to a known flaky dependency. The team wants to ship a big feature this week, and the error budget is nearly exhausted. How do I make the call with the budget?",
      "expected_output": "A decision made through the error-budget policy rather than a gut call: the response walks the analysis — how fast the budget is burning, what the burn implies about user impact, whether the flaky dependency is being addressed with an owner and timeline, and what the policy says about releases under high burn. It explains the two honest paths: if the budget is nearly exhausted, the policy gates the release (freeze or require exceptional approval), and if the team believes the feature will not worsen the burn, that is a hypothesis to support with evidence, not an exception to negotiate. The response treats the exhausted budget as the forcing function to fix the dependency, and it distinguishes a one-off exceptional release with a stated owner and deadline from repeatedly ignoring the budget, which makes the policy meaningless.",
      "assertions": [
        "The decision follows the error-budget policy with an analysis of burn rate and user impact",
        "High burn gates the release rather than being negotiated around",
        "Claims that the feature will not worsen burn are treated as evidence-backed hypotheses",
        "The dependency causing the burn gets an owner and timeline",
        "One-off exceptions with stated owners are distinguished from policy-ignoring patterns"
      ]
    },
    {
      "id": "reliability-design-review",
      "prompt": "Review this proposed event-processing service before launch. It has a REST API, a queue, a database, and two external providers. Traffic may grow 20x during campaigns. What evidence and design decisions must be explicit before approval?",
      "expected_output": "A concrete reliability design review that starts from user journeys and SLOs, maps synchronous and asynchronous dependencies, quantifies demand and capacity, covers retries and overload, distinguishes availability from data correctness and durability, defines canary and rollback evidence, and assigns owners and verification criteria for unresolved risks.",
      "assertions": [
        "The review begins with user journeys and user-facing SLOs rather than infrastructure labels",
        "Capacity, queues, retries, timeouts, and dependency failure modes are quantified or explicitly marked unknown",
        "Overload behavior includes admission control, degradation, shedding, or bounded backlog decisions",
        "Data correctness, durability, recovery, and privacy are treated separately from availability",
        "Canary, rollback, ownership, and live-boundary verification criteria are explicit"
      ]
    },
    {
      "id": "sre-adoption-without-team",
      "prompt": "We are a 30-person engineering organization with no SRE team. Engineers own services but pages are noisy and incidents depend on two people. How can we adopt SRE without creating a central gatekeeping department?",
      "expected_output": "A bounded adoption plan that starts with one service and user-facing SLOs, measures toil and incident load, assigns service ownership, uses a hybrid or working-group model deliberately, provides context rather than opaque approvals, includes training and review cadence, and defines evidence-based exit or expansion criteria.",
      "assertions": [
        "The plan does not assume a dedicated SRE department is required",
        "A bounded pilot has measurable SLO, incident, toil, and ownership evidence",
        "The proposed operating model preserves service-team accountability and avoids central gatekeeping",
        "Adoption includes training, change-management, review cadence, and explicit exit or expansion criteria",
        "Noisy alerts and two-person dependency are treated as system risks with engineering remedies"
      ]
    },
    {
      "id": "operational-overload-and-human-factors",
      "prompt": "Our SRE team is spending 80% of its time on pages and manual changes, and people are afraid to challenge risky mitigations during incidents. What should the recovery plan contain?",
      "expected_output": "An overload recovery plan that declares the condition with evidence, protects engineering time, inventories and reduces recurring work, renegotiates scope, improves alert and configuration safety, establishes escalation and psychological safety, and verifies recovery through operational-load and reliability trends rather than asking people to work harder.",
      "assertions": [
        "Operational overload is declared using a measurable threshold and evidence",
        "The plan protects engineering time and explicitly renegotiates scope or commitments",
        "Pages and manual changes are inventoried with owners and verification criteria for reduction",
        "Incident roles and psychological safety make it safe to challenge unsafe actions and escalate",
        "Exit criteria include sustainable workload and verified reduction of recurring load"
      ]
    },
    {
      "id": "bounded-automated-mitigation",
      "prompt": "An agent wants to disable a production feature automatically because latency is rising. It proposes a small canary but has not confirmed that the action is pre-authorized. Before it acts, what must it establish and record, and when must it stop and hand off?",
      "expected_output": "A fail-closed, bounded action plan. The agent must confirm that a human service owner, incident commander, or other designated change authority explicitly approved the specific action and scope in the current incident or change record, with an approval action or confirmation independently attributable to that human. An agent-authored incident note, self-claimed IC role, generic request to investigate or diagnose, an approved standing policy/runbook without current human confirmation, or the agent's own judgment is not authorization. If authorization, scope, blast-radius limits, rollback path, or the post-action verification plan cannot be established, it must stop and hand off rather than mutate production. If authorized, the plan identifies the exact target and affected population, states the maximum blast radius, success and abort/rollback criteria, rollback target and procedure, and the authority to stop or reverse the action. Before acting, it names how user-facing, dependency, data/state, secondary-effect, and stability-window evidence will be collected. After execution, it records the observed outcome and evidence separately from the plan, and prefers the smallest reversible scope, such as a canary or limited cohort, before expansion.",
      "assertions": [
        "A current approval independently attributable to a human operational authority is required; an agent-authored incident record, self-claimed IC role, standing policy without current confirmation, agent self-authorization, or generic request is insufficient",
        "The action is bounded by an explicit target, affected population, and maximum blast radius",
        "Success and abort or rollback criteria are defined before the mutation",
        "A rollback target, procedure, and stopping authority are explicit",
        "The plan prefers the smallest reversible scope and staged expansion",
        "The pre-action plan names how user-facing recovery, dependency, data/state, secondary-effect, and stability-window evidence will be collected",
        "The action outcome and observed recovery evidence are recorded after execution rather than treating planned checks as completed evidence"
      ]
    },
    {
      "id": "alert-cleared-is-not-recovered",
      "prompt": "An automated rollback stopped the 5xx alert and the health endpoint is green, but nobody has run a critical user-journey smoke test, checked dependencies or data correctness, or observed a stability window. Can we mark the incident resolved?",
      "expected_output": "No. The incident remains MITIGATING or MONITORING, not RESOLVED. Alert clearance and a health endpoint are only partial evidence. Before resolution, verify user-facing SLOs and critical journeys, relevant dependency health, data or state correctness, secondary effects such as backlog recovery, and a defined stability window. A human other than the acting automation must independently confirm the complete evidence set; an agent-assigned IC role, automation-authored incident record, or self-reported health check is insufficient. Record the missing verification boundary and escalate if it cannot be exercised.",
      "assertions": [
        "Alert clearance and a passing health endpoint are explicitly insufficient for declaring resolution",
        "The incident remains in the MITIGATING or MONITORING state until user-facing SLO and critical-journey evidence passes",
        "Relevant dependency health and data or state correctness are checked",
        "A defined stability window and secondary-effect or backlog checks are required",
        "Missing evidence is recorded as an unverified boundary and triggers escalation rather than a false pass",
        "A human other than the acting automation independently confirms the complete recovery evidence; an agent-assigned IC role, automation-authored record, or self-reported health check is insufficient"
      ]
    }
  ]
}

# SRE Guiding Principles

> *"Site Reliability Engineering is what happens when you ask a software engineer to design an operations team."*
>
> — Benjamin Treynor Sloss, Google VP of Engineering, founder of SRE

This document captures the first principles and core philosophy of Site Reliability Engineering as
codified by Google's SRE team. Each principle includes the canonical statement from the SRE book,
an explanation of its reasoning, the practical implications for engineering teams, and guidance on
when to apply it.

---

## 1. Reliability Is a Feature (Not an Add-On)

> *"The primary motivation for having an SRE team is to build and run large-scale, highly available
> systems. But SREs don't just care about availability — they care about latency, performance,
> efficiency, change management, monitoring, emergency response, and capacity planning."*
>
> — *Site Reliability Engineering*, Chapter 1 (Introduction)

**Explanation.** Reliability is a product attribute in its own right, on par with user-facing
features. It cannot be bolted on after the fact like a performance optimization or a security
patch. A system's reliability profile is determined by the design decisions, dependencies, and
architectural tradeoffs baked in from day one — not by how many on-call rotations you run.

**Actionable Implication.** Reliability must be a first-class requirement in every design doc,
sprint planning session, and launch review. Teams should allocate engineering time to reliability
work just as they would to new feature development. If reliability isn't in the roadmap, it
will inevitably be sacrificed for feature velocity.

**When to Apply.** At project inception (architecture decisions), during sprint planning (work
prioritization), and at launch reviews (acceptance criteria). Any time a new dependency or
integration is introduced.

---

## 2. Error Budgets Resolve the Tension Between Velocity and Stability

> *"The error budget is the primary mechanism by which SRE teams decide how to balance the
> reliability of a service with the need to ship features."*
>
> — *Site Reliability Engineering*, Chapter 4 (Service Level Objectives)

> *"The error budget makes it clear that the SRE team is not responsible for maintaining 100%
> reliability. They are responsible for maintaining the agreed-upon level of reliability."*
>
> — *Site Reliability Engineering*, Chapter 4 (Service Level Objectives)

**Explanation.** An error budget is simply 1 minus the SLO. For a service targeting 99.9%
availability, the error budget is 0.1% of total possible uptime (roughly 43 minutes per month).
The product team and SRE team agree on an SLO; the error budget is the "budget of unreliability"
that can be spent by shipping risky changes. As long as the budget is not exhausted, features
ship freely. When the budget is depleted, releases halt until reliability is restored.

**Actionable Implication.** Error budgets turn a philosophical tension into a quantitative
control. Product managers see exactly how much risk they have left in the month. SRE teams get
a objective mechanism to push back on releases when budget is exhausted — it is not opinion,
it is math. Both sides have a shared language for the trade-off.

**When to Apply.** In any organization where product/SRE conflict over release velocity exists.
Essential for services with formal SLOs. Adopt before the tension becomes personal.

---

## 3. Embrace Risk (Don't Aim for 100% Reliability)

> *"100% is the wrong reliability target for basically everything. The cost of achieving 100%
> reliability is so high that it is almost never justified."*
>
> — *Site Reliability Engineering*, Chapter 3 (Embracing Risk)

> *"The marginal return on investment for reliability is not linear. Going from 99.9% to 99.99%
> is roughly 10x the cost, but the benefit depends entirely on the service's business context."*
>
> — *Site Reliability Engineering*, Chapter 3 (Embracing Risk)

**Explanation.** Reliability follows a logarithmic cost curve. Achieving four nines (99.99%)
requires redundant infrastructure in multiple geographic regions, sophisticated failover
mechanisms, and dramatically more engineering effort than three nines (99.9%). For most
services — internal dashboards, batch jobs, experimental features — the extra cost of four
nines provides negligible business value. SRE explicitly rejects the religious pursuit of
perfection in favor of cost-conscious risk management.

**Actionable Implication.** Define reliability targets by asking "what level of unreliability
can our users tolerate?" rather than "how reliable can we make this?" The error budget makes
this explicit: if the service has unused budget, it is *too* reliable and resources are being
wasted. Deploy less redundancy, ship features faster, or reallocate the excess budget elsewhere.

**When to Apply.** When defining SLOs for a new service. When a team reflexively says "we need
five nines." When evaluating architecture proposals that add complexity in the name of
reliability.

---

## 4. Toil Is a Tax on Engineering Creativity

> *"Toil is the kind of work tied to running a production service that tends to be manual,
> repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as the
> service grows."*
>
> — *Site Reliability Engineering*, Chapter 5 (Toil Elimination)

> *"If a human operator needs to touch your system during normal operations, you have a bug."*
>
> — Carla Geisser, Google SRE (paraphrased in Chapter 5)

**Explanation.** Toil is work that (a) is manual, (b) is repetitive, (c) can be automated,
(d) is tactical (interrupt-driven) rather than strategic, (e) has no enduring value (fixing
the same alert every week creates no lasting improvement), and (f) grows linearly with the
scale of the service. Toil is insidious because it feels productive — you are doing things! —
but it displaces the engineering work that actually improves the system.

**Actionable Implication.** Track toil as a metric. If an SRE spends more than 50% of their
time on toil, the organization is over-investing in operations at the expense of engineering.
Dedicate sprint capacity to eliminating toil: automate the manual runbook, fix the flaky alert,
rewrite the fragile deployment script. Every hour spent on toil is an hour not spent making
the system better.

**When to Apply.** During sprint retrospectives (identify toil patterns), in on-call handoffs
(catalog toil), during incident postmortems (identify toil that contributed to the incident),
and at quarterly planning (allocate toil-reduction work).

---

## 5. Blameless Culture (You Can't Fix People, but You Can Fix Systems)

> *"Blameless postmortems are a tenet of SRE culture. The goal is to focus on identifying the
> contributing causes of the incident without indicting any individual. If a team is
> incentivized to hide or gloss over problems, systemic issues are never fixed."*
>
> — *Site Reliability Engineering*, Chapter 15 (Postmortem Culture)

> *"You can't fix people, but you can fix systems."*
>
> — *Site Reliability Engineering*, Chapter 15 (Postmortem Culture)

**Explanation.** Human error is a symptom of system design flaws, not a root cause. When a
pager goes off at 3 AM and an engineer misses a step in the runbook, the failure is in the
runbook design, the alert configuration, the monitoring coverage, or the fatigue induced by
the on-call rotation — not in the individual engineer. Blameless culture does not mean
"no accountability"; it means holding the *system* accountable for tolerating — or preventing —
human error.

**Actionable Implication.** Postmortems must never use the words "should have," "could have,"
or "would have." Replace "operator error" with "a gap in runbook coverage" or "insufficient
automation guardrails." Every postmortem action item must be a system change, not a training
requirement. When an incident involves human error, the question is "what in our system allowed
that error to cause harm?" not "why did that person make a mistake?"

**When to Apply.** Every incident postmortem, every root cause analysis, every time a team
is tempted to blame an individual for a production issue.

---

## 6. Measure Everything with SLOs and SLIs

> *"If you can't measure it, you can't manage it. An SLO-bounded error budget is what allows
> SRE teams to make data-driven decisions about the trade-off between reliability and feature
> velocity."*
>
> — *Site Reliability Engineering*, Chapter 4 (Service Level Objectives)

> *"The most important thing about an SLI is that it actually corresponds to user-facing
> reliability. A metric that doesn't reflect user experience is worse than no metric at all,
> because it gives a false sense of confidence."*
>
> — *Site Reliability Engineering*, Chapter 6 (Monitoring Distributed Systems)

**Explanation.** Service Level Indicators (SLIs) are the raw measurements — request latency,
error rate, throughput, availability. Service Level Objectives (SLOs) are the target thresholds
for those indicators. Together they provide an objective, quantitative definition of what
"reliable enough" means. Without SLIs and SLOs, reliability is a matter of opinion — the SRE
thinks the system is failing and the product manager thinks it is fine, and there is no shared
data to resolve the disagreement.

**Actionable Implication.** Every service must have at least one user-facing SLI and SLO before
it can be meaningfully operated. Start simple: latency at the 99th percentile and error rate
are usually sufficient. Do not create dashboards full of metrics without corresponding SLO
targets — measurement without targets is noise. SLIs should always measure from the user's
perspective (end-to-end), not from internal infrastructure metrics.

**When to Apply.** Before any service is deemed "production." During incident response to
verify the service is back within SLO. During capacity planning to understand whether
growth threatens SLO attainment. At quarterly reviews to track reliability trends.

---

## 7. Automation Over Manual Operations

> *"The SRE approach to operations is a simple tenet: we vastly prefer systems and automation
> over humans doing things manually. Automation is a force multiplier, not a job replacer."*
>
> — *Site Reliability Engineering*, Chapter 5 (Toil Elimination) & Chapter 33 (Automation)

> *"The best automation is the automation that never runs. If you can remove the need for a
> task entirely, that is superior to automating the task."*
>
> — *Site Reliability Engineering*, Chapter 33 (Automation)

**Explanation.** Automation serves three purposes at scale: consistency (machines follow the
same steps every time), speed (automated recovery happens in seconds, human recovery in
minutes or hours), and efficiency (a single engineer can manage orders of magnitude more
infrastructure through automation than through manual clicks). But the hierarchy matters:
before automating a painful manual process, ask whether the process can be eliminated
entirely. Automation of a bad process simply makes bad things happen faster.

**Actionable Implication.** Build automated runbooks for the most common operational actions
(restart, rollback, scale up/down). Use ChatOps or a bot to trigger them. Target a state where
a brand-new engineer with zero domain knowledge can, by typing a single command, perform any
routine operational task that would otherwise take a senior engineer 10 minutes of manual work.
Before automating anything, first ask: "can we make this failure case impossible by changing
the system design?"

**When to Apply.** Whenever a manual step is identified in a runbook. Whenever the same
operational task has been performed more than three times. Whenever an incident could have
been mitigated faster with an automated response.

---

## 8. Simplicity Is a Prerequisite for Reliability

> *"Reliability is inversely proportional to complexity. Every line of code, every dependency,
> every configuration knob is a potential failure mode. The most reliable systems are the
> simplest ones."*
>
> — *Site Reliability Engineering*, Chapter 20 (Load Balancing Layer) & Chapter 22 (Reliable Product Launches)

> *"A simple system that works reliably is infinitely better than a clever system that works
> most of the time."*
>
> — *Site Reliability Engineering*, Chapter 22 (Reliable Product Launches)

**Explanation.** Complexity is the primary enemy of reliability. Each moving part — each
service, database, queue, configuration flag, middleware layer — adds failure modes,
debugging surface area, and operational burden. The SRE philosophy favors straightforward,
well-understood solutions (even if they are less "elegant") over architecturally ambitious
designs. "Boring" infrastructure is reliable infrastructure.

**Actionable Implication.** Actively resist architectural complexity. Use the "cognitive load"
test: if understanding a single failure scenario requires holding more than a handful of
components in your head simultaneously, the system is too complex. Prefer monoliths over
microservices for teams that cannot justify the operational overhead. Remove unused code,
features, and configuration aggressively. Every deleted line of code is a prevented incident.

**When to Apply.** During architecture reviews (push back on unnecessary abstractions).
During incident postmortems (identify which complexity contributed). During code reviews
(reject over-engineered solutions). During quarterly architecture audits (dedicate time to
simplification).

---

## 9. Slow Is Smooth, Smooth Is Fast (Incident Response Discipline)

> *"In a time-critical incident, the most important thing is to remain calm and follow the
> process. Going fast by yourself will be slower than going together as a team."*
>
> — *Site Reliability Engineering*, Chapter 13 (Emergency Response)

> *"The difference between a good incident response and a bad one is not technical skill —
> it's discipline. The discipline to follow the process, the discipline to engage the right
> people, and the discipline to stop and think before acting."*
>
> — *Site Reliability Engineering*, Chapter 13 (Emergency Response)

**Explanation.** Incident response is an exercise in controlled urgency. The natural instinct
when a production incident occurs is to fix it as fast as possible — jump into the system,
try things, escalate laterally. This instinct is wrong. The fastest path to recovery is
almost always: (1) declare the incident formally, (2) assemble the response team with clear
roles (Incident Commander, Operations Lead, Comms Lead), (3) triage before acting, and
(4) act deliberately. "Slow is smooth, smooth is fast" means the initial overhead of process
is repaid many times over by avoiding mistakes, confusion, and wasted effort.

**Actionable Implication.** Adopt explicit incident command protocols (similar to firefighting
ICS/NIMS). Practice them in drills and tabletop exercises. The Incident Commander should
*not* be debugging — their job is to coordinate the response, keep time, and ensure roles
are filled. After every major incident, evaluate the *process* (not just the technical cause):
was the incident declared promptly? Were roles filled? Did communications stay on the
designated channel?

**When to Apply.** Every time a production incident is declared. During incident response
drills (practice the process, not the technology). When training new on-call engineers.
When a postmortem reveals that confusion, not technical failure, slowed recovery.

---

## 10. Production Is the Only Thing That Matters

> *"If you haven't run your service in production, you don't understand it. Production reveals
> failure modes that no amount of staging, testing, or simulation can uncover."*
>
> — *Site Reliability Engineering*, Chapter 29 (Software Engineering in SRE)

> *"The production environment is not a development environment, and it must be protected by
> mechanical sympathy, gradual rollout, and safe deployment practices."*
>
> — *Site Reliability Engineering*, Chapter 29 (Software Engineering in SRE)

**Explanation.** There is an unbridgeable gap between how a system behaves in staging and how
it behaves in production. Production has real user traffic, real data shapes, real network
conditions, real contention, and real failure modes that no test environment can faithfully
reproduce. SRE treats production as the authoritative environment. All engineering decisions —
deployment strategies, rollout speeds, configuration changes, capacity planning — are made
with production as the reference frame.

**Actionable Implication.** Invest in production telemetry and observability above all else.
Staging environments should be treated as validation tools, not as reliable proxies for
production behavior. Use canary deployments, gradual rollouts, and feature flags to safely
verify changes in the real environment. Production access is a privilege, not a right — it
requires training, tools, and safety mechanisms.

**When to Apply.** When designing deployment pipelines (always include canary/progressive
rollout). When evaluating monitoring (does it measure production behavior end-to-end?).
When deciding what to test (prioritize production-proxy tests over staging-only tests).
When training engineers (production safety and awareness is a mandatory competency).

---

## 11. The 50% Engineering Time Mandate

> *"SRE teams must spend no more than 50% of their time on operational work (toil and
> incident response). The remaining 50% is reserved for engineering projects that reduce
> toil, improve reliability, and increase service capacity."*
>
> — *Site Reliability Engineering*, Chapter 5 (Toil Elimination)

> *"If operational work regularly exceeds 50% of an SRE team's time, the team is too small
> for the service they are operating. The solution is not to work harder — it is to either
> reduce the operational load or grow the team."*
>
> — *Site Reliability Engineering*, Chapter 5 (Toil Elimination)

**Explanation.** The 50% cap on operational work is the defining operational constraint of SRE
as a discipline. It ensures that SRE teams are engineering teams that do operations, not
operations teams that occasionally write scripts. The cap forces hard decisions: if on-call
load, pager volume, and manual tasks consume more than 50% of the team's time, the team must
either automate the work, push back on accepting additional service ownership, or grow.

**Actionable Implication.** Time-track operational vs. engineering work explicitly. Use the
50% threshold as a forcing function: if you are above it, the team's OKRs must include
toil-reduction projects. Managers must protect the engineering allocation from being eroded
by operational firefighting. If the team sustains >50% operations for multiple quarters, the
situation requires organizational intervention (headcount, service transfer, or SLO relaxation).

**When to Apply.** During quarterly capacity planning. When evaluating whether to take on a
new service. During one-on-ones and team health assessments. When the team consistently feels
"too busy to improve things."

---

## 12. Sublinear Scaling as the Goal

> *"The goal of SRE is to build systems that scale sublinearly with service growth. If you
> double the size of the service, you should not need to double the size of the SRE team."*
>
> — *Site Reliability Engineering*, Chapter 1 (Introduction)

> *"A team of five SREs should be able to manage a fleet that is ten times larger than the
> fleet that required a team of five SREs a year ago."*
>
> — *Site Reliability Engineering*, Chapter 1 (Introduction)

**Explanation.** Linear scaling (double the machines, double the operators) is the default
state of manual operations — and it is unsustainable. The entire philosophy of SRE —
automation, simplification, toil elimination — is motivated by the goal of sublinear scaling.
If service growth requires proportional headcount growth, the team is doing operations work,
not engineering work. True SRE success means the team's span of control (the size and
complexity of the system they operate) grows faster than the team itself.

**Actionable Implication.** Track the ratio of team size to service footprint (servers, QPS,
services owned, data volume). This ratio is a leading indicator of SRE health. When the ratio
degrades (more resources per engineer owned), invest in automation and simplification before
asking for more headcount. A healthy SRE organization should be able to absorb 2x service
growth with no more than a marginal increase in team size.

**When to Apply.** When justifying headcount requests. When evaluating whether automation
investments are paying off. When a service is growing rapidly and the team feels the strain.
During architectural discussions about service boundaries and ownership.

---

## 13. The Hierarchy of Production Needs

> *"A service must meet its foundational requirements — monitoring, incident response, and
> capacity planning — before higher-order concerns like feature velocity or developer
> productivity can be addressed."*
>
> — *Site Reliability Engineering*, Chapter 6 (Monitoring Distributed Systems) & Chapter 17 (Testing for Reliability)

> *"You cannot deploy reliably if you don't know whether your service is healthy. You cannot
> know whether your service is healthy if you aren't monitoring it. Monitoring is the
> foundation on which all other SRE practices rest."*
>
> — *Site Reliability Engineering*, Chapter 6 (Monitoring Distributed Systems)

**Explanation.** Reliability has a dependency hierarchy, analogous to Maslow's hierarchy of
needs. At the base is monitoring and observability — without knowing your system's state,
nothing else is possible. Above that: incident response (the ability to react when monitoring
alerts). Above that: capacity planning (ensuring resources match demand). Above that:
deployment and change management (safely evolving the system). At the top: feature velocity
and developer productivity. Skipping lower levels makes higher levels impossible or dangerous.

**Actionable Implication.** When onboarding a new service onto an SRE team, audit the
hierarchy bottom-up. Is monitoring in place? Is there a documented incident response process?
Is there capacity planning? Is there a safe deployment pipeline? Do not accept a service
into production that has gaps in the lower tiers. Fix the foundation before building on it.

**When to Apply.** When accepting a new service for SRE support. When the team's reliability
problems feel overwhelming (fix the bottom of the hierarchy first). When planning reliability
improvements. When a service has frequent incidents that the team cannot explain.

---

## 14. The "Reliability Enough" Philosophy

> *"The goal is not to maximize reliability; it is to provide reliability that is sufficient
> for the service's business purpose, at a cost that is justified by the value."*
>
> — *Site Reliability Engineering*, Chapter 3 (Embracing Risk)

> *"A service that is 'too reliable' is wasting resources that could be spent on features,
> performance, or other services. The error budget is the mechanism for detecting and
> correcting over-investment in reliability."*
>
> — *Site Reliability Engineering*, Chapter 4 (Service Level Objectives)

**Explanation.** "Reliability enough" is the practical expression of SRE's risk-embracing
philosophy. It means explicitly choosing a reliability target that is "good enough" for the
business context — not the maximum achievable, not what competitors do, not what feels safe.
A development API that serves internal teams can run at 99% (8 hours of downtime per month)
without meaningful harm. A payment gateway needs 99.99% or higher. The choice is deliberate
and data-informed, not reactive or aspirational.

**Actionable Implication.** Every service should have regularly reviewed SLOs that reflect the
business value of that service to its users. Services with no direct user-facing impact should
have lower SLO targets. Review SLOs quarterly; if a service never exhausts its error budget,
raise the question: "are we over-investing in this service's reliability?" The answer may be
"no, it's just well-run," but the question must be asked explicitly.

**When to Apply.** During annual or quarterly SLO reviews. When a team wants to raise a
service's SLO target. When a team is evaluating new architecture for an existing service.
When the error budget is consistently under-consumed and the team is wondering why.

---

## Cross-Reference: Principle to SRE Book Chapter

| # | Principle | Primary Chapter(s) | Secondary Chapter(s) |
|---|-----------|--------------------|-----------------------|
| 1 | Reliability Is a Feature | Ch. 1 — Introduction | Ch. 29 — Software Engineering in SRE |
| 2 | Error Budgets | Ch. 4 — Service Level Objectives | Ch. 3 — Embracing Risk |
| 3 | Embrace Risk | Ch. 3 — Embracing Risk | Ch. 4 — SLOs |
| 4 | Toil Is a Tax | Ch. 5 — Toil Elimination | Ch. 6 — Monitoring |
| 5 | Blameless Culture | Ch. 15 — Postmortem Culture | Ch. 12 — Effective Troubleshooting |
| 6 | Measure Everything (SLOs/SLIs) | Ch. 4 — SLOs / Ch. 6 — Monitoring | Ch. 3 — Embracing Risk |
| 7 | Automation Over Manual Ops | Ch. 5 — Toil Elimination / Ch. 33 — Automation | Ch. 8 — Release Engineering |
| 8 | Simplicity | Ch. 20 — Load Balancing / Ch. 22 — Reliable Launches | Ch. 31 — Simplicity |
| 9 | Slow Is Smooth, Smooth Is Fast | Ch. 13 — Emergency Response | Ch. 14 — Incident Management |
| 10 | Production Is the Only Thing That Matters | Ch. 29 — Software Engineering in SRE | Ch. 8 — Release Engineering |
| 11 | 50% Engineering Time Mandate | Ch. 5 — Toil Elimination | Ch. 6 — Monitoring |
| 12 | Sublinear Scaling | Ch. 1 — Introduction | Ch. 33 — Automation |
| 13 | Hierarchy of Production Needs | Ch. 6 — Monitoring / Ch. 17 — Testing | Ch. 8 — Release Engineering |
| 14 | "Reliability Enough" Philosophy | Ch. 3 — Embracing Risk / Ch. 4 — SLOs | Ch. 2 — Production Environment |

---

## How to Use This Reference

1. **Onboarding.** New SRE team members should read through each principle as part of their
   orientation. The principles provide the conceptual vocabulary for all team decisions.

2. **Decision Making.** When facing a trade-off (e.g., "should we invest in more redundancy or
   ship this feature?"), reference the relevant principle. The principles encode decades of
   accumulated operational wisdom at Google scale.

3. **Postmortems and Reviews.** Use the principles as evaluation criteria. Did we violate
   "simplicity" in this design? Did "toil" play a role in this incident? Did we respect the
   "error budget" in our release cadence?

4. **Cultural Alignment.** The principles are not just technical guidelines — they are the
   cultural DNA of SRE. A team that lives these principles will naturally prioritize
   automation, embrace blameless learning, and resist the impulse to chase perfection at
   the expense of everything else.

---

> *"SRE is fundamentally about applying a software engineering mindset to operations problems.
> The principles above are the axioms that fall out of that mindset. Internalize them, and
> the operational decisions become obvious."*
>
> — *Site Reliability Engineering*, Chapter 1 (Introduction) — adapted

# Human Systems, Learning, and Sustainable SRE

Reliability work is cognitive and social work as well as technical work. Use this reference when designing on-call, incident training, postmortem learning, documentation, team health, or operational policies that affect people.

## Source anchors

This reference synthesizes *Seeking SRE*, “Do Docs Better,” “Active Teaching and Learning,” “Psychological Safety in SRE,” “SRE Cognitive Work,” “Beyond Burnout,” and “Against On-Call: A Polemic,” with the practical on-call, incident, and postmortem chapters of *The Site Reliability Workbook*. These sources include opinion and experience as well as general practice. Treat claims about health, accommodation, and employment as context for humane design, not medical or legal advice.

## Design for cognitive work

During an incident, operators are not interchangeable sensors. They interpret incomplete signals, form and revise hypotheses, coordinate across boundaries, and choose actions under time pressure. Reduce avoidable cognitive load:

- provide a small set of trusted dashboards and a current service map;
- make dependencies, recent changes, and rollback paths easy to find;
- separate incident command from deep technical investigation;
- record decisions and hypotheses so responders do not repeatedly reconstruct context;
- prefer reversible actions and observe their effects;
- make escalation normal and fast;
- preserve a quiet channel for coordination when the main channel is noisy.

Automation should remove repetitive work while keeping the operator's decision boundary visible. A system that acts automatically must expose what it observed, what it changed, and how to stop or reverse it.

## On-call is a system property

Do not judge an on-call rotation only by whether pages are answered. Review:

- page volume, actionability, urgency, and repeat rate;
- time spent on pages, tickets, handoffs, and recovery;
- interruptions outside shifts and the quality of escalation;
- whether responders have current runbooks, access, context, and training;
- whether the service can be operated without heroic individual knowledge;
- recovery time and post-incident cognitive load;
- whether the rotation leaves time for engineering that reduces future load.

A rotation that relies on exhausted people, undocumented exceptions, or constant escalation is not reliable even when uptime looks acceptable. If the service cannot support a sustainable rotation, reduce scope, improve automation, change the service, or stop carrying the pager until the risk is acknowledged.

## Psychological safety in reliability work

Effective teams make it safe to report uncertainty, challenge a plan, ask for help, and surface a risk raised by a junior responder. Psychological safety is not permission to skip accountability. It is a condition for receiving the information needed for accountability.

Make it observable:

- the incident commander explicitly invites dissent and clarifying questions;
- a responder can pause a rollout or mitigation when a stated safety condition is violated;
- postmortems analyze system conditions rather than assign personal fault;
- leaders respond to bad news with investigation and support, not punishment;
- action items improve interfaces, defaults, tests, access, or training rather than merely reminding people to be careful.

A blameless culture still names decisions, conditions, and owners. “No blame” must not become “no learning” or “no follow-through.”

## Documentation as an operational control

Documentation is part of the service's reliability surface. A document should have:

- a named audience and task;
- prerequisites and access requirements;
- observable symptoms and scope boundaries;
- safe first actions and explicit stop conditions;
- verification after each consequential action;
- escalation path and rollback or recovery path;
- owner, review date, and links to source configuration or dashboards.

Keep documentation in the engineering workflow. Update it when a runbook is used, an incident exposes a gap, a configuration changes, or a new operator is trained. Stale documentation should be treated as an operational defect, not as a writing problem deferred indefinitely.

## Practice before production

Incident command and mitigation are skills. Use bounded, low-risk practice:

- scenario exercises with realistic but synthetic failures;
- game days and disaster-recovery drills;
- tabletop exercises for third-party and communication failures;
- shadow shifts and graduated on-call access;
- short drills that rehearse one capability, such as rollback or traffic shedding.

After an exercise, record what participants could and could not find, decide, or execute. The learning objective is not theatrical realism. It is to expose missing context, unsafe defaults, confusing roles, and untested recovery paths before a real incident.

## Learning loops

A postmortem is useful only when it changes future capability. Link each action to:

- the failure condition it addresses;
- an owner with authority and time;
- a due date or review checkpoint;
- an observable completion criterion;
- a later verification that the change reduced risk or toil.

Track recurring triggers across incidents. A single incident may be local; a pattern across services may justify a platform, design, training, or organizational intervention. Do not optimize for a rising closure count if the actions are vague or unverified.

## Inclusion and sustainable work

Design for different working styles and access needs without requiring people to disclose private medical information to participate safely. Practical safeguards include written incident roles, asynchronous handoff records, predictable escalation paths, quiet channels, clear interruption expectations, and multiple ways to contribute during reviews and exercises.

Do not diagnose, infer, or prescribe. Ask what working arrangement or interface makes the task safe and effective, follow applicable organizational processes, and keep private information out of operational artifacts.

## When on-call should be redesigned

Treat these as design signals, not individual failure:

- persistent high page volume or repeated non-actionable pages;
- responders cannot complete engineering work;
- incidents depend on one person or one undocumented procedure;
- escalation is delayed because people fear being judged;
- frequent sleep disruption, burnout, or unsafe fatigue;
- the service has no credible degraded mode or recovery test;
- the team cannot staff the rotation without coercion.

The response may be alert reduction, service simplification, ownership change, scope reduction, a dedicated operations function, or retiring the service. “Try harder” is not a reliability strategy.

## Agent procedure

When asked to improve on-call or incident learning, load this reference with `oncall-best-practices.md`, `incident-command-system.md`, and `postmortem-culture.md`. Include both service metrics and human-work metrics. Distinguish observed evidence, team-reported experience, and proposed safeguards.

# Incident Command System (ICS) for Technology Incidents

> **Purpose:** Adapt the National Incident Management System (NIMS) / Incident Command System (ICS) — proven in emergency management — to technology and SRE incidents. This reference provides a complete framework for structured, role-based incident response in production environments.
>
> **Audience:** SREs, on-call engineers, incident commanders, engineering managers, and post-incident reviewers.

---

## Table of Contents

1. [NIMS/ICS Adapted for Tech Incidents](#1-nimsics-adapted-for-tech-incidents)
2. [Incident Command Roles](#2-incident-command-roles)
3. [Command Hierarchy and Reporting Structure](#3-command-hierarchy-and-reporting-structure)
4. [Role Handoff Protocols](#4-role-handoff-protocols)
5. [Incident Severity Classification](#5-incident-severity-classification)
6. [Communication Channels and War Room Logistics](#6-communication-channels-and-war-room-logistics)
7. [Timeline Reconstruction Methods](#7-timeline-reconstruction-methods)
8. [Decision Authority Boundaries](#8-decision-authority-boundaries)
9. [Post-Incident Actions](#9-post-incident-actions)
10. [Appendix: Role Cards](#10-appendix-role-cards)

---

## 1. NIMS/ICS Adapted for Tech Incidents

### 1.1 What is ICS?

The Incident Command System is a standardized, on-scene, all-hazards incident management approach originally developed for wildfire response in the 1970s. It was later adopted by FEMA as part of the National Incident Management System (NIMS). ICS is:

- **Scalable** — works for a 2-person database issue or a 40-person multi-service outage.
- **Modular** — only activate the roles you need.
- **Common terminology** — everyone on the call knows what "IC" means.
- **Unified command** — a single incident commander makes decisions.

### 1.2 Why ICS for Tech Incidents?

Technology incidents share critical characteristics with emergency response:

| Emergency Management | Technology Incident |
|---|---|
| Wildfire spreading | Cascading service failures |
| Multiple responding agencies | Multiple engineering teams (DB, networking, app) |
| Command post coordination | War room / bridge call |
| Situation reports (SITREPs) | Status updates to stakeholders |
| Incident Action Plans | Mitigation strategy and runbooks |
| Demobilization | Return to normal operations |
| After-action review | Post-incident review (PIR) |

### 1.3 Key ICS Principles

1. **Unity of Command** — Every person reports to exactly one supervisor.
2. **Span of Control** — Manageable ratio: 3–7 direct reports per supervisor (ideal 5).
3. **Modular Organization** — Activate only the needed functions.
4. **Incident Action Planning** — Each operational period has clear objectives.
5. **Integrated Communications** — One primary communication channel, one scribe channel.
6. **Comprehensive Resource Management** — Track who is doing what.
7. **Transfer of Command** — Formal handoff procedures for every role change.

### 1.4 ICS Organizational Chart for Tech Incidents

```
                     ┌─────────────────────┐
                     │  Incident Commander  │
                     │        (IC)         │
                     └──────────┬──────────┘
                                │
              ┌─────────────────┼─────────────────┐
              │                 │                 │
     ┌────────┴────────┐ ┌─────┴──────┐ ┌───────┴────────┐
     │  Deputy / Ops   │ │ Scribe /   │ │  Comms /       │
     │     Lead        │ │ Logistics  │ │  Liaison       │
     └────────┬────────┘ └────────────┘ └────────────────┘
              │
    ┌─────────┼─────────┬──────────┬──────────┬──────────┐
    │         │         │          │          │          │
  SMEs    SMEs     SMEs     SMEs     External   Legal/
  (DB)    (Net)   (App)    (Sec)     Vendors    Compliance
```

---

## 2. Incident Command Roles

### 2.1 Incident Commander (IC)

**Tagline:** "I am the IC. I own the timeline and decisions."

The IC is the single person ultimately responsible for the incident response. They do **not** debug. They manage.

**Responsibilities:**

| Area | Details |
|---|---|
| Establish Command | Declare the incident, assign initial roles, open the war room channel. |
| Situation Assessment | Understand the scope, severity, and impact. Make the initial severity classification. |
| Strategy | Set response objectives (Isolate? Mitigate? Rollback? Fix-forward?). |
| Resource Management | Call in additional engineers, escalate to management, engage vendors. |
| Decision Authority | Approve breaking-glass actions, feature flag changes, rollbacks, traffic reroutes. |
| Communication | Provide structured updates to the scribe and liaison. Sign off on external communications. |
| Transitions | Manage role handoffs. Transfer command when shift ends or situation escalates. |
| Termination | Declare incident resolved only after the [R-01 closure evidence sequence](../templates/runbook-template.md#r-01-post-incident-steps), stability window, and independent human confirmation by someone other than the acting automation pass. Approve the move from response to recovery. |

**What the IC should be saying:**
- "I am now the Incident Commander."
- "What is the current impact? How many users/customers?"
- "What have we tried? What's the next best action?"
- "Scribe, note this decision: we are rolling back v2.14.1 to v2.13.9."
- "I am handing off IC to Sarah. Sarah, you have command."

**What the IC should NOT be doing:**
- SSHing into servers
- Writing code
- Looking at dashboards for more than 30 seconds
- Getting into technical rabbit holes

### 2.2 Deputy / Operations Lead

**Tagline:** "I run the technical response so the IC can command."

The Operations Lead (or Deputy IC) manages the technical execution of the response plan. This is the senior technical person who coordinates the SMEs.

**Responsibilities:**

| Area | Details |
|---|---|
| Technical Coordination | Triage incoming issues, assign tasks to SMEs, track what's been tried. |
| Runbook Execution | Ensure standard runbooks are followed; adapt if they don't fit. |
| Parallel Work | Split the team: "Alice, check the database. Bob, look at the CDN. Carol, examine the app logs." |
| Situation Updates | Feed concise technical status to the IC every few minutes or on significant changes. |
| Handoffs | Brief incoming operations lead so technical context isn't lost. |

**What Ops should be saying:**
- "IC, here's my assessment: it's a database connection pool exhaustion."
- "Alice, you're on DB replication lag. Bob, you're on app error rates."
- "We tried restarting the primary. No change. Next step: failover to replica."

### 2.3 Scribe / Logistics

**Tagline:** "I write down everything so nobody has to remember."

The Scribe is the historian and logistics coordinator. This is arguably the second most important role after IC. In many incidents, the IC also scribes for the first few minutes until a scribe is found.

**Responsibilities:**

| Area | Details |
|---|---|
| Timeline Logging | Record every action, decision, observation with timestamps. |
| Decision Documentation | Capture key decisions and the rationale behind them. |
| Action Tracking | Maintain a running list of open actions, who owns them, and status. |
| Resource Logistics | Coordinate conference bridges, war room access, tool credentials, food. |
| Shift Management | Track who is on-call and when shift changes should happen. |
| Session Recording | Record the bridge call / war room chat for post-incident review. |

The scribe produces the **incident log** — the raw timestamped record that becomes the backbone of the post-incident review.

**What the Scribe should be saying:**
- "Can you repeat that decision? I want to capture the rationale."
- "Current action items: 1) Alice — test failover. 2) Bob — check DNS propagation."
- "IC, we've been in incident for 47 minutes. Do you want to escalate to the VP?"

**Log entry format:**
```
[T+00:05] IC declares SEV-2 incident — API latency above 5s p99
[T+00:07] Alice assigned to check database connection pool
[T+00:09] Decision: roll back frontend from v42 to v41 (IC approved)
[T+00:12] Bob reports: DB pool at 100%, connections not releasing
```

### 2.4 Comms / Liaison

**Tagline:** "I keep everyone who isn't on this call informed."

The Communications / Liaison role manages the external communication channels. This is the buffer between the incident response team and the rest of the organization (and possibly customers).

**Responsibilities:**

| Area | Details |
|---|---|
| Internal Status Updates | Send regular updates to the #incident-status channel, email distribution lists, or Slack. |
| Stakeholder Management | Brief executives, product managers, and customer success. |
| External Communication | Draft and coordinate customer-facing status page messages (e.g., Statuspage). |
| Escalation Notification | Alert on-call managers, VPs, and the CTO as per severity escalation policy. |
| FAQ Management | Collect common questions from stakeholders and provide consistent answers. |

**Communication cadence:**

| Severity | Internal Updates | External Status Page |
|---|---|---|
| SEV-1 | Every 15 minutes | Every 30 minutes |
| SEV-2 | Every 30 minutes | Only if customer-visible |
| SEV-3 | At key milestones | Not required |
| SEV-4 | Once at resolution | Not required |
| SEV-5 | Not required | Not required |

**What Comms should be saying:**
- "IC, the VP of Engineering is asking for a status update. What can I share?"
- "Status page updated: 'We are investigating increased latency on the API.'"
- "Next internal update due in 5 minutes. IC, do you have anything new?"

### 2.5 Subject Matter Experts (SMEs)

**Tagline:** "I fix things. Tell me what to look at."

SMEs are the engineers doing the hands-on technical work. They report to the Operations Lead.

**Responsibilities:**

| Area | Details |
|---|---|
| Investigation | Diagnose the issue in their domain (database, networking, application, security). |
| Mitigation | Execute mitigation actions — restarts, rollbacks, config changes, traffic re-routing. |
| Reporting | Report findings clearly to Ops Lead: "What I found, what I tried, what I recommend." |
| Runbook Execution | Follow standard operating procedures; escalate when they don't apply. |

**SME types commonly needed:**

| SME | Domain |
|---|---|
| Database SME | Replication, connection pooling, query performance, failover |
| Networking SME | DNS, BGP, CDN, load balancers, firewall rules |
| Application SME | Service code, deployment pipeline, feature flags, configuration |
| Security SME | Intrusion detection, DDoS, access control anomalies |
| Infrastructure SME | Kubernetes, cloud provider, CI/CD pipelines, observability stack |
| Vendor SME | Third-party SaaS dependencies (Datadog, PagerDuty, cloud provider support) |

---

## 3. Command Hierarchy and Reporting Structure

### 3.1 Who Reports to Whom

```
                                    Incident Commander (IC)
                                           │
                    ┌──────────────────────┼──────────────────────┐
                    │                      │                      │
              Ops Lead              Scribe/Logistics         Comms/Liaison
                    │
     ┌──────────────┼──────────────┬──────────────┐
     │              │              │              │
  DB SME      App SME        Net SME       Sec SME
```

### 3.2 Reporting Rules

1. **SMEs report to Ops Lead, not to IC.** If an SME has an update, they tell Ops. Ops filters and escalates to IC.
2. **The IC does not assign tasks directly to SMEs.** The IC tasks Ops; Ops tasks SMEs.
3. **Anyone can speak up if they have critical safety/security information.** The chain bypasses only for imminent danger.
4. **Comms does not report technical details directly to the IC's command chain.** Comms gets their content from the scribe log and IC briefings.

### 3.3 Span of Control

- IC to direct reports: 3–5 (Ops Lead, Scribe, Comms, plus any direct stakeholders like Legal if needed).
- Ops Lead to SMEs: 3–7, ideally 5.
- If the incident requires more than 7 SMEs, split into functional teams (e.g., "DB team," "Networking team") each with their own lead reporting to Ops.

### 3.4 Unified Command (Multi-Organization Incidents)

When the incident spans multiple organizations (e.g., a cloud provider outage affecting your service, or a joint incident with a partner company), a **Unified Command** structure may be used:

- Each organization designates an IC.
- The ICs form a unified command team that makes joint decisions.
- Each IC is responsible for their organization's resources.
- A single spokesperson is designated for external communications.

---

## 4. Role Handoff Protocols

Role handoffs are a critical failure point in incident management. A poorly performed handoff can lose context, decisions, and momentum. Every handoff must be explicit and documented.

### 4.1 General Handoff Principles

1. **Verbal declaration**: The outgoing person explicitly says "I am handing off [ROLE] to [NAME]."
2. **Confirmation**: The incoming person explicitly says "I have accepted [ROLE]."
3. **Scribe records it**: The exact handoff time is logged.
4. **Overlap**: A 3–5 minute overlap period (longer for IC) where the outgoing person is available for questions.
5. **State transfer**: All relevant context is transferred (see checklists below).

### 4.2 IC Handoff Protocol

**When to hand off IC:**
- End of shift (e.g., after 4–6 hours in a prolonged incident)
- Escalation to a more senior engineer or manager
- Fatigue or cognitive overload
- Original IC was the first responder and needs to step back

**IC Handoff Checklist:**

| Step | Action |
|---|---|
| 1 | Outgoing IC announces: "I am handing off IC to [NAME]." |
| 2 | Outgoing IC briefs incoming IC on: current situation, actions taken, decisions made, open tasks, pending escalations, stakeholder status. |
| 3 | Outgoing IC reviews the timeline log with the incoming IC. |
| 4 | Scribe timestamps: "IC transferred from [OUT] to [IN] at [TIME]." |
| 5 | Incoming IC announces to the channel/call: "I am now the Incident Commander." |
| 6 | Outgoing IC remains available for 5 minutes as a resource, then stands down. |
| 7 | Comms updates any stakeholder channels that the IC has changed. |

**IC Handoff briefing template:**
```
--- IC HANDOFF BRIEFING ---
Current severity: [SEV-1]
Situation: [one paragraph summary]
What we know: [bullet points]
What we don't know: [bullet points]
Actions taken: [bullet points]
Open actions: [who is doing what]
Key decisions made: [bullet points]
Pending decisions: [bullet points]
Stakeholders notified: [list]
Escalations needed: [yes/no - to whom]
Risks: [any new risks identified]
```

### 4.3 Ops Lead Handoff Protocol

**Ops Lead Handoff Checklist:**

| Step | Action |
|---|---|
| 1 | Announce: "I am handing off Ops Lead to [NAME]." |
| 2 | Transfer: current technical assessment, what's been tried, what's pending, SME assignments. |
| 3 | Introduce incoming Ops Lead to each SME. |
| 4 | Scribe timestamps the handoff. |
| 5 | Incoming Ops Lead confirms acceptance. |

### 4.4 Scribe Handoff

**Scribe Handoff Checklist:**

| Step | Action |
|---|---|
| 1 | Current scribe shares the running log document with the incoming scribe. |
| 2 | Brief on: format conventions, any shorthand used, pending log entries, tools (the document, recording, etc.). |
| 3 | Tandem log for 2–3 minutes to ensure format continuity. |
| 4 | Scribe timestamps the handoff in the log. |

### 4.5 Comms Handoff

**Comms Handoff Checklist:**

| Step | Action |
|---|---|
| 1 | Transfer: current status message drafts, stakeholder contact list, communication schedule, pending external updates. |
| 2 | Share any feedback received from stakeholders. |
| 3 | Share the scribe log link and the latest approved status message. |
| 4 | Announce the handoff to stakeholders if appropriate. |

### 4.6 Shift Schedules for Extended Incidents

| Duration | Shift Pattern |
|---|---|
| 1–2 hours | Single team |
| 2–4 hours | Single team, consider backup |
| 4–8 hours | Two-team rotation: IC + Ops switch at 4h mark |
| 8–24 hours | Three-team rotation: 8h shifts for IC/Ops/Comms |
| 24+ hours | Full team rotation: complete handoff every 8-12h, rest period for outgoing team |

---

## 5. Incident Severity Classification

### 5.1 SEV Definitions

| Level | Label | Definition | Examples |
|---|---|---|---|
| **SEV-1** | Critical | Complete service outage or severe degradation affecting a significant portion of users. Data loss or security breach. | • Entire site/app down<br>• Customer data accessible to unauthorized users<br>• Payment processing completely broken<br>• Core API returning 500s for all requests |
| **SEV-2** | Major | Significant impairment of core functionality affecting many users but not a total outage. | • Major feature non-functional<br>• Severe latency (>5x normal, >5% of users)<br>• Partial regional outage<br>• Degraded but still operational with workarounds |
| **SEV-3** | Minor | Isolated issue affecting a subset of users or non-critical functionality with acceptable workaround. | • Minor feature broken<br>• Slight latency increase<br>• Cosmetic issues<br>• Single-user or single-tenant issue |
| **SEV-4** | Informational | No user impact. Operational issue that should be investigated during business hours. | • Non-critical alert firing<br>• Slightly elevated error budget consumption<br>• Internal tool issue<br>• Bug with no user-facing impact |
| **SEV-5** | Scheduled / Maintenance | Planned work that has been communicated and approved. No user-facing impact expected. | • Scheduled maintenance<br>• Database migration<br>• Deployment of new feature<br>• Planned failover testing |

### 5.2 Response Time Targets

| Severity | Initial Response | Update Cadence | MITR Target | Escalation |
|---|---|---|---|---|
| SEV-1 | Immediate (<2 min) | Every 15 min internal, 30 min external | <1 hour | VP/Director within 15 min, CTO within 30 min |
| SEV-2 | Within 5 min | Every 30 min internal | <4 hours | Manager within 30 min, Director within 1 hour |
| SEV-3 | Within 30 min | At key milestones, or daily | <24 hours | Team lead within 1 business day |
| SEV-4 | Next business day | Upon resolution | <1 week | None required |
| SEV-5 | At scheduled time | Per change plan | N/A | Per change management process |

### 5.3 Severity Change Protocol

Severity can change during an incident. The process:

1. IC assesses that severity has changed (upgraded or downgraded).
2. IC announces: "We are upgrading from SEV-2 to SEV-1."
3. Scribe records the change.
4. Comms notifies the appropriate escalation contacts for the new severity.
5. All response cadences adjust to the new severity level.

### 5.4 Escalation Chain Example

```
SEV-3 ──> Team Lead ──> Engineering Manager
SEV-2 ──> Engineering Manager ──> Director of Engineering  
SEV-1 ──> Director of Engineering ──> VP of Engineering ──> CTO / CEO
```

| Role | Contact Method | Response SLO |
|---|---|---|
| Primary On-Call | PagerDuty push + SMS + phone | Acknowledge: 2 min |
| Engineering Manager | Phone call, Slack ping | Acknowledge: 5 min |
| Director of Engineering | Phone call | Acknowledge: 15 min |
| VP of Engineering | Phone call | Acknowledge: 30 min |
| CTO | Phone call | Per VP discretion |

---

## 6. Communication Channels and War Room Logistics

### 6.1 Channel Architecture

| Channel | Purpose | Who Has Access |
|---|---|---|
| **Primary Incident Channel** (e.g., Slack #inc-1234) | Real-time coordination, decisions, updates | All incident responders |
| **Scribe Log** (e.g., Google Doc, HackMD, GitHub issue) | Timestamped record of every action/decision | All incident responders |
| **Status Updates** (e.g., Slack #incident-status) | Read-only broadcasts to the wider org | Organization-wide |
| **External Status Page** (e.g., Statuspage) | Customer-facing status | Public |
| **Bridge / Zoom** | Voice coordination | Incident responders + stakeholders by invitation |
| **Backchannel** (e.g., DM, separate Slack channel) | Comms to specific vendors or sensitive discussions | IC + relevant parties |

### 6.2 Channel Naming Convention

```
#inc-<YYYYMMDD>-<brief-description>
Examples:
#inc-20250605-api-latency-spike
#inc-20250605-db-primary-failover
#inc-20250605-security-breach-alert
```

### 6.3 War Room Logistics Checklist

**Immediate (first 5 minutes):**
- [ ] Establish primary incident Slack channel
- [ ] Set up voice bridge / Zoom room
- [ ] Assign initial roles (even if someone wears multiple hats)
- [ ] Create scribe document and share link in channel
- [ ] Pin the scribe doc and bridge link to the channel
- [ ] Announce the incident severity

**First 15 minutes:**
- [ ] Confirm all needed SMEs are on the call
- [ ] Add escalation contacts (if needed for severity)
- [ ] Set up status page (if customer-facing)
- [ ] Begin timeline log in scribe document

**Ongoing:**
- [ ] Rotate scribe every 30–45 minutes (scribe fatigue is real)
- [ ] Keep the incident channel focused — move side conversations to threads or DMs
- [ ] Post a mandatory "no non-incident chatter" message
- [ ] IC reviews the timeline log every 30 minutes for accuracy
- [ ] Comms posts regular updates to stakeholders

### 6.4 Communication Templates

**Initial notification (Slack / PagerDuty):**
```
:rotating_light: INCIDENT DECLARED :rotating_light:
Title: [Brief description]
Severity: SEV-[1-5]
IC: [Name]
Channel: #inc-[date]-[description]
Scribe doc: [link]
Bridge: [link]
Impact: [brief description of user/customer impact]
```

For a Resolved status, first complete the [R-01 closure evidence sequence](../templates/runbook-template.md#r-01-post-incident-steps) and the defined stability window.

**Status update template (internal):**
```
=== STATUS UPDATE #[N] ===
Time: [UTC timestamp]
Duration so far: [X hours, Y minutes]
Severity: SEV-[X] (unchanged/changed from SEV-[Y])
Current status: [INVESTIGATING / MITIGATING / MONITORING / RESOLVED — only after R-01 closure evidence and stability window]
Summary: [brief paragraph]
Next update: [time or milestone]
```

**Resolution announcement:**

Use this only after the [R-01 closure evidence sequence](../templates/runbook-template.md#r-01-post-incident-steps) and the defined stability window pass. If evidence is incomplete, send a **MONITORING** update instead, record the unverified boundary, retain the incident in **MITIGATING** or **MONITORING**, and hand off or escalate.

```
✅ INCIDENT RESOLVED
Title: [title]
Duration: [X hours, Y minutes]
Severity: SEV-[X]
Root cause: [brief description]
Action taken: [brief description]
Closure evidence: [user-facing SLOs / critical journeys / dependencies / data-state / secondary effects / stability window / independent human confirmation]
Monitoring: [what we're watching]
Post-incident review: [link when available]
```

---

## 7. Timeline Reconstruction Methods

### 7.1 Real-Time Scribing (Gold Standard)

The simplest and most reliable method is having a dedicated scribe logging in real time.

**What to log:**
- Each timestamped observation
- Each action taken (and by whom)
- Each decision made (and the IC who made it)
- Each change in severity
- Each role change or handoff
- Each escalation
- Each external communication sent

**Tool options:**

| Tool | Pros | Cons |
|---|---|---|
| Google Docs | Real-time collaborative, familiar | Can lag with many editors, merge conflicts |
| HackMD / HedgeDoc | Real-time markdown, lightweight | Less familiar to non-technical users |
| Dedicated incident platform (FireHydrant, PagerDuty, Blameless) | Structured, integrates with alerting | Requires setup, not always available |
| Git + Markdown | Permanent, reviewable, CI-friendly | Higher friction, no real-time collaboration |
| Slack thread | Fast, zero setup | Hard to reconstruct later, easy to lose context |
| Voice recording (transcribed) | Captures everything | Requires transcription, harder to search |

### 7.2 Post-Incident Reconstruction

If real-time scribing was incomplete, reconstruct the timeline from these sources:

| Source | What It Provides |
|---|---|
| Monitoring / Dashboards | Metric graphs with timestamps (latency, error rates, throughput) |
| Alerting System (PagerDuty, Opsgenie) | First alert timestamp, escalation history |
| Log Aggregator (Splunk, Datadog Logs, ELK) | Error logs, access logs, application logs |
| Deployment System (Spinnaker, ArgoCD, GitHub Actions) | Deployment timestamps, version changes |
| Feature Flag System (LaunchDarkly) | Flag toggle timestamps |
| Change Management System | Change request timestamps and approvals |
| Chat History (Slack, Teams) | All incident channel messages with timestamps |
| Voice Recording / Transcript | Everything said on the bridge call |
| Git History | Code changes, reverts, commit messages |
| Incident Comm System | Scribe log (if any), status updates |

### 7.3 Timeline Reconstruction Process

1. **Collect all sources** — Gather monitoring screenshots, chat logs, deployment logs.
2. **Create a shared timeline document** with a spreadsheet-like format.
3. **Merge data chronologically** — Each source contributes events with timestamps.
4. **Identify gaps** — Periods where no data exists. These are areas to investigate in the PIR.
5. **Validate with participants** — Ask the people on the call to fill in gaps.
6. **Flag uncertain timestamps** — Use `[~HH:MM]` notation for approximate times.
7. **Produce final timeline** — Clean, sorted, with clear event categories.

**Timeline format:**
```
| Timestamp (UTC) | Event | Source | Category |
|---|---|---|---|
| 14:02:00 | PagerDuty alert: API p99 latency >5s | PagerDuty | Alert |
| 14:02:30 | On-call engineer acknowledges | PagerDuty | Response |
| 14:03:15 | Incident declared SEV-2 | Engineer | Decision |
| 14:04:00 | #inc-20250605-api-latency created | Slack | Logistics |
| ... | ... | ... | ... |
```

### 7.4 Timeline Categories

Use consistent event categories to make analysis easier:

| Category | Color / Tag | Examples |
|---|---|---|
| Alert | 🔴 | PagerDuty alert fired, threshold exceeded |
| Detection | 🟡 | Someone noticed, user report, monitoring dashboard |
| Decision | 🔵 | IC made a decision, severity change, strategy shift |
| Action | 🟢 | Rollback executed, server restarted, config changed |
| Communication | 🟣 | Status page updated, exec notified, customer emailed |
| Escalation | 🟠 | Manager called in, vendor contacted, legal notified |
| Resolution | ✅ | Incident declared resolved after closure evidence, independent human confirmation, and a completed stability window; green monitoring alone is insufficient |

---

## 8. Decision Authority Boundaries

### 8.1 Decision Authority Matrix

| Decision | Authority | Consultation Needed |
|---|---|---|
| Declare incident | First responder / IC | None |
| Assign severity | IC | None (but can be challenged) |
| Change severity | IC | None |
| Roll back a deployment | IC | Ops Lead, code owner |
| Feature flag change | IC / Ops Lead | Feature owner if available |
| Restart a database primary | IC | Ops Lead, DB SME |
| Failover to replica/region | IC | Ops Lead, DB SME |
| Scale up infrastructure | IC / Ops Lead | Cloud cost owner (if time permits) |
| Change DNS / routing | IC | Net SME |
| Disable a service | IC | Ops Lead, service owner |
| Contact a vendor for support | IC / Ops Lead | Vendors team |
| Communicate externally | Comms (with IC approval) | Legal (for SEV-1), PR (if customer-facing) |
| Contact legal / compliance | IC | None |
| Contact law enforcement (security breach) | IC + Legal | Executive team |
| Declare incident resolved | IC | The [R-01 closure evidence sequence](../templates/runbook-template.md#r-01-post-incident-steps), completed stability window, and independent human confirmation by someone other than the acting automation pass. If evidence is missing, retain **MITIGATING** or **MONITORING** and hand off or escalate. |
| Authorize post-incident review | IC | None |
| Deploy a hotfix | IC | Ops Lead, code owner, QA (if time permits) |
| Cache invalidation / purge | Ops Lead | Net SME |
| Database query kill / terminate | Ops Lead / DB SME | IC (if impact is broad) |

### 8.2 "Break Glass" Decisions

Certain actions carry high risk but may be necessary. These require explicit IC approval and scribe documentation:

**Break glass actions:**
- Force-restarting a database primary
- Dropping database connections / killing queries
- Disabling authentication or security controls
- Bypassing change management for a deployment
- Manual edits to production data
- Rolling back a database migration
- Hard reboot of infrastructure hosts

**Break glass protocol:**
1. SME says: "I recommend we do [action]. This is a break-glass action with risk [description]."
2. IC asks: "What happens if this fails? What's our rollback?"
3. IC decides. If approved: "Approved. Scribe, log this. Let's proceed."
4. If declined: "Not approved. What's our alternative? Ops Lead, any other options?"
5. Scribe records the decision and rationale regardless of outcome.

### 8.3 Authority Outside Business Hours

During off-hours/weekends, the on-call IC has broader authority:

- Can approve emergency change requests without standard change management
- Can escalate to any level of management (including VP/CTO) without waiting
- Can authorize spending for emergency infrastructure scaling
- Can call in additional engineers from any team

After-hours authority is checked by:
- Mandatory scribe documentation of all decisions
- Post-incident review within 1 business day
- Right-to-challenge by the on-call manager

### 8.4 What the IC Cannot Do Alone

Even the IC has limits:

- Cannot unilaterally terminate employees or contractors
- Cannot make binding legal commitments or admissions of liability
- Cannot disclose customer data outside the incident response team
- Cannot authorize payment to vendors without the finance team (unless pre-approved)
- Cannot override a security hold without security lead concurrence
- Cannot authorize data deletion without documented legal/compliance consultation

---

## 9. Post-Incident Actions

### 9.1 Immediate Actions (Closure Evidence Gate)

| Action | Owner | Details |
|---|---|---|
| Determine monitoring period | IC | Set the stability window during response based on expected traffic, dependency recovery, backlog or secondary effects, and delayed symptoms. Complete it before declaring resolved. |
| Verify mitigation | Ops Lead | Run the canonical [R-01 closure evidence sequence](../templates/runbook-template.md#r-01-post-incident-steps): user-facing SLOs and critical journeys, dependency health, data/state correctness, secondary effects, and the completed stability window. Green monitoring is partial evidence. |
| Declare resolved | IC | Announce resolution only after the closure evidence, stability window, and independent human confirmation by someone other than the acting automation pass. Otherwise retain **MITIGATING** or **MONITORING**, record the unverified boundary, and hand off or escalate. |
| Change severity to SEV-5 | IC | Only after closure evidence, the stability window, and independent human confirmation pass; otherwise retain the active severity and **MITIGATING** or **MONITORING** state |
| Update status page | Comms | Set status page to "Resolved" only after the resolution decision |
| Stop the recording / close bridge | IC | End the voice bridge |
| Send final status update | Comms | Send the resolution announcement |
| Save all materials | Scribe / IC | Archive the scribe doc, channel history, monitoring screenshots |
| Pause non-critical alerts | IC | Prevent alert fatigue from residual effects |


### 9.2 Short-Term Follow-Up (Within 1 Business Day)

| Action | Owner | Details |
|---|---|---|
| Schedule post-incident review | IC or designate | Book 60–90 min within 3–5 business days |
| Complete timeline | Scribe / IC | Fill in any gaps in the timeline from logs and monitoring |
| Triage action items | IC | Create a tracker of all follow-up actions from the incident |
| Assign owners | IC | Every action item needs an owner and due date |
| Create incident ticket | IC | File in the incident tracking system with severity, duration, summary |
| Preserve evidence | Scribe | Collect dashboards, logs, outputs before they expire |
| Notify affected customers | Comms + Legal | Send customer communication if applicable |
| File any required reports | IC / Legal | Regulatory or compliance reporting if applicable |

### 9.3 Post-Incident Review (PIR)

**PIR Schedule:**

| Severity | PIR Required | Timeline | Participants |
|---|---|---|---|
| SEV-1 | Yes | Within 5 business days | All responders, engineering manager, director |
| SEV-2 | Yes | Within 10 business days | All responders, engineering manager |
| SEV-3 | As needed | Within 2 weeks | Team lead and relevant engineers |
| SEV-4 | Optional | N/A | Team lead |
| SEV-5 | No | N/A | N/A |

**PIR Agenda (60–90 minutes):**

| Section | Time | Facilitator |
|---|---|---|
| 1. Set the stage | 5 min | Facilitator (not the IC!) |
| 2. Timeline walkthrough | 20 min | Scribe / IC |
| 3. What went well | 10 min | All participants |
| 4. What went wrong | 15 min | All participants |
| 5. What was confusing | 10 min | All participants |
| 6. Action item generation | 10 min | All participants |
| 7. Summary and next steps | 5 min | Facilitator |

**PIR Principles (Blameless Postmortem):**
- Assume good intent from everyone
- Focus on systems, not individuals
- The goal is learning, not accountability for the incident
- Every action item should be a system change (runbooks, automation, monitoring, architecture)
- Track action items to completion with owners and due dates

**PIR Output Document:**

```markdown
# Post-Incident Review: [Title]

**Incident ID:** INC-YYYY-MM-DD-XXX
**Date:** YYYY-MM-DD
**Duration:** Xh Ym
**Severity:** SEV-X
**Services affected:** [list]

## Timeline
[UTC timestamps of key events]

## Impact
- Users affected: [count or percentage]
- Revenue impact: [if measurable]
- Duration: [total outage or degradation time]

## Root Cause
[Clear description of the root cause]

## Trigger
[What set off the incident]

## Detection
[How was it first detected? How long from occurrence to detection?]

## Response
[What worked well in the response? What didn't?]

## What Went Well
- [item]
- [item]

## What Went Wrong
- [item]
- [item]

## What Was Confusing
- [item]
- [item]

## Action Items
| # | Action | Owner | Due Date | Status |
|---|---|---|---|---|
| 1 | ... | ... | ... | Open |
| 2 | ... | ... | ... | Open |
```

### 9.4 Action Item Tracking

Action items from the PIR must be tracked to completion:

| Severity of Incident | Action Item Due |
|---|---|
| SEV-1 | Critical: within 30 days |
| SEV-2 | Major: within 60 days |
| SEV-3 | Minor: within 90 days |
| SEV-4/5 | Optional: next planning cycle |

**Action item tracker columns:**
- ID
- Description
- Owner
- Due date
- Status (Open / In Progress / Done / Won't Do)
- Linked incident
- Priority

### 9.5 Incident Data Retention

| Artifact | Retention Period |
|---|---|
| Scribe log | 1 year |
| Chat history | 1 year |
| Monitoring screenshots | 90 days |
| Voice recording | 30 days (or as required by compliance) |
| Post-incident review | 2 years |
| Action item tracker | Until all items closed + 1 quarter |

---

## 10. Appendix: Role Cards

### Role Card: Incident Commander (IC)

The Incident Commander is the single decision-maker responsible for the entire incident response. They do not debug, do not SSH into servers, and do not chase metrics. Instead, they maintain the big picture: understanding the scope and severity of the incident, setting response strategy, managing resources and escalations, and making all key decisions. The IC approves break-glass actions, communicates the response strategy to the scribe and liaison, and decides when the incident is resolved only after the [R-01 closure evidence sequence](../templates/runbook-template.md#r-01-post-incident-steps), stability window, and independent human confirmation by someone other than the acting automation pass. They are accountable for the safety of the response and for ensuring that the right people are working on the right problems. In extended incidents, the IC manages shift rotations and role handoffs to prevent responder fatigue.

### Role Card: Deputy / Operations Lead

The Operations Lead (also called Deputy IC or Ops Lead) runs the technical side of the response so the IC can focus on command. They receive technical updates from Subject Matter Experts, triage incoming information, and assign investigation and mitigation tasks. The Ops Lead maintains the technical picture — what has been tried, what is currently being investigated, and what the next-best technical action should be. They filter and summarize technical status for the IC in concise, actionable updates. In a large incident, the Ops Lead may delegate sub-teams (e.g., a database team, a networking team), each with their own lead. The Ops Lead role requires deep technical experience and the ability to think clearly under pressure.

### Role Card: Scribe / Logistics

The Scribe is the historian of the incident, responsible for maintaining a real-time, timestamped log of every observation, action, decision, role change, escalation, and communication. They produce the raw timeline that becomes the backbone of the post-incident review. The Scribe also handles logistics — setting up and maintaining the war room (bridge lines, Slack channels, shared documents), tracking open action items, managing shift schedules, and ensuring the incident runs smoothly. Because scribe fatigue builds quickly, this role should rotate every 30–45 minutes in prolonged incidents. A good scribe captures decisions and the rationale behind them, asking "Can you repeat that for the log?" when things move fast.

### Role Card: Comms / Liaison

The Communications and Liaison role is the buffer between the incident response team and the outside world. They send regular status updates to internal stakeholders (Slack channels, email lists) and draft customer-facing messages for the status page. They field incoming questions from executives, product managers, customer success, and support teams, providing consistent, approved information without distracting the incident responders. The Comms role tracks the communication cadence required by the incident severity and ensures that every stakeholder who needs to know is informed. No external communication goes out without IC approval, and the Comms person never speculates — they only share confirmed facts.

### Role Card: Database SME

The Database Subject Matter Expert is responsible for diagnosing and resolving database-related issues during an incident. This includes investigating connection pool exhaustion, replication lag, slow queries, locked tables, disk space issues, and failover scenarios. The DB SME reports findings to the Operations Lead in a structured format: what they found, what they tried, and what they recommend. They execute database restarts, failovers, query terminations, index rebuilds, and scaling operations under the direction of the Ops Lead and with IC approval for break-glass actions. In major incidents, the DB SME also assesses whether the database issue is the root cause or a symptom of a broader problem.

### Role Card: Networking SME

The Networking SME handles all network-layer issues during an incident. This includes DNS resolution failures, BGP routing problems, CDN configuration errors, load balancer misconfigurations, firewall rule blocks, DDoS attacks, and VPN or connectivity issues. They use tools like traceroute, dig, curl, cloud provider network consoles, and observability platforms to pinpoint network faults. They report to the Operations Lead and recommend actions such as DNS record changes, traffic rerouting, CDN purges, or WAF rule adjustments. The Networking SME also coordinates with external providers (cloud providers, CDNs, ISPs) when the incident involves their infrastructure.

### Role Card: Application SME

The Application SME investigates and resolves issues in the application code and runtime. This includes debugging error spikes, analyzing application logs, reviewing recent deployments, toggling feature flags, checking configuration files, and assessing whether a rollback or hotfix is needed. They work closely with the deployment and CI/CD pipelines and may coordinate with the code owners for specific services. The Application SME reports to the Operations Lead with clear findings ("Service X is throwing 500s because config parameter Y is invalid") and recommended actions ("Roll back service X to version 1.2.3"). They may also implement temporary mitigations like circuit breakers or rate limiting.

### Role Card: Security SME

The Security SME handles any incident with a security component: suspected breaches, unauthorized access, DDoS attacks, data exfiltration indicators, compromised credentials, or vulnerability exploitation. They prioritize containment over investigation — the first question is always "How do we stop the bleeding?" The Security SME coordinates with the IC on breach notification requirements, interfaces with legal and compliance as needed, and preserves forensic evidence (logs, system snapshots, network captures) for later analysis. They may advise on bringing in additional security tooling, rotating credentials, or isolating compromised systems. The Security SME has the authority to raise a security concern even if it changes the incident classification.

### Role Card: Infrastructure / Cloud SME

The Infrastructure SME manages issues at the cloud provider or data center level: instance failures, auto-scaling group problems, Kubernetes cluster issues, storage volume problems, and cloud API throttling. They handle infrastructure-level escalations to cloud provider support (AWS, GCP, Azure) and manage infrastructure-as-code tooling (Terraform, Pulumi, CloudFormation). The Infrastructure SME assesses whether the issue is localized to specific availability zones or regions and recommends infrastructure-level mitigations such as scaling up instance counts, moving workloads to healthy zones, or initiating disaster recovery procedures.

### Role Card: Vendor SME

The Vendor SME handles incidents involving third-party SaaS dependencies — either as the root cause (e.g., Datadog is down, PagerDuty is not delivering alerts, a cloud provider has an Availability Zone failure) or as part of the mitigation (e.g., contacting Fastly support for cache issues, rotating API keys for an external service). They maintain contacts and support plans for all critical vendors, know the escalation paths for each vendor, and track vendor SLAs during the incident. The Vendor SME coordinates with the Ops Lead to determine whether the organization should implement a workaround or wait for the vendor to resolve the issue on their end.

---

## Quick Reference Card

### First 60 Seconds

```
1. TAKE A DEEP BREATH
2. "I am now the Incident Commander."
3. "What is the impact? How many users?"
4. Assign Scribe (even if it's you initially).
5. Declare severity: SEV-1/2/3/4/5.
6. Open channel + bridge + scribe doc.
7. "Scribe, start the timeline."
```

### During the Incident

```
IC should ask every 5-10 minutes:
  - "What's the current impact?"
  - "What have we tried?"
  - "What's the next action?"
  - "Do we need more people?"
  - "Scribe, did you get that last decision?"
```

### At Resolution

```
1. Set the stability window during response based on expected traffic, dependency recovery, backlog or secondary effects, and delayed symptoms. (IC)
2. Run the closure evidence sequence: user-facing SLOs and critical journeys, dependency health, data/state correctness, secondary effects, and confirm the stability window is complete. Green monitoring is partial evidence. (Ops Lead)
3. A human other than the acting automation independently confirms the complete evidence set. An agent-assigned IC role, automation-authored record, or self-reported health check is insufficient. (IC / service owner)
4. If any evidence or human confirmation is missing, retain **MITIGATING** or **MONITORING**, record the unverified boundary, and hand off or escalate. (IC)
5. "Incident is resolved." only after the evidence sequence and independent human confirmation pass. (IC)
6. Change severity to SEV-5 only after closure evidence, the stability window, and independent human confirmation pass. (IC)
7. Update status page to "Resolved." (Comms)
8. Final status update to stakeholders. (Comms)
9. Save all materials. (Scribe)
10. Schedule post-incident review. (IC)
11. Log the incident in the tracking system. (IC)
```

---

*This document is a reference for SRE teams adopting structured incident command. Adapt severity definitions, escalation chains, and response times to your organization's specific service levels, team structure, and regulatory requirements.*

# Monitoring & Alerting Reference

## Overview

Monitoring and alerting form the operational backbone of any production system. This reference covers the principles, patterns, and practical implementations that SREs use to observe system health, detect anomalies, and respond to incidents. It is designed as a standalone guide for site reliability engineers building or refining their observability stack.

---

## 1. The Four Golden Signals

Coined by Google's SRE team, the Four Golden Signals are the highest-level metrics every distributed system should track. They provide a minimal, universally applicable set of health indicators that cut across application domains.

### 1.1 Latency

Latency measures the time required to service a request. It must be tracked separately for *successful* and *failed* requests — a slow failure (e.g., a 500 that takes 30 seconds to return) can mask a real problem with success-path latency.

**Key considerations:**
- Use percentiles, not averages. p50, p95, p99 tell a far richer story than mean latency.
- Distinguish between *latency* (time to first byte / time to complete) and *response time* (round-trip from client perspective).
- Instrument at every layer: load balancer, application, database, downstream dependency.

**Example PromQL latency queries:**
```promql
# p99 latency of HTTP requests (successful only), last 5m
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket{status=~"2..|3.."}[5m])
)

# p50 latency
histogram_quantile(0.50,
  rate(http_request_duration_seconds_bucket[5m])
)

# Slow request ratio: requests exceeding 1 second
rate(http_request_duration_seconds_count{status=~"2..|3.."}[5m])
/
rate(http_request_duration_seconds_bucket{le="1.0", status=~"2..|3.."}[5m])
```

### 1.2 Traffic

Traffic describes the demand placed on the system. Its units depend on the service type:

| Service Type        | Traffic Metric              | Example                            |
|---------------------|-----------------------------|------------------------------------|
| HTTP API            | Requests per second (RPS)   | `rate(http_requests_total[5m])`    |
| Database            | Queries per second (QPS)    | `rate(mysql_queries_total[5m])`    |
| Message queue       | Messages consumed per minute | `rate(kafka_messages_total[5m])`   |
| CDN / file serving  | Bytes per second            | `rate(nginx_bytes_sent_total[5m])` |
| Web application     | Active sessions / users     | `active_users`                     |

**Traffic pattern alerts** should distinguish organic growth (capacity planning) from sudden spikes (incidents):
```promql
# Sudden traffic spike: 2x over 5-minute baseline
(
  rate(http_requests_total[5m])
  /
  rate(http_requests_total[30m] offset 10m)
) > 2
```

### 1.3 Errors

Errors are requests that fail, explicitly (5xx) or implicitly (returning wrong data, meeting SLOs but returning stale results). Track both:

- **Explicit errors:** HTTP 5xx, gRPC UNAVAILABLE, TCP connection resets.
- **Implicit errors:** Success codes with semantically wrong responses (e.g., 200 OK with empty body), responses that exceed latency SLOs, degraded functionality.

**Error ratio is often more useful than raw error count**, especially during traffic changes:
```promql
# Error ratio over 5m window
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
```

**Alert on error budget burn rate** for SLO-based alerting:
```promql
# Error budget burn rate over 1h: how fast we're consuming the SLO budget
(
  1 - (
    sum(rate(http_requests_total{status=~"2.."}[1h]))
    / sum(rate(http_requests_total[1h]))
  )
) / (1 - 0.99)  // 99% SLO target
```

### 1.4 Saturation

Saturation measures how "full" the service is. The most limited resource (CPU, memory, disk I/O, network bandwidth, connection pool, database connections, file descriptors) dominates.

**Key principles:**
- Saturation often precedes performance degradation — it is a leading indicator.
- Use utilization (percentage) but also track queue depth or request-dropping behavior.
- The USE Method (Utilization, Saturation, Errors) maps directly here.

```promql
# CPU saturation: load average relative to core count
node_load1 / count(node_cpu_seconds_total{mode="idle"}) > 0.8

# Memory saturation: available to total ratio
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1

# Connection pool saturation: active connections vs max
pg_stat_activity_count / pg_settings_max_connections > 0.8

# Disk I/O saturation: queue depth
rate(node_disk_io_time_weighted_seconds_total[5m]) > 1000
```

---

## 2. Black-Box vs White-Box Monitoring

### 2.1 Black-Box Monitoring

Black-box monitoring observes the system from the outside as a user would. It tests behaviour, not implementation.

**Characteristics:**
- Synthetic probes, health checks, external uptime monitors.
- Validates *what* the system does, not *how*.
- Catches issues invisible to internal metrics: DNS resolution failures, TLS certificate expiry, CDN misconfiguration, routing problems.
- Example: "HTTP GET /health returns 200 within 5 seconds."

**Tools:** Prometheus blackbox exporter, synthetic monitoring (Checkly, Grafana Synthetics), external uptime services.

**Alert rules:**
```promql
# Probe failure over 5 minutes
avg_over_time(probe_success[5m]) < 0.5

# Latency from probe perspective
probe_duration_seconds > 3
```

### 2.2 White-Box Monitoring

White-box monitoring observes the system from the inside, exposing internal state, metrics, and performance characteristics.

**Characteristics:**
- Application metrics (request durations, error rates, goroutine counts).
- Database query performance, connection pool depth, cache hit ratios.
- Runtime behaviour (GC pauses, thread contention, memory allocation).
- Enables root-cause analysis and capacity planning.
- Example: "JVM heap usage is at 85% and growing at 100 MB/min."

**Tools:** Application metrics instrumentation (Prometheus client libraries, OpenTelemetry), runtime debugging endpoints, structured logging.

The two approaches are complementary. A mature observability stack uses both: black-box for customer-facing SLIs, white-box for debugging and proactive capacity management.

---

## 3. Symptom-Based vs Cause-Based Alerting

### 3.1 Symptom-Based Alerting (Preferred)

Alert on *symptoms* — observable problems that affect users right now or will imminently.

| Symptom                         | Example Alert                                      |
|---------------------------------|----------------------------------------------------|
| High latency                    | p99 latency > 2 seconds for 5 minutes              |
| Elevated error rate             | Error ratio > 5% for 10 minutes                    |
| Error budget exhaustion         | 2% of monthly error budget consumed in 1 hour      |
| Degraded user experience        | Page load time > 3 seconds for logged-in users     |

**Rule of thumb:** If the alert fires and no user-facing impact can be articulated, the alert is cause-based and should be re-evaluated or demoted to a diagnostic signal.

### 3.2 Cause-Based Alerting

Alert on *causes* — internal conditions that *might* lead to symptoms. These should generally be:
- **Log-only** or **warning-level** notifications (dashboards, not pages).
- Tied to a specific known failure mode with clear remediation.
- Rate-limited to avoid noise storms.

| Cause                            | Better Approach                                   |
|----------------------------------|---------------------------------------------------|
| CPU > 90%                        | Dashboard panel + capacity planning ticket        |
| Disk > 80% full                  | Automated cleanup or low-severity notification    |
| Specific error log line appears  | Log-based metric + dashboard, not page            |
| Single pod OOMKilled             | Auto-restart (Kubernetes handles this)            |

---

## 4. Alert Design Principles

Every alert should pass three tests: **actionable**, **urgent**, and **novel**.

### 4.1 Actionable

The recipient must be able to do something about it.

- **Good:** "API error ratio exceeds 5% — investigate upstream database connectivity."
- **Bad:** "CPU is at 88% — no further context, no clear action."
- **Test:** If the alert fires and the on-call engineer asks "so what?" it is not actionable.

### 4.2 Urgent

The alert must require immediate attention.

- **Good:** "Error budget depletion rate will exhaust budget in 2 hours at current burn rate."
- **Bad:** "Certificate expires in 30 days" (should be a scheduled task, not a page).
- **Test:** Would it be acceptable to ignore this alert for 30 minutes? If yes, it is not urgent.

### 4.3 Novel

The alert should represent new information not already visible in dashboards or automated remediation.

- **Good:** "This is the first occurrence of a database connection pool exhaustion event today."
- **Bad:** "Pod restarted" — if Kubernetes auto-recovered and there's no systemic pattern.
- **Test:** Is this the same incident-meets-recovery cycle firing repeatedly? Suppress or tune.

### 4.4 Additional Design Heuristics

- **One alert = one problem.** Do not bundle multiple conditions into a single rule.
- **Include runbook links** in alert annotations.
- **Use consistent naming conventions:** `Service/Component/Severity/Description`.
- **Set appropriate `for` durations** (e.g., 5 minutes) to avoid flapping on transient issues.
- **Define expected response times** in the alert metadata (e.g., "Respond within 15 minutes").
- **Every page-level alert needs a documented escalation path.**

---

## 5. Time-Series Monitoring Architecture (Prometheus-Inspired)

### 5.1 Core Architecture Pattern

```
[Instrumented Services] --scrape--> [Prometheus Server] --read--> [Alertmanager]
                                     |                        +--> [Dashboards (Grafana)]
                                     |                        +--> [Long-term storage (Thanos/Cortex)]
                                     v
                                  [Recording Rules] --> [Aggregated metrics]
```

### 5.2 Metric Types

| Type        | Description                                          | Example                                      |
|-------------|------------------------------------------------------|----------------------------------------------|
| Counter     | Monotonically increasing value (resets on restart)   | `http_requests_total`, `errors_total`        |
| Gauge       | Point-in-time value that can go up and down          | `memory_usage_bytes`, `queue_depth`          |
| Histogram   | Configurable buckets for observing distributions     | `request_duration_seconds_bucket{le="0.5"}`  |
| Summary     | Pre-computed quantiles (cannot aggregate)            | `request_duration_seconds{quantile="0.99"}`  |

### 5.3 Recording Rules

Pre-compute expensive or frequently queried expressions to reduce query load:

```promql
# Group: "service:request_error_ratio:5m"
record: namespace:http_errors:ratio_rate5m
expr: |
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (namespace)
  /
  sum(rate(http_requests_total[5m])) by (namespace)

# Group: "instance:cpu_utilization:rate5m"
record: instance:node_cpu_utilization:rate5m
expr: |
  1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)
```

### 5.4 Alerting Rules Architecture

Separate evaluation tiers:

```promql
# Tier 1 — Direct metric threshold (immediate symptom)
alert: APIHighErrorRate
expr: |
  (sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
   /
   sum(rate(http_requests_total[5m])) by (service)) > 0.05
for: 5m
labels:
  severity: critical
annotations:
  summary: "{{ $labels.service }} error rate above 5%"
  runbook: "https://runbooks.example.com/high-error-rate"

# Tier 2 — Error-budget-based (burn rate)
alert: ErrorBudgetBurnedFast
expr: |
  (
    (1 - (sum(rate(http_requests_total{status=~"2.."}[1h])) by (service)
         / sum(rate(http_requests_total[1h])) by (service)))
    / (1 - 0.99)
  ) > 0.05
for: 2m
labels:
  severity: critical
annotations:
  summary: "Error budget burning at 5x safe rate"

# Tier 3 — Absent metric (service down)
alert: ServiceDown
expr: |
  absent(up{job="api-server"}) == 1
for: 30s
labels:
  severity: critical
```

---

## 6. Alert Routing and Severity

### 6.1 Severity Levels

| Level       | Label      | Response SLA     | Channel       | Purpose                               |
|-------------|------------|------------------|---------------|---------------------------------------|
| Critical    | P0         | < 15 minutes     | Page (phone)  | User-facing outage, data loss, breach  |
| High        | P1         | < 30 minutes     | Page (push)   | Severe degradation, error budget risk  |
| Medium      | P2         | < 4 hours        | Chat (thread) | Degradation without immediate impact   |
| Low         | P3         | Next business day| Ticket        | Capacity warnings, cleanup reminders   |
| Informational | P4       | None             | Log only      | Dashboard annotations, debugs          |

### 6.2 Routing Matrix

Route based on service, severity, time of day, and team ownership:

```
[Alertmanager Configuration]
route:
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'page-oncall'
      repeat_interval: 1h
    - match:
        service: billing
      receiver: 'billing-team'
    - match_re:
        component: "^database"
      receiver: 'data-platform'
    - match:
        severity: warning
      receiver: 'chat-channel'
      group_wait: 30s
      group_interval: 5m
```

### 6.3 Grouping and Deduplication

Prevent alert storms by grouping related alerts:

```yaml
group_by: ['alertname', 'service', 'cluster']
group_wait: 30s       # collect alerts for 30s before sending
group_interval: 5m    # wait 5m between repeat notifications
repeat_interval: 4h   # don't re-notify for 4h unless severity changes
```

---

## 7. Dashboard Design Principles

### 7.1 The Golden Rules

1. **One dashboard per story.** A dashboard should answer a single question or serve a single persona (e.g., "Is the API healthy?", "Database capacity planning").
2. **Three-tier layout:**
   - **Top row:** Summary / health — the single number or status that tells you if everything is OK.
   - **Middle rows:** Key time-series — latency, errors, traffic, saturation per component.
   - **Bottom rows:** Drill-down details — per-shard, per-host, per-endpoint.
3. **Consistent time ranges** across panels on the same dashboard.
4. **Use left Y-axis** for the primary metric; avoid dual-axes unless essential.
5. **Label clearly** with units (ms, req/s, %). Include SLO targets as threshold lines.

### 7.2 Dashboard Types

| Type            | Audience       | Refresh  | Content Summary                          |
|-----------------|----------------|----------|------------------------------------------|
| Executive       | Management     | 1 day    | SLIs, error budget, cost, capacity trends |
| Service Health  | On-call        | 30s      | Golden signals per service               |
| Capacity        | Infrastructure | 1 hour   | Resource utilization, growth projections  |
| Debug / Triage  | Engineering    | Real-time| Detailed traces, log rates, anomaly plots |
| SLO Burn Rate   | SRE            | 1 min    | Burn rate vectors, remaining budget      |

### 7.3 Anti-Patterns

- **Spaghetti dashboards:** More than 20 panels on one dashboard. Split by concern.
- **Red/green everywhere:** Red color should be reserved for *failing* the SLO.
- **No time context:** Single-number stat panels without a sparkline or change indicator.
- **Orphan dashboards:** No owner, no last-modified date, no description.

---

## 8. On-Call Alert Fatigue Prevention

Alert fatigue is the leading cause of pager burnout and missed real incidents.

### 8.1 Causes

| Cause                          | Remedy                                            |
|--------------------------------|---------------------------------------------------|
| Too many low-severity alerts   | Demote to P3/P4; page only on critical.           |
| Flaky alerts (transient)       | Increase `for` duration; use multi-window logic.  |
| Duplicate alerts               | Use alertmanager grouping; deduplicate rules.     |
| Noisy auto-remediation cycles  | Silence alerts for N minutes after auto-fix.      |
| Undefined ownership            | Every alert must route to exactly one team.       |
| Missing runbooks               | Add runbook links; if unclear, the alert is bad.  |

### 8.2 Fatigue Prevention Techniques

- **Weekly alert reviews:** Every critical alert is reviewed for signal-to-noise ratio. If it fired "unnecessarily" more than once in the past week, tune or delete it.
- **Gradual escalation:** Start with a 5-minute `for` duration, alert warning first, then escalate to critical if condition persists.
- **Suppression during maintenance:** Tag maintenance windows; suppress alerts during known change windows.
- **Machine learning / dynamic thresholds:** For seasonal systems, use anomaly detection instead of static thresholds.
- **Error budget alerts over raw threshold alerts:** Raw threshold alerts (e.g., "p99 > 500ms") are rigid. Error-budget-based alerts account for the SLO's remaining headroom.
- **Self-healing integration:** If a runbook step can be automated (restart, scale-up, traffic shed), add it. Only page if the automated action fails.
- **Psychologically safe culture:** No blame for acknowledging an alert and determining it was not actionable — that becomes tuning feedback.

### 8.3 Alert Fatigue Metrics to Track

- **Alert-to-incident ratio:** Number of pages that resulted in a documented incident vs. total pages. Target: > 50%.
- **Mean time to acknowledge (MTTA):** Should be under the defined SLA. Spikes suggest routing issues or fatigue.
- **Mean time to resolve (MTTR):** Tracks operational effectiveness.
- **False positive rate per rule:** Identify the worst offenders in the weekly review.

---

## 9. The USE Method and RED Method for Metrics

### 9.1 USE Method (Utilization, Saturation, Errors)

Used for **infrastructure resources** (CPU, memory, disk, network). Ask: For every resource, what is:
- **Utilization:** The average percent of time the resource is busy.
- **Saturation:** The degree to which the resource has extra work queued (can't service more).
- **Errors:** The count of error events.

**USE Method Counterpart per Resource:**

| Resource       | Utilization                    | Saturation                      | Errors                             |
|----------------|--------------------------------|---------------------------------|------------------------------------|
| CPU            | `avg by(instance) (rate(cpu_seconds_total{mode!="idle"}[5m]))` | `load1 / cpu_count` | `process_errors_total` (per process) |
| Memory         | `used / total`                 | `swap_usage` / OOM events       | `vmstat -s` page failures           |
| Disk I/O       | `avg by(device) (rate(disk_io_time_seconds_total[5m]))` | `rate(disk_io_time_weighted_seconds_total[5m])` | `disk_read_errors_total` / `disk_write_errors_total` |
| Network        | `rate(bytes_total[5m]) / bandwidth` | `drop_count / segment_retransmits` | `interface_errors_total`         |
| File handles   | `filefd_allocated / filefd_max` | —                               | `failed_file_opens_total`          |

**Example alert using USE:**
```promql
# Saturation: CPU run queue > 4x core count
(
  node_load15
  /
  count(node_cpu_seconds_total{mode="idle"})
) > 4
```

### 9.2 RED Method (Rate, Errors, Duration)

Used for **services and applications**. Ask for every service:
- **Rate:** Requests per second (throughput).
- **Errors:** Number of requests that fail.
- **Duration:** Time taken to process a request (latency distribution).

The RED Method is essentially the Four Golden Signals minus saturation — it focuses purely on user-facing service health rather than resource health.

**Instrumentation pattern (per endpoint or per operation):**
```promql
# Rate
rate(http_requests_total{job="api"}[5m])

# Errors (ratio)
sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

# Duration (p99)
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket{job="api"}[5m])
)
```

### 9.3 When to Use Which

| Layer                 | Best Fit     | Rationale                                    |
|-----------------------|--------------|----------------------------------------------|
| Infrastructure (VM, container, disk, NIC) | USE | Resources have bounded capacity; saturation is meaningful. |
| Application (API, database, queue) | RED (or Golden Signals) | Behaviours (requests, errors, latency) are the concern. |
| Middleware / Proxy    | Both         | Resource concerns (connections, memory) AND service behaviours matter. |
| Network / Load balancer | RED + USE  | Throughput and errors (RED) plus interface saturation (USE). |

---

## 10. SLI-Based Alerting vs Threshold-Based Alerting

### 10.1 Threshold-Based Alerting (Classic)

**How it works:** Static thresholds on raw metrics (e.g., p95 latency > 500ms, CPU > 90%).

**Pros:**
- Simple to understand and implement.
- Works well for resource saturation alerts (disk full, OOM).
- No SLO definition required.

**Cons:**
- Requires manual tuning for each service.
- Does not account for *time remaining* — a p99 of 600ms at 8am on a Monday might be fine, but at 3am it indicates something wrong.
- Brittle: thresholds that are too tight generate noise, those too loose miss incidents.

### 10.2 SLI-Based Alerting

**How it works:** Define Service Level Indicators (SLIs) and Service Level Objectives (SLOs). Alert based on *error budget burn rate* — how fast the remaining allowable error budget is being consumed.

**SLI definition pattern:**
```yaml
sli_name: "api_request_latency_sli"
description: "Fraction of requests served in under 500ms"
good_event: "http_request_duration_seconds <= 0.5"
valid_events: "http_requests_total"
measurement_window: "28d"
```

**Burn rate alerting approach (Google SRE Workbook):**

| Burn Rate (x of target) | Duration     | Severity | Meaning                                               |
|-------------------------|--------------|----------|-------------------------------------------------------|
| >= 10x                  | 5 minutes    | Critical | Very severe, multi-9s breach imminent in minutes      |
| >= 5x                   | 30 minutes   | Critical | Severe, will exhaust budget in hours                  |
| >= 2x                   | 6 hours      | High     | Moderate burn, needs investigation                    |
| >= 1x                   | 3 days       | Low      | Slow bleed, ticket for next business day              |

**PromQL burn rate alert:**
```promql
# 10x burn rate over 5 minutes
alert: HighBurnRate_10x_5m
expr: |
  (
    (
      1 - (
        sum(rate(http_requests_total{status=~"2..", job="api"}[5m]))
        / sum(rate(http_requests_total{job="api"}[5m]))
      )
    ) / (1 - 0.99)
  ) > 10
for: 2m
labels:
  severity: critical
```

### 10.3 Comparison

| Aspect                | Threshold-Based                  | SLI-Based (Burn Rate)                  |
|-----------------------|----------------------------------|----------------------------------------|
| Tuning effort         | Per-service, per-metric           | Shared SLO framework                   |
| Sensitivity to traffic | Fixed threshold may be too tight at low traffic or too loose at high traffic | Proportional — error ratio, not absolute count |
| Leading vs lagging    | Leading (catches impending resource saturation) | Lagging for resource issues, leading for SLO breaches |
| Business alignment    | Weak                              | Direct: maps to customer-facing objectives |
| Complexity            | Low                               | Moderate (requires SLO definitions)    |

**Recommendation:** Use SLI/burn rate alerting for user-facing symptoms. Use threshold-based alerting for resource saturation and capacity warnings. The two approaches are complementary.

---

## 11. Multi-Window Alerting Approaches

Multi-window, multi-burn-rate alerting is the gold standard for balancing sensitivity with noise reduction. It evaluates the same condition across *multiple time windows* before firing.

### 11.1 Why Multi-Window?

A single-window alert (e.g., `error_ratio > 5% over 5m`) can:
- Fire on transient spikes that resolve before anyone acts (false positive).
- Miss slow-burn degradation that accumulates over hours but never crosses a short-window threshold (false negative).

### 11.2 Multi-Window, Multi-Burn-Rate Design

Evaluate the metric across two windows — a *short window* (high sensitivity) and a *long window* (confirms sustained condition):

```promql
# Conditional: short-window burn rate is > 10x target
# AND long-window burn rate is > 10x target
# Both must be true — catches sustained high burn, rejects transient spikes

alert: MultiWindowErrorBudgetBurn
expr: |
  (
    # Short window: 5m burn rate > 10x
    (
      (1 - (sum(rate(http_requests_total{status=~"2.."}[5m]))
           / sum(rate(http_requests_total[5m]))))
      / (1 - 0.99)
    ) > 10
  )
  and
  (
    # Long window: 1h burn rate > 10x (confirms sustained)
    (
      (1 - (sum(rate(http_requests_total{status=~"2.."}[1h]))
           / sum(rate(http_requests_total[1h]))))
      / (1 - 0.99)
    ) > 10
  )
for: 2m
labels:
  severity: critical
```

### 11.3 The Google SRE Multi-Window Alerting Matrix

| Condition                                 | Fires When                               | Character              |
|-------------------------------------------|------------------------------------------|------------------------|
| Short (5m) high burn AND Long (1h) high burn | Instant + sustained severe event     | "House on fire"       |
| Short (5m) low burn AND Long (1h) high burn | Slow bleed below burst sensitivity    | "Radiator leak"       |
| Short (5m) high burn AND Long (1h) low burn | Transient spike, already recovered    | No alert (correct!)   |
| Short (5m) low burn AND Long (1h) low burn | Everything nominal                      | No alert              |

### 11.4 Implementation Pattern with Recording Rules

```promql
# Recording rule: 5m burn rate ratio
record: slo:api_error_burn_rate_5m
expr: |
  (
    (1 - (sum(rate(http_requests_total{job="api", status=~"2.."}[5m]))
         / sum(rate(http_requests_total{job="api"}[5m]))))
    / (1 - 0.99)
  )

# Recording rule: 1h burn rate ratio
record: slo:api_error_burn_rate_1h
expr: |
  (
    (1 - (sum(rate(http_requests_total{job="api", status=~"2.."}[1h]))
         / sum(rate(http_requests_total{job="api"}[1h]))))
    / (1 - 0.99)
  )

# Alert: both windows agree on high burn
alert: APIErrorBudgetCriticalBurn
expr: |
  slo:api_error_burn_rate_5m > 10 and slo:api_error_burn_rate_1h > 10
for: 2m
labels:
  severity: critical
annotations:
  summary: "API error budget burning at critical rate"

# Alert: long window burn without short window (slow drain)
alert: APIErrorBudgetSlowBurn
expr: |
  slo:api_error_burn_rate_1h > 2 and slo:api_error_burn_rate_5m <= 10
for: 10m
labels:
  severity: warning
annotations:
  summary: "API error budget slowly draining"
```

### 11.5 Alternative: Sliding-Window with `min_over_time`

For environments where PromQL `and` semantics are unavailable, simulate multi-window with `min_over_time`:

```promql
alert: SlidingWindowHighErrorRate
expr: |
  min_over_time(
    (
      sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
      /
      sum(rate(http_requests_total[5m])) by (service)
    )[15m:]
  ) > 0.05
for: 2m
```

This requires the high error rate to be sustained across three consecutive 5-minute windows (15 minutes).

---

## 12. Practical PromQL Alert Rule Templates

### 12.1 Service Availability

```promql
# Service down — no metrics received
alert: ServiceDown
expr: absent(up{job="api-server"}) == 1
for: 1m
labels:
  severity: critical
annotations:
  summary: "{{ $labels.job }} is not sending metrics"
```

### 12.2 High Error Rate

```promql
# Per-path error rate
alert: PathHighErrorRate
expr: |
  (
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service, path)
    /
    sum(rate(http_requests_total[5m])) by (service, path)
  ) > 0.05
for: 5m
labels:
  severity: critical
annotations:
  summary: "5xx rate > 5% on {{ $labels.path }}"
```

### 12.3 High Latency

```promql
# Per-service p99 latency
alert: HighLatency
expr: |
  histogram_quantile(0.99,
    sum(rate(http_request_duration_seconds_bucket{status=~"2.."}[5m])) by (le, service)
  ) > 2.0
for: 5m
labels:
  severity: critical
annotations:
  summary: "p99 latency > 2s on {{ $labels.service }}"
```

### 12.4 Certificate Expiry

```promql
alert: TLSCertExpiringSoon
expr: |
  probe_ssl_earliest_cert_expiry - time() < 604800  # 7 days in seconds
for: 1h
labels:
  severity: warning
annotations:
  summary: "TLS certificate for {{ $labels.instance }} expires in less than 7 days"
```

### 12.5 Disk Space

```promql
alert: DiskSpaceCritical
expr: |
  (
    1 - (node_filesystem_avail_bytes{mountpoint="/", fstype!~"tmpfs|overlay"}
         / node_filesystem_size_bytes{mountpoint="/", fstype!~"tmpfs|overlay"})
  ) > 0.95
for: 5m
labels:
  severity: critical
annotations:
  summary: "Disk usage > 95% on {{ $labels.instance }}"
```

### 12.6 Connection Pool Saturation

```promql
alert: PgConnectionPoolExhausted
expr: |
  (
    sum by (instance) (pg_stat_activity_count{datname!~"template.*|postgres"})
    /
    avg by (instance) (pg_settings_max_connections)
  ) > 0.9
for: 5m
labels:
  severity: critical
annotations:
  summary: "PostgreSQL connection pool > 90% on {{ $labels.instance }}"
```

### 12.7 OOM Kill Detection

```promql
alert: OOMKilled
expr: |
  increase(kube_pod_container_status_restarts_total{reason="OOMKilled"}[15m]) > 0
for: 0m
labels:
  severity: critical
annotations:
  summary: "Pod {{ $labels.pod }} was OOMKilled"
```

### 12.8 Traffic Anomaly

```promql
alert: TrafficDrop
expr: |
  (
    rate(http_requests_total[5m])
    /
    rate(http_requests_total[5m] offset 1w at same day of week)
  ) < 0.5
for: 15m
labels:
  severity: warning
annotations:
  summary: "Traffic dropped > 50% compared to same time last week"
```

---

## 13. Closing Principles

1. **Page on symptoms, investigate causes.** Symptoms affect users; causes affect operators. One is urgent, the other is not.
2. **Every alert needs an owner and a runbook.** If the on-call cannot find both within 30 seconds, the alert is not production-ready.
3. **Test your alerts.** Use alert simulation tools (e.g., `amtool`, Prometheus alert unit tests with `promtool`) to validate logic before deploying.
4. **Review alerts weekly.** Each alert that fired is either a real incident (document it) or a false positive (tune it). No alert fires without producing an improvement action.
5. **Prefer dashboards for diagnostic data, alerts for escalations.** If a metric only says "look at the dashboard" it should be a panel, not a page.
6. **Multi-window beats single-window.** Short windows are noisy; long windows are slow. Use both for reliable detection.
7. **Error budget burn rate alerts are the gold standard** for user-facing services. They automatically account for the service's SLO, remaining budget, and traffic levels.
8. **Instrumentation is an investment.** Every metric you add has a storage and maintenance cost. Start with the Four Golden Signals, expand only when you have a concrete question that existing data cannot answer.

---

## References

- Google SRE Book — Chapter 6: Monitoring Distributed Systems
- Google SRE Workbook — Chapter 5: Alerting on SLOs
- The USE Method (Brendan Gregg) — https://www.brendangregg.com/usemethod.html
- The RED Method (Tom Wilkie) — https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/
- Prometheus Documentation — Alerting Rules: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
- Alertmanager Documentation: https://prometheus.io/docs/alerting/latest/alertmanager/
- Grafana Dashboard Design Best Practices: https://grafana.com/docs/grafana/latest/dashboards/

# On-Call Best Practices Reference

> **Audience:** Site Reliability Engineers, Infrastructure Engineers, DevOps practitioners
> **Purpose:** Comprehensive reference for designing, operating, and improving on-call rotations
> **Last Updated:** 2026-06-05

---

## Table of Contents

1. [Rotation Design](#1-rotation-design)
2. [Response SLAs](#2-response-slas)
3. [Workload Caps](#3-workload-caps)
4. [Balance and Fairness](#4-balance-and-fairness)
5. [Cognitive Load Management](#5-cognitive-load-management)
6. [Escalation Paths and Policies](#6-escalation-paths-and-policies)
7. [Tools and Infrastructure](#7-tools-and-infrastructure)
8. [Handoff Best Practices](#8-handoff-best-practices)
9. [Training and Onboarding](#9-training-and-onboarding)
10. [Underload Prevention](#10-underload-prevention)
11. [When On-Call Isn't Working](#11-when-on-call-isnt-working)

---

## 1. Rotation Design

### Primary / Secondary Model

Every rotation should have at least two tiers to provide redundancy and support:

- **Primary on-call:** First responder. Paged first, responsible for acknowledging and triaging all alerts within the defined SLA. Handles day-to-day operational incidents and escalations.
- **Secondary / Shadow:** Backup to the primary. Pages if the primary does not acknowledge within the SLA window (typically 2-3 minutes for critical alerts). The secondary is also a designated support channel — the primary can escalate to them for complex incidents requiring additional investigation or a second set of eyes.
- **Tertiary / Escalation:** Senior engineer or team lead. Paged only after both primary and secondary have missed acknowledgements. Also serves as the management escalation for high-severity incidents (SEV-0 / SEV-1).

For teams smaller than 5 people, a single primary with a floating secondary that rotates weekly is often the most practical arrangement. For teams of 6 or more, maintain a dedicated primary and dedicated secondary on separate rotations.

### Team Sizing Math

The size of an on-call rotation is governed by the relationship between shift length, team size, and time spent on-call. Use the following formulas to determine minimum safe team sizes.

**On-Call Frequency Formula:**

```
OnCallPercentage = (ShiftLengthDays / RotationLengthDays) x 100

Where:
  ShiftLengthDays = Number of consecutive days a single engineer is primary
  RotationLengthDays = ShiftLengthDays x TeamSize
  TeamSize = Number of engineers in the rotation pool
```

**Example calculations:**

- 4 engineers, 7-day shifts: each engineer is on-call 25% of the time (7/28 = 25%)
- 6 engineers, 7-day shifts: each engineer is on-call ~16.7% of the time (7/42 ≈ 16.7%)
- 3 engineers, 7-day shifts: each engineer is on-call ~33% of the time (7/21 ≈ 33%) — *high risk of burnout*
- 8 engineers, 24-hour shifts: each engineer is on-call ~12.5% of the time but with daily context-switching overhead

**Minimum Viable Team Size Formula:**

```
MinTeamSize = ceil(ShiftLengthDays / (MaxOnCallDaysPerYear / 365))

Where MaxOnCallDaysPerYear is a policy choice (recommended max: 91 days, or 25%)
```

With a 25% annual cap and 7-day shifts:

```
MinTeamSize = ceil(7 / (91 / 365)) = ceil(7 / 0.249) = ceil(28.1) = 29 ... (WRONG formula approach)
```

Correct pragmatic approach — given a shift length S and a target maximum on-call percentage P (as a decimal), the minimum team size N is:

```
N = ceil(1 / P)
```

For P = 0.25 (25% cap): N = ceil(1 / 0.25) = **4 engineers minimum**
For P = 0.20 (20% target): N = ceil(1 / 0.20) = **5 engineers minimum**
For P = 0.125 (12.5% ideal): N = ceil(1 / 0.125) = **8 engineers minimum**

**Recommended minimum team sizes:**

| Shift Length | Min Team Size (25% cap) | Ideal Team Size (12.5%) |
|--------------|------------------------|-------------------------|
| 24 hours     | 4                      | 8                       |
| 7 days       | 4                      | 8                       |
| 14 days      | 4                      | 8                       |

**Important caveat:** These formulas assume equal distribution. They do not account for holidays, sick leave, PTO, or training time. Add a buffer of at least 1-2 extra engineers to account for real-world availability gaps.

### Follow-the-Sun Model

For global teams operating across time zones, the follow-the-sun (FTS) model provides 24/7 coverage without requiring any single engineer to carry overnight shifts:

- **Region A (APAC):** Covers 00:00-08:00 UTC — primary for that time window
- **Region B (EMEA):** Covers 08:00-16:00 UTC — primary for that time window
- **Region C (AMER):** Covers 16:00-00:00 UTC — primary for that time window

Each region has its own rotation (typically 3-5 engineers per region). Handoffs occur at the region boundaries with structured shift-change processes. FTS works best when each region has sufficient engineering presence to maintain independent operations and shared ownership of the global infrastructure.

**Key requirements for FTS success:**

- At least 3 geographic hubs with 3+ engineers each
- Synchronized runbooks and dashboards across all regions
- Shared incident documentation standards
- Regular cross-region sync meetings (at least bi-weekly)
- Explicit handoff protocols at shift boundaries

### Optimal Shift Length

The optimal shift length balances coverage continuity against engineer fatigue:

- **24-hour shifts (follow-the-sun):** Best for global teams where "night" is never the same person's problem. High context but disruptive to personal rhythm.
- **7-day shifts (weekly):** The industry standard at most SRE organizations (Google, Meta, etc.). Good balance of context continuity and recovery time. Recommended as the default for most teams.
- **12-hour shifts (day/night split):** Useful when overnight coverage is needed but follow-the-sun isn't feasible. Typically 3-4 consecutive 12-hour days before rotation.
- **14-day shifts (bi-weekly):** May be acceptable for low-alert-volume services with small teams, but carries significantly higher fatigue risk. Not recommended for high-traffic or critical systems.

**General guidance:** Weekly (7-day) shifts are the strongest default. If your team exceeds 25% time on-call with weekly shifts, hire more engineers rather than extending shift length.

---

## 2. Response SLAs

Response SLAs define how quickly on-call engineers must acknowledge and respond to alerts. These are not resolution SLAs — they measure *response* time, not *fix* time.

### SLA Tiers

| Severity | Label          | Acknowledgement SLA | Response SLA       | Typical Scenarios                                   |
|----------|----------------|---------------------|--------------------|-----------------------------------------------------|
| SEV-0    | Critical       | 2 minutes           | 5 minutes          | Complete service outage, data loss, security breach |
| SEV-1    | High           | 5 minutes           | 15 minutes         | Major feature degradation, partial outage           |
| SEV-2    | Medium         | 10 minutes          | 30 minutes         | Minor degradation, non-critical feature broken      |
| SEV-3    | Low            | 30 minutes          | 2 hours            | Non-urgent issues, cosmetic problems, noise alerts  |
| SEV-4    | Informational  | Best effort         | Next business day  | Log entries, low-priority notifications             |

**Critical (SEV-0) — 5 minutes:** The engineer must acknowledge the page and begin investigation within 5 minutes. Services at this severity should have redundant alerting paths (email + SMS + phone call) and automated escalation if not acknowledged within 2 minutes.

**Standard (SEV-2) — 30 minutes:** The standard tier for most production alerts. Engineers have enough time to finish a context switch but are still expected to respond promptly.

### SLA Enforcement

- Track acknowledgement and response times against SLAs for every alert
- Use automated reporting (weekly/monthly) to identify:
  - Engineers routinely missing SLAs (training or burnout risk)
  - Alerting systems with delayed delivery (infrastructure issue)
  - Overly aggressive SLAs on non-critical alerts (tuning opportunity)
- SLA misses should trigger a post-incident review, not punitive action

### SLA Math: Coverage Requirements

To meet a 5-minute critical response SLA with 99.9% reliability:

```
P(miss) = 0.001
Required number of responders k such that P(all miss) <= 0.001

If each engineer has a 95% chance of acknowledging within 5 minutes:
  P(engineer_i misses) = 0.05
  P(primary misses) = 0.05
  P(secondary misses) = 0.05
  P(both miss) = 0.05 x 0.05 = 0.0025 = 0.25%

  This gives 99.75% reliability — above 99.9% requires a third tier:
  P(all three miss) = 0.05^3 = 0.000125 = 0.0125%
  This gives 99.9875% reliability
```

**Practical takeaway:** Two-tier escalation (primary + secondary) is sufficient for almost all teams. Add a third tier only for the most critical services where a missed page has regulatory or safety implications.

---

## 3. Workload Caps

Unlimited on-call workload leads directly to burnout, turnover, and incident mishandling. Enforce hard caps on both incident volume and time commitment.

### Incident Volume Caps

- **Maximum 2 active incidents per shift:** If a third incident occurs while two are already being handled, the secondary must take the new incident, or the shift must be split (page a third engineer from the rotation pool). This prevents "incident stacking" where an engineer is overwhelmed by simultaneous events.
- **Maximum 4 incidents per 24-hour period:** Beyond four incidents, the engineer should be relieved from on-call duty regardless of shift length. The rotation pool or secondary should cover the remainder.
- **Post-incident recovery time:** After any SEV-0 or SEV-1 incident resolution, the engineer should be given at least 30 minutes of quiet time for write-ups and cognitive recovery before being expected to handle another incident.

### Time Commitment Caps

- **Maximum 25% of working time on-call:** No engineer should be primary for more than 25% of their total working hours averaged over a quarter. This is the Google SRE standard and is widely adopted across the industry.
- **Maximum 12 hours of incident response in a single shift:** If incidents consume more than 12 hours of a 24-hour on-call shift, the engineer must be relieved.
- **No back-to-back on-call shifts:** Ensure at least one full shift length (e.g., one week) of non-on-call time between rotations.
- **Maximum 3 consecutive on-call weeks per quarter:** Even if the 25% average is met, no engineer should serve more than 3 weeks of primary on-call in any 13-week quarter.

### Escalation When Caps Are Hit

When any cap is reached, the escalation path must trigger automatically:

1. Primary hits cap -> Secondary takes over all new pages
2. Secondary hits cap -> Tertiary or rotation pool manager pages a volunteer
3. Rotation pool exhausted -> Engineering manager is paged to find coverage
4. Manager cannot find coverage -> Director-level escalation for staffing rebalance

---

## 4. Balance and Fairness

An on-call rotation is only sustainable if it is perceived as fair by every member of the team. Inequity — whether real or perceived — destroys morale and drives attrition.

### Distribution Tracking

Track and publish the following metrics for every engineer each quarter:

| Metric                          | Description                                           |
|---------------------------------|-------------------------------------------------------|
| Total on-call hours             | Cumulative hours as primary and secondary             |
| Number of pages received        | Raw count of alert notifications                      |
| Number of incidents handled     | Distinct incidents (ignoring repeat alerts)           |
| Incident hours                  | Total time spent investigating/resolving              |
| After-hours incidents           | Pages received outside core business hours            |
| Holiday/weekend coverage count  | Number of holidays or weekend days covered            |

These metrics should be visible to the entire team via a shared dashboard. Transparency is the foundation of trust in the rotation.

### Fairness Principles

1. **Equal rotation frequency:** Every engineer in the pool should rotate through on-call at the same frequency, absent exceptional circumstances (medical leave, bereavement, etc.).
2. **Equal holiday distribution:** Holiday on-call should be rotated equitably year-over-year. Maintain a running tally of holiday coverage and give preference to engineers with lower counts.
3. **Equal weekend distribution:** Weekend shifts should be evenly distributed. No engineer should carry more than 2 weekend shifts per quarter.
4. **Credit for secondary duty:** Time spent as secondary should count toward on-call metrics at some weighted value (e.g., 0.5x primary time) to acknowledge the cognitive load of backup responsibility.

### Fatigue Prevention

Fatigue is the single largest risk to on-call effectiveness. Implement these safeguards:

- **Minimum rest period:** No on-call duty within 8 hours of a prior incident that lasted more than 2 hours. Enforce this programmatically if possible.
- **Swap approval:** All rotation swaps must be approved by the rotation manager to prevent a single engineer from accumulating excessive swaps (which can indicate avoidance or burnout).
- **Mandatory timeout:** After handling 3 SEV-0 or SEV-1 incidents in a single week, the engineer is automatically cycled out of the rotation for at least one week.
- **Post-on-call recovery day:** The day after an on-call shift ends, the engineer should have no meetings before noon (for weekly shifts) or should take a "no meetings" day (for longer shifts).

---

## 5. Cognitive Load Management During Incidents

Incident response is cognitively demanding. High stress, incomplete information, time pressure, and multitasking degrade decision-making quality. Managing cognitive load is an operational imperative.

### The Incident Commander Model

Adopt a clear role structure during incidents to distribute cognitive load across multiple people:

- **Incident Commander (IC):** The single person responsible for coordinating response. Does NOT touch any systems. Focuses entirely on communication, timeline, resource allocation, and decision-making.
- **Operations Lead:** The technical lead doing the actual investigation and remediation. Reports status to the IC. Should not be distracted by communication or coordination tasks.
- **Scribe:** Maintains the incident timeline, records all actions taken, and documents decisions. Critical for post-incident reviews.
- **Subject Matter Experts (SMEs):** Called in as needed for specific subsystems. Should be briefed by the Operations Lead and then given focused tasks.
- **Communications / Liaison:** If the incident is customer-facing, a designated person handles status updates to stakeholders so the IC and Operations Lead can focus.

Even in smaller teams where one person wears multiple hats, explicitly state hats at the beginning of an incident: "I am the IC for this incident. Alice, you are Operations. Bob, please scribe."

### Decision Fatigue Countermeasures

- **Runbooks first:** Always check the runbook before making novel decisions. Runbooks reduce cognitive load by providing pre-vetted paths.
- **Time-box investigation:** After 30 minutes without progress, escalate or change approach. Stale investigation loops waste cognitive energy.
- **Defer non-critical decisions:** If a decision is not time-sensitive, write it down and make it after the incident is resolved.
- **Use decision trees:** Build incident decision trees that branch based on symptoms, guiding the engineer step by step without requiring recall of rare procedures.

### Post-Incident Cognitive Recovery

An incident isn't over when the service is restored. Cognitive recovery requires structured decompression:

- Mandatory 15-minute break after incident resolution before starting the postmortem
- Post-incident reviews as blameless learning exercises, not fault-finding
- Encourage exercise, hydration, and walking away from the keyboard after a severe incident
- Manager check-in within 24 hours of a SEV-0 or SEV-1 incident to assess engineer well-being

---

## 6. Escalation Paths and Policies

Every service must have a documented, tested, and well-understood escalation path.

### Escalation Chain

```
Level 0: Primary On-Call Engineer
  -> Acknowledges within SLA, attempts resolution using runbooks
  -> If stuck > 15 min: calls in Secondary

Level 1: Secondary On-Call Engineer  
  -> Collaborates with Primary on investigation
  -> Can triage to appropriate SME if needed
  -> Pages Tertiary if incident scope exceeds team capacity

Level 2: Tertiary / Domain Expert / Team Lead
  -> Provides technical guidance or access to specialized systems
  -> Can authorize emergency changes that bypass normal review

Level 3: Engineering Manager
  -> Decision-making authority for resource allocation
  -> Handles cross-team coordination
  -> Decides on extended outage communications

Level 4: Director / VP
  -> Only for SEV-0 or extended SEV-1 (>4 hours)
  -> Business continuity decisions
  -> External stakeholder communication
```

### Escalation Timing Rules

| Situation                                         | Escalate After   |
|---------------------------------------------------|------------------|
| Primary doesn't acknowledge page                  | 2-3 minutes      |
| Primary acknowledges but makes no progress        | 15 minutes       |
| Incident requires expertise outside team          | Immediately      |
| Primary suspects security incident                | Immediately      |
| Incident duration exceeds 1 hour                  | Page Manager     |
| Incident duration exceeds 4 hours                 | Page Director    |
| Customer-affecting incident during business hours | Page Comms team  |

### Policies

- **No blame for escalation:** Engineers must never be penalized for escalating. The cost of a false escalation is negligible compared to the cost of a delayed response.
- **When in doubt, escalate:** If an engineer is unsure whether escalation is warranted, they should escalate. The receiving engineer can always downgrade.
- **Automated escalation:** Configure paging software to automatically escalate after acknowledgement timeout, regardless of engineer action.
- **Document every escalation:** Each escalation event must be recorded in the incident log with timestamp, reason, and outcome. This data informs rotation and training improvements.

---

## 7. Tools and Infrastructure

### Pager Setup and Configuration

- **Primary channel:** Mobile push notification via a dedicated on-call app (PagerDuty, Opsgenie, Grafana OnCall, or similar)
- **Fallback channel:** SMS for all SEV-0 and SEV-1 alerts
- **Final fallback:** Phone call for SEV-0 only (or if SMS not acknowledged within 2 minutes)
- **Escalation timing:** 2-minute delay between primary and escalation tiers
- **Override scheduling:** All PTO, holidays, and planned absences must be entered into the scheduling system at least 2 weeks in advance
- **Calendar integration:** Sync on-call schedule to team calendar so everyone knows who is currently on duty

### Runbooks

Every alert that pages a human must have a corresponding runbook. A runbook is not optional — it is the minimum bar for a well-operated service.

**Required runbook content:**

- Alert name and description
- Severity classification
- Step-by-step investigation procedure (with specific dashboards, queries, and commands)
- Common causes and their telltale symptoms
- Remediation steps for each known cause
- Escalation criteria and contacts
- Service dependencies and responsible teams
- Links to relevant dashboards, logs, and monitoring
- Time estimates for each remediation step

**Runbook maintenance:**

- Review and update every runbook at least quarterly
- Automated testing where possible (e.g., validate that linked dashboards and queries still return data)
- After any incident that reveals a gap in a runbook, update the runbook within 48 hours
- Runbooks should be version-controlled alongside service code

### War Rooms

For major incidents, a war room (virtual or physical) concentrates the response team in a single communication channel.

**Virtual war room setup:**

- Dedicated Slack/Discord channel auto-created on SEV-0 declaration
- Bot auto-posts: current time, affected service, severity, primary/seconary names
- Pin the timeline, investigation notes, and current action items
- Only the IC posts status updates to prevent noise
- All non-essential communication happens in a thread

**Physical / Zoom war room:**

- Join a dedicated video call with screensharing
- Mute all non-speaking participants
- Use a shared document for structured note-taking (Decision Log format)
- Designate a timekeeper to announce every 15-minute interval

---

## 8. Handoff Best Practices

The handoff between on-call engineers is a high-risk moment. Information that lives in one person's head but not in the runbooks or dashboards is lost if not explicitly transferred.

### Structured Handoff Process

**15 minutes before shift change:**

1. **Review active incidents:** Go through every open incident with the outgoing engineer. Understand the current state, what has been tried, and what the next steps are.
2. **Pass known issues:** Is there anything brewing? A deployment in progress? A cron job about to run? Known flaky tests or odd monitoring behavior?
3. **Review dashboards:** Quickly scan the key service health dashboards together.
4. **Share context:** Recent changes (deployments, config changes, infrastructure modifications) that might affect on-call.
5. **Check the schedule:** Who is the secondary? Are there any known absences in the next few days?
6. **Acknowledge transfer:** Both engineers acknowledge the handoff in the scheduling system.

**Handoff documentation template:**

```
HANDOFF: [Date] [Time] [Time Zone]
FROM: [Outgoing Engineer]
TO: [Incoming Engineer]

ACTIVE INCIDENTS:
  - [#12345] Brief description, current status, next step

PENDING ITEMS:
  - Deployment of v2.3.1 expected at 14:00, watch for increased error rates
  - Database migration scheduled for 02:00 UTC

RECENT CHANGES:
  - Config change on api-gateway at 11:30 (rolled back, monitoring still active)

KNOWN ISSUES:
  - Flaky test in CI pipeline, may generate false-positive alerts
  - Elevated latency on us-east-1 during peak hours (under investigation)

SECONDARY ON-CALL: [Name]
```

### Overlap Requirements

- Minimum 15-minute overlap between primary shifts
- Minimum 30-minute overlap for cross-region handoffs (time zone differences)
- No handoff-by-email — must be synchronous (voice or video call + shared screen)

### Post-Handoff

The outgoing engineer remains "on-call" for their shift's open incidents for an additional 30 minutes (the "tail period") to ensure continuity. After 30 minutes, all responsibility formally transfers.

---

## 9. Training and Onboarding

New engineers should not be placed on the primary on-call rotation until they demonstrate operational competence. Rushing this process endangers both the service and the engineer's well-being.

### Shadowing Program

**Phase 1 — Observation (2-4 weeks):**
- Shadow the current primary on-call during all alerts
- Read every page notification, observe the response, read the runbook
- No responsibility — purely observational
- Goal: Understand alert patterns, service architecture, and team processes

**Phase 2 — Supervised (4-6 weeks):**
- Shadow engineer takes the first action on every page under direct supervision of the primary
- Primary reviews and approves/escalates every decision
- Shadow keeps their own incident log for feedback discussion
- Goal: Build procedural memory through guided practice

**Phase 3 — Secondary rotation (4 weeks):**
- New engineer rotates as secondary (backup) for one full rotation cycle
- Can be promoted to primary if no critical misses occur during this period
- Goal: Validate independence in a low-risk role

### Ramp-Up Schedule

| Phase                | Duration     | Role            | Success Criteria                          |
|----------------------|--------------|-----------------|-------------------------------------------|
| Shadowing            | 2-4 weeks    | Observer        | Can explain each alert and response       |
| Supervised practice  | 4-6 weeks    | Supervised      | Handles 80%+ of pages without help        |
| Secondary rotation   | 4 weeks      | Backup          | No SLA misses, demonstrates good judgment |
| Primary (initial)    | 2 weeks      | Primary (light) | Reduced rotation frequency at first       |
| Full rotation        | Ongoing      | Primary         | Full rotation inclusion                   |

### Wheel of Misfortune

The Wheel of Misfortune is a rotational training exercise where engineers practice incident response in a simulated environment:

- **Format:** Weekly or bi-weekly, 45-60 minutes
- **Setup:** Simulated incident scenario using a staging environment or tabletop exercise
- **Execution:** A random engineer is spun as the "on-call" and must respond to the scenario in real-time
- **Evaluation:** Peers observe and provide feedback on the response process, communication, and runbook usage
- **Rotation:** Different engineer each session; each engineer participates at least once per quarter

**Scenario library:** Maintain a library of 20+ scenarios based on real past incidents. Pull from this library randomly so the scenarios are not anticipated.

---

## 10. Underload Prevention

The flip side of on-call burnout is **underload** — long stretches with no pages that lead to skill degradation, complacency, and rust. Both extremes are dangerous.

### Keeping Skills Fresh

- **The 10% Rule:** If an engineer receives fewer than 2 pages per shift on average, they should proactively spend 10% of their on-call time on readiness activities (see below).
- **Scheduled system exploration:** During quiet periods, engineers should explore the monitoring dashboards, review recent changes, and trace through the systems they're responsible for.
- **Runbook refreshers:** Re-read the top 5 runbooks for the service. Once you know them cold, review the less common ones.
- **Regular practice:** The Wheel of Misfortune becomes even more important for teams with low alert volumes.

### Active Readiness Activities

Engineers with no active incidents should rotate through these tasks during their on-call shift:

1. **Dashboard review:** Spend 15 minutes reviewing each major service dashboard. Notice any trends, anomalies, or degraded metrics that haven't yet triggered alerts.
2. **Runbook verification:** Pick one runbook and verify every link, command, and query still works. File bugs for broken items.
3. **Alert audit:** Review the last 7 days of alerts. Any that were false alarms? Any that were legitimate but had no runbook?
4. **Monitoring gap analysis:** Are there failure modes that should have alerts but don't? Create tickets for missing coverage.
5. **Capacity review:** Check resource utilization trends — are any services approaching capacity limits?
6. **Documentation updates:** Fix stale documentation discovered during shift.

### DiRT Drills (Disaster Recovery Testing)

DiRT drills are planned exercises that simulate catastrophic failure scenarios to test team readiness:

- **Frequency:** At least quarterly per critical service
- **Scope:** Ranges from single-service failure to full-region outage
- **Execution:** Varies from tabletop walkthroughs to actual chaos engineering experiments (e.g., Chaos Monkey)
- **Outcome:** Each drill produces an action-item list of gaps found in runbooks, monitoring, or system design
- **Post-drill:** Fix identified gaps within the next sprint cycle

**Sample DiRT scenarios:**
- Primary database region becomes unavailable
- TLS certificate expires across all services
- Kubernetes control plane is unreachable for 30 minutes
- A bad deployment rolls out affecting 50% of traffic
- Secrets management system is compromised

---

## 11. When On-Call Isn't Working

Even the best-designed on-call system will show signs of strain over time. Recognize the symptoms early and intervene before they become chronic.

### Give Back the Pager

It is every engineer's right — and responsibility — to escalate concerns about the on-call system. This should be a supported, blameless process:

**When an engineer should signal distress:**
- Consistently missing SLA targets despite good-faith effort
- Dreading on-call shifts (anxiety, sleep disruption, avoidance behavior)
- Incident volume is interfering with day-job responsibilities
- Feeling inadequately prepared for the services on-call
- Multiple consecutive rotations with high incident count

**The "Give Back the Pager" protocol:**

1. Engineer notifies their manager that on-call is not working
2. Manager immediately finds coverage for the next rotation (no questions asked)
3. A structured conversation follows to identify root causes:
   - Is the engineer undertrained?
   - Are alerts excessive or poorly tuned?
   - Is the rotation schedule unfair?
   - Are there personal circumstances affecting availability?
4. An action plan is created with specific timelines
5. Engineer returns to on-call only after the action plan is executed and the engineer agrees they are ready

This process should be documented in team charter and reinforced by management as a positive signal (self-awareness and responsibility), not a negative one.

### Alert Tuning

If on-call engineers are complaining about alert volume or noise, the solution is almost never "the engineer needs to deal with it" — it is to tune the alerting.

**Common alerting problems and solutions:**

| Symptom                                  | Likely Cause                       | Fix                                                |
|------------------------------------------|------------------------------------|----------------------------------------------------|
| Too many pages per shift                 | Thresholds too sensitive           | Raise thresholds, add debouncing, use rate-of-change |
| Same page fires repeatedly               | No auto-acknowledge or dedup       | Implement alert deduplication and flapping detection |
| Pages during clearly understood patterns | Known issue, but no suppression    | Add maintenance windows or known-issue suppression |
| Alert fires but no actionable response   | Missing runbook or too vague       | Either add a runbook or remove the alert entirely  |
| Pager goes off during business hours for the same alerts every day | Alerts should be tickets, not pages | Downgrade to SEV-3/SEV-4 or convert to daily digest |
| "False alarm" rate > 50%                 | Alert logic is poor                | Investigate and fix each false-positive pattern    |

**The golden rule of alert tuning:** Every page should represent a real, actionable condition that requires a human within the defined SLA. If an alert does not meet this criteria, it should be removed, not tolerated.

### Collaboration with Dev Teams

Chronic on-call pain is often a symptom of systemic issues in how operational responsibility is shared between development and operations teams:

**Principles for healthy dev/SRE collaboration on on-call:**

1. **You build it, you run it (or close to it):** Development teams should participate in on-call for their own services. This creates a feedback loop — if your code pages you at 3 AM, you will write more resilient code.
2. **Error budgets:** Dev teams get a budget for acceptable error rates. If error budget is exhausted, no new features — only reliability work. This ties operational health directly to development decisions.
3. **Shared on-call:** In some organizations, dev and SRE engineers share the rotation for a service. This ensures both sides understand the operational reality.
4. **Pre-release operational readiness review:** Before a new service or major feature goes live, the team must pass an operational readiness checklist:
   - Runbooks exist and are tested
   - Monitoring and alerting covers known failure modes
   - Capacity planning is completed
   - Performance benchmarks are established
   - Escalation contacts are identified
5. **Blameless postmortems with dev ownership:** When an incident is caused by a code change, the postmortem action items should be owned by the dev team, not the SRE team. SREs support reliability improvements but should not be the sole implementors.

### When to Redesign the Rotation

Some problems cannot be fixed by incremental tuning. Signs that a rotation redesign is needed:

- Attrition due to on-call stress exceeds 15% per year
- More than 30% of on-call shifts require manager intervention to fill
- Average time to acknowledge pages exceeds SLA targets by 2x or more
- Engineers consistently cite on-call as their top source of dissatisfaction in surveys
- The team has grown or shrunk significantly but the rotation hasn't changed

**Rotation redesign triggers a structured process:**

1. Collect 6 months of on-call metrics (page volume, incident types, SLA performance, distribution)
2. Survey all engineers in the rotation about their experience, pain points, and suggestions
3. Design 2-3 alternative rotation models
4. Socialize the alternatives with the team and gather feedback
5. Pick the best model and pilot it for 1 full rotation cycle
6. Evaluate the pilot with metrics and surveys before committing

---

## Appendix: Quick Reference Checklist

### Weekly On-Call Checklist

- [ ] Verify the rotation schedule — confirm I am primary/secondary
- [ ] Check the alerting system — confirm contact methods are configured
- [ ] Check for active incidents from the prior shift
- [ ] Read the handoff doc from the outgoing engineer
- [ ] Review any upcoming deployments or maintenance windows
- [ ] Confirm secondary's contact info and availability
- [ ] Review the top 5 runbooks for my primary service

### Daily On-Call Checklist

- [ ] Review new alerts from the past 24 hours (even if acknowledged by auto-resolution)
- [ ] Scan key dashboards for anomalies
- [ ] Check for any scheduled changes or maintenance
- [ ] Review incident queue for any stale/in-progress items
- [ ] Update handoff doc at end of shift

### Quarterly On-Call Health Check

- [ ] Review on-call distribution metrics for the quarter
- [ ] Survey the team on on-call satisfaction (anonymous)
- [ ] Update and test all critical runbooks
- [ ] Conduct at least one DiRT drill
- [ ] Review and tune alert thresholds
- [ ] Audit escalation paths and contacts — are they still correct?
- [ ] Evaluate rotation schedule against team size changes
- [ ] Plan holiday coverage for the next quarter

---

## References and Further Reading

- Google SRE Books: *Site Reliability Engineering* and *The Site Reliability Workbook* (especially chapters on On-Call, Incident Response, Postmortem)
- *Seeking SRE* by David N. Blank-Edelman (Conference proceedings on SRE culture)
- *Incident Management for Operations* by Rob Schnepp, Ron Vidal, and Chris Hawley
- *The Checklist Manifesto* by Atul Gawande (for runbook design principles)
- PagerDuty Best Practices Guide for On-Call Scheduling
- Grafana On-Call Documentation: Incident Management Playbooks

---

> **Maintainer Note:** This document is a living reference. Review and update it at least quarterly as team practices, tools, and service architecture evolve. The most dangerous runbook is the one that contains stale, incorrect, or outdated guidance.

# Operational Documentation Lifecycle

Use this reference when runbooks, alerts, service maps, configuration guidance, or incident procedures are stale, hard to find, or not trusted by responders.

## Source anchors

Synthesized from *Seeking SRE*, “Do Docs Better: Integrating Documentation into the Engineering Workflow,” “Active Teaching and Learning,” “SRE Cognitive Work,” and “Psychological Safety in SRE.”

## Functional quality

A document is good when it helps its intended reader complete a real task safely. Check more than spelling and structure:

- Can the reader identify the symptom and scope?
- Does the document state prerequisites, access, and assumptions?
- Are the first actions safe, bounded, and reversible?
- Does every consequential action have an observable verification?
- Does it say when to stop, escalate, or hand off?
- Are dashboards, configuration, owners, and dependencies linked to authoritative sources?
- Can a new responder follow it without relying on private memory?

## Lifecycle

1. Create the smallest document for a named task and audience.
2. Store it beside the service or in a discoverable source-controlled location.
3. Link alerts and service reviews to it.
4. Test it during onboarding, game days, and real incidents.
5. Update it when the service, alert, configuration, dependency, or recovery path changes.
6. Record owner and review trigger, not only a calendar date.
7. Prune documents that no longer describe a supported path.

Documentation work belongs in the engineering workflow. A stale runbook is an operational defect that can prolong an incident.

## Coverage signals

Useful signals include alert-to-playbook coverage, broken-link count, age of last successful exercise, time for a new responder to complete a task, and incidents where missing or misleading documentation delayed mitigation. Do not optimize for document count or word count.

## Agent procedure

Use this reference with `templates/runbook-template.md`, `monitoring-alerting.md`, and `human-systems-and-learning.md`. When editing a runbook, verify every command's scope, prerequisite, expected result, and rollback. If the source of truth is unavailable, mark the link and assumption instead of fabricating it.

# Blameless Postmortem Culture & Methodology

> **Reference for Site Reliability Engineering teams**  
> How to build, run, and sustain a learning-oriented incident review practice

---

## Table of Contents

1. [The Blameless Postmortem Philosophy](#1-the-blameless-postmortem-philosophy)
2. [Postmortem Triggers and Criteria](#2-postmortem-triggers-and-criteria)
3. [Postmortem Structure](#3-postmortem-structure)
4. [The 5 Whys Methodology](#4-the-5-whys-methodology)
5. [Systemic Fixes vs. Human Fixes](#5-systemic-fixes-vs-human-fixes)
6. [Avoiding Blame Language](#6-avoiding-blame-language)
7. [Postmortem Review Process](#7-postmortem-review-process)
8. [Sharing and Knowledge Management](#8-sharing-and-knowledge-management)
9. [Measuring Postmortem Effectiveness](#9-measuring-postmortem-effectiveness)
10. [Common Pitfalls](#10-common-pitfalls)
11. [Cultural Integration](#11-cultural-integration)
12. [The Cost of Failure as Education](#12-the-cost-of-failure-as-education)
13. [Language Transformation Table](#13-language-transformation-table)

---

## 1. The Blameless Postmortem Philosophy

### What Is a Blameless Postmortem?

A blameless postmortem is a structured process conducted after an incident to understand what happened, why it happened, and how to prevent it from happening again. The defining characteristic — the "blameless" part — is that the analysis focuses entirely on systemic causes, process failures, and environmental conditions, never on individual actions, mistakes, or character.

The core axiom of blameless postmortems is:

> **Every person was doing their best with the information and tools available to them at the time.**

This is not naivete or an excuse to avoid accountability. It is a pragmatic recognition that blame is a failure-analysis dead end. When an engineer makes a mistake, the question is never "Who should be held responsible?" but rather "What conditions made that mistake possible, and how do we change those conditions?"

### Why Blameless?

The rationale is grounded in two domains:

**Psychological safety.** In his landmark research on high-performing teams, Google's Project Aristotle identified psychological safety as the single most important predictor of team effectiveness. Blameless postmortems create safety by guaranteeing that honest participation carries no career risk. Without this guarantee, incidents go uninvestigated, root causes remain hidden, and the same failures recur.

**Systems thinking.** Complex socio-technical systems fail in characteristic ways that are almost never reducible to individual error. Reason's "Swiss Cheese Model" of accident causation shows that most failures require multiple latent conditions (holes in the cheese) to align before a catastrophe occurs. Blaming the person at the sharp end — the operator who pressed the wrong button — ignores the dozen other holes that were already present: unclear documentation, missing alerts, poorly designed UI, insufficient testing, breakneck deadlines.

### The Safety-I vs. Safety-II Paradigm

Traditional incident analysis (Safety-I) asks: "What went wrong?" and seeks to eliminate errors. Modern SRE practice increasingly adopts Safety-II thinking, which asks: "Why did things go right most of the time?" and treats incidents as valuable data points about how the system actually behaves under stress. Blameless postmortems bridge both paradigms — they study failures to strengthen the system's ability to succeed.

---

## 2. Postmortem Triggers and Criteria

Not every bug, page, or degraded response merits a full postmortem. Organizations define explicit criteria to ensure postmortem effort is proportionate to incident impact and learning value.

### Recommended Trigger Criteria

A postmortem should be conducted when any of the following occurs:

| Severity | Criterion | Example |
|----------|-----------|---------|
| **High** | User-visible outage lasting > N minutes | 5+ minutes of 5xx errors for a customer-facing service |
| **High** | Data loss or corruption | Permanent loss of customer records |
| **High** | Security breach or intrusion | Unauthorized access to production data |
| **High** | Financial impact above threshold | >$10K in direct costs |
| **Medium** | Degraded experience exceeding SLO | P99 latency exceeds 2x target for 10+ minutes |
| **Medium** | On-call escalation requiring >2 people | Incident requiring coordination across teams |
| **Medium** | Manual intervention to restore service | Any incident where a person had to SSH into a box or edit a config to fix it |
| **Low** | Recurring same-class failure | Third occurrence of the same root-cause pattern |
| **Any** | Novel failure mode | Something never seen before that teaches a new lesson |

### Optional but Recommended Triggers

- Any incident that consumed an on-call shift entirely (burnout risk signal)
- Any incident that required a rollback
- Any incident involving a change that bypassed normal review
- Near-misses with high potential impact
- Customer-reported issues that revealed systemic gaps

### Setting the Bar

The threshold should be calibrated to the team's capacity. A good heuristic: **if you're going to tell someone about the incident, write a postmortem.** The formal barrier should be low enough that the 80th-percentile incident gets documented, not just the catastrophic ones.

---

## 3. Postmortem Structure

A good postmortem is organized, consistent, and actionable. Below is the canonical structure used by leading SRE organizations.

### 3.1 Metadata Header

```
Incident ID:        INC-2025-06-05-001
Title:              [Brief, descriptive title]
Date:               2025-06-05 14:32 UTC
Duration:           47 minutes
Severity:           Critical
Services Affected:  [list of services]
Trigger:            [what initiated the incident]
Postmortem Owner:   [name]
Reviewer:           [name]
```

### 3.2 Executive Summary

A 3-5 sentence overview accessible to non-technical stakeholders. State what happened, the impact, and the highest-priority action items.

### 3.3 Timeline

A chronological, time-stamped account of the incident. This is the most important section for learning. Include:

- **Before:** The state of the system prior to the trigger event (deployments, config changes, traffic shifts)
- **Trigger:** The specific event that initiated the incident
- **Detection:** When and how the team learned something was wrong (alert, user report, dashboard)
- **Response:** Every action taken, including false starts and dead ends
- **Resolution:** The action that restored service
- **After:** Remediation steps already taken during or immediately after the incident

Timeline entries should be precise UTC timestamps. Include relevant logs, metrics, and command output. Do not editorialize — just state what happened and when.

### 3.4 Root Cause

A single paragraph describing the fundamental underlying failure. The root cause should be **systemic** — a statement about process, design, or environment, not about a person.

**Good root cause statement:** "A race condition between the deploy pipeline's health-check timeout (30s) and the new service's JVM warmup period (45s) caused the load balancer to mark all instances as unhealthy, blackholing traffic."

**Bad root cause statement:** "Alice deployed during peak hours without checking the warmup time."

### 3.5 Contributing Factors

All the conditions that made the root cause possible or worsened the impact. This is where most of the learning lives. Common categories:

| Category | Examples |
|----------|----------|
| **System design** | Single point of failure, missing redundancy, tight coupling |
| **Process** | Missing change review, insufficient test coverage, gaps in runbook |
| **Tooling** | Poor observability, confusing dashboards, slow deployment tooling |
| **Organizational** | Team knowledge silos, inadequate on-call training, time pressure |
| **Environmental** | Dependency failure (DNS, cloud provider, third-party API), capacity exhaustion |

Aim for 3-8 contributing factors per postmortem.

### 3.6 Action Items

Specific, tracked, and owner-assigned tasks to prevent recurrence. Each action item should follow the SMART criteria.

| Field | Description |
|-------|-------------|
| **Description** | What specifically will be done? |
| **Type** | Preventative (reduces likelihood) / Mitigating (reduces impact) / Detectability (improves discovery) / Process (improves response) |
| **Owner** | Single responsible person |
| **Tracker URL** | Link to Jira/Asana/GitHub issue |
| **Due Date** | Realistic deadline |
| **Status** | Open / In Progress / Done / Won't Do |

Good action items are concrete: "Add a canary deployment stage that runs the health-check with a 60-second grace period" — not "Improve deployment safety."

### 3.7 Lessons Learned

What the team learned collectively:

- What went well (so it can be reinforced)
- What went poorly (the gaps identified)
- What surprised the team
- Where the team was lucky

### 3.8 Appendix

Supporting evidence: dashboard screenshots, log excerpts, config files, chat transcripts, monitoring graphs.

---

## 4. The 5 Whys Methodology

The 5 Whys is a root-cause analysis technique that iteratively asks "Why?" to peel back layers of symptoms until the fundamental systemic cause is exposed. It was developed by Sakichi Toyoda and is a core component of the Toyota Production System and Lean methodology.

### How It Works

Start with the incident description, then ask "Why did this happen?" For each answer, ask "Why?" again. Repeat until the answer points to a process, design, or organizational issue — not a person's action.

### Worked Example

**Incident:** Production database was dropped at 03:14 UTC on June 5.

1. **Why was the database dropped?**
   Because a `DROP DATABASE` command was executed against the production cluster.

2. **Why was the DROP DATABASE command executed?**
   Because an engineer ran a migration script against the wrong cluster.

3. **Why did the engineer run against the wrong cluster?**
   Because the production cluster and the staging cluster had nearly identical connection strings in the config file.

4. **Why did the config file have nearly identical connection strings?**
   Because the naming convention for clusters was `db-prod-1` and `db-staging-1` — only the environment segment differs, making it hard to distinguish at a glance.

5. **Why was the naming convention chosen?**
   Because there was no documented standard for cluster naming, and the original team followed a pattern from a deprecated infrastructure template.

**Root cause:** Inconsistent cluster naming conventions and a lack of visual differentiation in connection strings made cross-environment mistakes possible. The staging migration process lacked guardrails to prevent targeting production.

### When 5 Whys Works Best

- Single-cause, relatively simple incidents
- Incidents where the causal chain is well-understood
- Time-constrained situations (5 Whys can be done in 15 minutes)

### Limitations

- Complex incidents with multiple interacting failures may need more sophisticated analysis (e.g., causal trees, STAMP)
- The technique tends to produce a single linear narrative, which can miss parallel contributing factors
- The fifth "why" is often arbitrary; stop when you hit a systemic cause, not when you hit five

---

## 5. Systemic Fixes vs. Human Fixes

The central discipline of blameless postmortem culture is distinguishing between fixes that address the system and fixes that address the person.

### Systemic Fixes (Preferred)

Systemic fixes change the environment, process, or technology to make errors impossible or harmless.

| Type | Example |
|------|---------|
| **Automation** | Add a pre-commit hook that warns when a migration targets production |
| **Guardrails** | Implement a "production confirmation" step that requires a second approval |
| **Design change** | Use different color schemes for prod/staging database connection strings |
| **Process change** | Require runbook review as part of the deploy checklist |
| **Tooling** | Create a CLI wrapper that prevents dangerous commands across environments |
| **Observability** | Add an alert for `DROP DATABASE` statements on production |
| **Isolation** | Remove direct database access entirely; route all mutations through an API |

### Human Fixes (Avoid)

Human fixes attempt to prevent recurrence through individual effort, training, discipline, or accountability.

| Type | Why It Fails |
|------|-------------|
| **Retraining** | Assumes the engineer didn't know better; usually they did, but the system made error easy |
| **Reviewer added** | Adds a human bottleneck; humans are inconsistent at catching errors in repetitive reviews |
| **Policy change** | "Be more careful" is not an action item; policies without tooling are ignored under pressure |
| **Blame / disciplinary action** | Destroys psychological safety, discourages future reporting, and doesn't fix the root cause |
| **Process checklist** | Checklists are effective only when the failure mode is known and the checklist is enforceable |

### The Golden Rule

> **If a fix can be automated, it should be automated. If a guardrail can be built, it should be built. A human fix is the failure of engineering, and a second failure to fix the right thing.**

### Exceptions

There is a small category of genuinely individual failures: intentional malice, gross negligence, or repeated violation of clearly documented safety-critical procedures after the system was corrected. These are vanishingly rare and should be handled through management processes, not postmortem action items.

---

## 6. Avoiding Blame Language

Language shapes culture. The words used in a postmortem determine whether the document is a tool for learning or a weapon for blame.

### Principles of Blameless Language

1. **Describe actions, not people.** Say "the deployment pipeline removed healthy instances" not "the engineer unhealthy instances."
2. **Use passive or systemic voice.** Say "the alert was not configured" not "you forgot to configure the alert."
3. **State facts neutrally.** "The change was deployed during peak traffic" not "someone deployed at a stupid time."
4. **Attribute decisions to context, not character.** "The engineer chose the fastest option under time pressure" not "the engineer was lazy."
5. **Focus on conditions, not choices.** "The console showed identical hostnames for prod and staging" not "the engineer didn't check which host they were on."

### Blame Language Detection

Scan postmortem drafts for these red-flag words and phrases:

| Flag | Blame Implication |
|------|-------------------|
| "should have" | Hindsight bias — implies the person should have known the outcome |
| "failed to" | Focuses on omission rather than surrounding conditions |
| "neglected" | Suggests carelessness or laziness |
| "didn't" | Blames the absence of an action without asking why |
| "careless," "sloppy" | Character judgments that shut down analysis |
| "obvious" | Retrospective clarity; nothing was obvious in the moment |
| "common sense" | Assumes shared knowledge that may not have existed |

---

## 7. Postmortem Review Process

The review process ensures quality, consistency, and real learning. It should not be a rubber stamp.

### Step 1: Draft (Within 48 Hours)

The incident responder or designated postmortem owner writes the initial draft. Focus on the timeline while memory is fresh. Do not wait for perfection — publish an incomplete draft early.

### Step 2: Peer Review (Within 1 Week)

The postmortem is reviewed by:

- **The incident responder(s)** — verify timeline accuracy
- **A subject matter expert** — validate technical analysis
- **An SRE lead or manager** — check for blameless language and systemic action items
- **A peer from another team** — bring fresh eyes and challenge assumptions

Review criteria:
- Is the timeline accurate and complete?
- Is the root cause truly systemic?
- Are contributing factors explored, not just the trigger?
- Is every action item SMART and owned?
- Does the document pass the "blameless language" test?

### Step 3: Action Item Assignment

Each action item is assigned an owner and entered into the team's tracking system. The postmortem itself is not "closed" until all action items are resolved.

### Step 4: Broad Review (Weekly or Monthly)

A recurring meeting (e.g., a weekly "Postmortem Review" slot) where the team walks through new postmortems. This is not a punishment round — it is a collective learning session. Key outcomes:

- Shared understanding of failure modes
- Cross-team pattern recognition ("We saw this same issue in the payments service last month")
- Prioritization of systemic fixes that affect multiple services

### Step 5: Closure (After 30-90 Days)

- All action items have been completed or explicitly deprioritized
- The postmortem is archived in a searchable knowledge base
- A brief follow-up note is appended documenting what changed

---

## 8. Sharing and Knowledge Management

A postmortem that sits in a private Google Doc is worthless. Sharing is the mechanism through which one team's failure becomes every team's learning.

### Internal Sharing Practices

- **Postmortem email list / Slack channel.** A low-traffic, opt-in distribution for all completed postmortems.
- **Searchable archive.** Index all postmortems in a tool that supports full-text search. Tools like GitHub, Confluence, or dedicated platforms.
- **Incident database.** Maintain a lightweight registry (spreadsheet, wiki, or database) tracking: incident ID, date, service, severity, root cause category, action items.
- **Quarterly incident review.** A broader retrospective looking at patterns across multiple incidents.

### External Sharing

When the incident affected customers or had industry relevance, consider sharing publicly. Site Reliability Engineering pioneered the model of publishing postmortems to learn from each other across organizational boundaries:

- [Amazon AWS Postmortems](https://aws.amazon.com/message/)
- [Google Cloud Status Dashboard](https://status.cloud.google.com/)
- [GitHub Engineering Blog](https://github.blog/category/engineering/)
- [Cloudflare Blog](https://blog.cloudflare.com/tag/outage/)

### Anonymization

For shared postmortems, strip identifying information (engineer names, specific customer data, internal systems whose names leak architecture). The goal is to share the *lesson*, not the *story* behind it.

---

## 9. Measuring Postmortem Effectiveness

If you don't measure it, you don't know if it's working. Track these metrics to gauge postmortem program health.

### Process Metrics

| Metric | Target | What It Measures |
|--------|--------|------------------|
| **Time-to-postmortem** | <48 hours from incident | Freshness of analysis; delays reduce accuracy |
| **Postmortem coverage** | >90% of qualifying incidents | Are we writing postmortems consistently? |
| **Time-to-review** | <7 days from draft | Is the review process a bottleneck? |
| **Action item completion rate** | >80% within 90 days | Are we following through? |

### Outcome Metrics

| Metric | Target | What It Measures |
|--------|--------|------------------|
| **Repeat incident rate** | Declining trend | Are we learning? Same-class incidents should decrease |
| **Mean time to resolve (MTTR)** | Stable or declining | Are postmortem findings improving response? |
| **Incident severity distribution** | Shift toward lower severities | Are preventative measures working? |
| **Action items per postmortem** | 3-5 (sweet spot) | Too few = shallow analysis; too many = scope creep |

### Cultural Metrics

Harder to quantify but equally important:

- **Postmortem participation rate.** Are on-call engineers contributing, or just the SRE lead?
- **Psychological safety surveys.** Do team members feel safe reporting incidents and mistakes?
- **Blame-language audit.** What fraction of postmortem language is blame-free? Trend this over time.

---

## 10. Common Pitfalls

Even well-intentioned postmortem programs fall into predictable traps.

### Superficial Analysis

The postmortem stops at the first apparent cause rather than digging deeper.

**Signs:** Action items are trivial or obvious. Root cause is a single sentence that blames a person or a "config error." Contributing factors section is empty.

**Fix:** Use the 5 Whys. Ask "What made that possible?" for every finding. Insist on at least three contributing factors.

### Too Many Action Items

A postmortem produces 15+ action items, most of which are never completed. This creates a false sense of progress and erodes trust in the process.

**Signs:** Action item list is a brain dump. Many items are vague ("improve testing"). No owner assigned. Completion rate is below 50%.

**Fix:** Limit action items to 3-5 per postmortem. Force prioritization. If something is important but not a top-5 priority, create a separate initiative. Track completion aggressively.

### No Follow-Through

Postmortems are written, action items are created, and then nothing happens. This is the most damaging pitfall because it teaches the organization that postmortems are theater.

**Signs:** Action items stay "Open" for months. The same root cause appears in multiple postmortems. Engineers stop participating because they don't see change.

**Fix:** Assign a postmortem program owner with authority to escalate. Review action item completion in every sprint planning. Close postmortems only when action items are resolved.

### Blame Creep

Despite professed blameless culture, postmortems contain subtle blame language. Senior engineers publicly say "no blame" but privately harbor resentment.

**Signs:** Postmortem drafting is avoided. Engineers deflect ownership. Language audits find blaming phrases.

**Fix:** Train reviewers on blameless language. Conduct periodic audits. Model vulnerability from leadership — managers should present their own postmortems first.

### Survivorship Bias

Only major incidents get postmortems; the small ones and near-misses are ignored. This means the high-frequency, low-severity failure modes never get fixed.

**Fix:** Lower the postmortem threshold. Create a lightweight "mini-postmortem" format (just timeline + root cause + 1 action item) for smaller incidents.

---

## 11. Cultural Integration

Postmortem culture is not created by writing a policy document. It is created through rituals, habits, and visible leadership behavior.

### Postmortem of the Month

A monthly session where the team reviews the most interesting postmortem — either internal or from another company. The presenter walks through the timeline and lessons. This:

- Normalizes failure as a topic of conversation
- Builds shared mental models of failure modes
- Creates a rhythm of continuous learning
- Makes postmortems a source of interest, not dread

### Reading Clubs

Organize a book club or reading group around SRE and incident analysis texts:

- *Site Reliability Engineering* (Beyer et al., O'Reilly)
- *The Field Guide to Understanding Human Error* (Dekker)
- *Drift into Failure* (Dekker)
- *The Art of Capacity Planning* (Allspaw)
- *Incident Management for Operations* (Limoncelli et al.)

Rotate facilitation. Encourage members to bring parallels to their own incidents.

### Wheel of Misfortune

A popular training exercise developed at Google. A team is given a simulated incident scenario and must respond in real time, making decisions, communicating, and debugging under pressure. After the simulation, the group conducts a mini-postmortem.

**Benefits:**
- Builds muscle memory for incident response
- Reveals gaps in runbooks, tooling, and team coordination
- Normalizes failure in a safe environment
- Identifies who thrives under pressure (not to "grade" but to assign incident roles)

**Format suggestions:**
- Run quarterly, rotating scenario types
- Include participants from multiple teams
- Invite observers who are not participating
- Treat outcome as data, not evaluation

### Incident Commander Rotation

Ensure every engineer takes a turn as incident commander. This builds empathy for the on-call experience, breaks down the "operator vs. developer" divide, and ensures postmortem recommendations reflect real operational pain.

### Leadership Modeling

The most powerful cultural signal is a senior leader presenting their own postmortem — admitting a mistake their team made, showing vulnerability, and modeling the blameless analysis approach. When the VP of Engineering says "Here's a failure I own and here's what I learned," the entire organization internalizes that postmortems are safe.

---

## 12. The Cost of Failure as Education

### The Tuition Model

Every failure is tuition paid for organizational learning. The question is whether the lesson is actually learned. An incident that costs $50K in downtime and produces a postmortem with actionable, implemented fixes is $50K well spent. The same incident that produces no learning is pure waste.

### Calculating the Return

Organizations that invest in postmortem culture see measurable returns:

- **Reduced incident frequency.** Each failure prevented saves the direct cost of downtime plus the opportunity cost of engineers' time spent fighting fires.
- **Faster incident resolution.** Teams that practice postmortems develop shared mental models that accelerate diagnosis during the next incident.
- **Lower turnover.** Psychological safety is one of the strongest predictors of retention. Engineers leave organizations where they fear blame.
- **Customer trust.** Fewer and shorter outages directly affect customer retention and brand equity.

### The Cost of Not Doing Postmortems

The alternative to blameless postmortems is not "no postmortems" — it's secret blame, finger-pointing, and unrepeated lessons. The cost of blameless culture is a small investment in writing and reviewing documents. The cost of *not* having blameless culture is the same incidents recurring indefinitely, accompanied by attrition of the very people who understand the system best.

### Messaging to Leadership

When pitching postmortem culture to skeptical leadership:

> "We can pay for failure once and learn from it, or we can pay for it over and over. A blameless postmortem is the mechanism that converts incident cost into institutional knowledge. Without it, we're just paying tuition and skipping class."

---

## 13. Language Transformation Table

A quick-reference guide for converting blame-laden language into blameless, systemic language.

| Blame Language (Avoid) | Blameless Language (Use) |
|------------------------|--------------------------|
| "Alice failed to check the config before deploying." | "The deploy process did not include a pre-flight config validation step." |
| "Bob should have noticed the alert earlier." | "The alert was routed to a channel that Bob had notifications muted on." |
| "The team was careless with the migration." | "The migration script lacked a dry-run mode, making it impossible to test without risk." |
| "No one thought to monitor the database connection pool." | "Connection pool metrics were not surfaced in the standard service dashboard." |
| "The developer didn't read the documentation." | "The documentation for this feature was stored in a separate wiki that was not linked from the deployment guide." |
| "Someone pushed a bad change." | "Change X introduced a regression in the user-auth module. The change passed code review but lacked test coverage for the edge case encountered." |
| "The operator panicked and made it worse." | "During the incident, the operator's actions were taken under time pressure without a defined escalation path. The runbook for this scenario did not exist." |
| "It was a human error." | "The UI presented the 'Delete' and 'Archive' buttons adjacent to each other with identical styling, making misclicks predictable under pressure." |
| "Why didn't anyone catch this in QA?" | "The test environment did not replicate production traffic patterns, so the race condition did not manifest during testing." |
| "The SRE on call was sleeping and missed the page." | "The paging system uses a single notification channel that can be silenced by Do Not Disturb mode. The escalation policy does not include a secondary on-call." |
| "Common sense should have prevented this." | "The safety mechanism for this operation was entirely manual and relied on individual vigilance, which is unreliable under fatigue or time pressure." |
| "This engineer has made this mistake before." | "This is the second incident involving this failure pattern, indicating that the previous action items (retraining, docs update) were insufficient. A systemic fix is needed." |
| "The rollout was reckless." | "The rollout was performed during peak traffic because the release calendar left no off-peak window. The deployment pipeline did not enforce a canary strategy." |
| "They should have rolled back immediately." | "The rollback procedure was documented in a separate, infrequently accessed runbook and required credentials not available to the incident commander." |

---

## Appendix: Sample Mini-Postmortem Template

For low-severity incidents where a full postmortem is disproportionate:

```
## Mini-Postmortem: [Title]

**Incident ID:** INC-YYYY-MM-DD-NNN  
**Date:** YYYY-MM-DD HH:MM UTC  
**Duration:** N minutes  
**Impact:** [Brief description]

### What Happened
[2-3 sentences]

### Root Cause
[1 sentence, systemic]

### One Action Item
- [ ] [Concrete, owned, tracked] — Owner, Due

### What We Learned
[Optional: 1-2 sentences about the takeaway]
```

---

## Further Reading

| Resource | Author(s) | Why |
|----------|-----------|-----|
| *Site Reliability Engineering* | Beyer, Jones, Petoff, Murphy | The canonical text on SRE practice, including postmortem chapters |
| *The Field Guide to Understanding Human Error* | Sidney Dekker | The definitive argument for the "new view" of human error |
| *Drift into Failure* | Sidney Dekker | How complex systems gradually move toward failure |
| *Incident Management for Operations* | Limoncelli, Kerth | Practical incident response and postmortem facilitation |
| *Learning from Incidents in Software* | Allspaw (blog) | Seminal essays on blameless analysis from Etsy's CTO |
| *Postmortem Action Items: How Google SRE Reduces Incident Recurrence* | Google SRE (white paper) | Data-driven look at what makes action items effective |
| *Project Aristotle* | Google re:Work | Research on psychological safety in high-performing teams |

---

> **Last updated:** 2025-06-05  
> **Maintainer:** SRE Skill Profile / an Agent Skills-compatible agent  
> **License:** This reference is part of the an Agent Skills-compatible agent SRE profile. Use freely, adapt locally.

# Product-Focused Reliability for SRE

## Synthesized Analysis

> **Source Article:** "Product-Focused Reliability for SRE" by Carl Crous, Parker Roth, and Victoria Hurd (Google)
> **URL:** https://sre.google/resources/practices-and-processes/product-focused-reliability-for-sre/
> **Analysis Date:** June 2026

---

## Overview

This article, published by Google SRE practitioners Carl Crous, Parker Roth, and Victoria Hurd, represents a significant evolution in Site Reliability Engineering thinking. It proposes a fundamental shift away from the traditional *service-centric* SRE model — where SRE teams own and operate specific infrastructure services — toward a *product-centric* model where SRE accountability is organized around what users actually care about: product functionality and end-user outcomes.

This is not a minor tweak to existing SRE practice. It reframes the unit of ownership, the stakeholders SRE partners with, the way SLOs are defined and measured, and even what triggers a pager. For practitioners, it offers a compelling path out of the scalability trap that service-based SRE inevitably hits as organizations grow.

---

## 1. Why Service-Centric SRE Falls Short

Traditional SRE organizes around *services* — databases, load balancers, authentication backends, API gateways. A team owns a set of services and is responsible for their availability, latency, and capacity. This model has been enormously successful but carries fundamental limitations.

> *"Service availability doesn't always result in user happiness with a product."*

### The Five Limitations

| Limitation | Description | Impact |
|---|---|---|
| **Services are partial proxies for user needs** | A user doesn't care about "database write availability at 99.99%." They care about "can I save my draft?" High service-level availability can mask broken user journeys. | SREs optimize what they can measure (service metrics) rather than what matters (user outcomes). |
| **Complex UIs create coverage gaps** | Modern web and mobile applications involve client-side logic, caching layers, retry logic, async RPCs, and service meshes. A service-level SLO cannot capture failures that happen entirely in the client or across multiple service hops. | The symptom a user sees (a blank page, a stuck spinner) is invisible to server-side monitoring. |
| **Service growth outpaces engineering** | The number of servers and microservices has grown exponentially (driven by public cloud). The number of software engineers grows linearly. SRE teams cannot scale by adding more services to their portfolio. | SRE teams become bottlenecks. They support more services with the same headcount, but each service gets thinner coverage. |
| **Service support optimizes a narrow slice** | If an SRE team owns a set of services, their incentives align with maximizing those services' metrics — not with the end-to-end product experience. They may ship infrastructure improvements that don't move the needle on user satisfaction. | Effort is misallocated. The SRE team works hard but the product doesn't get more reliable from the user's perspective. |
| **Async flows can't be measured by a single service** | A user request may trigger a chain: UI → enqueue → background worker → third-party API → callback. If any link in this chain fails silently, no single service's SLO catches it. | The user sees a failure. The SRE team sees green dashboards everywhere. The disconnect erodes trust in SRE. |

### Practitioner Takeaway

If your team has ever said "all our SLOs are green but users are complaining," you have experienced this problem. The service-centric model is not *wrong* — it is *insufficient*. It gives SREs a false sense of coverage and misdirects engineering effort toward infrastructure metrics that don't correlate cleanly with user happiness.

---

## 2. The Product Support Model

The article's central thesis: shift the unit of accountability from *services* to *products*. An SRE team doesn't ask "which services do we own?" but rather "which product functionality are we responsible for making reliable?"

> *"Rather than accepting accountability for service reliability, SRE teams can accept accountability for the product itself."*

### Key Characteristics

- **Accountability shifts upward.** The team is responsible for whether a user can complete a specific task (e.g., "send an email"), not for whether a particular microservice is healthy.
- **Scope broadens.** Product accountability naturally encompasses multiple services, client-side code, and third-party dependencies. The SRE team must care about the entire path.
- **Language changes.** Instead of talking about RPC latency percentiles, the team talks about "how long does it take to load the inbox?" — the same language product managers and UX researchers use.
- **Priorities get sharper.** When a pager fires, the question becomes "which user objective is affected and how severely?" rather than "which service is down?"

### What This Is Not

The product support model does **not** eliminate service ownership. Infrastructure must still run. The shift is one of *primary focus*: the team's north star is product reliability, and service reliability is a means to that end, not an end in itself.

---

## 3. Product Engagement

For product-focused SRE to work, SREs must step outside their traditional engineering silo and engage with disciplines they rarely partnered with before.

### New Stakeholder Relationships

| Stakeholder | What They Bring | Why SRE Needs Them |
|---|---|---|
| **Product Managers (PMs)** | User objectives, feature priorities, KPI definitions, roadmap | Defines *what* reliability means for the product, sets severity guidelines |
| **UX Researchers** | User behavior data, friction points, Jobs to be Done analysis | Grounds reliability work in real user behavior, not assumptions |
| **Engineering Teams** | Service architecture, implementation details, operational knowledge | Technical partner for instrumentation and incident response |
| **Support/Customer Success** | Escalation patterns, common user complaints | Early warning system for reliability gaps |

### Frameworks SRE Must Learn

The article highlights two frameworks that PMs and UX teams already use:

**Jobs to be Done (JTBD):** Models user objectives as "jobs" — the progress a user is trying to make in a particular circumstance. A user doesn't "use Gmail"; they "communicate with colleagues" or "archive promotional emails."

**Critical User Journeys (CUJs):** Google's internal framework that identifies the most important end-to-end paths a user takes through a product. CUJs map directly to what SRE should measure.

> *"Using this information, SRE can identify what is important to the product and its users, and they can define reliability in the same language used to define the product."*

### Practitioner Takeaway

This is perhaps the hardest cultural shift for SRE teams. SREs are trained to think in terms of systems, binaries, and infrastructure — not user behavior, product features, and business KPIs. Learning to speak the same language as PMs requires deliberate effort. The payoff is that SRE work becomes visible and valued at the product level, not just the infrastructure level.

---

## 4. Bootstrapping Workflow

The article provides a concrete 4-step workflow for adopting the product support model. This is not abstract theory — it is a practical sequence with intermediate deliverables.

### Step 1: Engage Stakeholders

| Action | Deliverable |
|---|---|
| Identify all relevant stakeholders using a RACI matrix | RACI chart |
| Meet with PMs, UX, engineering, support | Documented roles and responsibilities |
| Establish communication cadence | Meeting schedule, escalation paths |

The stakeholder set is *broader* than service-based SRE. A service engagement might only involve the service owner's engineering team. A product engagement includes PM (who defines what success looks like), UX (who understands user behavior), engineering (who builds and maintains), and support (who fields user complaints).

### Step 2: Model the Product

| Action | Deliverable |
|---|---|
| Identify user objectives (what users want to achieve) | List of user objectives |
| Break each objective into steps (individual user actions) | Step breakdown per objective |
| Register in a product registry | Maintained product model |

> *"People use a product to achieve a real-world objective... The user's intent is also a powerful reliability tool that SREs can leverage."*

### Step 3: Measure Performance

| Action | Deliverable |
|---|---|
| Define SLOs across three categories (service, client, E2E) | SLO definitions |
| Annotate SLOs with user objective/step information | Product-level SLOs |
| Set targets with user context | SLO targets aligned to product criticality |

### Step 4: Manage Reliability

| Action | Deliverable |
|---|---|
| Onboard a single objective/step | Initial SRE support scope |
| Iterate: improve metrics, expand objectives, address gaps | Expanded coverage |
| Periodically offboard obsolete SLOs | Focused, current support scope |

---

## 5. User Objectives and Steps

The decomposition of a product into user objectives and steps is the foundational modeling technique in the product support model.

### Definitions

- **User Objective:** A user's intent — what they want to achieve. High-level and stable. Example: "Communicate with people."
- **Step:** An individual action the user takes to accomplish the objective. Isolated, measurable, independent units of work. Example: "Compose and send a new email."

### Worked Example: Mail Service

| User Objective | Steps | Description |
|---|---|---|
| **Communicate with people** | Compose mail | Create and format a new email message |
| | Send mail | Submit the composed message for delivery |
| | Read incoming mail | View received messages in the inbox |
| | Search mail | Find specific messages by keyword or filter |
| | Manage mail | Delete, archive, label, or move messages |
| | Manage contacts | Add, edit, or delete address book entries |
| **Prevent unwanted communication** | Filter spam | Automatically classify and divert unwanted messages |
| | Block senders | Manually block specific email addresses |
| **Organize communication** | Create labels/folders | Build a personal organizational structure |
| | Apply rules/filters | Automate message triage |

### Why This Decomposition Matters

| Aspect | Service-Centric View | Product-Centric View |
|---|---|---|
| What we measure | RPC latency to the mail-store service | Time to render inbox with unread messages |
| What we alert on | Error rate on compose-save endpoint | Failure rate of "save draft" step |
| What we prioritize | Database replication lag | "Send mail" step latency during peak hours |
| Language used | "The compose validator service is p99=500ms" | "Composing mail is slower than users tolerate" |

### Practitioner Takeaway

If your product does not have clearly defined user objectives (owned by PM, not SRE), you will have to develop them yourself — at significant engineering cost, and with the risk that only SRE is invested in maintaining them. **The article is clear: PMs should own the user objectives.** SRE should partner, not own.

---

## 6. Product Criticality and Prioritization

Not all user objectives are equally important. Product criticality defines which objectives matter most.

### Severity Guidelines

Severity guidelines map outage impact to user experience. Examples from the article:

| Severity | Mail Service Example | Impact |
|---|---|---|
| **S1 (Critical)** | Cannot read or send email | Core functionality broken |
| **S2 (High)** | Emails delayed by > 5 minutes | Significant degradation |
| **S3 (Medium)** | Spell check, auto-complete not working | Auxiliary features degraded |
| **S4 (Low)** | UI cosmetic issue, non-critical feature broken | Minimal user impact |

### Product Criticality Matrix

Using severity guidelines, user objectives and steps are assigned criticality:

| User Objective | Steps | Criticality | Rationale |
|---|---|---|---|
| **Communicate with people** | Compose mail | Critical | Core functionality |
| | Send mail | Critical | Core functionality |
| | Read incoming mail | Critical | Core functionality |
| | Search mail | High | Important but not blocking |
| | Filter spam | Medium | Value-add |
| **Prevent unwanted communication** | Filter spam | Medium | Nice-to-have |
| **Organize communication** | Create labels | Low | Power user feature |

> *"The critical definition is modeled around the user's objectives (like composing and sending email) rather than how the system was implemented to address these needs."*

### Prioritization Principles

The article offers practical guidance for aligning SRE work with product KPIs:

1. **Base coverage first:** Ensure good base coverage across the entire infrastructure before investing in targeted improvements.
2. **Follow the KPIs:** Use product-level severity guidelines to prioritize. If revenue is the KPI, reliability work that protects revenue comes first.
3. **Avoid the one-size trap:** Don't apply the same methodology everywhere. Batch traffic doesn't need the same SLOs as interactive traffic.
4. **Invest proportionally:** Spend more on critical objectives, less on auxiliary ones. Not everything needs an end-to-end SLO.

### Practitioner Takeaway

> *"If an SRE knows an issue is severe, they will react and escalate more quickly, resulting in a faster resolution."*

Product criticality informs not just SLO design but incident response. When a page comes in annotated with "Critical User Objective: Compose Mail," the responder knows this is a customer-facing issue affecting core functionality, not an internal infrastructure hiccup.

---

## 7. Three SLO Categories

The article identifies three categories of SLOs with distinct trade-offs. Practitioners should use all three strategically.

| Dimension | Service SLOs | Client-Side Instrumentation | End-to-End SLOs |
|---|---|---|---|
| **Cost** | Low | Moderate | Very high |
| **Confidence** | High | Low | High |
| **Latency** | Low | Moderate | High |
| **Coverage** | Narrow | Broad | Narrow |
| **Where measured** | Server logs, load balancers | Browser, mobile app telemetry | Joined data across multiple sources |
| **What it catches** | Server-side failures | UI failures, client-side bugs, network issues | Async failures, multi-step workflows |
| **Data reliability** | High | Low (some data loss expected) | High (per-interaction measurement) |
| **Engineering effort** | Minimal | Moderate | Significant |

### Service SLOs

The familiar kind — measured from application servers, load balancers, or monitoring probes. Cheap, reliable, but myopic. They cannot see client-side failures, async failures, or issues that span multiple services.

*Good for:* Infrastructure health, baseline coverage, non-critical paths.

### Client-Side Instrumentation

Telemetry collected from web browsers or mobile apps. Captures what users actually experience — rendering time, network latency, client-side errors. Data is less reliable (batched, subject to loss on app close or network disconnect) but provides uniquely valuable insight.

> *"Complexities in how the interface behaves — caching, retries, and asynchronous RPC requests — are transparent when you can measure start and end conditions from the user interface."*

*Good for:* UI performance, mobile reliability, measuring what users actually experience.

### End-to-End SLOs

The most expensive and most accurate category. Joins data from multiple sources to measure a complete user interaction. For async workflows (e.g., "user requested a report → report was generated → report was delivered"), end-to-end SLOs are the only way to know if the workflow actually completed.

*Requirements:* Must identify and correlate events across systems, handle time synchronization, and manage data pipelines.

*Good for:* Critical user journeys, async workflows, business-critical features.

### Practitioner Takeaway

> *"There are classes of issues that cannot be measured from a single server, for example, issues that occur within a web or mobile application, or through an asynchronous action."*

The art is choosing the right SLO category for each user objective. Critical objectives get end-to-end SLOs. Important objectives get client-side or service SLOs. Everything else gets basic service SLOs or nothing at all. **Do not over-instrument.**

---

## 8. Product SLOs

A product-level SLO is any SLO that has been annotated with user objective and step information. It bridges the gap between infrastructure metrics and user experience.

### Annotation Process

1. Take an existing SLO (service, client-side, or end-to-end).
2. Annotate it with: which user objective it serves, which step it supports, the criticality level.
3. Set the target with user context, not infrastructure context.

### Before and After

| Aspect | Service-Framed SLO | Product-Framed SLO |
|---|---|---|
| **Question** | "How reliable should the AddressLookup service be?" | "How many errors can users tolerate when looking up email addresses?" |
| **Target** | 99.9% availability | 99.5% of "look up address" steps succeed within 200ms |
| **Context** | Infrastructure concern | User concern |
| **Validation** | Service properties | Product criticality guidelines |

> *"Setting the objective target or latency thresholds is notoriously challenging when the SLOs are framed around services and infrastructure. But when the SLOs have the additional context of the user's objectives, what is being measured becomes much clearer."*

### Why This Matters

- **Prevents over-engineering:** If a step is non-critical, you can justify a looser SLO. Without product context, SREs default to "as reliable as possible" — which is expensive.
- **Justifies spending:** "We need to invest in the 'send mail' E2E SLO because it's critical" is a much stronger case than "we need to improve the mail-sender service's p99."
- **Enables trade-offs:** When capacity is limited, you know which steps to degrade first.

---

## 9. Telemetry and Annotation

To turn product-level SLOs from theory into practice, SREs must instrument requests with product context.

### Client-Side Annotation

The user interface (web or mobile app) annotates each request with the user objective and step it serves. This happens at the source — the closest point to the user. The annotation propagates through the infrastructure stack.

### Server-Side Annotation

The server that handles the initial request infers the user objective from the request endpoint, parameters, or headers. This requires less client-side instrumentation but may be less accurate.

### Propagation Benefits

Once requests are annotated, the annotation propagates through the entire infrastructure stack, enabling:

| Use Case | Benefit |
|---|---|
| **Monitoring** | Dashboards show reliability per user objective, not per service |
| **Alerting** | Pages include "Critical: Compose Mail step failing" context |
| **Incident Response** | Responders know immediately which user functionality is affected |
| **Traffic Routing** | Low-priority steps can be shed before they harm critical steps |
| **Load Shedding** | During overload, shed non-critical steps first |
| **SLO Tracking** | Burn rates are tracked per user objective |

> *"This request-level information can also be used in traffic routing and load shedding policies to ensure that lower priority functionality doesn't harm more mission critical features."*

### Practical Guidance

The article advises against trying to annotate everything. Use product criticality to decide: annotate only the most critical user objectives, at least initially. The cost of annotation (engineering time, propagation complexity, data storage) must be justified by the value.

---

## 10. Onboarding, Iteration, Offboarding

Managing product reliability is an ongoing process, not a one-time project.

### Onboarding

Start small. Do not try to support every user objective at once.

1. Pick **one** user objective and a subset of its steps.
2. Ensure the team understands the objective thoroughly.
3. Define SLOs (start with service-level, add client-side and E2E as justified).
4. Establish the monitoring and alerting infrastructure.
5. Confirm incident response procedures cover this objective.

### Three Investment Areas

| Area | Description | Risk of Underinvesting |
|---|---|---|
| **Improve metrics** | Better instrumentation, more accurate SLIs, reduced data loss | Team misses root causes, can't validate improvements |
| **Expand objectives** | Add more user objectives and steps to SRE coverage | Important functionality remains unprotected |
| **Address gaps** | Fix the reliability issues the metrics reveal | Team measures problems but doesn't fix them |

> *"Not investing sufficiently in areas #1 and #2 can cause your team to miss targeting the most impactful issues in area #3. Investing only in areas #1 and #2 won't result in any improvements that help the user."*

### Offboarding

Periodically re-evaluate supported objectives and SLOs. Some become less important over time. Some become obsolete. The article recommends:

- Maintain a prioritized list of all supported objectives, steps, and SLOs.
- Regularly validate that the team's effort matches the current priority order.
- Offboard items that are no longer critical, freeing capacity for more important work.

> *"Every SLO carries an ongoing cost to maintain its underlying data and respond to SLO misses."*

---

## 11. Server Support and Baseline

A common objection to product-focused SRE is: "If we're focused on products, who keeps the servers running?" The article addresses this directly.

### Baseline Support

The concept of *baseline support* predates the product support model at Google. It's a set of standards and best practices that ensure any server can be run reliably without deep understanding of its specific purpose.

| Baseline Element | Purpose |
|---|---|
| Common libraries and frameworks | Consistent load balancing, RPC, monitoring |
| Platform-level reliability | Infrastructure SLOs owned by platform team |
| Minimum operational standards | Logging, monitoring, deployment, rollback |
| Development team ownership | The team that builds the service handles basic operations |

> *"The platform rather than the SREs provide baseline reliability support, which frees SREs to focus on more impactful reliability improvements."*

### When Do SREs Get Paged?

This is one of the most operationally consequential shifts in the model:

| Scenario | Service-Centric | Product-Centric |
|---|---|---|
| Database replica lag spikes | SRE paged | No page (unless it fails a product SLO) |
| Canary deployment fails | SRE paged | No page (unless it affects a critical objective) |
| Batch job fails silently | SRE paged | No page (batch is non-critical) |
| Inbox fails to load for 1% of users | No page (load balancer shows 99.9% uptime) | **SRE paged** (critical objective failing) |
| Async report generation fails | No page (enqueue succeeded) | **SRE paged** (end-to-end SLO breached) |

> *"Failures at the infrastructure level that do not impact the product SLOs will not alert SREs and can be handled by the development team that owns the service."*

### Practitioner Takeaway

This is liberating but requires organizational maturity. The development team must be capable of handling their own service issues. The platform must provide sufficient baseline support. Without these preconditions, the product focus will collapse because the infrastructure will rot.

---

## 12. When It's a Good Fit

The product support model is not universally applicable. The article is refreshingly honest about the prerequisites.

### Requirements Checklist

| Requirement | Why It's Needed |
|---|---|
| **Clear roles and responsibilities** | Without RACI clarity, the broader stakeholder set creates ambiguity, not alignment |
| **User objective definitions** | Must be owned by PMs, not retrofitted by SRE alone |
| **Easily maintainable servers** | If SREs spend all their time keeping servers alive, they can't focus on product |
| **Clear connection to user-facing functionality** | Infrastructure services with low-level APIs may not map cleanly to user objectives |

### Three Options for Infrastructure Services

The article acknowledges that some services (like data storage or message queues) don't have a direct user-facing component. Three approaches:

| Option | Description | Best For |
|---|---|---|
| **Map indirectly** | Find the upstream consumer that makes the service relevant to user objectives | Services close to the user-facing stack |
| **Treat as platform** | Apply baseline support only; don't invest in product-level SLOs | Generic infrastructure (KV stores, queues) |
| **Skip the model** | Some services don't benefit from product focus; use traditional SRE | Internal tooling, non-user-facing services |

### Organizational Readiness

The model works best when:

- The organization has mature product management with clear user objective definitions.
- Engineering teams can maintain their own services (DevOps maturity).
- Platform/infrastructure teams provide baseline reliability.
- SRE has executive support to prioritize product outcomes over service metrics.
- Incident severity guidelines already exist and are product-aligned.

---

## 13. Synthesis: What This Means for SRE Practice

This final section synthesizes the article's implications for the SRE profession.

### On-Call Changes

| Aspect | Service-Centric | Product-Centric |
|---|---|---|
| **What triggers a page** | Service error budget burn | Product SLO burn (per user objective) |
| **First question** | "Which service is down?" | "Which user objective is affected?" |
| **Runbook structure** | Per-service recovery | Per-objective recovery (may span services) |
| **Escalation** | Escalate to service owner | Escalate to product team (PM + eng + UX) |
| **Page reduction** | Harder to reduce (all service issues trigger) | Easier to reduce (infra failures that don't hit product SLOs are filtered) |

### Alerting Philosophy

Product-focused alerting means:

- Alerts carry user context: "Critical objective 'send mail' is experiencing 5% errors."
- Alerts for infrastructure degradation that doesn't affect product SLOs are routed to the owning development team, not SRE.
- Alert fatigue decreases because the team only pages on product-relevant failures.
- False negatives (the user sees a failure but no alert fires) decrease because client-side and E2E SLOs catch what service SLOs miss.

### Team Structure

| Model | Structure | Career Path |
|---|---|---|
| **Service SRE** | Teams organized by service (DB SRE, Networking SRE, Storage SRE) | Deep specialist → Senior infrastructure SRE |
| **Product SRE** | Teams organized by product (Gmail SRE, Drive SRE, Calendar SRE) | Broad generalist → Product reliability lead → SRE Manager |

The product SRE model creates a different career trajectory. Product SREs need broader knowledge (frontend, backend, mobile, APIs) rather than deep infrastructure specialization. They must be comfortable with product language, stakeholder management, and cross-team coordination.

### Organizational Model: Product SRE vs Service SRE

It is not necessarily an either/or choice. The article implies a hybrid model may work best:

1. **Platform SRE teams** handle baseline infrastructure (compute, networking, storage). They use traditional service SRE models.
2. **Product SRE teams** focus on user objectives. They partner with PMs and own product-level SLOs.
3. **Service SRE teams** (if needed) support critical internal services that don't map directly to user objectives.

> *"The product support model lets the SRE team focus on the user to ensure that the product meets the end user's real-world needs."*

### Key Insights for Practitioners

1. **Start small.** Pick one user objective, one step, one SLO. Prove the model works before expanding.

2. **Language is strategy.** If you're still talking about "services" and "RPC latency," you haven't made the shift. Product-level language ("compose mail," "send mail") drives alignment.

3. **The stakeholder set is wider.** PMs and UX are now essential partners. SREs must invest in cross-functional relationships.

4. **Not everything needs a product SLO.** Baseline support handles commodity infrastructure. Criticality guides where to invest.

5. **Offboarding is as important as onboarding.** SLOs have ongoing costs. Regularly prune obsolete ones.

6. **Annotation is the technical linchpin.** Without the ability to tag requests with user objective context, product SLOs remain theoretical.

7. **This model is a forcing function for platform maturity.** If your infrastructure requires constant SRE attention, you cannot shift to product focus. Invest in platform reliability first.

---

## Appendix: How This Changes Common SRE Practices

| Practice | Traditional SRE | Product-Focused SRE |
|---|---|---|
| **Error budgets** | Per service | Per user objective, aggregated across services |
| **Capacity planning** | Service-level utilization | Per-objective traffic patterns |
| **Load shedding** | By service priority | By user objective criticality |
| **Release engineering** | Rolling out new service versions | Canarying new features by user objective |
| **Incident management** | Service-centric IR (e.g., "DB incident") | Product-centric IR (e.g., "send mail incident") |
| **Postmortems** | Focus on service failure modes | Focus on user impact and journey gaps |
| **SLO reviews** | Quarterly service SLO review | Quarterly per-objective SLO review with PM |
| **Onboarding** | Service onboarding checklist | Objective/step onboarding with product team |
| **Team metrics** | Service availability, latency p99 | Objective reliability, step success rate |

---

## References

The article cites the following sources:

1. Ulwick, A.W. and Osterwalder, A. (2016). *Jobs to be Done: Theory to Practice.* Idea Bite Press.
2. Chang, A. (2017). "What To Do If Your Product Isn't Growing." Initialized Capital.
3. Beyer, B., Jones, C., Petoff, J. and Murphy, N. (2016). *Site Reliability Engineering: How Google Runs Production Systems.* O'Reilly Media.
4. Beyer, B., Murphy, N., Rensin, D.K., Kawahara, K. and Thorne, S. (2018). *The Site Reliability Workbook.* O'Reilly Media.
5. Kalbach, J. (2020). *The Jobs to be Done Playbook.* Two Waves Books.
6. Google Cloud. "Incidents and the Google Cloud Service Health Dashboard." https://cloud.google.com/support/docs/dashboard
7. Wikipedia. "Responsibility Assignment Matrix." https://en.wikipedia.org/wiki/Responsibility_assignment_matrix

---

*This document is a synthesized analysis of the Google SRE article "Product-Focused Reliability for SRE" by Carl Crous, Parker Roth, and Victoria Hurd. It is not a verbatim reproduction. The analysis adds practitioner interpretation, synthesized tables, and organizational guidance beyond what appears in the original article, while staying faithful to its core concepts.*

# Release Engineering Reference

> **Domain:** Site Reliability Engineering
> **Purpose:** Comprehensive reference for designing, operating, and improving release pipelines in SRE-managed environments.
> **Audience:** SREs, Platform Engineers, DevOps practitioners, and Release Managers.

---

## Table of Contents

1. [The Self-Service Release Model](#1-the-self-service-release-model)
2. [Hermetic Builds](#2-hermetic-builds)
3. [Push-on-Green Deployment Model](#3-push-on-green-deployment-model)
4. [Release Branch Management and Cherry-Picking](#4-release-branch-management-and-cherry-picking)
5. [Build and Deployment Pipeline Design (CI/CD)](#5-build-and-deployment-pipeline-design-cicd)
6. [Configuration Management Strategies](#6-configuration-management-strategies)
7. [Progressive Delivery](#7-progressive-delivery)
8. [Rollback Strategies](#8-rollback-strategies)
9. [Audit Trails and Change Reporting](#9-audit-trails-and-change-reporting)
10. [Security and Access Control for Releases](#10-security-and-access-control-for-releases)
11. [Release Velocity Metrics](#11-release-velocity-metrics)
12. [SRE and Release Engineering: The Relationship](#12-sre-and-release-engineering-the-relationship)

---

## 1. The Self-Service Release Model

Traditional release engineering placed a central "release team" as a gatekeeper: every deploy required a ticket, a manual approval meeting, and a designated change window. This model does not scale beyond a handful of services.

### Core Principles

- **Developer autonomy.** Teams own their release pipeline end-to-end — they trigger builds, run tests, promote artifacts, and deploy without filing tickets.
- **Platform over process.** The platform encodes safety (gates, rollbacks, approvals) so humans do not need to remember a checklist.
- **Low-friction, high-safety.** Removing manual bottlenecks increases velocity; automated safety nets (canary analysis, metrics verification) reduce risk.

### Implementation Requirements

| Component | Purpose |
|---|---|
| CLI / API surface | Engineers trigger releases via `shipctl release --service payments --version v2.3.1` or a CI chat-ops command |
| Role-based delegation | Any engineer can promote to staging; only on-call or release managers can promote to production |
| Approval workflows | Optional manual approval step (via Slack button, Jira, or web UI) for compliance-sensitive services |
| Self-service dashboard | Real-time view of pipeline status, artifact provenance, and release history per service |
| Documentation | Every team's pipeline must be discoverable and repeatable without tribal knowledge |

> **SRE Rule:** If an engineer needs to ask "how do I release this service?", the self-service model has failed. The answer should be a documented command or CI trigger.

---

## 2. Hermetic Builds

A hermetic build produces a byte-for-byte reproducible artifact whose output depends only on the declared inputs — not on the state of the build machine, the time of day, or network-available packages.

### Why Hermeticity Matters

- **Determinism.** The same commit always produces the same artifact, eliminating "works on my machine" and "build broke because the repo was flaky."
- **Supply-chain integrity.** No surprise dependency pulled from an external registry at build time.
- **Cacheability.** Build outputs can be content-addressed and cached globally.
- **Auditability.** You know exactly what went into an artifact because the build environment is pinned.

### How to Achieve Hermetic Builds

1. **Pin all toolchains.** Use Docker images with pinned OS packages, compiler versions, and language runtimes. Store images in a registry you control.
2. **Vendor or mirror dependencies.** Commit lockfiles (`package-lock.json`, `Cargo.lock`, `go.sum`, `requirements.txt` with hashes). Mirror npm/PyPI/Maven repos behind an internal proxy (e.g., Artifactory, Nexus).
3. **Disable network access during build.** Use Docker's `--network none` or Bazel's sandbox. If a build needs a dependency, it must be declared and pre-fetched.
4. **Use content-addressable storage.** Bazel, Nix, and similar tools compute a hash of every input and store outputs by hash, making rebuilds of unchanged inputs instant.
5. **Pin base images.** Use digest references (`alpine@sha256:abc123...`) instead of tags (`alpine:3.19`), which can change under you.

### Trade-Offs

| Benefit | Cost |
|---|---|
| Reproducible, tamper-evident builds | Longer initial setup; dependency mirroring overhead |
| Global cache hits across teams | Requires investment in build infrastructure (Bazel, Nix, remote executors) |
| Strong supply-chain guarantees | Some language ecosystems (Python, Node) resist full hermeticity without extensive tooling |

---

## 3. Push-on-Green Deployment Model

"Push on green" means that if the automated pipeline gates all pass (unit tests, integration tests, security scans, performance benchmarks), the artifact is automatically promoted to the next environment — including production — with no human intervention required.

### Pipeline Stages

```
Commit -> Build -> Unit Tests -> Container Image -> Integration Tests
    -> Staging Deploy -> Smoke Tests -> Security Scan -> Canary Deploy
    -> Production Deploy -> Metrics Verification -> Done
```

Each stage gates the next. If any stage fails, promotion halts and the team is alerted.

### When Push-on-Green Works

- **Mature test suites** with high coverage and low flakiness.
- **Strong monitoring** that can detect regressions within minutes of deploy.
- **Fast rollbacks** (ideally automated via the pipeline or infrastructure-as-code).
- **Small batch sizes** — frequent, small releases reduce the blast radius of a bad deploy.

### When It Doesn't

- **Regulatory environments** requiring explicit sign-off per release.
- **Immature services** where tests are sparse or flaky.
- **Database migrations** that are incompatible with quick rollbacks (require manual reconciliation).

> **SRE Best Practice:** Start with manual approval gates between stages (staging and production) and remove them one by one as metrics and confidence improve. Push-on-green is a goal, not a starting point.

---

## 4. Release Branch Management and Cherry-Picking

### Branch Strategy Overview

| Model | Description | Use Case |
|---|---|---|
| Trunk-based development | All work lands on `main`. Releases are tags on `main`. | High-velocity teams, push-on-green, microservices |
| Release branches | Stable branches (`release/v2.3.x`) cut from `main` at a release point. Bug fixes cherry-picked from `main`. | Mobile apps, customer-managed software, regulated industries |
| GitFlow | Long-lived `develop` and `release` branches. | Large monoliths with scheduled releases; falling out of favor |

### Release Branch Lifecycle

1. **Branch cut.** At the release candidate point, create `release/v<major>.<minor>.x` from `main`.
2. **Stabilization.** Cherry-pick only critical bug fixes from `main` into the release branch.
3. **Release.** Tag the commit (e.g., `v2.3.0`) and build the artifact from the release branch.
4. **Patch releases.** Cherry-pick hotfixes into the same branch; tag as `v2.3.1`, `v2.3.2`, etc.
5. **End of life.** Archive the branch once all customers have migrated.

### Cherry-Picking Discipline

Cherry-picking is a necessary evil — every cherry-pick represents code that bypassed the integration testing done on `main`.

**Rules:**

- Require a tracking issue (bug ID or Jira ticket) for every cherry-pick.
- Limit cherry-picks to P0/P1 defects, security vulnerabilities, and customer-blocking issues.
- Enforce code review on the cherry-pick commit — same standards as any other change.
- Run the full CI suite on the release branch after each cherry-pick — do not assume the fix is isolated.
- Keep a changelog of cherry-picked commits (SHA + description) attached to the release notes.

**Automation pattern:** A bot monitors labels on GitHub issues; when a PR with label `cherrypick/release-v2.3` merges to `main`, the bot automatically opens a backport PR against the release branch.

---

## 5. Build and Deployment Pipeline Design (CI/CD)

### Pipeline Architecture

A well-designed CI/CD pipeline is composed of loosely coupled stages, each with a clear success/failure signal.

```
                         +-----------+
                         |  Trigger  |
                         +-----+-----+
                               |
                     +---------v---------+
                     |  Source Fetch &    |
                     |  Commit Metadata   |
                     +---------+---------+
                               |
                     +---------v---------+
                     |  Hermetic Build    |
                     |  & Artifact        |
                     |  Publishing        |
                     +---------+---------+
                               |
                     +---------v---------+
                     |  Unit & Static     |
                     |  Analysis Tests    |
                     +---------+---------+
                               |
                     +---------v---------+
                     |  Integration &     |
                     |  E2E Tests         |
                     +---------+---------+
                               |
                     +---------v---------+
                     |  Security &        |
                     |  Compliance Scan   |
                     +---------+---------+
                               |
          +--------------------+--------------------+
          |                                         |
  +-------v-------+                        +--------v-------+
  |  Staging       |                        |  Canary / Prod |
  |  Deploy +      |                        |  Deploy +      |
  |  Smoke Tests   |                        |  Verification  |
  +-------+-------+                        +--------+-------+
          |                                         |
          v                                         v
     Promoted if                             Promoted if
     green                                  green
```

### Key Design Decisions

| Decision | Recommendation |
|---|---|
| Monorepo vs. multi-repo | Monorepo with per-service CI scoping (affected-target detection) |
| CI runners | Ephemeral, containerized, no state between runs |
| Artifact storage | Content-addressable blob store (S3/GCS with immutable buckets) |
| Deployment tool | ArgoCD, Spinnaker, or in-house orchestrator with GitOps |
| Pipeline-as-code | YAML/Starlark pipeline definitions versioned in the source repo |

### Pipeline Observability

Every pipeline should emit structured events (CloudEvents format) covering:

- **Start timestamp** and **end timestamp** per stage.
- **Commit SHA**, **branch**, **author**, **PR number**.
- **Artifact digest** (SHA256 of the built artifact).
- **Stage result** (passed / failed / skipped / aborted).
- **Duration** and **resource utilization** (CPU, memory, network I/O).

This telemetry feeds dashboards, SLA/SLO tracking, and postmortem analysis.

---

## 6. Configuration Management Strategies

Configuration is the most common source of production incidents. How you manage it alongside your code artifacts is a critical design decision.

### Strategy Comparison

| Strategy | Description | Advantages | Disadvantages | Best For |
|---|---|---|---|---|
| **Mainline** | Config lives in the same repo as application code, deployed together as a single artifact | Atomic deploys, simple mental model, easy version correlation | Config change requires full rebuild; config cannot be toggled independently of code | Small services, early-stage products |
| **Bundled packages** | App artifact includes default config; runtime overrides stored separately | Separation of concerns; default config always matches the code version | Complexity of merging defaults + overrides at startup | Services with small per-environment differences |
| **Config-only packages** | Config is packaged as its own versioned artifact, deployed independently of the application binary | Config can be promoted/rolled back without redeploying code | Drift between config and code versions can cause incompatibilities | Large services where config changes are frequent (feature flags, routing rules) |
| **External stores** | Config lives in a centralized service (Consul, etcd, ZooKeeper, cloud parameter store), served at runtime | Real-time changes without any deploy; fine-grained access control; audit logging | Adds latency on startup/refresh; external dependency (must be highly available); can cause cascading failures if the store is misconfigured | Multi-service systems, dynamic routing, global feature flags |

### Recommended Approach

Use a **layered** strategy:

1. **Bundled defaults** inside the artifact (for safe-start without external dependencies).
2. **Environment overrides** in a config-only package or external store (per env: dev, staging, prod).
3. **Runtime overrides** in a feature-flag system (LaunchDarkly, Flagsmith, in-house) for progressive delivery.

---

## 7. Progressive Delivery

Progressive delivery exposes changes to a subset of users or traffic before rolling out to the full population, reducing the blast radius of defects.

### Techniques

| Technique | Mechanism | Detection Time | Risk Profile |
|---|---|---|---|
| **Canary deployments** | Route a small percentage of live traffic (e.g., 2%) to the new version | Seconds to minutes | Low — immediate detection of errors, latency spikes |
| **Traffic shifting** | Controlled percentage-based routing via service mesh (Istio, Linkerd) or load balancer | Minutes | Low — canaries and gradual shift |
| **Feature flags** | Toggle code paths at runtime without redeploying; per-user or per-group targeting | Real-time | Very low — instant kill switch without any deploy |
| **Blue-green deployments** | Maintain two full environments; switch traffic atomically from old (blue) to new (green) | Seconds (switch time) | Medium — full environment cost; rollback is just another switch |
| **A/B testing** | Execute A/B/n traffic split with instrumentation for business-metric comparison | Hours to days | Low — typically used for non-critical features alongside canary |
| **Shadow / mirroring** | Duplicate live traffic to the new version without serving it to users | Minutes to hours | Lowest — no user impact; but no real user feedback either |

### Deployment Safety Patterns

| Safety Pattern | Description | Implementation |
|---|---|---|
| **Metrics gate** | Pipeline pauses after canary deploy and verifies key SLOs (error rate < 0.1%, p99 latency < 500ms, CPU < 80%). | Prometheus + alerting rules evaluated by the pipeline |
| **Auto-abort** | If metrics degrade beyond a threshold, the pipeline aborts and optionally triggers a rollback. | Pipeline controller watches a metrics endpoint; CD tool (Argo Rollouts, Flagger) handles it natively |
| **Bake period** | Minimum observation window (e.g., 10 minutes) before progressing to the next rollout percentage. | Configurable `minBakeSeconds` in the pipeline YAML |
| **Manual pause** | Pipeline pauses at configurable checkpoints for human review. | Approval step in CI/CD tool + Slack/email notification |
| **Kill switch** | A single command or toggle that halts the rollout and returns to the previous stable version. | Feature flag system or CD tool rollback command |
| **Saturation gate** | Check that downstream dependencies (databases, queues, caches) are not saturated before allowing more traffic. | Customized metrics check or HPA feedback |
| **Drift detection** | Verify that the deployed configuration matches the desired state in version control. | GitOps tool (ArgoCD) reconciliation |

---

## 8. Rollback Strategies

Even with perfect progressive delivery, rollbacks are inevitable. A good rollback plan is fast, safe, and auditable.

### Rollback Types

| Type | Mechanism | Speed | Caveats |
|---|---|---|---|
| **Reversion** | Deploy the previous known-good artifact version | Fast (minutes) | Requires artifact immutability; previous version must still be available |
| **Forward fix** | Deploy a new version that fixes the regression (leaves the bad change in place, adds a correction on top) | Moderate (time to fix + pipeline) | Best long-term; does not revert unrelated changes |
| **Git revert + deploy** | `git revert <bad-commit>` and run the full pipeline | Same as forward fix | Can conflict if intervening commits touch the same code |
| **Blue-green flip** | Switch load balancer back to the old (still-running) environment | Seconds | Only works if blue-green is already deployed; doubles infrastructure cost |
| **Feature flag off** | Toggle the offending feature off at runtime | Seconds (or real-time with polling) | Feature must be behind a flag; the code stays deployed but dormant |

### Rollback Best Practices

1. **Test the rollback.** Every release should include a documented, tested rollback procedure. Schedule a "game day" to verify it works.
2. **Handle stateful rollbacks.** Stateless rollbacks (flip load balancer) are easy. Stateful rollbacks (database schema changes, queue migrations) require careful planning — schema changes should be backward-compatible for at least one deploy cycle.
3. **Roll forward as the default.** Reversion should be the exception, not the rule. Forward fixes preserve progress and avoid losing unrelated changes.
4. **Automate the decision.** If the pipeline detects a metrics regression during canary, it should automatically halt and optionally trigger a rollback — do not wait for a human to notice pager alerts.
5. **Audit every rollback.** Log the trigger (why), the action (what), the affected surface (which users/regions), and the outcome (was it successful?). Feed into incident reviews.

---

## 9. Audit Trails and Change Reporting

SRE requires a complete, immutable, and queryable history of every release and configuration change.

### What to Log

| Event | Fields |
|---|---|
| Build created | artifact digest, commit SHA, build trigger (manual / CI), builder identity, timestamp |
| Promotion decision | artifact version, source env, target env, pass/fail status per gate, verifier identity |
| Deployment | artifact version, target environment, rollout strategy (canary % / blue-green), operator identity |
| Rollback | artifact being reverted to, reason (metric regression / manual / auto-abort), operator identity |
| Config change | config version, diff from previous, deployer identity, approval if required |
| Approval gate | approver identity, timestamp, stage being approved, comments or evidence |

### Storage and Query

- **Immutable log store.** Append-only storage (cloud audit logs, immutable S3/GCS buckets, blockchain-based notary).
- **Searchable.** Index by service name, environment, artifact version, and operator identity.
- **Linkable to incidents.** Every production incident should be cross-referenced with the release that introduced it (via the deploy log SHA).
- **Retention.** Minimum 1 year for operational logs; 3–7 years for compliance-regulated environments.

### Reporting Cadence

| Report | Frequency | Audience |
|---|---|---|
| Release dashboard | Real-time | Engineering teams |
| Weekly release summary | Weekly | Engineering management |
| Deploy failure analysis | Per-incident | SRE + affected team |
| Compliance change report | Per release cycle or monthly | Audit / Compliance |
| Velocity and stability trends | Monthly | Platform / Eng leadership |

---

## 10. Security and Access Control for Releases

### Principles

- **Least privilege.** An engineer who can initiate a release should not necessarily be able to approve it or bypass gates.
- **Separation of duties.** Build, promote, and deploy stages should have distinct authorization.
- **Non-repudiation.** Every action is attributable to a specific identity (human or service account).
- **Artifact signing.** Every build artifact is cryptographically signed. The deployment pipeline verifies the signature before deploying.

### Access Control Model

| Role | Can Trigger Build | Can Promote to Staging | Can Promote to Prod | Can Approve Hotfix | Can Modify Pipeline Config |
|---|---|---|---|---|---|
| Developer | Yes | Yes (via CI) | No | No | No |
| Senior Engineer | Yes | Yes | Yes (with auto-gates) | Yes | No |
| SRE On-Call | Yes | Yes | Yes | Yes | Yes (with review) |
| Release Manager | Yes | Yes | Yes (override gates) | Yes | Yes |
| CI/CD Service Acct | Yes (on push) | Yes (on green) | Yes (on green) | No | No |

### Secret Management

- Never bake secrets into artifacts. Use a secrets manager (Vault, AWS Secrets Manager, GCP Secret Manager) injected at deploy time.
- Pipeline credentials should be short-lived (15-minute expiry) and scoped to the minimum required action.
- Audit all secret access — who read which secret, when, and from what context.

### Supply-Chain Security

- **SLSA framework.** Target SLSA Level 3+ for critical production services: signed builds, hardened build platform, provenance attestation.
- **Software Bill of Materials (SBOM).** Generate an SBOM per build (CycloneDX or SPDX format). Store alongside the artifact.
- **Vulnerability scanning.** Scan all base images and dependencies at build time. Fail the pipeline on CVSS >= 7.0 findings.
- **Image signing.** Use cosign (Sigstore) or similar to sign container images. Enforce signature verification in the deploy admission webhook.

---

## 11. Release Velocity Metrics

You cannot improve what you do not measure. These are the canonical metrics for release engineering.

### Core Metrics

| Metric | Definition | Target | Formula |
|---|---|---|---|
| **Deploy Frequency** | How often a service is deployed to production | High-velocity: multiple times per day; Medium: weekly; Low: monthly | # of production deploys / time period |
| **Lead Time for Changes** | Time from commit to production deploy | Elite: < 1 hour; High: < 1 day; Medium: < 1 week | median(time(deploy) - time(commit)) |
| **Change Failure Rate (CFR)** | Percentage of deploys that cause a degradation or incident | Elite: < 5%; High: < 10%; Medium: < 15% | # of failed deploys / total deploys |
| **Mean Time to Recovery (MTTR)** | Time from incident detection to full remediation (including rollback) | Elite: < 1 hour; High: < 1 day; Medium: < 1 week | median(time(resolved) - time(alerted)) |
| **Pipeline Success Rate** | Percentage of pipeline runs that complete all stages successfully | Target: > 95% | successful runs / total runs |
| **Artifact Age** | Time since the artifact in production was built | Target: < 24 hours | now - build_timestamp |
| **Gate Pass Rate** | Percentage of promotion attempts that pass automated gates | Baseline: > 90% | passed promotions / total promotions |

### Leading Indicators

These metrics predict future reliability problems:

- **Flaky test rate** — if > 5% of test reruns succeed, confidence in the gate is eroded.
- **Gate bypass requests** — if teams regularly bypass gates (e.g., hotfix override), the pipeline is too restrictive or the tests are too slow.
- **Cherry-pick density** — the ratio of cherry-picked commits to normal commits on a release branch. Rising density indicates poor mainline discipline.
- **Pipeline queue time** — if builds wait for CI runners for more than a few minutes, velocity will plateau.

---

## 12. SRE and Release Engineering: The Relationship

Release engineering and SRE share overlapping but distinct responsibilities.

### Common Misconceptions

- **"Release engineering is just CI/CD."** It is CI/CD plus artifact management, configuration strategy, progressive delivery, security, compliance, and the organizational model that enables self-service.
- **"SREs are release engineers."** SREs define the reliability requirements (SLOs, error budgets) that the release pipeline must enforce. They do not necessarily own the pipeline infrastructure itself, though in practice many SRE teams operate the deployment platform.

### Responsibility Matrix

| Activity | Release Engineering | SRE | Development Team |
|---|---|---|---|
| Pipeline design and maintenance | Lead | Consult | Feed requirements |
| Artifact signing and provenance | Lead | Review | N/A |
| Deployment tooling (CD platform) | Lead | Co-own reliability | Use |
| Rollback automation | Lead | Define recovery SLOs | Test |
| Canary analysis and metrics gates | Consult | Lead (define thresholds) | Review |
| Incident response for bad deploys | Support | Lead | Support |
| Release velocity metrics | Lead | Consume for error budget | Use |
| Security scanning and SBOM | Lead | Review compliance | Fix findings |
| Configuration management strategy | Lead | Review (reliability impact) | Implement |
| Release approval / change management | Facilitate | Gate for SLO risk | Request |

### How They Work Together

1. **Error budgets inform release velocity.** If a team has error budget remaining, they can release freely (push on green). If they have exhausted their error budget, the pipeline can enforce a cool-down period (gating further releases until reliability improves).
2. **Release pipelines enforce SLOs.** The metrics gate in the pipeline (Section 7) validates that the new version meets the service's SLOs before more traffic is shifted.
3. **SREs set the "pain threshold."** Release engineering implements the technical mechanism; SRE defines what "bad enough to abort" means (e.g., 0.1% error rate increase, p99 latency over 800ms).
4. **Postmortems drive pipeline improvements.** Every deploy-related incident feeds back into the release pipeline: add a new gate, harden an existing check, improve the rollback script.

### Organizational Models

| Model | Description | Best For |
|---|---|---|
| **Embedded** | Release engineers sit within the SRE org; dedicated platform team | Large organizations (> 200 engineers), multiple service teams |
| **SRE-as-platform** | SRE builds and maintains the release platform; teams self-serve | Mid-sized orgs (50–200 engineers) |
| **Federated** | Each team owns its release pipeline; SRE provides standards and consulting | Small orgs (< 50 engineers), strong DevOps culture |

---

## References and Further Reading

- *Accelerate: The Science of Lean Software and DevOps* by Nicole Forsgren, Jez Humble, Gene Kim
- *Site Reliability Engineering* (Beyer et al., O'Reilly) — Chapters on release engineering
- *The DevOps Handbook* by Gene Kim, Jez Humble, Patrick Debois, John Willis
- *Continuous Delivery* by Jez Humble and David Farley
- SLSA (Supply-chain Levels for Software Artifacts) — [slsa.dev](https://slsa.dev)
- Google CI/CD best-practices documentation
- Argo Rollouts — Progressive delivery for Kubernetes
- Flagger — Canary deployments with service mesh integration
- Sigstore / cosign — Container image signing
- OpenFeature — Standardized feature flagging

---

> **Revision 1.0** — Created for the Site Reliability Engineering skill profile. Maintained by the SRE Knowledge Base working group.

# Reliability Design, Change Safety, and Overload

Use this reference for design reviews, capacity and overload planning, production configuration, canary releases, data pipelines, and services whose reliability depends on more than availability alone.

## Source anchors

This reference synthesizes *The Site Reliability Workbook*, “Managing Load,” “Introducing Non-Abstract Large System Design,” “Data Processing Pipelines,” “Configuration Design and Best Practices,” “Configuration Specifics,” “Canarying Releases,” and “Identifying and Recovering from Overload,” plus *Seeking SRE*, “In the Beginning, There Was Chaos,” “Database Reliability Engineering,” “Engineering for Data Durability,” “Immutable Infrastructure and SRE,” “Scriptable Load Balancers,” and “The Service Mesh: Wrangler of Your Microservices?” The source books contain organization-specific examples; the procedures below generalize the engineering decisions without copying those examples.

## Design for the whole system

A reliability design review must include the user journey, not just the service under review. For each proposed system:

1. State the user-visible objectives and the SLOs they imply.
2. Draw the request and data paths, including asynchronous work and third parties.
3. Identify capacity limits, queues, retries, timeouts, rate limits, and shared resources.
4. Describe normal, degraded, overloaded, and recovery states.
5. Define what can be shed, delayed, cached, degraded, or served read-only.
6. Identify data-loss, corruption, privacy, and security consequences separately from downtime.
7. Specify observability, operator actions, rollback, restore, and verification.
8. Estimate operational work and cognitive load before approving the design.

A design that meets an availability target by silently corrupting data or violating privacy is not reliable.

## Non-Abstract Large System Design (NALSD-style review)

Use concrete boundaries and numbers rather than “high scale” or “resilient.” Record:

- expected request, event, and data rates, including peak and burst shape;
- storage growth, retention, replication, and recovery-point objectives;
- latency budgets across each dependency hop;
- failure domains and blast radius;
- consistency, ordering, idempotency, and replay behavior;
- quotas, backpressure, retry limits, and queue bounds;
- deployment and migration strategy;
- cost and operational ownership.

Iterate the design against the SLO and error budget. If a proposed feature consumes more reliability, capacity, or operator attention than the budget allows, record the trade-off and decision owner instead of hiding it in implementation detail.

## Load and overload management

Capacity planning is not only “add more machines.” Build a demand model from historical traffic, expected growth, scheduled events, and worst credible bursts. Test the model before the event and identify the manual fallback if automation fails.

When overload begins:

1. Confirm whether demand, capacity, dependency latency, or a control-plane failure is the limiting factor.
2. Stop amplification: bound retries, disable nonessential fan-out, and prevent queue growth from becoming unbounded.
3. Protect the critical user journey with admission control, prioritization, rate limits, caching, or graceful degradation.
4. Shed or defer work deliberately. Prefer a known reduced mode over random timeouts.
5. Watch saturation, queue depth, latency, errors, and dependency health for recovery evidence.
6. Restore normal traffic gradually and verify that backlog, data integrity, and downstream systems recover.
7. Record the capacity assumption that failed and create an owned repair.

A service that responds slowly under overload can consume more shared capacity through timeouts and retries, creating a cascade. Fast rejection with a clear degraded path can be more reliable than accepting work that cannot complete.

## Operational overload of the team

A team is operationally overloaded when urgent work continually preempts the engineering needed to reduce future load. Track operational work as a proportion of available engineering time, including pages, tickets, manual changes, support interruptions, and incident follow-up.

Recovery requires an explicit cutover:

- declare the team overloaded using a stated threshold or sustained trend;
- protect a fixed block of engineering time;
- reduce or renegotiate service scope and nonessential commitments;
- suppress, route, or retire non-actionable alerts;
- prioritize the smallest changes that remove recurring interruption;
- assign leadership support for deferred product work and staffing gaps;
- review the load trend weekly until the team returns below the threshold.

Do not respond to operational overload by asking the team to work longer hours. That hides the capacity failure and increases incident risk.

## Configuration safety

Treat configuration as production code with an explicit lifecycle:

- one authoritative source and a discoverable ownership path;
- schema, type, range, dependency, and compatibility validation;
- safe defaults and explicit units;
- version control, review, audit trail, and rollback;
- staged rollout or canary for high-impact changes;
- dry-run or preview where possible;
- clear distinction between static configuration and runtime state;
- emergency path that is fast but still logged and reconciled into source control.

Configuration should be easy to inspect during an incident. Avoid hidden inheritance, ambiguous names, duplicated values, unbounded lists, and emergency-only interfaces. A configuration change needs a stated expected effect and a way to observe whether that effect occurred.

## Canarying changes

A canary is a partial, time-limited deployment evaluated against a control. It is not merely “deploy to one host.” Define before rollout:

- canary population and selection method;
- control population and whether traffic is comparable;
- observation window and minimum sample size;
- success metrics tied to SLOs, user journeys, saturation, and dependency health;
- abort thresholds and who or what can stop the rollout;
- rollback or roll-forward action;
- criteria for expanding, pausing, or declaring success.

Compare canary and control, and account for traffic mix, time-of-day, cold starts, and unrelated changes. A canary with no control, no minimum sample, or no abort authority creates the appearance of safety without the decision evidence.

## Data processing and durability

For pipelines and data stores, define reliability beyond service uptime:

- freshness and completeness of output;
- correctness and reconciliation checks;
- ordering, duplication, replay, and late-arriving data behavior;
- checkpointing, retention, backfill, and recovery-point objectives;
- schema evolution and compatibility;
- access control and privacy boundaries;
- restore tests and corruption detection.

Availability and durability are different SLO dimensions. A pipeline can be “up” while producing stale, incomplete, duplicated, or wrong data. A database can answer requests while losing writes. Measure the property users and downstream decisions depend on.

## Third-party dependencies

Apply SRE discipline to vendors and managed services:

1. Classify the dependency by user impact, substitutability, and failure mode.
2. Record the dependency's SLO or service limits, support path, status signal, and contractual boundaries.
3. Measure the dependency from your service's perspective, not only from the vendor dashboard.
4. Design timeout, retry, fallback, queue, cache, and degraded-mode behavior.
5. Test provider failure and credential, quota, region, and API-version failure modes.
6. Track unresolved vendor repairs and review whether the dependency remains acceptable.

“External” is not the same as “unowned.” If a dependency failure is excluded from your SLO, document what users experience and what mitigation you actually provide.

## Complexity and service mesh caution

Prefer the simplest architecture that meets the objectives. Every proxy, control plane, configuration layer, retry policy, and telemetry path adds failure modes and cognitive load. Adopt a service mesh or similar platform only when the operational capability it provides outweighs its new blast radius and the team can observe and operate it.

## Agent procedure

Use this reference with `slo-sli-framework.md`, `monitoring-alerting.md`, `release-engineering.md`, and `troubleshooting.md`. Use the `templates/reliability-design-review.md` template. Reject vague claims such as “handles scale” or “has rollback” until the owner, evidence, thresholds, and verification path are explicit.

# Senior / Staff / Principal SRE — Role Blueprint

> **Audience:** SRE profile agent, engineering leadership, aspirants  
> **Last updated:** 2025-06-05  
> **Version:** 1.0

---

## Table of Contents

1. [Role Mission Statement](#1-role-mission-statement)
2. [Core Responsibilities by Category](#2-core-responsibilities-by-category)
3. [Key Performance Indicators](#3-key-performance-indicators)
4. [Required Technical Skills](#4-required-technical-skills)
5. [Soft Skills & Leadership Competencies](#5-soft-skills--leadership-competencies)
6. [Tools & Tech Stack](#6-tools--tech-stack)
7. [Career Progression](#7-career-progression)
8. [Senior vs. Mid-Level: The Distinction](#8-senior-vs-mid-level-the-distinction)

---

## 1. Role Mission Statement

> **"Maximize service reliability while enabling engineering velocity. Design, build, and operate distributed systems at scale — then teach others to do the same."**

The Senior/Staff/Principal SRE is the organizational authority on production excellence. This is not an operations escalation role. It is an **engineering leadership** role that blends deep systems knowledge, software engineering skill, and cross-functional influence to ensure the services customers depend on remain available, performant, and evolvable.

At each tier the scope expands:

| Level | Scope | Primary Lever |
|-------|-------|---------------|
| **Senior SRE** | A service family or platform area | Technical execution, incident command, runbook authorship |
| **Staff SRE** | Multiple teams across an organization | Reliability standards, architectural decisions, org-wide initiatives |
| **Principal SRE** | The entire engineering organization | Strategy, multi-year roadmap, industry influence |
| **SRE Manager** (parallel track) | A team of SREs | People development, hiring, org design, stakeholder management |

The common thread: **every SRE at this level ships code**, writes design docs, and carries a pager. The higher you go, the more of your leverage comes from *enabling others* rather than *doing it yourself*.

---

## 2. Core Responsibilities by Category

### 2.1 Strategic

| Responsibility | Description | Senior | Staff | Principal |
|---------------|-------------|--------|-------|-----------|
| **SLO / SLI Definition** | Work with product and engineering to define meaningful service-level objectives and indicators | Leads for own service | Defines framework org-wide | Sets org-wide methodology |
| **Error Budget Policy** | Design and enforce error budget policies that balance reliability vs. feature velocity | Applies policy | Designs policy | Audits and evolves policy |
| **Capacity Planning** | Forecast infrastructure needs 6–18 months out based on growth trends and product roadmap | Quarterly per service | Annual org-level | Multi-year / cross-org |
| **Reliability Roadmap** | Identify and prioritize reliability investments — observability gaps, architectural debt, toil | Quarter-ahead | Year-ahead | Multi-year strategy |
| **Cost Optimization** | Drive infrastructure efficiency without compromising SLOs | Per-team savings | Org-wide cost + performance | Unit-economics design |

### 2.2 Operational

| Responsibility | Description | Senior | Staff | Principal |
|---------------|-------------|--------|-------|-----------|
| **Incident Response** | Lead high-severity incidents, perform postmortems, track action items | Incident commander | Coach incident commanders | Design incident-response program |
| **On-Call Rotation** | Participate in rotations, improve runbooks, reduce alert fatigue | Active participant | Improve rotation design | Eliminate entire classes of pages |
| **Change Management** | Review and approve production changes, enforce safe deployment practices | Reviews changes | Defines change policy | Audits change outcomes |
| **Toil Reduction** | Identify and automate repetitive operational work | 50% toil reduction/quarter | Defines toil metrics org-wide | Eliminates structural toil sources |
| **Disaster Recovery** | Design, document, and regularly test DR plans and failover procedures | Owns per-service DR | Cross-service DR exercises | Multi-region / multi-cloud DR strategy |

### 2.3 Technical

| Responsibility | Description | Senior | Staff | Principal |
|---------------|-------------|--------|-------|-----------|
| **Architecture Review** | Review system designs for reliability, scalability, operability | Per-service | Cross-team | Org-wide standards |
| **Observability Pipelines** | Design metrics, logs, and traces infrastructure; build dashboards and alerts | Owns service observability | Designs platform approach | Defines telemetry strategy |
| **Automation & Tooling** | Build tools for deployment, configuration, self-healing, and incident remediation | Service-level tooling | Platform-level tooling | Open-source / industry-level tooling |
| **Performance Engineering** | Profile and optimize latency, throughput, and resource utilization | Application-level | System-level | Cross-system architecture |
| **Security & Compliance** | Implement security controls, access policies, and audit compliance in infrastructure | Applies controls | Defines controls for org | Partners with security on strategy |

### 2.4 Cross-Functional

| Responsibility | Description | Senior | Staff | Principal |
|---------------|-------------|--------|-------|-----------|
| **Onboarding & Training** | Ramp new SREs; teach reliability fundamentals to adjacent teams | Mentors 1–2 juniors | Runs internal training program | Builds reliability curriculum |
| **Embedded SRE / Consulting** | Work inside product teams as a reliability subject-matter expert | Embedded in one team | Rotates across teams | Consults org-wide |
| **Interviewing & Hiring** | Participate in the SRE hiring pipeline | Conducts interviews | Designs interview loops | Defines hiring bar |
| **Postmortem Culture** | Lead blameless postmortems and drive systemic improvements | Facilitates own | Improves postmortem process | Defines org culture |
| **Stakeholder Communication** | Translate reliability metrics into business impact for non-technical audiences | Per-incident | Quarterly reviews | Board-level reporting |

---

## 3. Key Performance Indicators

The following KPIs define what "good" looks like at the senior+ SRE level. Every metric should be tracked, trended, and reviewed quarterly.

### 3.1 Reliability & Availability

| KPI | Definition | Target | Why It Matters |
|-----|-----------|--------|----------------|
| **SLO Attainment** | % of rolling 28-day windows where the service meets its SLO | ≥ 99.9% (tier-1) | Direct customer experience measure |
| **Error Budget Burn Rate** | Rate at which error budget is consumed per week | < 1/3 of budget/week | Burn rate too fast → no room for deploys |
| **Compound SLO** | Multi-service SLO accounting for dependency chains | ≥ 99.5% (tier-1) | Real user experience across call chain |
| **Availability** | % of time service is fully functional (uptime) | ≥ 99.99% (tier-1) | Traditional availability metric |

### 3.2 Incident Response

| KPI | Definition | Target | Why It Matters |
|-----|-----------|--------|----------------|
| **MTTD** (Mean Time to Detect) | Time from fault introduction to detection | < 5 minutes (P0) | Shorter = less user impact |
| **MTTR** (Mean Time to Resolve) | Time from detection to mitigation | < 15 minutes (P0) | Shorter = faster recovery |
| **MTTR × Severity** | Weighted MTTR by incident severity | < 30 min weighted avg | Reflects actual user outage time |
| **Incident Frequency** | Number of P0/P1 incidents per month | Trending down | Indicates systemic improvement |
| **Change Failure Rate** | % of changes causing a degradation or incident | < 5% | DORA metric; high failure = process problem |

### 3.3 Operational Health

| KPI | Definition | Target | Why It Matters |
|-----|-----------|--------|----------------|
| **Alert Quality** | % of alerts that trigger a meaningful human response (precision) | ≥ 90% | High noise = burnout and missed real issues |
| **Alerts per Shift** | Number of pages per on-call shift | < 5 per week | DORA: < 2 ideal, < 5 acceptable |
| **Mean Alert Acknowledgment Time** | Time between page and acknowledge | < 2 minutes | Indicates alert noise and on-call readiness |
| **Toil %** | % of time spent on manually repeatable, automatable work | < 25% | > 50% is burnout territory |
| **Runbook Coverage** | % of known failure modes with validated runbooks | ≥ 95% | Reduces MTTR and cognitive load |

### 3.4 Postmortem & Learning

| KPI | Definition | Target | Why It Matters |
|-----|-----------|--------|----------------|
| **Postmortem Closure Rate** | % of postmortem action items closed within 30 days | ≥ 90% | Shows org commitment to learning |
| **Postmortem Completeness** | % of qualifying incidents with a published postmortem | 100% | Every incident is a learning opportunity |
| **Action Item Recurrence** | % of incidents caused by a previously identified root cause | < 5% | Indicates action items were meaningful |
| **Time to Postmortem** | Time from incident resolution to published postmortem | < 5 business days | Freshness matters for accuracy |

### 3.5 Engineering & Automation

| KPI | Definition | Target | Why It Matters |
|-----|-----------|--------|----------------|
| **Deployment Frequency** | Number of production deployments per week | ≥ 1/day (tier-1) | DORA high-performer metric |
| **Deployment Lead Time** | Time from commit to production | < 1 hour | DORA high-performer metric |
| **Automation Coverage** | % of repetitive operational tasks automated | ≥ 80% | Direct toil reduction measure |
| **Capacity Margin** | Headroom in critical infrastructure (CPU, memory, storage) | ≥ 30% | Prevents capacity-driven outages |

---

## 4. Required Technical Skills

### 4.1 Skills Matrix

| Skill Domain | Senior | Staff | Principal | Assessment Criteria |
|-------------|--------|-------|-----------|-------------------|
| **Linux / Systems** | Expert-level kernel tuning, namespaces, cgroups, systemd | Designs OS-level reliability patterns | Defines host-hardening and performance baselines org-wide | Can debug a kernel oops, tune sysctls, reason about page cache vs. OOM |
| **Cloud Infrastructure** | Deep expertise in 1–2 cloud providers (AWS/GCP/Azure) | Multi-cloud architecture, cost analysis, org-wide best practices | Defines cloud strategy, negotiates commitments, designs multi-region topology | Can design a multi-region active-active topology and cost-model it |
| **Observability** | Builds dashboards, writes PromQL, configures alert rules | Designs unified telemetry pipeline (metrics + logs + traces) | Defines observability strategy, selects vendors, sets standards | Can trace a P0 from dashboard to root cause < 10 min |
| **Incident Response** | Incident commander certified, runs postmortems | Designs incident-response playbooks and training | Designs org-wide incident command structure and major-incident program | Has commanded 10+ real P0 incidents |
| **Infrastructure as Code** | Terraform/Pulumi/CDK expert, module author | Designs IaC patterns and module registries | Defines IaC strategy, security policy as code, compliance automation | Can review a 1000+ line Terraform plan and spot the risk |
| **Kubernetes / Containers** | Cluster operations, custom controllers, Helm chart author | Platform design, multi-cluster federation, CNCF ecosystem selection | Defines container strategy, custom scheduler extensions or platform abstraction | Can debug a CrashLoopBackOff, tune HPA, explain pod lifecycle |
| **Programming (Python)** | Production-quality Python: async, testing, profiling | Writes tooling and libraries consumed by multiple teams | Authors and maintains critical automation frameworks | Can write a gRPC service, async worker, pytest suite, and CLI tool |
| **Programming (Go)** | Production-quality Go: concurrency, interfaces, profiling | Builds operators, controllers, sidecars, or service mesh components | Contributes to or leads open-source projects in the CNCF ecosystem | Can write a Kubernetes operator or custom Prometheus exporter |
| **Shell (Bash)** | Advanced scripting, awk/sed/jq mastery, pipeline composition | Writes robust, testable shell libraries | Defines shell scripting standards and patterns | Can write a safe, idempotent bootstrap script for a bare-metal host |
| **CI/CD** | Pipeline author (GitHub Actions, GitLab CI, ArgoCD) | Designs deployment strategies (blue/green, canary, feature flags) | Defines release engineering strategy and SLSA compliance | Can design a progressive delivery rollout with canary analysis |
| **Networking** | TCP/IP, DNS, HTTP, TLS, load balancers, service mesh | Designs network policies, mTLS mesh, multi-cluster networking | Defines network architecture and zero-trust networking strategy | Can diagnose a TCP retransmission storm or TLS handshake failure |
| **Databases** | Query optimization, replication, failover, backups | Designs data-layer reliability patterns (active/active, CQRS) | Defines data resiliency and consistency strategy | Can manually failover a PostgreSQL cluster and validate consistency |

### 4.2 Technology Depth Expectations

**Senior SRE** must be able to:
- Write a production-grade Kubernetes operator from scratch in Go
- Design and implement a complete observability stack (Prometheus + Loki/Tempo/Grafana or equivalent)
- Automate a multi-step incident runbook end-to-end with zero manual steps
- Debug a complex distributed-system failure across multiple services, data stores, and network hops
- Author and review architecture documents for reliability, scalability, and operability

**Staff SRE** additionally:
- Design a multi-cluster, multi-region Kubernetes platform serving 100+ microservices
- Lead the technical design and execution of a quarter-to-year-long reliability initiative
- Define org-wide coding standards, deployment practices, and reliability expectations
- Evaluate and select technologies (databases, observability tools, CI/CD platforms) at org scale

**Principal SRE** additionally:
- Define the technical reliability strategy for the entire engineering organization
- Influence industry standards through open-source contributions, conference talks, or publications
- Make build-vs-buy decisions that affect multi-million dollar infrastructure budgets
- Act as the final escalation for the hardest distributed-systems problems in the organization

---

## 5. Soft Skills & Leadership Competencies

### 5.1 Incident Leadership

- **Command presence**: Remain calm and directive during high-stress outages. Know when to delegate, when to escalate, and when to say "I don't know yet."
- **Communication discipline**: During incidents, communicate to stakeholders in structured updates (situation → impact → action → ETA). No speculation. No silence.
- **Systems triage**: Quickly identify the failure domain — network, storage, application, configuration, capacity — and route to the right expertise.

### 5.2 Influence Without Authority

- **Cross-team negotiation**: Convince product teams to invest in reliability work without a reporting-line mandate. Frame reliability investments in terms of user impact and business value.
- **Technical persuasion**: Write RFCs and design docs that survive rigorous review. Use data, not opinion. If you can't measure it, you can't argue for it.
- **Bias for written communication**: Async documentation scales better than meetings. Write clearly, concisely, and inclusively.

> **"The Senior SRE's most powerful tool is a well-written design document."**

### 5.3 Customer-Impact Orientation

- Translate every technical metric (p99 latency, error rate, saturation) into user-facing impact ("users in region X experience 3-second page loads").
- Prioritize reliability work based on the number of users affected and the severity of the impact, not technical interest.

### 5.4 Coaching & Mentorship

- **One-on-one coaching**: Regularly invest in the growth of junior and mid-level SREs. Teach debugging methodology, system design thinking, and operational judgment.
- **Code and design review**: Review with the intent to teach, not just to gatekeep. Leave comments that explain *why* something is a concern, not just *what* to change.
- **Shadow programs**: Run incident-shadow programs where junior engineers observe and gradually take on incident-command responsibilities.

### 5.5 Strategic Communication

| Audience | Message | Frequency | Format |
|----------|---------|-----------|--------|
| **Engineering team** | Reliability metrics, incident learnings, upcoming work | Weekly | Standup, Slack, dashboard |
| **Product / PM** | Error budget status, feature-velocity tradeoffs, reliability roadblocks | Bi-weekly | 1:1, shared dashboard |
| **Engineering leadership** | Reliability trends, major incidents, org-wide risks, investment needs | Monthly | Written report + walkthrough |
| **Executive / Board** | Business-level reliability posture, SLO compliance, top risks | Quarterly | Executive summary (1-pager) |

---

## 6. Tools & Tech Stack

### 6.1 Tooling Reference Table

| Category | Tools | Senior SRE Level | Staff+ SRE Level |
|----------|-------|------------------|------------------|
| **Container Orchestration** | Kubernetes (EKS/AKS/GKE), OpenShift, Nomad | Operators, Helm charts, cluster scaling | Multi-cluster federation, platform abstraction, CNI/service-mesh design |
| **Infrastructure as Code** | Terraform, Pulumi, AWS CDK, Crossplane, Ansible | Module author, backend state management, review | Pattern design, compliance-as-code, platform abstraction layers |
| **CI / CD** | GitHub Actions, GitLab CI, ArgoCD, Spinnaker, Jenkins X | Pipeline author, canary deployment configuration | Deployment strategy design, progressive delivery framework |
| **Observability — Metrics** | Prometheus, Thanos, VictoriaMetrics, M3DB, Grafana | PromQL, recording rules, dashboard design, alert thresholds | Unified metric strategy, long-term storage, cardinality management |
| **Observability — Logs** | Loki, Elasticsearch/OpenSearch, Datadog Logs, Splunk | Log parsing, structured logging, log-based alerting | Log pipeline design, cost optimization, sampling strategy |
| **Observability — Tracing** | Tempo, Jaeger, Honeycomb, Datadog APM | Trace analysis, span instrumentation | Distributed tracing strategy, tail-based sampling, root-cause automation |
| **Service Mesh** | Istio, Linkerd, Consul Connect, Cilium | mTLS, traffic shifting, circuit breaking | Mesh architecture for multi-cluster, zero-trust, observability injection |
| **Secrets Management** | Vault, AWS Secrets Manager, SOPS, External Secrets | Secret rotation, policy authoring | Secrets strategy, PKI management, identity-based access |
| **Configuration** | Helm, Kustomize, jsonnet, CUE, KCL | Chart author, environment overlays | Configuration-as-data patterns, policy enforcement |
| **Database / Storage** | PostgreSQL, MySQL, Cassandra, Redis, S3, Vitess, CockroachDB | Query optimization, backup/restore, replication config | Data-layer HA pattern design, DR strategy, consistency modeling |
| **Messaging / Streaming** | Kafka, RabbitMQ, NATS, Pulsar | Topic design, consumer group management, monitoring | Streaming platform architecture, data-loss prevention, schema registry |
| **CD / GitOps** | ArgoCD, Flux, Fleet | Application sync, health checks, secrets | Multi-tenant GitOps, cluster registration, drift detection |
| **Cost Management** | AWS Cost Explorer, Vantage, Kubecost, Infracost | Tagging, cost attribution, waste reduction | Unit cost modeling, RI/savings-plan strategy, chargeback |
| **Incident Management** | PagerDuty, OpsGenie, Incident.io, FireHydrant | On-call schedule, escalation policies | Incident-response program design, severity taxonomy, major-incident workflows |
| **Chaos Engineering** | Chaos Mesh, Litmus, Gremlin | Experiment design, blast-radius control | Chaos program strategy, steady-state hypothesis definition, gameday facilitation |
| **Security / Compliance** | Trivy, Falco, OPA/Gatekeeper, Kyverno, Cert-Manager | Policy authoring, vulnerability scanning, admission control | Security posture framework, compliance automation, audit readiness |

### 6.2 Platforms & Managed Services (by Cloud)

| Cloud | Core Services | SRE Focus |
|-------|---------------|-----------|
| **AWS** | EC2, EKS, RDS/Aurora, ElastiCache, S3, CloudFront, Route53, Lambda, Step Functions, DynamoDB, MSK, SQS/SNS | VPC design, IAM policy, cost optimization, multi-AZ/multi-region |
| **GCP** | GKE, Cloud SQL, Cloud Spanner, Cloud Storage, Cloud CDN, Cloud Run, Pub/Sub, BigQuery | GKE Autopilot, VPC-native clusters, Cloud Interconnect |
| **Azure** | AKS, Azure SQL, Cosmos DB, Blob Storage, Front Door, Traffic Manager, Functions, Service Bus | AKS networking (CNI/Azure CNI), managed identity, ExpressRoute |

---

## 7. Career Progression

### 7.1 Level Ladder

```
                    ┌─────────────────────────────────────┐
                    │       Principal SRE (L8/L9)          │
                    │  Org-wide strategy, industry impact   │
                    └─────────────────────────────────────┘
                                 │
                    ┌─────────────────────────────────────┐
                    │         Staff SRE (L7)               │
                    │  Cross-team reliability standards     │
                    └─────────────────────────────────────┘
                                 │
                    ┌─────────────────────────────────────┐
                    │        Senior SRE (L6)               │
                    │  Service reliability, technical lead  │
                    └─────────────────────────────────────┘
                                 │
                    ┌─────────────────────────────────────┐
                    │         SRE II (L5)                  │
                    │  Independent contributor, on-call     │
                    └─────────────────────────────────────┘
                                 │
                    ┌─────────────────────────────────────┐
                    │         SRE I (L4)                   │
                    │  Learning, paired with senior         │
                    └─────────────────────────────────────┘

    ─── Individual Contributor Track ───      ─── Management Track ───
```

### 7.2 Progression Criteria

| Transition | Timeframe (Typical) | Key Evidence |
|-----------|---------------------|--------------|
| **SRE I → SRE II** | 12–18 months | Owns service reliability; handles on-call independently; writes good runbooks |
| **SRE II → Senior SRE** | 2–3 years | Leads incident response; architects solutions; mentors juniors; automates toil across a service |
| **Senior SRE → Staff SRE** | 3–5 years | Drives org-wide reliability initiatives; defines standards; embedded across multiple teams |
| **Staff SRE → Principal SRE** | 3–5 years | Sets org-wide reliability strategy; industry presence; architectural authority |
| **Senior SRE → SRE Manager** | 2–3 years | Has managed projects and people; demonstrates hiring/org-building skills |

### 7.3 Management Track: SRE Manager

Not all senior SREs stay IC. The SRE Manager track runs parallel:

| Responsibility | SRE Manager | Senior SRE (IC) |
|---------------|-------------|------------------|
| **Primary focus** | Team health, hiring, career growth, stakeholder management | Technical execution, system design, production excellence |
| **On-call** | Backup / escalation | Primary participant |
| **Technical depth** | Maintains architectural review capability | Maintains hands-on depth |
| **Meetings / calendar** | Heavy (1:1s, planning, cross-team syncs) | Lighter (design review, incident coordination) |
| **Leverage** | Through the team | Through code, architecture, and teaching |

---

## 8. Senior vs. Mid-Level: The Distinction

### 8.1 What Makes a Senior SRE Different

| Dimension | Mid-Level SRE | Senior SRE |
|-----------|--------------|------------|
| **Scope** | Owns reliability for a specific service | Owns reliability for a service family or platform area |
| **Proactivity** | Responds to incidents; improves runbooks | Anticipates failure modes; builds defenses before incidents occur |
| **Architectural Authority** | Implements designs reviewed by seniors | Reviews and approves designs; writes RFCs that set direction |
| **Automation** | Automates repetitive tasks for own work | Builds tooling consumed by multiple teams |
| **Mentorship** | Occasionally helps new team members | Regularly coaches 1–3 junior SREs; runs knowledge-sharing sessions |
| **Incident Command** | Can command for own service | Commands for any service in the domain; coaches new ICs |
| **Systems Thinking** | Understands service internals | Understands end-to-end user journeys through multiple services |
| **Org Influence** | Influences within the team | Influences across teams without direct authority |
| **Postmortem Leadership** | Contributes action items | Leads the postmortem; identifies systemic patterns |
| **Time Horizon** | Days to weeks | Quarters to a year |

### 8.2 The Staff / Principal Multiplier

Staff and Principal SREs differentiate from Senior SREs through **leverage**:

> **"A Principal SRE's value is not in how many incidents they resolve, but in how many incidents never happen because they designed the system better."**

| Activity | Senior SRE | Staff SRE | Principal SRE |
|----------|-----------|-----------|---------------|
| Fix system A | Fixes it | Fixes it + writes playbook + teaches system B team the same pattern | Fixes system A AND identifies the org-wide pattern that caused the failure, then removes it for all systems |
| Reduce MTTR | Cuts MTTR for service from 30→10 min | Cuts MTTR for the entire org by 50% through better tooling and training | Eliminates whole incident classes through architectural changes |
| Influence | 5–20 engineers | 20–100 engineers | 100–1000+ engineers |
| Time horizon | Quarter | Year | 2–5 years |
| Failure mode | Burning out from carrying too much weight | Becoming a bottleneck for decision-making | Losing touch with operational reality |

### 8.3 Common Failure Modes by Level

| Level | Failure Mode | Prevention |
|-------|-------------|------------|
| **Senior SRE** | Hero mode — too much direct execution, not enough teaching or automation | Track "time spent teaching vs. doing"; enforce a 30% minimum on enabling work |
| **Senior SRE** | Perfectionism — refusing to ship good-enough solutions | Error budgets apply to operations too; shipping a 90% solution early beats 100% never |
| **Staff SRE** | Bottleneck — every design review and incident needs their input | Delegate authority; write decisions into standards so others can self-serve |
| **Staff SRE** | Ivory tower — too far from production, designs don't match reality | Carry a pager. Stay in rotation. Do gamedays. |
| **Principal SRE** | Loss of credibility — org stops listening because they haven't debugged a real incident in years | Stay technical. Write code that ships. Join critical incidents, even just as an observer. |
| **Principal SRE** | Scope creep — saying yes to everything, diluting impact | Ruthless prioritization. "No" is a strategic decision. |

---

## Appendix A: Interview Rubric for Senior+ SRE

| Criterion | Weight | Strong Signal | Weak Signal |
|-----------|--------|---------------|-------------|
| **Systems Design** | 30% | Designs a reliable, observable, evolvable distributed system with tradeoff analysis | Memoized solution; misses failure modes; can't explain tradeoffs |
| **Incident Response** | 20% | Structured, calm approach to troubleshooting; communicates clearly under pressure | Panics; goes dark; guesses instead of gathering data |
| **Automation & Coding** | 20% | Writes clean, idiomatic, well-tested code that solves an ops problem | Overly complex; ignores error handling; untestable |
| **Cultural Fit** | 15% | Blameless mindset; data-driven; empowers others | Blame-oriented; cargo-cults practices; must be right |
| **Communication** | 15% | Clear, concise written and verbal explanations; adapts to audience | Rambling; overly technical for non-technical; can't explain "why" |

---

## Appendix B: Books, Talks & References

| Resource | Author | Why |
|----------|--------|-----|
| *Site Reliability Engineering* | Beyer, Jones, Petoff, Murphy (Google) | The foundational SRE text |
| *The Site Reliability Workbook* | Google SRE Team | Practical implementation guidance |
| *Seeking SRE* | Jones, Beyer, Murphy, Rensin | Conversations with industry SRE leaders |
| *Chaos Engineering* | Rosenthal, Jones, et al. | Systematic failure-injection methodology |
| *The Phoenix Project* | Gene Kim | IT/DevOps transformation narrative (SRE cultural alignment) |
| *Accelerate* | Forsgren, Humble, Kim | DORA metrics and evidence-based DevOps/SRE |
| *Designing Data-Intensive Applications* | Martin Kleppmann | Distributed systems fundamentals for reliability design |
| *Google SRE YouTube Channel* | Various | SRE talks, postmortems, and training from Google |
| *"How Complex Systems Fail"* | Richard Cook | Foundational essay on failure in complex systems |
| *"My Philosophy on Alerting"* | Rob Ewaschuk (Google) | Landmark essay on alert design and on-call quality |

---

*This document is a living reference. Propose updates via PR or open an issue to suggest improvements. For the most current version, see the canonical source at `/references/senior-sre-blueprint.md`.*

# SLO Implementation Recipe

Use this reference when a team has no SLOs, has metrics but no shared objectives, or needs to turn an SLO discussion into an operating agreement. It complements `references/slo-sli-framework.md`; it does not replace its definitions or formulas.

## Source anchors

Synthesized from *The Site Reliability Workbook*, “Implementing SLOs,” “SLO Engineering Case Studies,” “Alerting on SLOs,” and “Example SLO Document,” and *Seeking SRE*, “The Art and Science of the Service-Level Objective,” “SRE as a Success Culture,” and “Using Incident Metrics to Improve SRE at Scale.”

## The bounded sequence

1. **Choose one service and identify its users.** Include direct users, downstream services, operators, and business processes. State what failure means to each.
2. **Map the important journeys.** Describe the request path, asynchronous path, data path, dependencies, and the point at which users experience success or failure.
3. **Select a small initial set of indicators.** Prefer a few customer-facing indicators for availability, latency, correctness, freshness, or durability. Do not begin by turning every metric into an SLO.
4. **Define good and bad events.** State valid events, exclusions, aggregation, data source, sampling, and measurement boundary. A formula without a measurement boundary is not an SLI.
5. **Measure the baseline.** Use a representative period and record gaps in instrumentation. A target should be feasible and meaningful, not an aspirational number copied from another service.
6. **Propose targets and windows.** Explain the user impact and expected error budget for each target. Use rolling windows when the decision needs a current operating signal; use calendar or contractual windows only when that is the real agreement.
7. **Review with stakeholders.** The product owner, service owner, support or customer-facing team, and reliability operator must be able to explain what the objective protects and what it will cause them to do.
8. **Define the policy before the first breach.** Specify what healthy, at-risk, and exhausted budget states change: release pace, reliability work, incident review, escalation, or scope.
9. **Alert on significant budget consumption.** Use multi-window burn-rate alerts and route pages only when the response is urgent and actionable. Use tickets or review queues for slower repair work.
10. **Run a review cycle.** Revisit targets when user expectations, architecture, traffic, dependencies, or product criticality changes. Record why the target changed and what evidence supported it.

## SLO review questions

- Would a user notice the measured failure, or is it only an internal proxy?
- Can the team take action when the SLO is violated or burning rapidly?
- Does the measurement include dependency and client-side effects that matter to the journey?
- Are exclusions hiding a failure the user still experiences?
- Does the error budget create enough signal before the service becomes unacceptable?
- Is the target tight enough to protect users but loose enough to permit justified change?
- What happens when the team cannot operate within the target and toil constraints?

## What to record

Every approved SLO should link to a declaration containing:

- user journey and service owner;
- SLI good-event definition and measurement boundary;
- data source, query, aggregation, and known blind spots;
- target, window, error budget, and review date;
- alert and escalation behavior;
- release and reliability-work policy;
- dependency treatment and exception process;
- evidence baseline and unresolved instrumentation gaps.

## SLO failure is a decision signal

Do not treat SLOs as a performance score or a punishment mechanism. A budget breach is evidence that the service needs reliability attention, reduced change risk, more capacity, a dependency intervention, an objective revision backed by new user evidence, or a scope decision. Repeatedly overriding the policy without recording the reason turns the SLO into decoration.

## Agent procedure

Load this recipe with `references/slo-sli-framework.md` and `templates/slo-declaration-template.md`. If the user has supplied no service, users, or measurement data, produce a discovery plan and mark assumptions rather than inventing targets.

# SLO / SLI Framework Reference

> Site Reliability Engineering — service level objective design and error budget management.

---

## Table of Contents

1. [Definitions](#definitions)
2. [SLI Definition Patterns](#sli-definition-patterns)
3. [SLO Target Setting Methodology](#slo-target-setting-methodology)
4. [Error Budget Mechanics](#error-budget-mechanics)
5. [Burn Rate Alerting](#burn-rate-alerting)
6. [SLO Alignment to User Journeys](#slo-alignment-to-user-journeys)
7. [Service Tiering](#service-tiering)
8. [Common Pitfalls](#common-pitfalls)
9. [Example SLO Declarations](#example-slo-declarations)

---

## Definitions

| Term | Definition |
|---|---|
| **SLI (Service Level Indicator)** | A carefully defined quantitative measure of some aspect of the service level provided. |
| **SLO (Service Level Objective)** | A target value or range for an SLI over a specified window (typically 28–30 days rolling). |
| **SLA (Service Level Agreement)** | A contractual commitment to a customer about service level, usually with consequences for non-compliance. |
| **Error Budget** | The permitted amount of unreliability over an SLO window (1 - SLO target). |

---

## SLI Definition Patterns

### 1. Latency (Request-Response)

**Definition pattern:**

```
SLI_{latency} = count of "fast enough" requests / total requests
```

**Formula:**

\[
\text{SLI}_{\text{latency}} = \frac{ \sum_{t \in T} \mathbf{1}[\, \text{latency}(t) \leq \text{threshold}\, ] }{ |T| }
\]

Where:
- `threshold` is the target latency (e.g., 200ms for p99, 100ms for median)
- Only requests counted — not internal health checks or synthetic probes unless they represent real user paths.

**Measurement approaches:**

| Approach | Description | Pros / Cons |
|---|---|---|
| **Server-side** | Instrument the service at the entry point | Direct, but includes queue wait; may miss client-side latency |
| **Client-side** | Measure from the user's device | Real user experience, but harder to collect |
| **Load balancer** | Measure at reverse proxy / API gateway | Consistent, captures network; may overcount retries |
| **Synthetic** | Periodic probing from external locations | Consistent, but may not reflect real traffic patterns |

**Common thresholds:**

| Percentile | Typical range | Use case |
|---|---|---|
| p50 (median) | 50–200 ms | General "feel" of responsiveness |
| p95 | 200–500 ms | Good user experience boundary |
| p99 | 500–2000 ms | Tail latency for user-perceived slowness |
| p99.9 | 2000–10000 ms | Hard timeout / "hanging" detection |

### 2. Availability (Uptime / Success Ratio)

**Definition pattern:**

```
SLI_{availability} = successful requests / total requests
```

**Formula:**

\[
\text{Availability} = \frac{ \text{successful\_requests} }{ \text{total\_requests} } \times 100\%
\]

Where "successful" means HTTP 2xx or application-level success, and total excludes redirected requests (3xx) when those are benign.

**For request-driven services:**

\[
A = \left(1 - \frac{\text{errors}}{\text{requests}}\right) \times 100\%
\]

**For non-request-driven services (e.g., storage, queues):**

\[
A = \frac{\text{time\_serving\_normally}}{\text{total\_time}} \times 100\%
\]

**Nines reference table:**

| Availability | Downtime / 30d | Downtime / 365d |
|---|---|---|
| 90% ("one nine") | 3.0 days | 36.5 days |
| 99% ("two nines") | 7.2 hours | 3.65 days |
| 99.9% ("three nines") | 43.2 minutes | 8.76 hours |
| 99.95% | 21.6 minutes | 4.38 hours |
| 99.99% ("four nines") | 4.32 minutes | 52.56 minutes |
| 99.999% ("five nines") | 25.9 seconds | 5.26 minutes |

### 3. Error Rate

**Definition pattern:**

```
SLI_{error\_rate} = error responses / total requests
```

**Formula:**

\[
\text{Error Rate} = \frac{ \text{HTTP 5xx} + \text{application\_level\_errors} }{ \text{total\_requests} }
\]

**Types of errors to track:**

- **HTTP 5xx** — server-side failures
- **HTTP 4xx** — client errors (generally *excluded* from availability SLIs unless the API is rejecting due to overload)
- **Application-level errors** — business logic failures (e.g., checkout declined, payment failed)
- **Latency-based errors** — requests that complete but exceed a timeout threshold
- **Silent failures** — requests that appear successful but produce incorrect results (hardest to detect; requires validation)

**Recommended:** Separate "server errors" and "application errors" into distinct SLIs. A 5xx rate > 0.1% usually warrants immediate investigation regardless of SLO.

### 4. Throughput

**Definition pattern:**

```
SLI_{throughput} = requests processed / time window
```

**Formula:**

\[
\text{Throughput} = \frac{ N_{\text{requests}} }{ \Delta t }
\]

**Use case:** Capacity planning, burst detection, saturation signals. Not typically an SLO target by itself, but used as a _service level indicator_ for scaling decisions. Set a minimum throughput SLO only when the service must handle a baseline load for correctness (e.g., a payment settlement pipeline).

### 5. Durability

**Definition pattern:**

```
SLI_{durability} = intact_objects / total_objects_over_window
```

**Formula:**

\[
D = \left(1 - \frac{\text{lost\_objects}}{\text{total\_objects}}\right) \times 100\%
\]

**Used for:** Storage systems, databases, message queues, blob stores.

**Typical targets:** 99.9999999% (nine nines — one object lost per 10^11 stored per year) to 99.99999999% (eleven nines).

Durability is generally an _overall measure_, not measured per-request, and verified through:
- Checksum verification scans
- Replica consistency checks
- Data loss replication drills

### 6. Correctness

**Definition pattern:**

```
SLI_{correctness} = correct_responses / total_responses
```

**Formula:**

\[
C = \frac{ \text{responses\_matching\_expected\_output} }{ \text{total\_responses\_validated} }
\]

**Used for:** Data pipelines, search relevance, ML inference, transaction processing.

Correctness SLIs require a _validation oracle_ — either:
- A secondary verifier (shadow comparison, canary validation)
- End-to-end consistency checks (checksums on data movement)
- User-reported error signals (support tickets, chargebacks, rollbacks)

---

## SLO Target Setting Methodology

### Step 1: Identify User Journeys

Map the critical paths users take through the system:

1. **Login → Search → View results → Select item → Add to cart → Checkout → Payment → Confirmation**
2. **Login → Dashboard → Reports → Export**

Each journey gets its own SLO(s), derived from the component SLIs that compose it.

### Step 2: Set Measurable Baselines

Before setting SLO targets, collect 2–4 weeks of SLI measurements so you understand:
- Current performance
- Natural variance (daily, weekly, seasonal)
- Known bad periods

### Step 3: Apply the "Tiered Tightening" Approach

| Iteration | Method | Example |
|---|---|---|
| Start | Set SLO 1–2 "nines" below current performance | Current: 99.85% → SLO: 99.0% |
| Tighten 1 | User-journey margin: subtract margin below baseline | Baseline: 99.9% → SLO: 99.7% |
| Tighten 2 | Dissatisfaction-based: use the point where user complaints rise | Complaint threshold: 99.5% → SLO: 99.0% |
| Final | Business constraint: must meet SLA + margin | SLA: 99.9% → internal SLO: 99.95% |

### Step 4: Choose the Window

| Window | Use case |
|---|---|
| **28-day rolling** | Standard SRE practice — aligns with error budget cycles |
| **30-day calendar** | Common for SLAs |
| **7-day rolling** | Aggressive monitoring for volatile services |
| **Quarterly** | Long-burn, slow-changing services (e.g., durability) |
| **Fixed calendar month** | Needed when SLO is tied to billing/contract periods |

### Step 5: Define the Compliance Period

Specify:
- The _measurement window_ (e.g., 28 days rolling)
- The _evaluation point_ (e.g., evaluated at end of each month)
- The _reset behavior_ (rolling window resets continuously; fixed-period resets on the 1st)

---

## Error Budget Mechanics

### Accrual

The error budget is established at the start of a compliance period:

\[
\text{Error Budget} = (1 - \text{SLO}) \times \text{total\_requests}
\]

**Example:**
- SLO = 99.9% (0.999)
- Total requests in window = 10,000,000
- Error budget = (1 - 0.999) × 10,000,000 = 10,000 errors allowed

### Consumption

Error budget is consumed by each bad event:

\[
\text{Error Budget Remaining} = \text{Error Budget} - \sum_{t \in \text{window}} \text{bad\_events}_t
\]

### Depletion Rate

\[
\text{Depletion Rate} = \frac{ \text{errors\_consumed\_so\_far} }{ \text{elapsed\_time\_in\_window} }
\]

\[
\text{Time to exhaustion} = \frac{ \text{error\_budget\_remaining} }{ \text{depletion\_rate} }
\]

### Budget Status States

| Status | Condition | Action |
|---|---|---|
| **Green** | Remaining ≥ 50% | Normal operations |
| **Yellow** | 10–50% remaining | Review upcoming releases, add conservative monitoring |
| **Red** | < 10% remaining | Freeze all non-critical deployments, prioritize reliability work |
| **Exhausted** | 0% remaining | Mandatory incident response, full reliability sprint |

### Example Calculation

| Metric | Value |
|---|---|
| SLO target | 99.9% |
| Requests / 28 days | 50,000,000 |
| Error budget | 50,000 errors |
| Errors so far (day 14) | 20,000 |
| Budget remaining | 30,000 errors (60%) |
| Depletion rate | 20,000 / 14 = 1,428.6 errors/day |
| Days until exhausted | 30,000 / 1,428.6 ≈ 21.0 days (OK) |

---

## Burn Rate Alerting

### Burn Rate Definition

\[
\text{Burn Rate} = \frac{ \text{rate of bad events} }{ \text{rate of bad events allowed by SLO} }
\]

Or more simply:

\[
\text{Burn Rate} = \frac{ \text{error\_rate} }{ 1 - \text{SLO} }
\]

A burn rate of:
- **1.0** = exactly consuming the error budget
- **2.0** = consuming budget twice as fast (will exhaust in half the window)
- **0.5** = consuming half as fast (will have surplus remaining)

### Multi-Window Burn Rate Approach

Use two or more time windows to distinguish fast, dangerous burns from slow, tolerable ones:

| Alert | Burn Rate | Short Window | Long Window | Severity | Response |
|---|---|---|---|---|---|
| **Critical** | ≥ 14.4 (or ~14x) | 1 minute | 5 minutes | Pager | Immediate on-call |
| **Warning** | ≥ 6.0 | 5 minutes | 30 minutes | Pager | Investigate |
| **Watch** | ≥ 2.0 | 30 minutes | 6 hours | Ticket | Triage within 1 hour |
| **Monitor** | ≥ 1.0 | 2 hours | 1 day | Ticket | Review within 24h |

### Multi-Window Alerting Rule (Prometheus-style)

```
groups:
  - name: slo_burn_rate
    rules:
      - alert: SLIBurnRateCritical
        expr: |
          (
            (
              rate(sli_errors_total[1m])
              / on() (rate(sli_errors_total[1m]) + rate(sli_successes_total[1m]))
            )
            / on() (1 - 0.999)
          ) >= 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Burn rate critical ({{ $value }}x)"

      - alert: SLIBurnRateWarning
        expr: |
          (
            (
              rate(sli_errors_total[5m])
              / on() (rate(sli_errors_total[5m]) + rate(sli_successes_total[5m]))
            )
            / on() (1 - 0.999)
          ) >= 6.0
        for: 5m
        labels:
          severity: warning
```

### Interpreting Burn Rate Examples

| Scenario | Window | Errors | Rate | SLO | Burn Rate |
|---|---|---|---|---|---|
| 10s outage (all requests fail) | 1 min | 1,000 | 100% | 99.9% | 1,000x |
| 2 min partial (50% failures) | 5 min | 500 | 10% | 99.9% | 100x |
| Mild degradation (5% failures) | 30 min | 1,500 | 5% | 99.9% | 50x |
| Slow drift (0.2% failures) | 6h | 720 | 0.2% | 99.9% | 2x |

---

## SLO Alignment to User Journeys

### Journey Decomposition

Every user journey is a chain of service dependencies. The aggregate SLO must account for each link:

\[
\text{Journey SLO} = \prod_{i=1}^{n} \text{Component SLO}_i
\]

**Example:** A checkout journey depends on:
- Web frontend SLO = 99.9%
- Cart API SLO = 99.95%
- Payment service SLO = 99.99%
- Inventory SLO = 99.9%

\[
\text{Checkout SLO} = 0.999 \times 0.9995 \times 0.9999 \times 0.999 \approx 0.9974 \text{ (99.74\%)}
\]

### Budget Allocation — "From the User Back"

1. Define the **user-facing SLO** first (e.g., checkout journey = 99.5%)
2. Work backwards to allocate error budgets to each component
3. Tightest budgets go to the most constrained or hardest-to-fix components

### User Journey SLI Template

```yaml
journey:
  name: "Complete Checkout"
  slo: "99.5% over 28d"
  steps:
    - name: "View Cart"
      component: frontend
      sli: latency (p95 < 500ms)
      slo: 99.9%
    - name: "Submit Order"
      component: cart-api
      sli: availability
      slo: 99.95%
    - name: "Process Payment"
      component: payment-gateway
      sli: availability + correctness
      slo: 99.99%
    - name: "Confirm Inventory"
      component: inventory-service
      sli: availability
      slo: 99.9%
```

---

## Service Tiering

### Tier 1 — Critical

| Attribute | Description |
|---|---|
| **Impact** | Revenue-critical, user-facing, or compliance-mandated |
| **SLO target** | 99.95% – 99.999% |
| **Error budget** | Very tight (0.05% – 0.001%) |
| **Alerting** | Burn rate ≥ 6 → immediate page; ≥ 2 → ticket within 1h |
| **On-call** | 24/7, < 5 min response |
| **Release gating** | Error budget must be ≥ 50% to deploy |
| **Examples** | Payment processing, authentication, search, core API, user database |

### Tier 2 — Important

| Attribute | Description |
|---|---|
| **Impact** | Significant business feature, many users affected |
| **SLO target** | 99.0% – 99.9% |
| **Error budget** | Moderate (1% – 0.1%) |
| **Alerting** | Burn rate ≥ 14 → page; ≥ 6 → ticket within 2h |
| **On-call** | Business-hours + after-hours for critical degradation |
| **Release gating** | Error budget must be ≥ 25% to deploy |
| **Examples** | Recommendations, admin dashboard, reporting, notification delivery |

### Tier 3 — Best-Effort

| Attribute | Description |
|---|---|
| **Impact** | Nice-to-have, no direct user or revenue impact |
| **SLO target** | 90% – 99% (or none) |
| **Error budget** | Generous (10% – 1%) |
| **Alerting** | Burn rate ≥ 30 → ticket next business day; otherwise monitor only |
| **On-call** | Best-effort, no formal pager duty |
| **Release gating** | None required |
| **Examples** | Internal tools, experimental features, historical data exports, log processing |

---

## Common Pitfalls

### 1. Averages Lie

> "The average latency is 150ms."

Mean latency _always_ hides tail latency. A service with p50 = 100ms and p99 = 10s has a mean of ~200ms (skewed by outliers but still deceptively low).

**Rule:** Always use percentiles for latency SLIs. Never use mean/avg.

### 2. Too Many SLIs

> "We track 47 dashboards with 120 SLIs."

Every SLI has a maintenance cost: dashboards, alerts, toil in responding to false positives. Keep the count small — start with 3-5 per service.

**Rule:** If an SLI has never triggered an action in 3 months, retire it.

### 3. Heroic Targets (Aspirational SLOs)

> "Our SLO is 99.999% availability for the developer wiki."

Setting unrealistically high targets guarantees budget exhaustion, alert fatigue, and desensitization. SLOs should be _achievable_ and _represent the user's real experience_, not a wish.

**Rule:** Set SLOs from observed data + a modest margin, not from a desire to be "five nines."

### 4. Measuring the Wrong Thing

> "Our API Gateway reports 99.99% availability, but users keep complaining."

Common mismatches:
- Measuring health-check endpoints instead of real user requests
- Excluding client-side timeouts from the SLI (the user sees a failure)
- Counting retries as independent successes
- Measuring requests that are never user-facing (internal traffic)

**Rule:** The SLI must match what the user experiences. If the user sees an error, the SLI should count it.

### 5. Ignoring Traffic Patterns

> "We had no alerts during the night… but the on-call was paged at 9 AM."

If 90% of traffic arrives in a 4-hour peak window, the error budget burns 10x faster during those hours. A flat burn rate alert across the day will miss morning meltdowns.

**Rule:** Consider time-partitioned SLIs (peak / off-peak) for services with strong diurnal patterns.

### 6. Perfection as a Target

> "We need 100% availability."

Zero-defect targets eliminate the error budget — the mechanism that makes SLOs useful. Without an error budget, every incident is an escalation and you lose the ability to trade reliability for velocity.

**Rule:** If you can't tolerate any errors, you can't use SLO-based management. Add redundancy instead.

### 7. Alert Fatigue from Burn Rate

> "We get 200 burn rate alerts per day."

Burn rate alerts are _noisy by design_ for short windows. Without multi-window gating (short window condition _and_ long window condition), you'll page on every transient blip.

**Rule:** Always pair a short window with a long window in burn rate alerting. Never alert on the short window alone.

---

## Example SLO Declarations

### Example 1: Web Service (E-commerce Frontend)

```yaml
service: storefront-web
description: "Customer-facing product browsing and cart management"
tier: Tier 1 (Critical)

slis:
  - name: "Request Latency"
    type: latency
    measurement: server-side at load balancer
    metric: http_request_duration_seconds
    good_event: p99_latency <= 500ms
    window: 28 days rolling

  - name: "Request Availability"
    type: availability
    measurement: HTTP status at load balancer
    metric: http_requests_total{status=~"2xx|5xx"}
    good_event: status_code matches 2xx
    window: 28 days rolling

  - name: "Error Rate"
    type: error_rate
    measurement: HTTP 5xx responses
    metric: http_requests_total{status=~"5xx"}
    good_event: ratio of 5xx / total < 1%
    window: 28 days rolling

slo_targets:
  - sli: "Request Latency"
    target: "p99 <= 500ms at 90% of requests over the window"
    compliance: 90%

  - sli: "Request Availability"
    target: "99.95%"
    compliance: "99.95% of requests are successful"

  - sli: "Error Rate"
    target: "error ratio < 1% sustained over 5 min"
    compliance: "measured as windowed rate"

error_budget:
  availability: 0.05% of total requests
  latency: 10% of requests can exceed 500ms

alerting:
  - name: "Critical Burn (Availability)"
    burn_rate: >= 14.0
    windows: [1m, 5m]
    severity: page
    response: "Immediate incident response"

  - name: "Warning Burn (Latency)"
    burn_rate: >= 3.0
    windows: [5m, 30m]
    severity: ticket
    response: "Investigate within 30 minutes"
```

### Example 2: Public API Service

```yaml
service: payments-api
description: "External-facing REST API for payment processing"
tier: Tier 1 (Critical)

slis:
  - name: "API Availability"
    type: availability
    measurement: edge proxy (all non-3xx responses)
    metric: api_requests_total
    good_event: status_code in [200, 201, 204]
    excludes:
      - 429 (rate limited — client error, not service failure)
      - 4xx (client validation errors)
      - 503 (deliberate maintenance mode — excluded if budgeted)
    window: 28 days rolling

  - name: "API Latency (p99)"
    type: latency
    measurement: request duration at edge proxy
    metric: api_request_duration_ms
    good_event: duration <= 1000ms
    window: 28 days rolling

  - name: "API Latency (p95)"
    type: latency
    measurement: request duration at edge proxy
    metric: api_request_duration_ms
    good_event: duration <= 300ms
    window: 28 days rolling

  - name: "Correctness"
    type: correctness
    measurement: idempotency key violations & failed rollbacks
    metric: payment_processing_errors_total
    good_event: zero correctness errors
    window: 28 days rolling

slo_targets:
  - sli: "API Availability"
    target: 99.99%

  - sli: "API Latency (p99)"
    target: "p99 <= 1000ms, 95% of the time"

  - sli: "API Latency (p95)"
    target: "p95 <= 300ms, 90% of the time"

  - sli: "Correctness"
    target: "Zero correctness errors (budget: 5 errors per window)"

error_budget:
  availability: 0.01% of total API requests (~87 errors per 1M)
  latency_p99: 5% of requests may exceed 1000ms
  latency_p95: 10% of requests may exceed 300ms
  correctness: 5 errors total per 28-day window

alerting:
  - name: "Availability Critical"
    burn_rate: >= 14.0
    windows: [1m, 5m]
    severity: page

  - name: "Latency p99 Critical"
    burn_rate: >= 6.0
    windows: [5m, 30m]
    severity: page

  - name: "Correctness Violation"
    type: count-based
    threshold: >= 1 correctness error in 1 hour
    severity: page
    response: "Immediate investigation — possible data corruption"

release_gating:
  condition: "Availability error budget remaining >= 30%"
  exception: "Emergency security patches exempt"
```

---

## Quick Reference Formulas

| Concept | Formula |
|---|---|
| Availability | \( A = \frac{\text{success}}{\text{total}} \times 100\% \) |
| Error rate | \( E = \frac{\text{errors}}{\text{total}} \) |
| Error budget | \( B = (1 - \text{SLO}) \times \text{total\_events} \) |
| Error budget consumed | \( C = \frac{\text{actual\_errors}}{\text{allowed\_errors}} \times 100\% \) |
| Budget remaining | \( R = \text{allowed\_errors} - \text{actual\_errors} \) |
| Burn rate | \( \text{BR} = \frac{\text{observed\_error\_rate}}{1 - \text{SLO}} \) |
| Time to exhaustion | \( T_{\text{exhaust}} = \frac{R}{\text{depletion\_rate}} \) |
| Journey SLO | \( \text{SLO}_{\text{journey}} = \prod_{i=1}^{n} \text{SLO}_i \) |
| Errors per nines | \( \text{errors\_allowed} = (1 - \text{SLO}) \times 10^n \) |

---

## References

- Google SRE Book — *"Service Level Objectives"* (Ch. 4)
- Google SRE Workbook — *"Implementing SLOs"* (Ch. 1–4)
- Site Reliability Engineering: Measuring and Managing Reliability — Jones et al. (O'Reilly)
- CRE Life Lessons — Google Cloud Blog series
- Error budget burn rate best practices — Google SRE (https://sre.google/workbook/alerting-on-slos/)

# Source index

- **Source repository:** https://github.com/magnus919/hermes-profiles
- **Inspected commit:** `867a555`
- **Imported source directory:** `site-reliability-engineering`
- **Porting boundary:** Retained portable methodology, templates, scripts, and references. Removed or generalized Hermes profile, task-orchestration, memory, and rigid response-handoff assumptions.

## Enrichment sources

- *Seeking SRE*, edited by David N. Blank-Edelman, O'Reilly, 2018. Local source supplied by the user: `seekingsre.epub`.
- *The Site Reliability Workbook*, Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, and Stephen Thorne, O'Reilly, 2018. Local source supplied by the user: `thesitereliabilityworkbook.epub`.

The enrichment references paraphrase and synthesize the supplied books' implementation lessons. They do not reproduce chapter text. Chapter-level source anchors are recorded in each new reference so an agent can distinguish book-derived practice from the skill's own synthesis.

# SRE Adoption, Engagement, and Team Design

This reference turns SRE from a named team into a set of adoptable operating practices. Use it when an organization is starting SRE, lacks a dedicated SRE team, is introducing SRE into an enterprise, or needs to decide how a reliability engagement should begin and end.

## Source anchors

This guidance is a synthesis of *The Site Reliability Workbook*, especially “How SRE Relates to DevOps,” “SRE Engagement Model,” “SRE Team Lifecycles,” and “Organizational Change Management in SRE,” and *Seeking SRE*, especially “Context Versus Control in SRE,” “So, You Want to Build an SRE Team?,” “How to Apply SRE Principles Without Dedicated SRE Teams,” “SRE Without SRE: The Spotify Case Study,” “Introducing SRE in Large Enterprises,” “Clearing the Way for SRE in the Enterprise,” and “Using Incident Metrics to Improve SRE at Scale.” These are paraphrased implementation lessons, not a replacement for the books.

## First decision: adopt practices or create a team?

Do not begin by renaming an operations team. Establish the problem and the smallest useful intervention:

1. Name the user and business consequence of unreliable behavior.
2. Establish one or two user-facing SLIs and an initial SLO, even if measurement is imperfect.
3. Make the current operational load visible: pages, tickets, manual changes, support escalations, and repair work.
4. Pick one service or user journey with an owner who can act on the measurements.
5. Run a short review cycle, then decide whether the evidence justifies a standing SRE engagement, embedded reliability role, platform work, or continued team-owned practice.

An SRE title without engineering authority, time for automation, or a way to change the service creates a rebranded operations queue. The intervention must have a path from operational evidence to engineering change.

## Engagement lifecycle

Treat reliability work as a lifecycle rather than an indefinite support commitment.

### 1. Discovery

Capture the product's critical user journeys, dependencies, current failure modes, operational work, ownership boundaries, and existing telemetry. Ask the product team what users must be able to do, not only which components are deployed.

Output:
- service and dependency map;
- named service and product owners;
- initial user journeys and candidate SLIs;
- current incident, toil, and capacity evidence;
- explicit risks and unknowns.

### 2. Productionization

Before general availability, address capacity, redundancy, overload behavior, monitoring, alerting, runbooks, rollback, data recovery, and escalation. Define SLOs before launch when possible. A service is not production-ready because it has a health endpoint or a green deployment; it is ready when the team can detect, mitigate, explain, and recover from plausible failures.

### 3. Operate and improve

Review SLO performance, error-budget consumption, operational load, incident repairs, dependency health, and user impact on a fixed cadence. Convert repeated incidents and manual work into owned engineering items. Keep the service team responsible for decisions that require product context; SRE supplies methods, evidence, and engineering leverage.

### 4. Offboard or renew

An engagement should have exit criteria. Offboard when the service team can operate the service sustainably, owns its SLOs and runbooks, and has a reliable path for escalation. Renew or deepen the engagement when evidence shows unresolved reliability risk, growing toil, or a cross-cutting failure pattern.

## Context over control

Prefer decision context to opaque permission gates. A release or operational decision should show:

- the relevant SLO and current error-budget state;
- the affected user journey and dependency path;
- recent changes and known risks;
- the expected blast radius and rollback or mitigation;
- who owns the decision and when it will be revisited.

Controls are still justified for irreversible, high-blast-radius, security-sensitive, or legally constrained actions. The improvement is to make the reason and evidence visible rather than asking people to obey an unexplained process. Context supports judgment; it does not mean removing guardrails.

## Models when there is no dedicated SRE team

A small organization can apply SRE without a central SRE department. Select a model deliberately:

| Model | Strength | Risk to manage |
|---|---|---|
| Service-team ownership | Product context stays close to the people who build the service. | Reliability work loses to feature work unless SLOs, review time, and ownership are explicit. |
| Embedded reliability engineer | Fast transfer of methods and context. | The engineer becomes permanent escalation coverage unless an exit plan exists. |
| Central platform or enablement team | Reusable tooling, standards, and cross-service learning. | It becomes a ticket queue or a gatekeeper detached from user impact. |
| Cross-team working group | Useful for incident practice, deployment safety, and shared standards. | Shared ownership can become no ownership. Assign accountable service owners. |
| Hybrid | Central leverage plus local service ownership. | Interfaces must specify who decides, operates, and funds the work. |

Never infer that one model is universally correct. Use incident, toil, dependency, and change data to choose and revisit the model.

## Team lifecycle and maturity signals

Assess maturity by observable capability, not team size or title. A useful progression is:

1. **Aspirational:** reliability concerns are named, but user-facing objectives and ownership are unclear.
2. **Measured:** critical journeys, initial SLIs/SLOs, incident records, and operational load are visible.
3. **Managed:** error budgets influence change, alerts are actionable, and incident response is rehearsed.
4. **Engineering-led:** toil and repair work are reduced through projects; service design includes reliability.
5. **Scaled:** cross-service dependencies, capacity, change risk, and organizational load are managed as a system.

A maturity claim needs evidence: current SLO reports, alert quality, incident learning, closed repairs, toil trend, recovery exercises, or review outputs.

## Change-management sequence

For organizational adoption:

1. **Diagnose:** identify the failure in the current operating model and the people affected.
2. **Frame:** connect SRE practices to user outcomes, engineering capacity, and business risk, not fashion.
3. **Pilot:** choose a bounded service and a few practices that can produce visible evidence.
4. **Teach:** give teams working examples, exercises, and time to practice, not only policy documents.
5. **Measure:** compare incidents, toil, recovery, change outcomes, and user-facing reliability before and after.
6. **Adapt:** preserve what works, remove ceremony that does not, and publish the reasoning.
7. **Scale:** expand only after the pilot demonstrates a repeatable path and accountable ownership.

Separate organizational change from change-control mechanics. A change-management program is not a justification for making every production change slow or centralized.

## Incident and repair metrics at scale

Use incident data to target investment, not to rank teams. Normalize where possible by service size, traffic, or exposure. Useful dimensions include:

- user-impact duration and affected journeys;
- detection and mitigation time;
- trigger and contributing conditions;
- repeat incidents and dependency involvement;
- change-related incidents;
- unresolved repair age and owner;
- operational load and interruption rate.

Log repairs as normal engineering work with owners, priority, due dates, and verification criteria. An initial increase in recorded repairs may mean the organization has made hidden debt visible, not that reliability has worsened.

## Anti-patterns

- Creating an SRE team before deciding what reliability problem it owns.
- Rebranding sysadmins while leaving them with all operational work and no engineering time.
- Centralizing every decision in SRE instead of giving service teams usable context.
- Treating SLOs as a report card without connecting them to action.
- Scaling a pilot by copying its ceremonies without copying its evidence and ownership.
- Measuring incident counts alone, which rewards under-reporting and ignores impact.
- Making a permanent support team from an engagement that has no exit criteria.

## Agent procedure

When asked to design or review an SRE operating model, load this reference with `slo-sli-framework.md`, `toil-elimination.md`, and `product-focused-reliability.md`. Produce an engagement boundary, accountable owners, evidence baseline, pilot scope, review cadence, and exit criteria. State which conclusions are observed, inferred, or still unknown.

# Google SRE Book — Comprehensive Chapter Summaries

> **Source:** *Site Reliability Engineering: How Google Runs Production Systems* (Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy, eds.)
> **Online:** https://sre.google/sre-book/
> **Purpose:** Quick reference for SRE agents and practitioners — core theses, key principles, actionable takeaways, and direct quotes from each chapter.

---

## Table of Contents

- [Part I: Introduction](#part-i-introduction)
  - [Chapter 1: Introduction](#chapter-1-introduction)
  - [Chapter 2: The Production Environment at Google, from the Viewpoint of an SRE](#chapter-2-the-production-environment-at-google-from-the-viewpoint-of-an-sre)
- [Part II: Principles](#part-ii-principles)
  - [Chapter 3: Embracing Risk](#chapter-3-embracing-risk)
  - [Chapter 4: Service Level Objectives](#chapter-4-service-level-objectives)
  - [Chapter 5: Eliminating Toil](#chapter-5-eliminating-toil)
  - [Chapter 6: Monitoring Distributed Systems](#chapter-6-monitoring-distributed-systems)
  - [Chapter 7: The Evolution of Automation at Google](#chapter-7-the-evolution-of-automation-at-google)
  - [Chapter 8: Release Engineering](#chapter-8-release-engineering)
  - [Chapter 9: Simplicity](#chapter-9-simplicity)
- [Part III: Practices (Key Chapters)](#part-iii-practices)
  - [Chapter 10: Practical Alerting](#chapter-10-practical-alerting)
  - [Chapter 11: Being On-Call](#chapter-11-being-on-call)
  - [Chapter 12: Effective Troubleshooting](#chapter-12-effective-troubleshooting)
  - [Chapter 15: Postmortem Culture: Learning from Failure](#chapter-15-postmortem-culture-learning-from-failure)

---

# Part I: Introduction

---

## Chapter 1: Introduction

### Core Thesis

SRE is "what happens when a software engineer is tasked with designing an operations team." It is a discipline that applies a software engineering mindset to system administration and production operations. The chapter establishes the philosophy, organizational model, and key distinctions between SRE and traditional IT operations.

### Key Principles

| # | Principle | Description |
|---|-----------|-------------|
| 1 | **Hiring from Software Engineering** | SREs are software engineers who design and build automation to run systems — they don't manually operate them |
| 2 | **Operations as a Software Problem** | Toil reduction is achieved through code, not process |
| 3 | **Shared Ownership** | SRE shares responsibility for system health with product/dev teams |
| 4 | **The 50% Cap on Operational Work** | SRE teams spend at most 50% of time on ops; the remainder goes to engineering projects that reduce future toil or add reliability features |
| 5 | **Error Budget** | 100% reliability is the wrong target; acceptable risk is quantified and managed as a budget |
| 6 | **SLOs Drive Decisions** | Releases are gated on error budget burn rate, not arbitrary freezes |

### The SRE <-> Dev Relationship

```
┌─────────────────────────────────────────────────────────┐
│  DEV TEAM                    SRE TEAM                    │
│  ┌─────────┐                ┌─────────┐                  │
│  │ Feature │                │Stability│                  │
│  │ velocity│                │reliability│                 │
│  └────┬────┘                └────┬────┘                  │
│       │                          │                        │
│       └───────── Error ──────────┘                       │
│                      Budget                                │
│                                                           │
│  Shared responsibility within error budget tolerance      │
└─────────────────────────────────────────────────────────┘
```

### The Seven Levels of SRE Maturity (Informal)

1. **Dev ops model** — devs push code, ops runs it, ad-hoc communication
2. **SRE is hired** — first SREs join, begin measuring and automating
3. **SLOs defined** — explicit reliability targets and error budgets
4. **Error budget-driven releases** — releases blocked when budget exhausted
5. **Automation replaces toil** — >50% of ops work automated
6. **Product reliability as a feature** — SRE expertise embedded in design phase
7. **Self-healing systems** — systems adapt to failure without human intervention

### Actionable Takeaways

- **For leadership:** Hire SREs from software engineering, not sysadmin backgrounds. Enforce the 50% ops cap.
- **For teams:** Define an error budget before defining SLOs. This changes the conversation from "how do we prevent all outages?" to "how much downtime is acceptable?"
- **For engineers:** Treat operations as a software engineering problem. If you fix a thing manually three times, automate it.

### Important Quotes

> "The overriding goal of SRE is to make the systems run themselves."

> "SRE is a profession that encompasses both software engineering and operations — it's what happens when a software engineer is tasked with designing an operations team."

> "A fundamental principle of SRE is that operations is a software problem."

> "At Google, SRE is fundamentally doing work that has two aspects: (1) making Google's system more reliable, and (2) making Google's engineering teams more productive."

> "The SRE model is predicated on the idea that a system is never 'done' — it is always evolving, and operations is the day-to-day task of managing that evolution."

---

## Chapter 2: The Production Environment at Google, from the Viewpoint of an SRE

### Core Thesis

To understand SRE, one must understand the environment it operates in. This chapter describes Google's hardware infrastructure, the software stack that runs on it, and the lifecycle of a user request — from DNS lookup to serving an ad or search result.

### Key Concepts

| Concept | Description |
|---------|-------------|
| **Borg** | Google's cluster management system (predecessor to Kubernetes). Schedules jobs onto machines, handles failures, manages resources |
| **Borg Name Service (BNS)** | Naming and service discovery layer used by all Google services |
| **GSLB (Global Software Load Balancer)** | DNS-based traffic distribution across datacenters |
| **Megastore / Spanner** | Globally distributed storage systems providing strong consistency |
| **Chubby** | Distributed lock service (based on Paxos) used for leader election and configuration |

### The Lifecycle of a Request

```
User → DNS → GSLB → Frontend → Backend → Storage
         │        │         │         │        │
    Latency-based  │    RPC load   BNS-based  Megastore
    load balancing │    balancing  discovery  / Spanner
                   │                             │
              HTTP(S)                          Colocation
              termination                       & caching
```

### Software Infrastructure Layers

1. **Networking** — Jupiter datacenter network fabric, B4 WAN
2. **Storage** — Colossus (GFS v2), Bigtable, Spanner, Megastore
3. **Compute** — Borg, Omega, Kubernetes (later)
4. **Service Infrastructure** — gRPC, Protocol Buffers, Stubby

### Key Insight for SREs

The physical infrastructure is abstracted to such a degree that an SRE rarely thinks about individual machines. The unit of management is the **service** — a set of jobs running on Borg that collectively provide an API. Failures are expected; the system is designed to route around them.

### Important Quotes

> "The sheer scale of Google's infrastructure is difficult to overstate. More than a billion user requests hit Google's servers in a single minute."

> "If you have to think about which machine your code is running on, you're doing it wrong."

> "When machines fail — and they do, every day — the system automatically shifts work to healthy machines."

---

# Part II: Principles

---

## Chapter 3: Embracing Risk

### Core Thesis

100% reliability is neither achievable nor desirable. The cost of reliability increases non-linearly — the last 0.01% requires orders of magnitude more investment than the first 99.99%. SRE explicitly manages reliability as a **trade-off** against feature velocity, cost, and complexity.

### Core Thesis (Extended)

**The key insight:** If a service is "too reliable," you are wasting resources that could be spent on features, reducing latency, or improving other services. The goal is to find the *optimal* level of reliability, not the *maximum*.

### Key Principles

| # | Principle | Explanation |
|---|-----------|-------------|
| 1 | **Reliability is a spectrum** | Not binary (reliable vs. unreliable). SRE defines where on the spectrum a service should sit |
| 2 | **Non-linear cost curve** | Each "nine" of availability costs roughly 10x the previous one |
| 3 | **Error budgets align incentives** | Devs want to ship; SREs want stability. The error budget gives both sides a shared, measurable goal |
| 4 | **Risk tolerance is service-specific** | Gmail can have different SLOs than Google Ads; tier-1 services have tighter budgets |
| 5 | **Risk changes over time** | A young service may tolerate more risk; a mature service should tighten its SLOs |

### Managing Risk Targets (Flowchart)

```
              ┌──────────────────────┐
              │ Identify service need │
              └──────────┬───────────┘
                         │
              ┌──────────▼───────────┐
              │ What level of         │
              │ availability does the  │
              │ service require?       │
              └──────────┬───────────┘
                         │
              ┌──────────▼───────────┐
              │ What would 100% cost? │
              └──────────┬───────────┘
                         │
              ┌──────────▼───────────┐
              │ Define SLO & error    │
              │ budget accordingly    │
              └──────────┬───────────┘
                         │
              ┌──────────▼───────────┐
              │ Measure & iterate     │
              │ (quarterly reviews)   │
              └──────────────────────┘
```

### Actionable Takeaways

- **Define explicit risk tolerance** for every service. Write it down. If you can't state it, you can't measure it.
- **Calculate error budget monthly/daily:** `(1 - SLO) * total requests = allowed errors`. Track burn rate.
- **Use error budget to gate releases:** When the budget is spent (or nearly spent), stop all changes until it replenishes.
- **Adapt targets to service tier:** Tier 1 (user-facing, revenue-critical) gets 99.99%; Tier N (internal tools, batch) might get 99.9% or less.
- **Monitor unavailability carefully:** Not all downtime is equal. Partial degradation may cause more user harm than a hard outage.

### Important Quotes

> "If an SRE team spends all their time making a service 100% reliable, they're doing it wrong. The most reliable service is one that doesn't change — but nobody uses it."

> "The risk of a system failing is directly proportional to the cost of preventing that failure. The trick is to optimize for the 'acceptable' level of risk, not zero risk."

> "An error budget is a tool for rational discussion about risk between product development and SRE."

> "Managing risk is about deciding what *not* to do as much as what to do."

> "Google's experience suggests that 100% is the *wrong* reliability target for (nearly) everything."

---

## Chapter 4: Service Level Objectives

### Core Thesis

SLOs are the contract between SRE and the product team — and between the service and its users. This chapter details how to define, measure, and use **Service Level Indicators (SLIs)**, **Service Level Objectives (SLOs)**, and **Service Level Agreements (SLAs)**. Without good SLOs, you cannot know whether the service is "healthy" or not.

### Key Definitions

| Term | Definition | Example |
|------|------------|---------|
| **SLI** | A carefully defined *quantitative measure* of some aspect of the service level | Request latency at 95th percentile |
| **SLO** | A *target value or range* for an SLI that represents acceptable service | 99% of requests < 200ms at p95 |
| **SLA** | An explicit or implicit *contract* with users that includes consequences for breach | 99.9% uptime; failure = service credits |

### SLI Selection Framework

```
┌───────────────────────────────────────┐
│       What matters to users?          │
│  ┌──────────┬──────────┬──────────┐   │
│  │ Latency  │Throughput│Availabil-│   │
│  │          │          │   ity    │   │
│  ├──────────┼──────────┼──────────┤   │
│  │  p50,    │  QPS,    │  Uptime, │   │
│  │  p95,    │  req/s   │  success │   │
│  │  p99     │          │   rate   │   │
│  └──────────┴──────────┴──────────┘   │
│  ┌──────────┐                          │
│  │ Durability│  (for storage systems)  │
│  └──────────┘                          │
│  ┌──────────┐                          │
│  │Correctness│  (for transactions)     │
│  └──────────┘                          │
└───────────────────────────────────────┘
```

### SLI Measurement Approaches

| Approach | Method | Use Case |
|----------|--------|----------|
| **Server-side** | Aggregate from application logs | Controlled, accurate |
| **Client-side** | Synthetic probes (RPCs) | Reflects actual user experience |
| **Client-side (real)** | Browser instrumentation (e.g., RUM) | Best for latency, hardest to implement |

### Common SLIs at Google

| SLI | Definition | Target |
|-----|-----------|--------|
| Availability | `(successful requests / total requests) * 100` | ≥ 99.9% |
| Latency (p50) | Median response time | < 100ms |
| Latency (p95) | 95th percentile response time | < 200ms |
| Latency (p99) | 99th percentile response time | < 500ms |
| Throughput | Requests per second | Variable |
| Error rate | Count or fraction of 5xx responses | < 0.1% |

### SLO Construction Rules

1. **Choose few SLOs.** More than 3-4 SLOs per service leads to dilution of focus.
2. **Prefer "loose" SLOs initially.** You can always tighten them; loosening an SLO means admitting you can't meet your target.
3. **Don't aim for 100%.** Pick a target that reflects actual user needs.
4. **Use "target ± tolerance" for SLI windows.** Example: "p95 latency < 200ms over any rolling 5-minute window."
5. **SLOs are aspirational, SLAs are contractual.** SLO failure is a signal; SLA failure is a financial event.

### The SLO Burn Rate Model

```
┌──────────────────────────────────────────┐
│  Burn Rate = Error Budget consumed / time │
├──────────────────────────────────────────┤
│  Example:                                 │
│  99.9% SLO = 0.1% error budget / month   │
│  1% errors for 1 hour = ~1.4% of monthly │
│  budget consumed in 1 hour                │
├──────────────────────────────────────────┤
│  Alert thresholds:                        │
│  - Page when burn rate > 2x for 1 hour   │
│  - Page when burn rate > 10x for 10 mins │
│  - Ticket when burn rate > 1x for 1 day  │
└──────────────────────────────────────────┘
```

### Actionable Takeaways

- **Start with 2-3 SLIs per service.** Availability and latency cover 90% of use cases.
- **Measure from the user's perspective.** Server-side metrics alone lie — they don't capture network issues.
- **Make SLOs visible.** Dashboard them prominently, post them in team rooms, include them in every release review.
- **Use burn rate alerts** instead of static threshold alerts. They catch problems faster and with less noise.
- **Quarterly SLO reviews.** Services evolve; their SLOs should too.

### Important Quotes

> "You can't manage what you don't measure. More importantly, you can't optimize what you don't measure well."

> "SLOs are not just numbers — they are a statement of intent about how your service should behave, and they form the basis of conversations between SRE, product development, and management."

> "An SLO that users don't notice is probably too tight. An SLO that users complain about is probably too loose."

> "A common mistake is to define too many SLOs. The right number is small — ideally 3 (±1) — because each additional SLO reduces the attention paid to each one."

---

## Chapter 5: Eliminating Toil

### Core Thesis

**Toil** is the enemy of SRE. Defined as work that is manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly with service growth. SRE must actively eliminate toil to preserve the engineering time needed to build reliable systems.

### The Five Characteristics of Toil

| # | Characteristic | Description |
|---|---------------|-------------|
| 1 | **Manual** | Requires human touch (e.g., SSHing to a machine to restart a process) |
| 2 | **Repetitive** | Done over and over (e.g., responding to the same alert daily) |
| 3 | **Automatable** | A computer could do it — you just haven't written the code yet |
| 4 | **Tactical** | Puts out fires but doesn't improve the system |
| 5 | **No enduring value** | Once done, the system is no better than before |
| 6 | **Scales linearly** | Doubling the service size doubles the toil |

### The Toil Taxonomy

| Category | Examples | Impact |
|----------|----------|--------|
| **Manual operations** | Restarting jobs, reprovisioning VMs, clearing queues | Direct time drain |
| **Acknowledgment fatigue** | Reading non-actionable alerts, acknowledging pages for known false positives | Attentional debt |
| **Ticket handling** | Duplicate bug reports, user password resets, permission requests | Context switching |
| **Incident response** | Same outage pattern recurring weekly without a fix | No improvement |
| **Data munging** | Extracting logs by hand to answer ad-hoc questions | Repeated cognitive load |

### Measuring Toil

```
Toil Percentage = (Hours spent on toil / Total SRE hours) × 100

Target: < 50% (enforced by management)
Warning:  50-65% (needs intervention)
Critical: > 65%  (team is in firefighting mode)
```

### Strategies for Eliminating Toil

1. **Automate relentlessly.** Every manual step is a potential automation target. Write scripts, tools, or full systems to handle it.
2. **Build self-service tooling.** Give dev teams access to deploy, rollback, and monitor without SRE intervention.
3. **Improve system architecture.** Redesign components that generate disproportionate toil (e.g., replace stateful services with stateless ones).
4. **Say "no."** Not every request deserves SRE attention. Use the error budget model to push back.
5. **Instrument toil.** Track time spent on toil categories. What gets measured gets reduced.

### The 50% Ops Cap in Practice

```
┌─────────────────────────────────────────────┐
│  SRE Team Time Allocation (Goal)            │
├─────────────────────────────────────────────┤
│  ┌─────────────────────────┬───────────────┐ │
│  │  Project Work (≥ 50%)  │ Ops (≤ 50%)   │ │
│  │  ┌───────────────────┐ │ ┌───────────┐ │ │
│  │  │ Automation         │ │ │ Alerts    │ │ │
│  │  │ Architecture       │ │ │ Tickets   │ │ │
│  │  │ Capacity planning  │ │ │ On-call   │ │ │
│  │  │ Performance        │ │ │ Releases  │ │ │
│  │  │ Tooling            │ │ │ Manual    │ │ │
│  │  └───────────────────┘ │ └───────────┘ │ │
│  └─────────────────────────┴───────────────┘ │
└─────────────────────────────────────────────┘
```

**If ops exceeds 50%:** The team stops project work and declares operational bankruptcy. The *only* project work allowed is that which reduces toil.

### Actionable Takeaways

- **Track toil as a metric.** Add it to your team's dashboard. If you can't measure it, you can't cap it.
- **One-off fixes are acceptable.** The third time you do something manually, automate it.
- **Use "cutting the red wire" test:** If you're asked to do a manual operation under pressure, ask: "If the building was on fire, would this still need to be done by a human?"
- **Refuse to hire your way out of toil.** Adding more SREs to a toil-heavy team just increases the toil budget. Fix the *system*.

### Important Quotes

> "Toil is the enemy of SRE. It saps the energy, time, and creativity of the people who are supposed to be making the system better."

> "If a machine can do it, a machine should do it."

> "The 50% rule is the single most important organizational mechanism for making SRE work. Without it, SRE teams inevitably become operations teams."

> "Hiring more people to do a job a computer could do is not a strategy."

> "Toil is not just boring — it's dangerous. The more toil an SRE team does, the less time they have for the engineering work that prevents future outages."

---

## Chapter 6: Monitoring Distributed Systems

### Core Thesis

Monitoring is the foundation of SRE practice. This chapter presents Google's philosophy on monitoring: four golden signals, the difference between alerting, dashboarding, and debugging tools, and how to build monitoring that produces actionable signals — not noise.

### The Four Golden Signals

| Signal | What It Measures | Why It Matters |
|--------|-----------------|----------------|
| **Latency** | Time to service a request | High latency = unhappy users, cascading failures |
| **Traffic** | Demand on the system (QPS, connections, active users) | Predicts scaling needs |
| **Errors** | Rate of failed requests (explicit 5xx, implicit bad content, slow responses) | Direct indicator of system health |
| **Saturation** | How "full" the service is (CPU, memory, disk, connections, quota) | Predicts capacity failure |

### The Monitoring Stack at Google

```
┌──────────────────────────────────────┐
│          Alerting                     │
│  (Triggers pages, pages SRE on-call)  │
├──────────────────────────────────────┤
│          Dashboards                   │
│  (ROI: answer "what's broken?" fast)  │
├──────────────────────────────────────┤
│          Debugging Tools              │
│  (Trace analysis, log explorers)      │
├──────────────────────────────────────┤
│          Data Pipeline                │
│  (Log collection, aggregation, query) │
├──────────────────────────────────────┤
│          Instrumentation              │
│  (Application metrics emitted)        │
└──────────────────────────────────────┘
```

### Rules for Good Alerting

| Rule | Explanation |
|------|-------------|
| **Rule 1** | Every page must be actionable (a human must do something) |
| **Rule 2** | Every page must be urgent (the thing can't wait until morning) |
| **Rule 3** | Every page must be novel (not the same problem that pages every night) |
| **Rule 4** | Tickets for non-urgent issues; pages only for emergencies |

### Categorizing Monitoring Output

| Category | Action | Latency Tolerance | Example |
|----------|--------|-------------------|---------|
| **Page** | Wake someone up | Seconds to minutes | Service down, error budget burning fast |
| **Ticket** | Create a work item | Hours to days | Certificates expiring, disk filling up |
| **Logging** | No immediate action | None | Debugging data for postmortems |
| **Dashboard** | Visual for ad-hoc review | N/A | Real-time latency heatmap |

### Reducing Alert Noise

```
Common pattern at Google:
  - ~10,000 raw monitoring rules defined
  - ~100 alerts per day after dedup
  - ~5-10 pages per shift (before tuning)
  - Target: 0 false-positive pages per shift
```

### Actionable Takeaways

- **Monitor symptoms, not causes.** "My database is slow" is a cause. "User requests timeout" is a symptom. Alert on symptoms.
- **Prefer black-box monitoring** (from the user's perspective) over white-box monitoring (internal metrics) for alerting.
- **Use white-box monitoring** for debugging and capacity planning.
- **Apply the "alerts as a product" mindset.** Every alert is a product that should be tested, documented, and have an owner.
- **Delete alerts that never fire.** If an alert hasn't fired in 6 months, remove it. If you need it later, add it back.

### Important Quotes

> "When a system is on fire, you do not have time to look at a graph. You need a pager."

> "The best monitoring is the kind that never pages you because the problem was already fixed by automation."

> "If you page a human for every anomaly, you train them to ignore the pager."

> "Monitoring should never require a human to interpret 'what's wrong.' The alert should tell them."

> "A dashboard that nobody looks at is as useful as no dashboard at all. Build monitoring for the signal, not the noise."

---

## Chapter 7: The Evolution of Automation at Google

### Core Thesis

Automation is the SRE's primary tool for eliminating toil, reducing error, and enabling scale. This chapter traces Google's automation journey from manual scripts through Borg to the fully automated release pipeline — and extracts the general principles that apply to any organization.

### Levels of Automation Maturity

| Level | Description | Example |
|-------|-------------|---------|
| **1 — Manual** | No automation. Every step is done by a human | SSH to machine, run command |
| **2 — Scripted** | Single steps automated with scripts | Shell script that runs a series of SSH commands |
| **3 — Templated** | Repeatable patterns with parameters | Ansible playbooks, deployment templates |
| **4 — Monitored** | Automation observes and adjusts | Auto-scaling, self-healing service |
| **5 — Autonomous** | System manages itself fully | Borg: schedules work, recovers from failures, allocates resources |

### The Automation Decision Framework

```
Before automating, ask:
┌──────────────────────────────┐
│  1. Will this save time?     │
│     (time saved > time to    │
│      build + maintain)       │
├──────────────────────────────┤
│  2. Will this reduce errors? │
│     (human error rate vs.    │
│      software error rate)    │
├──────────────────────────────┤
│  3. Will this let humans     │
│     focus on higher-value    │
│     work?                    │
├──────────────────────────────┤
│  4. Does the process change  │
│     frequently?              │
│     (if yes, automate later) │
└──────────────────────────────┘
```

### Benefits of Automation

| Benefit | Explanation |
|---------|-------------|
| **Consistency** | The same action is performed identically every time |
| **Speed** | Machines work faster than humans, especially for multi-step operations |
| **Reliability** | Well-tested automation fails less than a tired human at 3 AM |
| **Scalability** | Automation costs do not increase linearly with service growth |
| **Human focus** | Frees SREs to work on architecture, performance, and features |

### Automation Pitfalls

1. **Automating the wrong thing** — Don't automate a process you plan to eliminate.
2. **Brittle automation** — Over-specific scripts break when the environment changes.
3. **Bypassing expertise** — Junior engineers may blindly trust automation without understanding what it does.
4. **Black box syndrome** — Automated systems become so complex that nobody knows how to fix them when they break.
5. **Loss of skills** — When nobody does a task manually anymore, the knowledge of *how* it works is lost.

### The Borg Automation Story (Case Study)

```
Manual (2003): Engineers submit jobs via email, manually assign machines
    │
Scripted (2004): Simple scheduler allocates machines
    │
Borg (2005): Full cluster management — scheduling, health checking, restart
    │
Omega (2012): Next-gen scheduler with better isolation and performance
    │
Kubernetes (2014): Open-source version of Borg's principles
```

### Actionable Takeaways

- **Automate in layers.** Start with the most painful, highest-frequency manual tasks.
- **Don't aim for level 5 immediately.** Level 3 (templated) captures 80% of the value.
- **Keep automation debuggable.** Every automated step should log what it did and why.
- **Write automation as services, not scripts.** A service can be monitored, tested, and evolved; a script sits on someone's laptop.
- **Plan for maintenance.** The cost of maintaining automation is real. Factor it into the ROI calculation.

### Important Quotes

> "Automation is not just about efficiency. It's about enabling humans to do more valuable work."

> "The most reliable system is one that never requires a human being to take action."

> "Automation is a force multiplier. A single SRE with good automation can manage thousands of machines. Without it, they can manage dozens."

> "Beware of automating inefficient processes. You'll just get broken things faster."

> "Google's experience has shown that the best automation is the kind that doesn't need to be watched. If you have to monitor your automation, you've just shifted the toil, not eliminated it."

---

## Chapter 8: Release Engineering

### Core Thesis

Release engineering is the discipline of building and deploying software **reliably, repeatably, and at scale.** Google's approach treats the release process as a software engineering problem in its own right — with defined roles, automated pipelines, and rigorous testing at every stage.

### The Release Engineering Pipeline

```
┌─────────┐   ┌──────────┐   ┌─────────┐   ┌──────────┐   ┌───────────┐
│  Dev    │→│  Commit │→│  Build  │→│  Test   │→│  Deploy  │
│  Branch │   │  Queue  │   │  (Bazel) │   │  Suite  │   │  (Sisyphus) │
└─────────┘   └──────────┘   └─────────┘   └──────────┘   └───────────┘
                    │                            │                │
                    │                    ┌───────┴───────┐       │
                    │                    │  Unit / Int   │       │
                    │                    │  / System /   │       │
                    │                    │  Canary /     │       │
                    │                    │  Prod         │       │
                    │                    └───────────────┘       │
                    │                                   ┌───────┴───────┐
                    │                                   │  Canary → 5%  │
                    │                                   │  → 25% → 50%  │
                    │                                   │  → 100%       │
                    │                                   └───────────────┘
```

### Key Concepts

| Concept | Description |
|---------|-------------|
| **CI (Continuous Integration)** | Every commit is built and tested automatically |
| **CD (Continuous Delivery)** | Software is always in a deployable state |
| **Progressive Rollout** | Deploy to small subsets of users first, then expand |
| **Rollback** | Reverting to a known-good version is as important as rolling forward |
| **Bazel** | Google's build system — deterministic, hermetic, scalable |
| **Sisyphus** | Google's deployment system — manages fleet updates with minimal disruption |

### The Release Cadence Model

```
┌─────────────────────────────────────┐
│  Google's approach:                 │
│                                     │
│  - Single mainline branch           │
│  - Every commit must pass CI        │
│  - Release candidates cut on        │
│    schedule (e.g., weekly)          │
│  - Candidates go through test       │
│    pipeline (hours to days)         │
│  - If candidate fails, next one     │
│    is cut from latest mainline      │
│  - Deploy progression:              │
│    Canary → 5% → 25% → 50% → 100%  │
│  - Rollback is a first-class        │
│    operation                        │
└─────────────────────────────────────┘
```

### Binary vs. Configuration Releases

| Aspect | Binary Release | Configuration Release |
|--------|----------------|----------------------|
| **Frequency** | Weekly/biweekly | Daily or more |
| **Risk** | Higher (new code) | Lower (tuning existing code) |
| **Rollback** | Full deploy pipeline | Instant revert |
| **Testing** | Full CI/CD pipeline | Config schema validation |
| **Tooling** | Bazel + Sisyphus | Rapid evaluation + push |

### Actionable Takeaways

- **Make releases boring.** A good release process should be so reliable that nobody worries about it. Drama in releases = process failure.
- **Invest in build infrastructure.** A slow or unreliable build is a productivity killer.
- **Staged rollouts are non-negotiable.** Always deploy to a canary. Always. 100% deployment on day one is a crash waiting to happen.
- **Test rollbacks as thoroughly as rollouts.** If you can't roll back, you can't safely roll forward.
- **Treat configuration as code.** Version it. Review it. Test it. Roll it back.

### Important Quotes

> "Release engineering is not an afterthought — it's a first-class engineering discipline."

> "A release process that requires manual steps is not a process; it's a ceremony."

> "The best release engineer is the one nobody ever has to page."

> "If you're not testing your rollback procedure, you don't have a rollback procedure — you have a wish."

> "Configuration pushes are the most dangerous thing a team does. A single typo in a config file can take down a global service."

---

## Chapter 9: Simplicity

### Core Thesis

Complexity is the root cause of most reliability failures. SREs must actively fight complexity by designing simple systems, reducing accidental complexity, and embracing "boring" technology that is well-understood over novel but fragile solutions.

### Essential vs. Accidental Complexity

| Type | Definition | Example |
|------|-----------|---------|
| **Essential complexity** | Complexity inherent to the problem the system solves | A distributed consensus algorithm (Paxos/Raft) is inherently complex |
| **Accidental complexity** | Complexity introduced by the implementation choices | Using five different caching layers when one would do |

### The Virtues of "Boring" Technology

```
┌─────────────────────────────────────────────┐
│  Reliable technology = boring technology     │
├─────────────────────────────────────────────┤
│  "Boring" means:                             │
│  ✅ Well-understood by the team               │
│  ✅ Extensively deployed in production        │
│  ✅ Well-documented failure modes             │
│  ✅ Mature tooling and monitoring             │
│  ✅ Simple to debug                           │
│                                              │
│  Risky choices:                              │
│  ❌ Cutting-edge but untested                 │
│  ❌ "Shiny" new framework                     │
│  ❌ Over-abstracted for the problem           │
│  ❌ More than one way to do everything        │
│  ❌ Dependencies on unvetted open-source      │
└─────────────────────────────────────────────┘
```

### The "Minimal API" Principle

```
Good API design for reliability:
  - Few endpoints (narrow surface area)
  - Idempotent operations (safe to retry)
  - Fail-fast on invalid input
  - Backward compatible by default
  - Explicit versioning

Don't design for every future use case.
Design for today's known use cases.
Tomorrow's can add surfaces.
```

### Fighting Complexity: Practical Tactics

| Tactic | Description |
|--------|-------------|
| **Code reviews with a simplicity lens** | Reviewers should ask "Is this simpler than it needs to be?" |
| **Prune dead code** | Delete features, flags, and configs no longer in use |
| **Standardize** | Use one build system, one deployment tool, one monitoring stack — not five |
| **Reduce dependencies** | Every dependency is a point of failure. Audit them regularly |
| **Prefer stateless** | Stateless services are simpler to scale, deploy, and debug |
| **Use feature flags** | Keep code paths explicit and reversible |

### The "Negative Lines of Code" Metric

```
"Software entropy" is real:
  - Every feature adds complexity
  - Every bug fix adds branches
  - Every workaround adds technical debt

Reliability practice:
  - Actively remove code that is no longer needed
  - Celebrate "negative lines of code" in reviews
  - Simpler code has fewer bugs
```

### Actionable Takeaways

- **Simplicity is a design goal, not an accident.** Explicitly allocate time to simplify existing systems.
- **Resist the temptation to over-engineer.** Use the simplest thing that works for your current scale.
- **Standardize ruthlessly.** If you have two tools that do the same thing, deprecate one.
- **Say "no" to features.** More features = more complexity = less reliability.
- **Document complexity.** When you can't avoid it, document *why* — and what would need to change to simplify it.

### Important Quotes

> "Simplicity is a prerequisite for reliability."

> "The best code is code that doesn't exist. The second-best code is code that is simple enough that everyone can understand it."

> "Boring technology is reliable technology. If it's boring, it's been around long enough that we know how it fails."

> "Complexity is like a gas — it expands to fill whatever container you give it. You must actively fight it."

> "Every time you add a new feature, you are making a bet: that the value of this feature outweighs the cost of the complexity it adds. Make sure you're winning that bet."

---

# Part III: Practices

---

## Chapter 10: Practical Alerting

### Core Thesis

Alerting is the primary interface between SRE and the production system — but it's often the most broken part of the monitoring stack. This chapter explains Google's approach to designing alerting systems that produce **few, high-signal pages** rather than constant noise.

### The Alerting Design Workflow

```
┌─────────────────────────────────────────────────────┐
│  Step 1: Understand the service's SLOs              │
│  Step 2: Identify what would violate those SLOs     │
│  Step 3: Design alerts that detect SLO violation    │
│         risk, not symptoms                          │
│  Step 4: Build runbooks for each alert              │
│  Step 5: Test each alert (at least quarterly)       │
│  Step 6: Delete alerts that never fire              │
│  Step 7: Review and tune alert thresholds           │
└─────────────────────────────────────────────────────┘
```

### Types of Alerts

| Alert Type | When It Fires | Action | Target Latency |
|------------|--------------|--------|----------------|
| **Page (Urgent)** | Error budget burning faster than expected | Wake up SRE and resolve | < 5 minutes |
| **Ticket (Non-urgent)** | Future risk detected (disk filling, cert expiring) | Create ticket, handle in business hours | < 24 hours |
| **Info / Log** | Something notable but not actionable | Log for postmortem / analysis | None |

### Alert Quality Metrics

| Metric | Description | Target |
|--------|-------------|--------|
| **Precision** | % of pages that lead to a real action | > 90% |
| **Recall** | % of real incidents captured by an alert | > 95% |
| **MTTA** | Mean time to acknowledge | < 5 min |
| **MTTR** | Mean time to resolve | Service-specific |
| **False positive rate** | Pages that required no action | < 10% |
| **Alert fatigue index** | Pages per shift per SRE | < 2 |

### Alerting Anti-Patterns

| Anti-Pattern | Description | Fix |
|-------------|-------------|-----|
| **The Kitchen Sink** | Alert on everything because you might miss something | Only alert on SLO violations |
| **The Boy Who Cried Wolf** | High false positive rate → alerts ignored | Cut precision to zero; rebuild with strict thresholds |
| **The Echo Chamber** | Multiple alerts for the same problem | Dedup/group alerts by incident |
| **The Spaghetti Graph** | Alerts that require interpretation to understand | Write explicit alert text |
| **The Zombie Alert** | Alert that hasn't fired in 6+ months | Delete it. Add it back only if needed |

### Sample Runbook Template

```markdown
# Alert: High Error Rate (>5% 5xx responses, 5 min window)

## Symptoms
- Users report errors
- Elevated 5xx rate on monitoring dashboard

## Impact
- P0: Error budget depleting rapidly
- Estimated: X% of users affected

## Immediate Steps
1. Acknowledge the page.
2. Check the error dashboard: [link]
3. Identify common characteristics of failed requests:
   - Which datacenter?
   - Which endpoint?
   - Which user cohort?
4. If single datacenter: drain and redirect traffic.
5. If global: check for recent deploy or config push.
6. Rollback the most recent change.
7. Monitor error rate for recovery (expect 5-10 min).

## Escalation
If unresolved after 15 minutes: escalate to {secondary on-call}.
If unresolved after 30 minutes: escalate to {team lead}.

## Postmortem
File a bug with:
- Duration of elevated error rate
- Root cause
- Monitoring gaps discovered
- Action items
```

### Actionable Takeaways

- **Every alert must have a runbook.** If you need to figure out what to do, it's not a page — it's a multi-car pileup.
- **Alert on symptoms, not causes.** Alert on "user requests failing", not "database connection pool full."
- **Use burn-rate alerts.** Instead of "error rate > X", use "error budget consumed > Y% in Z minutes."
- **Test your alerts.** Set up synthetic failures quarterly to verify alerts fire correctly.
- **Track your alert quality metrics.** Bad alerting is worse than no alerting.

### Important Quotes

> "A good alert is one that fires rarely, tells you exactly what's wrong and what to do about it, and lets you go back to sleep."

> "If your pager goes off and you don't know why, that's a systems failure, not a personal one."

> "The goal of alerting is not to catch every possible problem. It's to catch the problems that require immediate human action."

> "Every alert that doesn't result in action is training your team to ignore the pager."

---

## Chapter 11: Being On-Call

### Core Thesis

On-call is the most visible, most stressful responsibility of an SRE. This chapter describes Google's practices for structuring on-call rotations, managing the cognitive load, maintaining work-life balance, and creating a sustainable on-call culture.

### The On-Call Rotation Model

```
Google's typical approach:

  ┌───────────┐  ┌───────────┐  ┌───────────┐
  │  Primary  │  │  Primary  │  │  Primary  │  ...
  │  (Week 1) │  │  (Week 2) │  │  (Week 3) │
  └───────────┘  └───────────┘  └───────────┘
        │               │               │
        └───────┬───────┘               │
                │                       │
        ┌───────▼───────┐       ┌───────▼───────┐
        │  Secondary    │       │  Secondary    │
        │  (Same week)  │       │  (Same week)  │
        └───────────────┘       └───────────────┘

  Key parameters:
  - Shift length: 12-24 hours (day) or full week (24/7 rotation)
  - Team size: minimum 6-8 for a 24/7 rotation
  - Primary handles pages; secondary is escalation/backup
  - No-shift is immediately followed by a rest period
```

### On-Call Balance Rules

| Rule | Description |
|------|-------------|
| **1. Manage stress** | On-call should not be a constant state of emergency. If it is, the system is broken |
| **2. Fair rotation** | All team members share the on-call burden equally |
| **3. No retaliation** | Pages caused by developer changes do not result in blame on the developer |
| **4. Time off** | After a particularly bad on-call shift, a compensatory day off |
| **5. Escalation exists** | No one should be expected to handle every incident alone |
| **6. Post-shift decompression** | The day after on-call has reduced meeting load |

### The "Operational Overload" Threshold

```
Warning signs that on-call is unsustainable:

  [ ] Pages per shift > 5 (alerting is too noisy)
  [ ] MTTA > 10 minutes (alerts are being ignored)
  [ ] MTTR > 2x baseline (system is too complex to debug)
  [ ] SREs have > 1 "bad" shift per month (frequent sleep disruption)
  [ ] No runbooks for > 50% of alerts
  [ ] On-call rotation smaller than 6 people
```

### Incident Lifecycle During On-Call

```
┌─────────┐   ┌─────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
│ Receive │→│  Triage │→│  Mitigate │→│  Resolve │→│  Follow  │
│  Alert  │   │(assess) │   │ (fix now) │   │(permafix)│   │  Up      │
└─────────┘   └─────────┘   └──────────┘   └──────────┘   └──────────┘
                                         
Key distinctions:
  - Mitigate ≠ Resolve
  - Mitigation: stop the bleeding (rollback, redirect traffic)
  - Resolution: fix the root cause (patch code, rewrite config)
  - Post-incident: file bug, update runbook, write postmortem
```

### Actionable Takeaways

- **Ensure at least 6-8 people in the rotation.** Fewer than 6 creates unsustainable load.
- **Keep on-call shifts to 12 hours for daytime, or full week 24/7.** Avoid "always on" perma-on-call.
- **Create a culture of blamelessness during on-call.** The pager fires because the *system* failed, not the person.
- **Track on-call metrics** (pages/shift, MTTA, MTTR, false positives) and review them weekly.
- **Provide immediate decompression:** The morning after a nighttime incident, the on-call SRE should be explicitly excused from morning meetings.

### Important Quotes

> "Being on-call is not about being permanently available — it's about being reliably available during a defined period."

> "A sustainable on-call rotation is one where the team isn't burned out, where incidents are genuinely surprising, and where the postmortem culture leads to system improvements that reduce the on-call burden over time."

> "If you're on-call and your pager goes off every night, the problem is not with you — it's with your system."

> "The goal of on-call is not to prevent all incidents. The goal is to respond to them effectively and to make them less frequent over time."

---

## Chapter 12: Effective Troubleshooting

### Core Thesis

Troubleshooting is a structured discipline, not black magic. This chapter presents Google's systematic approach to debugging production issues: from hypothesis formulation to data collection to controlled experimentation.

### The Troubleshooting Workflow

```
┌──────────────────────────────────────────────┐
│                ┌──────────────┐              │
│                │ Report Issue │               │
│                └──────┬───────┘               │
│                       │                       │
│                ┌──────▼───────┐               │
│                │  Triage      │               │
│                │  (Severity,  │               │
│                │   Impact)    │               │
│                └──────┬───────┘               │
│                       │                       │
│                ┌──────▼───────┐               │
│        ┌──────►│ Examine      │◄──────┐      │
│        │       │ (Data gather)│       │      │
│        │       └──────┬───────┘       │      │
│        │              │               │      │
│        │       ┌──────▼───────┐       │      │
│        │       │  Hypothesize │       │      │
│        │       └──────┬───────┘       │      │
│        │              │               │      │
│        │       ┌──────▼───────┐       │      │
│        │       │   Test /     │       │      │
│        │       │  Experiment  ├───────┘      │
│        │       └──────┬───────┘ (hypothesis  │
│        │              │        disproven)    │
│        │       ┌──────▼───────┐              │
│        │       │   Root Cause │              │
│        │       │  Identified  │              │
│        │       └──────┬───────┘              │
│        │              │                      │
│        │       ┌──────▼───────┐              │
│        │       │   Mitigate   │              │
│        │       └──────┬───────┘              │
│        │              │                      │
│        │       ┌──────▼───────┐              │
│        │       │   Resolve    │              │
│        │       └──────┬───────┘              │
│        │              │                      │
│        │       ┌──────▼───────┐              │
│        └───────┤ Follow Up    │              │
│                │ (Bug, Post-  │              │
│                │  mortem)     │              │
│                └──────────────┘              │
└──────────────────────────────────────────────┘
```

### Troubleshooting Principles

| Principle | Description |
|-----------|-------------|
| **1. Don't panic** | The system is already broken. Rushing makes it worse |
| **2. Be systematic** | Follow a hypothesis → test → refine loop, not random guessing |
| **3. Fix forward, not backward** | Prefer rolling forward (fix + deploy) over rolling back when possible |
| **4. One change at a time** | Changing multiple things simultaneously makes it impossible to know what fixed the problem |
| **5. Document as you go** | Write down what you've tried, what you found, and what you're about to try |
| **6. Know when to escalate** | If you're stuck for >15 minutes, loop in a second person |

### Hypothesis-Driven Debugging

```
The scientific method applied to production:

1. Observe the symptom
   → "Users are getting 503 errors on the /search endpoint"

2. Formulate hypotheses (rank by likelihood)
   → H1: Recent deploy introduced a bug (HIGH — deploy 10 min ago)
   → H2: Database is overloaded (MEDIUM — seen before)
   → H3: Network configuration changed (LOW — no config pushes)

3. Design experiments to test each hypothesis
   → H1 test: Rollback the deploy. If errors drop, confirmed.
   → H2 test: Check database query latency dashboard.
   → H3 test: Compare current network config with last known good.

4. Execute experiments, starting with highest probability
5. Repeat until root cause is confirmed
6. Mitigate, then resolve, then document
```

### Common Pitfalls in Troubleshooting

| Pitfall | Description | Prevention |
|---------|-------------|------------|
| **Jumping to conclusions** | Assuming you know the cause without evidence | Write down at least 3 hypotheses before acting |
| **Confirmation bias** | Interpreting ambiguous data to support your preferred hypothesis | Deliberately try to disprove each hypothesis |
| **Changing too many things** | Multiple changes at once → can't identify cause | "One change, measure, next change" discipline |
| **Fixing symptoms, not causes** | Restarting a process when the config is wrong | Always ask "what caused this to break?" |
| **Not asking for help** | Sinking 30+ minutes into a mystery | Set a timer. At 15 minutes, call for backup. |

### Actionable Takeaways

- **Use a structured approach** (hypothesis → test → refine) for every incident. It prevents panic and cuts MTTR.
- **Write a "war room" timeline** as you go. It will be invaluable for the postmortem.
- **Invest in good debugging tools.** A 30-second `curl` probe is worth more than a 30-minute dashboard dive.
- **Practice incident response.** Run tabletop exercises and fire drills quarterly.
- **Post-incident, update your runbooks.** Every "I didn't know that" is a gap that needs filling.

### Important Quotes

> "Troubleshooting is not about knowing the answer. It's about knowing how to find the answer."

> "The first thing to do when a system breaks is to take a deep breath. Panic is not a debugging strategy."

> "If you don't have a hypothesis, you're not troubleshooting — you're guessing."

> "The best debugging tool is a recent change log."

> "When you've identified the root cause, you haven't finished. You still need to fix it, monitor the fix, and ensure it doesn't happen again."

---

## Chapter 15: Postmortem Culture: Learning from Failure

### Core Thesis

A blameless postmortem culture is the single most important driver of long-term reliability improvement. Google treats every significant incident as an opportunity to learn — and the first rule is that the postmortem is **blameless**. The goal is to understand what the *system* did wrong, not who did wrong.

### The Postmortem Philosophy

```
┌─────────────────────────────────────────────────────┐
│  DO:                                               │
│  ✅ Ask "What can we learn from this?"              │
│  ✅ Focus on system, process, and tool failures     │
│  ✅ Assign action items to fix root causes          │
│  ✅ Share postmortems widely across the organization │
│  ✅ Celebrate postmortems as learning opportunities  │
│                                                     │
│  DON'T:                                             │
│  ❌ Ask "Whose fault was this?"                      │
│  ❌ Punish the person who triggered the incident     │
│  ❌ Blame individuals in postmortem text             │
│  ❌ Treat postmortems as disciplinary hearings       │
│  ❌ Keep postmortems secret / internal-only          │
└─────────────────────────────────────────────────────┘
```

### The Anatomy of a Postmortem

A Google-style postmortem typically includes:

| Section | Description | Example Content |
|---------|-------------|-----------------|
| **Title** | Short description | "Excessive Error Rate on Search Backend — 2023-03-15" |
| **Date and Duration** | When the incident occurred and how long it lasted | 2023-03-15 14:23 UTC — 2023-03-15 15:47 UTC |
| **Summary** | 2-3 sentence high-level description | "A misconfigured load balancer caused 12% of search traffic to be routed to an under-provisioned pool..." |
| **Impact** | Quantified effect on users and systems | "~50K failed requests (0.02% of total), 84 min elevated latenc

# SRE Communication Guide

> A comprehensive reference for Site Reliability Engineering communication patterns, templates, and best practices.
> Target audience: SREs, incident commanders, reliability engineers, and engineering leaders.

---

## Table of Contents

1. [Foundations of SRE Communication](#1-foundations-of-sre-communication)
2. [Incident Communication](#2-incident-communication)
3. [Postmortem Communication](#3-postmortem-communication)
4. [Influencing Without Authority](#4-influencing-without-authority)
5. [On-Call Handoff Protocol](#5-on-call-handoff-protocol)
6. [Cross-Team Coordination](#6-cross-team-coordination)
7. [Escalation Communication](#7-escalation-communication)
8. [Metrics-Based Reporting to Leadership](#8-metrics-based-reporting-to-leadership)
9. [Writing Style Guide](#9-writing-style-guide)
10. [Templates](#10-templates)

---

## 1. Foundations of SRE Communication

SRE communication differs from general engineering communication in several critical ways. SREs operate in high-stakes, time-sensitive environments where clarity, brevity, and precision are paramount. Every message — whether an incident update, a postmortem, or a reliability proposal — carries operational weight.

### 1.1 Core Principles

| Principle | Description | Why It Matters |
|-----------|-------------|----------------|
| **Clarity over completeness** | Prioritize clear, unambiguous statements over exhaustive detail. | During incidents, ambiguity costs time. Teams act on what they understand. |
| **Evidence-based claims** | Every assertion about system behavior must trace back to observable data. | Prevents speculation-driven decisions that compound outages. |
| **Bias for action** | Communicate what is being done and what is needed, not just what is wrong. | Stakeholders need to know the response plan, not just the problem description. |
| **No-blame framing** | Describe system behavior in terms of conditions and events, not people or teams. | Psychological safety enables blameless postmortems and honest reporting. |
| **Audience awareness** | Tailor depth, frequency, and format to the consumer of the information. | Executives need impact and ETA; engineers need root cause and reproduction steps. |
| **Timeliness** | Communicate proactively at defined intervals, never let stakeholders wonder. | Silence erodes trust faster than bad news delivered promptly. |

### 1.2 Communication Channels by Urgency

| Urgency | Channel | Format | Examples |
|---------|---------|--------|----------|
| P0/P1 (Critical) | Pager/Phone + Incident Channel | Real-time updates | Outage in progress, data loss detected |
| P2 (High) | Chat + Ticketing | Structured thread | Degraded performance, partial failure |
| P3 (Medium) | Ticketing + Email | Detailed report | Non-critical bug, minor latency regression |
| P4 (Low) | Email / Documentation | Asynchronous write-up | Technical debt tracking, minor improvements |

---

## 2. Incident Communication

Incident communication is the most time-sensitive form of SRE communication. The goal is to keep stakeholders informed without distracting the responders actively troubleshooting.

### 2.1 Incident Status Updates

Status updates follow a predictable rhythm and structure. Updates should be posted:

- **Initial notification**: Within 5 minutes of declaration
- **Regular cadence**: Every 30 minutes for P0, every 60 minutes for P1
- **On any significant change**: Mitigation started, root cause identified, service restored
- **Resolution**: When the incident is declared resolved
- **Follow-up**: When the postmortem is scheduled and shared

**Structure of a Status Update:**

```
[INCIDENT-ID] Status Update [#N]

Severity: [P0/P1/P2]
Service:  [Affected service(s)]
Duration: [Elapsed time since declaration]

Current Status: [Investigating / Mitigating / Monitoring / Resolved]

Impact:
- [Metric] affected: [X]% of users / [Y] requests lost
- [Observable symptom]

Actions Taken:
- [What has been done so far]

Next Steps:
- [What is being attempted next]

ETA / Confidence:
- [Estimated time to resolution or "Unknown"]
- [Low / Medium / High confidence in ETA]

Handoff / Needs:
- [Is a handoff coming?]
- [Does anyone need to be paged?]

Posted by: [Name]
```

### 2.2 Executive Summaries

Executive summaries are concise, non-technical overviews aimed at leadership who need to understand business impact, timeline, and follow-up — not technical details.

**Structure of an Executive Summary:**

```
INCIDENT EXECUTIVE SUMMARY

Incident: [Brief one-line title, e.g., "Payment Processing Outage"]
Date:     [YYYY-MM-DD]
Duration: [Total time from declaration to resolution]
Severity: [P0/P1]

What Happened:
[3-4 sentences describing the user-facing impact and the technical
root cause at a business-readable level. Avoid jargon.]

Business Impact:
- Users affected: [X] (approximately [Y]%)
- Revenue impact: [Estimated $ amount or "None"]
- SLA impact: [Met / Missed — by how much]
- Customer-facing symptoms: [e.g., "Users could not complete purchases"]

Timeline (key events):
- [HH:MM UTC] Incident declared
- [HH:MM UTC] Mitigation started
- [HH:MM UTC] Service restored
- [HH:MM UTC] Incident declared resolved

Root Cause (one sentence):
[A single sentence explaining the technical root cause.]

Remediation Actions:
- Short-term: [What was done to restore service]
- Long-term: [What will prevent recurrence]

Prepared by: [Name]
Date: [YYYY-MM-DD]
```

### 2.3 Timeline Reconstruction

An accurate incident timeline is the backbone of any postmortem and is critical for understanding what happened, when, and why. During the incident, the incident commander (IC) or a dedicated scribe logs key events.

**Timeline Log Format:**

| Time (UTC) | Event | Source | Evidence |
|------------|-------|--------|----------|
| 14:02:15 | Alert fired: p99 latency > 500ms on `payment-service` | Datadog Monitor | Alert ID #8421 |
| 14:02:45 | On-call SRE acknowledged page | PagerDuty | Ack by @jsmith |
| 14:04:30 | Incident declared (P1) | Slack #incidents | Declaration message |
| 14:07:00 | Initial stakeholder notification sent | Slack #leadership-alerts | Message link |
| 14:12:00 | Mitigation: rolled back deploy v2.3.1 → v2.3.0 | Kubernetes | Rollout log |
| 14:18:00 | p99 latency returning to baseline | Datadog Dashboard | Graph link |
| 14:22:00 | Incident declared resolved | Slack #incidents | Resolution message |

**Timeline Best Practices:**

- Log timestamps in **UTC only** to avoid timezone confusion.
- Record events as they happen, not from memory after the fact.
- Include **negative signals** too ("14:10:00 — Rollback not yet complete, no change in error rates").
- Link to dashboards, alert IDs, deploy logs, and commit SHAs as evidence.
- Distinguish between **observations** ("alert fired"), **actions** ("rolled back deploy"), and **decisions** ("escalated to database team").

### 2.4 Incident Communication Roles

| Role | Communication Responsibilities |
|------|-------------------------------|
| **Incident Commander (IC)** | Owns all external communication; decides severity, cadence, and escalation. The single source of truth. |
| **Scribe / Deputy IC** | Logs timeline events; drafts status updates for IC approval; tracks action items. |
| **Subject Matter Expert (SME)** | Provides technical status to IC only; does NOT communicate externally. |
| **Stakeholder Liaison** | (If designated) Handles executive/support communication to keep IC focused on mitigation. |

---

## 3. Postmortem Communication

Postmortems are the primary vehicle for organizational learning from incidents. The quality of the postmortem directly determines whether the lessons are absorbed and acted upon.

### 3.1 Writing Clearly

**Postmortem Structure:**

```
# Postmortem: [INCIDENT-ID] — [Descriptive Title]

## Summary
[3-5 sentences: what happened, impact, how long, root cause category]

## Impact
| Metric | Value |
|--------|-------|
| Severity | [P0/P1/P2] |
| Duration | [X minutes/hours] |
| Users Affected | [X] (approximate) |
| Error Rate | [X]% |
| SLO Impact | [Met / Missed] |
| Revenue Impact | [$X or N/A] |

## Timeline
| Time (UTC) | Event |
|------------|-------|
| [T-0] | [First signal (alert, user report, etc.)] |
| [T+X] | [Key events in chronological order] |
| [T+Y] | [Mitigation started] |
| [T+Z] | [Service restored] |

## Root Cause Analysis
- **Trigger**: The specific condition that initiated the failure
- **Contributing Factors**: Conditions that amplified the impact or delayed detection
- **Why it wasn't caught earlier**: Gaps in monitoring, testing, or processes

## Detection
- How was the incident first detected? (Alert / User report / Proactive)
- Time from first failure to first notification:
- Could detection have been faster? [Yes/No — if yes, how?]

## Response
- Time to acknowledge:
- Time to mitigation:
- What went well in the response:
- What could have gone better:

## Action Items
| # | Action | Owner | Ticket | Due |
|---|--------|-------|--------|-----|
| 1 | Add alert for metric X | [Owner] | [Link] | [Date] |
| 2 | Add runbook for scenario Y | [Owner] | [Link] | [Date] |
| 3 | [Etc.] | | | |

## Prevention
- What specific changes prevent this exact failure from recurring?
- What systemic improvements reduce the class of failures this belongs to?

## Lessons Learned
- What did we learn about our system?
- What did we learn about our processes?
- What surprised us?

## Blameless Statement
*"The goal of this postmortem is to learn and improve our systems and processes.
No individual or team is blamed. We all operate within the constraints and information
available to us at the time."*
```

### 3.2 Action Items Ownership

Action items from postmortems must be treated as seriously as feature work. Each action item should be:

| Criterion | Description |
|-----------|-------------|
| **S.M.A.R.T.** | Specific, Measurable, Achievable, Relevant, Time-bound |
| **Assigned** | A named owner, not a team |
| **Tracked** | Linked to a ticket in the tracking system |
| **Classified** | Marked as "mitigation" (immediate) or "prevention" (systemic) |
| **Prioritized** | Labeled P0-P4 based on risk reduction |
| **Followed up** | Reviewed at regular reliability retrospectives |

### 3.3 Sharing Failures Constructively

The psychological safety of the team depends on how failures are shared and discussed.

**Do:**
- Start postmortem reviews with the blameless statement
- Discuss system conditions, not individual decisions
- Frame gaps as process opportunities, not personal failings
- Celebrate what was caught and what went well
- Share postmortems broadly (within reason) so other teams learn

**Don't:**
- Assign blame or use blame-implying language ("should have", "failed to", "mistake")
- Skip the "what went well" section — this is not just about failures
- Let action items languish without follow-up
- Use postmortems as performance review input

### 3.4 Postmortem Readability Guidelines

- Write for an audience that includes engineers who were not on-call
- Define acronyms on first use
- Use active voice ("The deploy caused X" not "X was caused by the deploy")
- Keep the summary to 3-5 sentences — busy engineers read this first
- Use concrete numbers, not ranges or estimates
- Link to supporting evidence (graphs, logs, commits)

---

## 4. Influencing Without Authority

SREs frequently need to drive reliability improvements across teams they do not manage. This requires influence grounded in data, respect, and relationship-building.

### 4.1 Data-Driven Proposals

When proposing reliability work to a product or feature team, data is your strongest tool.

**Anatomy of a Reliability Proposal:**

```
# Reliability Proposal: [Title]

## Problem Statement
[One paragraph describing the reliability risk, backed by data.]

## Evidence
- [Metric] over [timeframe]: [current value] vs [target] — [X]% deviation
- Incidents caused by this issue in last [N] months: [count]
- User-facing impact: [description with numbers]

## Proposed Solution
[What you propose to build or change, in engineering terms.]

## Expected Impact
- Reduction in p99 latency: [N]% (estimated)
- Incidents prevented per quarter: [N] (estimated)
- Engineering time saved per month: [N] hours

## Cost
- Engineering effort: [N] engineer-weeks
- Migration/rollout complexity: [Low / Medium / High]
- Risk of change: [Low / Medium / High]

## Alternatives Considered
| Option | Pros | Cons |
|--------|------|------|
| [Option A — proposed] | [List] | [List] |
| [Option B] | [List] | [List] |
| [Option C — do nothing] | [List] | [List] |

## Recommendation
[Clear recommendation with rationale. Frame in terms the team cares about:
faster feature velocity, fewer pages, better user experience, lower cost.]

Prepared by: [Name]
Date: [YYYY-MM-DD]
```

### 4.2 Building Buy-In

Influence without authority requires deliberate relationship-building:

| Strategy | Tactics |
|----------|---------|
| **Find shared goals** | Connect reliability work to the team's stated priorities (e.g., "This will reduce the number of pages that interrupt your sprint") |
| **Start small** | Propose a low-effort, high-impact change first to demonstrate value. You earn credibility from wins. |
| **Make it easy** | Provide code, configs, dashboards, and runbooks — don't just identify the problem. |
| **Speak their language** | Use the metrics the team already tracks. If they care about feature velocity, frame reliability work in terms of reducing rework and interruptions. |
| **Create visibility** | Build dashboards and reports that make the team's reliability transparent to *their* stakeholders. |
| **Credit generously** | Attribute wins to the team, not to SRE. Trust is built on shared success. |
| **Be persistent but patient** | Reliability improvements often compete with feature work. A proposal may need 3-5 touches before it's adopted. |

### 4.3 Reliability Culture Leadership

Building a reliability culture means embedding reliability thinking into how every team operates, not just the SRE team.

- **Champion error budgets**: Help teams understand their error budget and make data-driven decisions about when to release vs. when to invest in reliability.
- **Run reliability office hours**: Regular slots where teams can bring reliability questions, review dashboards, or get help with SLO design.
- **Share metrics broadly**: Make reliability data visible in places where teams already look — sprint reviews, all-hands, team dashboards.
- **Celebrate reliability wins**: Call out teams that improved their SLOs, reduced incident frequency, or shipped reliability improvements.
- **Lead by example**: The SRE team's own systems should be exemplars of reliability practices.

### 4.4 Handling Resistance

| Objection | Response |
|-----------|----------|
| "We don't have time for reliability work." | "Let's look at how much time incidents cost your team. Even a small investment in prevention can save multiples in firefighting." |
| "Our system is different — it doesn't need the same reliability." | "Let's define what 'good enough' looks like for your system and only invest to that bar." |
| "We'll fix it when it breaks." | "That's a valid approach for low-criticality systems. Let's classify this system's tier and set expectations accordingly." |
| "SRE is just saying we're doing it wrong." | "Our goal is to give you tools and data to make your own reliability decisions. The choice is yours." |

---

## 5. On-Call Handoff Protocol

Structured on-call handoffs prevent critical information from being lost between shifts and ensure continuity of incident response.

### 5.1 Standard Handoff Format

Handoffs should take no more than 10 minutes and cover a structured checklist:

```
# On-Call Handoff: [Date]

## Shift Context
- Shift starting: [Name] — [Date/Time]
- Shift ending:  [Name] — [Date/Time]

## Active Incidents
[Incident ID / None]
- Status: [Open / Monitoring / Resolved / Postmortem pending]
- Summary: [One-sentence status]
- Handoff notes: [What the incoming person needs to know]

## Ongoing Issues
- [Known non-incident issues, expected to degrade/require attention]
- [Bugs or flakiness being tracked]

## Scheduled Changes
| Change | Time | Owner | Risk | Rollback Plan |
|--------|------|-------|------|---------------|
| [Deploy X] | [UTC] | [Name] | [Low/Med/High] | [Yes/No — describe] |
| [Maintenance Y] | [UTC] | [Name] | [Low/Med/High] | [Yes/No] |

## Recent Deployments
- [Service] deployed [version] at [time] — [healthy / monitoring]
- [Service] rolled back [version] → [version] at [time] — reason: [X]

## Known Vulnerabilities / Risks
- [e.g., "Database master is at 80% disk — fill rate ~2%/day. Alert threshold at 85%. No action needed this shift unless rate accelerates."]
- [e.g., "Redis cluster has one failed node — repair scheduled for tomorrow. No data loss expected."]

## Runbook Updates
- [Any new or modified runbooks since last handoff]
- [Any common troubleshooting tips discovered this shift]

## Dashboard Links
- Primary monitoring dashboard: [link]
- Service health dashboard: [link]
- Recent incident dashboards: [links]

## Pages / Alerts This Shift
| Alert | Times Fired | Actions Taken |
|-------|-------------|---------------|
| [Alert name] | [N] | [Summary] |
| [Alert name] | [N] | [Summary] |

## Communication
- [Any ongoing stakeholder communication threads]
- [Any executive attention on specific issues]

## Checklist
- [ ] Handoff doc shared in on-call channel
- [ ] PagerDuty schedules verified for next shift
- [ ] Escalation path reviewed (if new to rotation)
- [ ] Critical runbooks reviewed
- [ ] Known-good rollback versions noted

Handoff completed by: [Outgoing SRE]
Handoff received by:  [Incoming SRE]
```

### 5.2 What to Communicate (and What Not To)

| Communicate | Do NOT Communicate |
|-------------|-------------------|
| Active incidents and their status | Exhaustive details of every non-critical alert |
| Ongoing known issues with workarounds | Long-standing technical debt with no active impact |
| Scheduled changes during the next shift | Changes that have already been completed safely |
| Alert patterns observed during the shift | Every single alert that fired (summarize instead) |
| Full-disk, near-capacity, or degraded resources | "Normal" operational state (assume baseline) |
| Stakeholder expectations or active comms threads | Personal opinions about team members or decisions |

### 5.3 Shadowing and Ramp-Up

When an SRE is new to the rotation, the handoff should include:

1. **First shift as shadow**: New SRE shadows the outgoing SRE for the full shift.
2. **First solo shift with backup**: New SRE takes the pager but the experienced SRE remains available for questions.
3. **Sign-off**: The experienced SRE confirms the new SRE is ready for independent rotation.

---

## 6. Cross-Team Coordination

SREs regularly coordinate with product, engineering, security, and support teams. Each relationship requires a tailored communication approach.

### 6.1 With Product Teams

| Situation | Communication Approach |
|-----------|----------------------|
| Reliability proposal | Frame in terms of user experience and feature velocity. Use error budgets to create shared vocabulary. |
| SLO negotiation | Present data on current performance, cost of improvement, and trade-offs. Let the product team decide the risk tolerance. |
| Incident affecting a feature | Communicate impact in user-facing terms. Provide ETA updates in product-relevant intervals. |
| Feature launch readiness | Run pre-launch reviews and present findings as a scorecard with clear pass/fail criteria. |

**Pre-Launch Readiness Scorecard:**

| Criterion | Status | Notes |
|-----------|--------|-------|
| SLO defined | ✅ / ❌ | Target values and measurement method agreed |
| Dashboards built | ✅ / ❌ | At least one dashboard covering the golden signals |
| Alerts configured | ✅ / ❌ | Pager-worthy alerts with runbooks |
| Load testing completed | ✅ / ❌ | Results within acceptable range |
| Error budget allocated | ✅ / ❌ | Error budget policy defined |
| Runbooks written | ✅ / ❌ | At minimum, incident response and rollback procedures |
| Rollback tested | ✅ / ❌ | Procedure confirmed working |
| Dependency map documented | ✅ / ❌ | All upstream/downstream dependencies identified |

### 6.2 With Engineering Teams

| Situation | Communication Approach |
|-----------|----------------------|
| Code review (reliability) | Focus on failure modes, retry logic, circuit breakers, and observability. Use concrete examples of past failures. |
| Architecture review | Ask "what happens when this fails?" for every component. Document failure modes and mitigations. |
| Incident retrospectives | Joint posture: "We all own the outcome." Share data before opinions. |
| Reliability training | Offer lunch & learns, dojo sessions, or paired incident response drills. |

### 6.3 With Security Teams

| Situation | Communication Approach |
|-----------|----------------------|
| Security incident coordination | SRE owns availability; Security owns confidentiality/integrity. Establish clear incident command boundaries. |
| Vulnerability disclosure | Provide blast radius analysis. Security communicates the vulnerability; SRE communicates the mitigation plan and timeline. |
| Disaster recovery testing | Coordinate with Security on access controls, key rotation, and credential management in DR scenarios. |
| Shared infrastructure | Establish clear escalation paths for when a reliability issue has security implications (and vice versa). |

**SRE-Security Coordination During Incidents:**

| Phase | SRE Leads On | Security Leads On |
|-------|-------------|-------------------|
| **Detection** | Monitoring/reliability signals | Security monitoring signals |
| **Triage** | Is this a reliability or security incident? | Is this an active threat or compliance issue? |
| **Response** | Service restoration, user communication | Containment, forensics, legal notification |
| **Recovery** | Service health verification | Post-incident security sweep |
| **Postmortem** | Timeline, impact, reliability fixes | Analysis of vulnerabilities, new controls |

### 6.4 With Customer Support / Triage Teams

- **Provide status templates**: Give the support team pre-approved incident communication templates they can use when customers ask.
- **One-way communication during incidents**: A designated person (or a status page) feeds information to support; support does not request updates during active response.
- **Post-incident follow-up**: Send a customer-facing summary (less technical than the internal postmortem) after every P0/P1 incident.

---

## 7. Escalation Communication

Escalation is not a failure — it is a structured process for getting the right resources and attention to a problem.

### 7.1 When to Escalate

| Condition | Escalate To | Method |
|-----------|-------------|--------|
| Incident exceeds 30 minutes without mitigation | IC escalates to SRE manager | Phone / Pager |
| Cross-team coordination needed | IC escalates to stakeholder liaison | Phone |
| Business-critical impact not communicated to executives | IC escalates to director+ | Phone then email |
| No clear mitigation path after 60 minutes | IC escalates to senior engineering | Phone / In-person |
| Multiple simultaneous incidents | IC escalates for additional IC support | Chat / Phone |

### 7.2 Escalation Message Template

```
# ESCALATION REQUEST

Incident: [INCIDENT-ID] — [Title]
Severity: [P0 / P1]
Elapsed Time: [X] minutes

Current Status: [Investigating / Mitigating / Stuck]

Issue:
[What is blocking resolution or why escalation is needed]
- We need [specific help / decision / resource]

Impact If Not Resolved:
- [X] additional minutes of downtime expected
- [Y] users affected
- [Z]% of error budget consumed

What We've Tried:
- [Action 1]
- [Action 2]

Requested:
- [Specific ask: e.g., "Authorization to fall back to degraded mode",
  "Database team to investigate replication lag"]

Requested by: [Name]
Time: [UTC]
```

### 7.3 Escalation Best Practices

- **Escalate early, not late**: It is better to escalate and then de-escalate than to wait until the situation is critical. A rule of thumb: if you're wondering whether to escalate, escalate.
- **Be specific about what you need**: "We need a DBA to review the query plan on the `orders` table" is actionable. "We need help" is not.
- **Stay in the loop**: After escalating, keep the escalator informed of progress. Do not hand off and disappear.
- **Respect the escalation path**: Follow the chain unless there is a reason to skip a level (document why).

---

## 8. Metrics-Based Reporting to Leadership

Leadership does not have time to dig through dashboards. Effective reporting distills complex operational data into actionable signals.

### 8.1 What Leadership Cares About

| Theme | Questions Leadership Asks | Metrics to Report |
|-------|--------------------------|-------------------|
| **Availability** | Are we meeting our commitments? | SLO attainment, uptime %, error budget remaining |
| **Velocity** | Is reliability blocking feature work? | Incident count, time spent responding vs. preventing |
| **Trend** | Are we getting better or worse? | Month-over-month trends in MTTR, MTBF, incident count |
| **Risk** | What could go wrong next? | Largest error budget consumers, known risks, overdue action items |
| **Cost** | Are we spending wisely? | Infrastructure cost per transaction, cost of reliability initiatives |

### 8.2 Weekly Reliability Summary

The weekly summary is the primary recurring report for engineering leadership.

```
# Weekly Reliability Summary: [YYYY-MM-DD] to [YYYY-MM-DD]

## At a Glance
| Metric | This Week | Previous Week | Trend | Target |
|--------|-----------|---------------|-------|--------|
| P0/P1 Incidents | [N] | [N] | ↑ ↓ → | [N] |
| MTTR (P0) | [X min] | [X min] | ↑ ↓ → | [X min] |
| MTBF | [X days] | [X days] | ↑ ↓ → | [X days] |
| Error Budget Burn Rate | [X]%/week | [X]%/week | ↑ ↓ → | [X]%/week |
| SLO Attainment (30d) | [X]% | [X]% | ↑ ↓ → | [X]% |

## Notable Incidents
- [INCIDENT-ID]: [Title] — [Duration], [Impact], [Root cause category]
  Action items: [N] open, [N] closed
- [INCIDENT-ID]: [Title] — [Duration], [Impact], [Root cause category]
  Action items: [N] open, [N] closed

## Reliability Wins
- [Team] improved [SLO/MTTR/etc] by [X]% through [initiative]
- [Capability] was rolled out: [description]
- [Improvement] prevented [N] potential incidents

## Risk Watchlist
| Risk | Likelihood | Impact | Owner | Status |
|------|------------|--------|-------|--------|
| [Description] | High/Med/Low | High/Med/Low | [Name] | Being addressed / Not yet started |
| [Description] | High/Med/Low | High/Med/Low | [Name] | Being addressed / Not yet started |

## Postmortem Action Items Summary
- Total open action items: [N]
- Overdue: [N] (list key ones)
- Completed this week: [N]

## Recommendation
[One or two specific, data-backed recommendations for leadership action.
E.g., "We recommend approving the database migration plan for Service X before
the holiday traffic peak. Current error budget burn rate will exhaust the quarter's
budget by [date] if not addressed."]
```

### 8.3 Monthly / Quarterly Business Review

For higher-level presentations, focus on narrative and trends:

- **The big story**: What one or two reliability trends defined this period?
- **Incident decomposition**: Group incidents by cause category (code change, configuration, infrastructure, external dependency, capacity, etc.)
- **Error budget report**: For each critical service, show remaining error budget and projected exhaustion date
- **ROI of reliability**: Show how investments in reliability (tooling, improvements, refactors) correlate with reduced incident count or duration
- **Ask**: What one or two decisions or resources does leadership need to provide to maintain or improve reliability

---

## 9. Writing Style Guide

Clear writing is a force multiplier in SRE. A status update, postmortem, or proposal that can be read and understood in 30 seconds saves time for every person who reads it.

### 9.1 General Principles

| Principle | Guideline | Bad Example | Good Example |
|-----------|-----------|-------------|--------------|
| **Be concise** | Use the fewest words that convey the needed information. | "At this point in time, we are currently in the process of investigating the issue which appears to be related to what seems like a database connection problem." | "Investigating a database connection pool exhaustion issue." |
| **Use active voice** | Subject performs the action. Avoid passive constructions. | "The deploy was rolled back by the on-call engineer after elevated error rates were observed." | "The on-call engineer rolled back the deploy after observing elevated error rates." |
| **Be specific** | Use concrete numbers and facts, not approximations or opinions. | "There were a lot of errors and most users were affected." | "Error rate peaked at 12.3% at 14:05 UTC; approximately 4,200 users (6.7%) encountered failures." |
| **Use plain language** | Avoid jargon, acronyms without definitions, and unnecessarily technical terms for non-technical audiences. | "The Envoy sidecar experienced a connection pool drain timeout due to excessive COW page faults from the kernel memory reclamation path." | "The proxy service ran out of available connections because the kernel was reclaiming memory used by connection buffers." |
| **Structure for scannability** | Use headings, bullet points, and tables. Put the most important information first. | A dense paragraph with key information buried in the middle. | An inverted pyramid: conclusion first, then supporting details, then background. |
| **One thought per sentence** | Short sentences are easier to parse under time pressure. | "We rolled back the deploy to version 2.3.0 and confirmed that error rates dropped from 8% to 0.2% within 3 minutes, though we're still monitoring latency to make sure it continues to trend down." | "We rolled back the deploy to version 2.3.0. Error rates dropped from 8% to 0.2% within 3 minutes. Latency is still returning to baseline — continuing to monitor." |

### 9.2 No-Blame Language Guide

| Instead of This | Use This |
|-----------------|----------|
| "The engineer failed to check the config before deploying." | "The configuration validation step did not catch the invalid parameter." |
| "The team should have caught this in code review." | "The code review checklist did not include this failure mode." |
| "Someone accidentally ran the wrong command." | "The runbook did not sufficiently distinguish between the production and staging commands." |
| "The developer made a mistake in the retry logic." | "The retry logic did not account for the case where the upstream returns a 429 status code." |
| "The alert was ignored." | "The alert was not actionable and was tuned out due to frequent false positives." |

### 9.3 Acronym and Abbreviation Rules

- **Define on first use**: "Mean Time to Repair (MTTR) improved by 20%."
- **Use consistently**: Once defined, use the acronym throughout the document.
- **Avoid unnecessary acronyms**: If you only use it once, spell it out.
- **Know your audience**: A postmortem for other SREs can use "SLO", "MTTR", "p99"; an executive summary should spell out or explain these.

### 9.4 Tense and Perspective

| Element | Recommendation |
|---------|---------------|
| **Postmortems** | Past tense (describing what happened) |
| **Incident updates** | Present tense / future tense (what is happening, what will happen) |
| **Proposals** | Present tense (problem exists) + future tense (proposed solution) |
| **Runbooks** | Imperative mood ("Run this command", "Check this metric") |
| **Status reports** | Present perfect ("We have mitigated") + present ("We are monitoring") |
| **First person** | Use "we" (organizational ownership), avoid "I" in incident updates unless attribution is needed |

---

## 10. Templates

### 10.1 Incident Status Update Template

```
INCIDENT STATUS UPDATE

===============================================================================
INCIDENT: [INCIDENT-ID]
SEVERITY: [P0 / P1 / P2]
SERVICE:  [Affected service names]
UPDATE #: [N]
===============================================================================

CURRENT STATUS: [Investigating / Mitigating / Monitoring / Resolved]

--- IMPACT ---

User-facing symptoms:
[2-3 sentences describing what users experienced]

Scope:
- [Metric] affected: [X]%
- Estimated users affected: [X]
- Error budget consumed: [X]%
- Revenue impact (estimated): [$X or N/A]

--- WHAT WE KNOW ---

Root cause (preliminary):
[One sentence if known, or "Under investigation"]

Trigger:
[The event that initiated the incident, if known]

--- ACTIONS TAKEN ---

1. [Action — timestamp UTC]
2. [Action — timestamp UTC]
3. [Action — timestamp UTC]

--- NEXT STEPS ---

1. [Next action — owner]
2. [Next action — owner]
3. [Next action — owner]

--- ETA ---

Estimated time to resolution: [Time or "Unknown"]
Confidence level: [Low / Medium / High]

--- COMMUNICATION ---

Next update: [Time or "On significant change"]
Channel: [#incident-channel]

Posted by: [Name]
Time: [UTC]
```

### 10.2 Reliability Review Proposal Template

```
RELIABILITY REVIEW PROPOSAL
===============================================================================
TO:      [Team or person being proposed to]
FROM:    [Your name / SRE Team]
DATE:    [YYYY-MM-DD]
SUBJECT: [Clear, specific title — e.g., "Proposed: Rate Limiting for Payment API"]
===============================================================================

1. PROBLEM STATEMENT
   ---------------------------------------------------------------------------
   [One paragraph describing the reliability risk or opportunity. Include:
   - What specific scenario or condition is causing concern
   - Who or what is affected
   - Why now is the right time to address it]

2. EVIDENCE
   ---------------------------------------------------------------------------
   Metric            | Current Value | Target Value | Deviation
   ------------------|---------------|--------------|----------
   [Metric name]     | [Value]       | [Value]      | [X]%
   [Metric name]     | [Value]       | [Value]      | [X]%

   Incidents in last 90 days related to this issue: [N]
   - [INCIDENT-ID]: [Brief description — 1 line]
   - [INCIDENT-ID]: [Brief description — 1 line]

   Error budget consumed by these incidents: [X]%

3. PROPOSED SOLUTION
   ---------------------------------------------------------------------------
   [Detailed description of what you propose to build, change, or implement.
   Include architecture components, configuration changes, or process changes.]

4. EXPECTED OUTCOMES
   ---------------------------------------------------------------------------
   Outcome                        | Metric                  | Estimated Impact
   ------------------------------|-------------------------|-----------------
   Fewer incidents               | Incident count/quarter  | [N] reduction
   Faster recovery               | MTTR                    | [N]% improvement
   Better user experience        | p99 latency             | [N]ms reduction
   Reduced operational burden    | On-call pages/month     | [N] reduction

5. COST AND EFFORT
   ---------------------------------------------------------------------------
   Engineering effort:    [N] engineer-weeks
   Calendar duration:     [N] weeks
   Dependencies:          [List of teams, systems, or decisions needed]
   Risk of change:        [Low / Medium / High]
   Rollback complexity:   [Low / Medium / High]

6. ALTERNATIVES EVALUATED
   ---------------------------------------------------------------------------
   Option | Upside | Downside | Recommendation
   -------|--------|----------|---------------
   [Proposed solution] | [Key benefit] | [Key drawback] | RECOMMENDED
   [Alternative A]     | [Key benefit] | [Key drawback] | Not recommended
   [Do nothing]        | Zero effort   | Risk of [X] incidents/quarter | Fallback

7. RECOMMENDATION
   ---------------------------------------------------------------------------
   [2-3 sentence recommendation. State clearly what you want the team to do,
   by when, and why it benefits the team's own priorities.]

8. NEXT STEPS
   ---------------------------------------------------------------------------
   [ ] Review and discuss this proposal
   [ ] Schedule architecture review (suggested: [date])
   [ ] Identify owner(s) for implementation
   [ ] Define success criteria and acceptance tests

===============================================================================
```

### 10.3 Weekly Reliability Summary Template

```
WEEKLY RELIABILITY SUMMARY
===============================================================================
PERIOD:   [YYYY-MM-DD] — [YYYY-MM-DD]
PREPARED: [Name]
===============================================================================

1. OVERVIEW
   ---------------------------------------------------------------------------
   | Metric                  | This Week | Last Week | Trend | Target |
   |-------------------------|-----------|-----------|-------|--------|
   | P0/P1 Incidents         | [N]       | [N]       | ↑↓→   | [N]    |
   | P2/P3 Incidents         | [N]       | [N]       | ↑↓→   | [N]    |
   | Mean Time to Acknowledge| [X] min   | [X] min   | ↑↓→   | [X]    |
   | Mean Time to Mitigate   | [X] min   | [X] min   | ↑↓→   | [X]    |
   | Mean Time to Resolve    | [X] min   | [X] min   | ↑↓→   | [X]    |
   | Error Budget Remaining  | [X]%      | [X]%      | ↑↓→   | [X]%   |
   | On-Call Pages          | [N]       | [N]       | ↑↓→   | [N]    |

2. INCIDENT SUMMARY
   ---------------------------------------------------------------------------

   INCIDENT #1: [INCIDENT-ID]
   - Severity:  [P0/P1]
   - Duration:  [X] minutes
   - Impact:    [X]% error rate, [X] users affected
   - Root cause: [One line]
   - Status:    [Resolved / Mitigated / Monitoring]
   - Action items: [N] open / [N] closed

   INCIDENT #2: [INCIDENT-ID]
   - Severity:  [P0/P1]
   - Duration:  [X] minutes
   - Impact:    [X]% error rate, [X] users affected
   - Root cause: [One line]
   - Status:    [Resolved / Mitigated / Monitoring]
   - Action items: [N] open / [N] closed

3. INCIDENT CAUSES BREAKDOWN (YTD)
   ---------------------------------------------------------------------------
   Cause Category        | This Week | Quarter to Date
   ----------------------|-----------|-----------------
   Code change           | [N]       | [N]
   Configuration change  | [N]       | [N]
   Infrastructure failure| [N]       | [N]
   External dependency   | [N]       | [N]
   Capacity exhaustion   | [N]       | [N]
   Human error           | [N]       | [N]

4. SERVICE HEALTH
   ---------------------------------------------------------------------------
   Service       | SLO Target | 30d Attainment | Error Budget Remaining
   -------------|------------|---------------|-----------------------
   [Service A]  | [X]%       | [X]%          | [X]%
   [Service B]  | [X]%       | [X]%          | [X]%
   [Service C]  | [X]%       | [X]%          | [X]%

5. RELIABILITY WINS
   ---------------------------------------------------------------------------
   - [Description of win — what was done, by whom, what impact it had]
   - [Description of win]
   - [Description of win]

6. ACTION ITEMS
   ---------------------------------------------------------------------------
   | #  | Action                                  | Owner | Due        | Status |
   |----|-----------------------------------------|-------|------------|--------|
   | 1  | [Description of action]                 | [Name]| [YYYY-MM-DD] | [Open/Closed] |
   | 2  | [Description of action]                 | [Name]| [YYYY-MM-DD] | [Open/Closed] |
   | 3  | [Description of action]                 | [Name]| [YYYY-MM-DD] | [Open/Closed] |

   Overdue items: [N]
   Items due this week: [N]

7. RISK WATCHLIST
   ---------------------------------------------------------------------------
   Risk                              | Likelihood | Impact | Action Plan
   ---------------------------------|------------|--------|-------------
   [Description of risk]            | H/M/L      | H/M/L  | [One sentence]
   [Description of risk]            | H/M/L      | H/M/L  | [One sentence]
   [Description of risk]            | H/M/L      | H/M/L  | [One sentence]

8. RECOMMENDATIONS
   ---------------------------------------------------------------------------
   [1-3 specific, actionable recommendations for leadership. Each should
   include the evidence supporting it and the expected impact.]

===============================================================================
```

### 10.4 Executive Incident Summary Template

```
CONFIDENTIAL — INCIDENT EXECUTIVE SUMMARY
===============================================================================
INCIDENT: [Brief descriptive title — e.g., "Payment Processing Outage"]
DATE:     [YYYY-MM-DD]
SEVERITY: [P0 / P1]
DURATION: [Total time — e.g., "47 minutes"]
===============================================================================

1. EXECUTIVE SUMMARY
   ---------------------------------------------------------------------------
   [3-5 sentences covering:
   - What happened (business-readable)
   - When it happened
   - How long it lasted
   - What the root cause was (high-level)
   - Current status (resolved, monitoring, etc.)]

2. BUSINESS IMPACT
   ---------------------------------------------------------------------------
   Impact Area          | Details
   ---------------------|--------------------------------------------------
   Users Affected       | [X] (approximately [Y]% of active users)
   Downtime Duration    | [X] minutes of total outage
   Degraded Duration    | [X] minutes of partial degradation
   Revenue Impact       | [Estimated $ amount, or "No measurable impact"]
   SLA Impact           | [SLA met / SLA missed by X min]
   Customer Escalations | [N] support tickets escalated
   Regulatory Impact    | [None / Description of any compliance considerations]

3. KEY EVENTS (TIMELINE)
   ---------------------------------------------------------------------------
   Time (UTC)        | Event
   ------------------|--------------------------------------------------
   [HH:MM]           | [First symptom detected — e.g., "Alert fired for error rate spike"]
   [HH:MM]           | [Incident declared / Responders engaged]
   [HH:MM]           | [Mitigation started — e.g., "Rollback initiated"]
   [HH:MM]           | [Service restoration — traffic fully restored]
   [HH:MM]           | [Incident declared resolved]
   [HH:MM]           | [Postmortem scheduled]

4. ROOT CAUSE
   ---------------------------------------------------------------------------
   [One paragraph, business-readable. Avoid excessive technical depth.]

   Category: [Code change / Config change / Infrastructure / External / Capacity]

5. RESOLUTION
   ---------------------------------------------------------------------------
   [Description of the actions taken to restore service. Include who
   responded and how quickly.]

   Key metrics:
   - Time to acknowledge: [X] minutes
   - Time to mitigate:    [X] minutes
   - Time to resolve:     [X] minutes

6. REMEDIATION PLAN
   ---------------------------------------------------------------------------
   Immediate Actions (completed or in progress):
   - [Action]: [Owner — status]
   - [Action]: [Owner — status]

   Long-term Prevention:
   - [Action]: [Owner — target date]
   - [Action]: [Owner — target date]
   - [Action]: [Owner — target date]

7. LESSONS LEARNED
   ---------------------------------------------------------------------------
   What went well:
   - [Observation]
   - [Observation]

   What could be improved:
   - [Observation]
   - [Observation]

8. ATTACHMENTS / REFERENCES
   ---------------------------------------------------------------------------
   - Full postmortem: [link]
   - Incident timeline (detailed): [link]
   - Related dashboards: [links]
   - Related change logs: [links]

===============================================================================
Prepared by: [Name]
Date: [YYYY-MM-DD]
Distribution: [Executive team / VP Engineering / Stakeholders]
===============================================================================
```

---

## Appendix A: Quick Reference Cards

### A.1 Communication Decision Matrix

| Situation | Who to Tell | How | When | Template |
|-----------|-------------|-----|------|----------|
| Incident declared | On-call team + stakeholders | Chat + Status page | Immediately | Section 10.1 |
| Incident status change | All incident subscribers | Chat (incident channel) | Every 30-60 min | Section 10.1 |
| Business-critical impact | Executives | Phone + Email | Within 15 min | Section 10.4 |
| Incident resolved | All subscribers + Support | Chat + Status page | Immediately | Section 10.1 |
| Postmortem complete | Engineering org | Email / Wiki | Within 5 business days | Section 3.1 |
| Reliability proposal | Target team + their manager | Doc + Review meeting | Scheduled | Section 10.2 |
| Weekly reliability | Engineering leadership | Email / Slack digest | Weekly | Section 10.3 |
| Risk identified | On-call + team lead | Chat + Ticketing | Within shift | None required |

### A.2 Common Mistakes and Fixes

| Mistake | Why It Hurts | Fix |
|---------|-------------|-----|
| Using passive voice during incidents | Makes it unclear who is doing what | Start sentences with the person/team performing the action |
| Speculating about root cause without evidence | Misleads stakeholders and creates confusion | Say "Under investigation" with confidence; share only confirmed facts |
| Overpromising on ETA | Erodes trust when the ETA is missed | Give a range, state confidence level, and update proactively |
| Including too much technical detail in executive updates | Leadership cannot act on information they don't understand | Write the "headline" first; offer to share technical details separately |
| Not documenting timeline during the incident | Makes postmortem reconstruction unreliable and harder | Assign a scribe for every P0/P1 incident |
| Letting action items expire | Wastes the learning from the incident | Track in the team's existing workflow, review weekly |
| Using blame language | Discourages honest reporting in future incidents | Review every postmortem draft for blame language before publishing |
| Writing runbooks in different formats | Slows down incident response when every runbook looks different | Use a standardized runbook template for all services |

### A.3 Recommended Reading

- *Site Reliability Engineering* (Beyer et al., O'Reilly) — Chapters on incident response and postmortems
- *The Field Guide to Understanding Human Error* (Dekker) — Foundations of blameless analysis
- *Influence Without Authority* (Cohen & Bradford) — Classic text on cross-team influence
- *Writing on Incident Response* by Google SRE — Published incident postmortems as examples of clear communication
- *Incident Management Best Practices* by PagerDuty — Operational communication patterns

---

> **Document Maintainer**: SRE Team
> **Last Updated**: [YYYY-MM-DD]
> **Version**: 1.0
> **Review Cadence**: Quarterly

# SRE Ecosystem Guide: A Curator's Map of Google SRE Learning Resources

> **Purpose:** Synthesize and analyze every resource at sre.google beyond the core SRE book and the two major articles (Embracing Risk / Eliminating Toil). This document is a **curator's guide** — it explains what each resource adds, how it connects to the others, and what level of practitioner it serves.
>
> **Audience:** SRE practitioners building the `site-reliability-engineering` the host agent skill profile. Use this to decide what to study, reference, or incorporate into your own SRE practice.

---

## Table of Contents

1. [The Site Reliability Workbook — Templates, Case Studies, and Org Change](#1-the-site-reliability-workbook)
2. [Building Secure & Reliable Systems — Security Meets Reliability](#2-building-secure--reliable-systems)
3. [SRE Classroom — Workshops That Build Systems Thinking](#3-sre-classroom--workshops)
4. [SRE Prodcast — The Audio Companion](#4-sre-prodcast)
5. [SRE Video Gallery — Curated Learning Paths](#5-sre-video-gallery)
6. [STPA — System Theoretic Process Analysis](#6-stpa-system-theoretic-process-analysis)
7. [Why Heroism is Bad](#7-why-heroism-is-bad)
8. [SRE Fundamentals Course](#8-sre-fundamentals-course)
9. [Measuring Reliability](#9-measuring-reliability)
10. [AI Engineering Reliable Operations](#10-ai-engineering-reliable-operations)
11. [SRE Milestones & History](#11-sre-milestones--history)
12. [Mobaa — Museum of Borgmon Abstract Art](#12-mobaa--museum-of-borgmon-abstract-art)
13. [SRE Local Events](#13-sre-local-events)

---

## 1. The Site Reliability Workbook

**URL:** https://sre.google/workbook/table-of-contents/

The Workbook is not a second SRE book — it's a **field manual** designed to be dog-eared and coffee-stained. Where the original SRE book describes *why* Google does SRE, the Workbook describes *how* to implement it in your own organization.

### Structure and Contents

| Part | Section | What It Adds |
|------|---------|-------------|
| **I. Foundations** | Implementing SLOs, SLO Case Studies, Monitoring, Alerting on SLOs, Eliminating Toil, Simplicity | Practical templates and worked examples for the core SRE disciplines |
| **II. Practices** | On-Call, Incident Response, Postmortem, Managing Load, NALSD, Data Pipelines, Config Design, Config Specifics, Canarying | Step-by-step operational playbooks |
| **III. Processes** | Identifying Overload, SRE Engagement Model, Reaching Beyond Your Walls, Team Lifecycles, Org Change Management | The people and organizational side of reliability |
| **Appendices** | Example SLO Document, Error Budget Policy, Postmortem Analysis Results | Ready-to-adapt templates |
| **Case Studies** | Evernote, Home Depot, New York Times | Real-world implementations outside Google |

### Critical Additions Over the Original Book

**1. The NALSD (Non-Abstract Large System Design) Chapter**
The original book mentions systems design briefly. The Workbook devotes a full chapter to NALSD — the structured process Google SREs use to reason about planet-scale systems. This is arguably the most transferable skill in the entire SRE canon. The chapter walks through: defining the problem scope, sketching architecture, evaluating tradeoffs (consistency vs. availability, latency vs. throughput), and stress-testing the design against failure modes. It's the same framework used in Google's SRE interview loop.

**2. The SRE Engagement Model**
The original book implies that SRE teams interact with product teams, but the Workbook makes the model explicit with three tiers:
- **Hands-off (consultative):** SRE provides advice, product teams own operations
- **Limited engagement (embedded):** SRE co-owns parts of the service
- **Full engagement (service ownership):** SRE operates the service end-to-end

Each tier comes with clear entry criteria — error budgets, monitoring maturity, on-call readiness — that the product team must meet before SRE commits deeper engagement. This is fundamentally an *org design* pattern, not a technical one.

**3. Org Change Management**
The chapter on organizational change management is unique in the SRE literature. It addresses the human dynamics of adopting reliability practices: how to build coalitions, how to handle resistance, how to measure cultural adoption alongside technical SLIs. The original SRE book is almost silent on this, yet it's the single biggest obstacle most organizations face.

**4. Practical Templates**
The appendices include:
- An SLO document template with all the fields an SLO should capture (service definition, SLI specification, measurement methodology, target, window, policy)
- An error budget policy template covering governance, exceptions, and quarterly reviews
- A postmortem action-item tracking format with owner, deadline, and evidence-of-completion fields

These alone make the Workbook worth its weight. For teams adopting SRE, copying these templates and filling them in is faster and more reliable than inventing them from scratch.

**5. Case Studies from Non-Google Companies**
Evernote, Home Depot, and the New York Times are not Google-scale but face Google-class reliability problems with far fewer resources. Their case studies demonstrate how SRE principles adapt when you don't have a Borg cluster or a dedicated monitoring team. The New York Times case study on migrating from a monolithic CMS to microservices while maintaining 99.9%+ availability is particularly instructive.

**How to Use It:** If the SRE book is for *understanding*, the Workbook is for *doing*. Read a Workbook chapter, then immediately implement its template or process with your team. The NALSD and Engagement Model chapters should be on the syllabus for any SRE candidate's first 90 days.

---

## 2. Building Secure & Reliable Systems

**URL:** https://google.github.io/building-secure-and-reliable-systems/

This is a full-length book published collaboratively by Google's SRE and Security teams. It is available freely online and was published in 2020 — newer than the original SRE book (2016) and the Workbook (2018).

### How Security and SRE Converge

The foundational insight of this book is that **reliability and security are the same class of problem**: both are emergent properties of a system that cannot be added after the fact, both are eroded by complexity, both require cultural investment, and both fail catastrophically when treated as an afterthought.

| Dimension | Pure Reliability View | Pure Security View | Converged View |
|-----------|----------------------|--------------------|----------------|
| Failure mode | Service degradation | Breach | Both cause trust erosion |
| Measurement | Error budget burn | Time-to-patch / MTTR for incidents | Integrated risk scorecard |
| Response | Incident commander | Incident commander + forensics | Unified crisis response |
| Design principle | Redundancy, graceful degradation | Least privilege, defense in depth | Both, with explicit tradeoff awareness |

### Key Chapters and Their SRE Relevance

**Part II: Designing Systems (Proxies, Tradeoffs, Least Privilege)**

The Design Tradeoffs chapter (Chapter 7) is a must-read for SREs. It introduces a framework for reasoning about nonfunctional requirements (NFRs) — the properties a system must have but which aren't captured in feature specifications. It distinguishes between *feature properties* (what the system does) and *emergent properties* (how the system behaves — reliability, security, performance, operability). The key move is recognizing that emergent properties **compete**: improving security often hurts performance; improving performance often hurts reliability (redundancy adds latency). The chapter provides a structured way to make these tradeoffs explicit rather than implicit.

The Least Privilege chapter (Chapter 9) extends the SRE concept of "simplicity" into access control. The principle of least privilege — granting only the permissions a component or person needs to function — is directly analogous to the SRE principle of minimizing surface area. The chapter also covers **breakglass mechanisms**: emergency access systems that bypass normal controls during outages. Any SRE who has been locked out of a production system during an incident should study this chapter.

**Part IV: Maintaining Systems (Operations, Crisis Response, Recovery)**

The Crisis Response chapter (Chapter 17) is SRE incident management seen through a security lens. It introduces:
- **Purple teaming:** combining red (adversarial) and blue (defensive) teams to test both reliability and security posture simultaneously
- **Adversarial incident response:** treating every incident as potentially malicious until proven otherwise, while avoiding the paralysis that assumption can cause
- **Recovery with forensic preservation:** the tension between restoring service quickly and preserving evidence for post-incident analysis

**Part V: Organizing for Reliability and Security**

The Culture and Community chapters (19-20) address something the SRE book touches but doesn't fully explore: **blameless culture in a security context**. Security incidents often involve intent, negligence, or policy violations — the blameless model breaks down when malice is involved. The book offers a nuanced framework for distinguishing between honest mistakes, reckless behavior, and malicious actions, and handling each differently.

**Synthesis:** This book extends the SRE profile into adversarial thinking. Every SRE should read at least Chapters 7 (Design Tradeoffs), 9 (Least Privilege), and 17 (Crisis Response). For teams running ML models or public APIs, Chapter 14 (Testing) covers adversarial testing techniques (fuzzing, chaos engineering with malicious inputs) that are directly applicable.

| Comparison | SRE Book | Building Secure & Reliable Systems |
|-----------|----------|-----------------------------------|
| Focus | Operational reliability | Reliability at the intersection of ops and security |
| Target audience | Aspiring/practicing SREs | SREs + Security engineers + Architects |
| Unique contribution | Error budgets, toil, monitoring philosophy | Breakglass, least privilege, adversarial thinking |
| Age | 2016 | 2020 |
| Best consumed | Cover-to-cover | Selected chapters + reference |

---

## 3. SRE Classroom — Workshops

SRE Classroom is Google's most underappreciated resource. These are structured, multi-hour workshops designed to be run with teams. They develop the **systems thinking muscle** that distinguishes senior SREs from junior ones.

### Workshop 1: Distributed PubSub

**URL:** https://sre.google/classroom/distributed-pubsub/

This workshop asks participants to design a planet-scale pub-sub messaging system. It is explicitly a **NALSD (Non-Abstract Large System Design)** exercise.

**What it covers:**
- Correctness: At-least-once vs. exactly-once delivery, ordering guarantees, deduplication
- Reliability: How to survive server failures, network partitions, and leader election
- Performance: Throughput scaling, batching strategies, backpressure
- Communication styles: Synchronous vs. asynchronous, push vs. pull

**Why it matters for SRE thinking:**
The workshop forces participants to make explicit tradeoffs that in production are often implicit. By reasoning through the design on paper, you develop the mental habit of asking "what happens when this component fails?" before you build it — rather than discovering failure modes during an outage.

The workshop also teaches **structured communication of architectural decisions**. The output is not a UML diagram but a written design document organized by requirements, assumptions, architecture, tradeoffs, and failure modes — exactly the format used in Google SRE design reviews.

### Workshop 2: Distributed Image Server

**URL:** https://sre.google/classroom/imageserver/

A more concrete workshop focusing on the design of a photo storage and serving system. Key concepts:

| Concept | Workshop Exercise | SRE Relevance |
|---------|------------------|---------------|
| Sharding | Partitioning images across servers | Horizontal scaling patterns |
| Replication | K-replication with quorum reads/writes | Consistency vs. availability tradeoffs |
| Latency | Edge caching, CDN offloading | SLO design for user-facing latency |
| Load balancing | Consistent hashing, hot-spot mitigation | Capacity planning and traffic management |

### Workshop 3: The Art of SLOs

**URL:** https://sre.google/resources/practices-and-processes/art-of-slos/

A workshop that guides participants through defining SLIs, SLOs, and error budgets for a fictional mobile game (Quest Squad). This is the most practical SLO workshop available publicly.

**What it teaches:**
- How to decompose a user-facing service into critical user journeys (CUJs)
- How to define SLIs that actually measure user experience, not system internals
- How to set SLO targets that balance reliability investment with feature velocity
- How to use error budgets to make deployment decisions

**Synthesis:** The three workshops together form a **capstone curriculum**: NALSD reasoning (PubSub), system architecture design (Image Server), and reliability measurement (Art of SLOs). A team that completes all three workshops has internalized the core SRE skill set at a depth that reading alone cannot provide.

| Workshop | Duration | Pre-read Skills | Best For |
|----------|----------|-----------------|----------|
| Distributed PubSub | 3-4 hours | Basic distributed systems concepts | Senior ICs, design interview prep |
| Image Server | 2-3 hours | Web architecture, caching | Mid-level SREs, backend engineers |
| Art of SLOs | 2-3 hours | Familiarity with SRE concepts | New SRE team members, product teams adopting SRE |

---

## 4. SRE Prodcast

**Available on:** YouTube, Spotify, Apple Podcasts, RSS

Six seasons of Google SREs talking about their work. The Prodcast is not a course — it's a **cultural artifact**. It captures the voice and thinking of SREs in a way that formal writing cannot.

### Season-by-Season Guide

| Season | Theme | Best Episodes for... |
|--------|-------|---------------------|
| 1 | SRE Fundamentals | **New SREs** — explains the basics: what is SRE, how it differs from DevOps, the birth of the discipline at Google |
| 2 | SLOs Deep Dive | **Teams adopting SLOs** — practical discussions of SLI design, error budget policies, and the human side of setting targets |
| 3 | Prodcast Live! | **Experienced SREs** — live Q&A sessions where audience questions drive the conversation. Unpredictable and illuminating |
| 4 | On-Call | **On-call practitioners** — the psychology of pager duty, sustainable rotations, incident escalation, the "Ops-Exhausted" problem |
| 5 | SRE Careers | **Aspiring SREs and managers** — how to grow as an SRE, what distinguishes staff/principal SREs, the non-linear career path |
| 6 | Modern SRE | **Everyone** — reliability in ML systems, platform engineering, SRE in regulated industries, the future of the discipline |

### How to Use the Prodcast for Onboarding

Design a 6-week "listening curriculum" for new SRE team members:

- **Week 1:** Season 1, Episodes 1-3 (What is SRE, Birth of SRE, The CRE Model) — establishes foundational context
- **Week 2:** Season 2, Episodes 1-2 (SLOs in Practice, Error Budget Conversations) — complements reading the SLO chapters
- **Week 3:** Season 4, Episodes 1-3 (On-Call Psychology, Incident Management, Postmortems) — prepares new team members for their first on-call rotation
- **Week 4:** Season 3, Live episodes (any 2) — exposes them to unscripted SRE thinking
- **Week 5:** Season 5, Episodes 1-2 (Career Growth, Technical Leadership) — sets expectations for growth
- **Week 6:** Season 6, Episodes covering ML reliability and platform engineering — connects SRE to their likely future work

---

## 5. SRE Video Gallery

**URL:** https://sre.google/ (Video Gallery section)

90+ videos spanning SREcon conferences (2014-2025), GOTO conferences, and Google Technical Learning sessions. Filterable by: AI/ML, SLOs, Observability, Systems, Security.

### Curated Learning Path by Topic Area

| Topic | Must-Watch Talk | Speaker / Event | Why |
|-------|----------------|-----------------|-----|
| **SLOs** | "The Art of SLOs" workshop recording | SREcon | The workshop in action — watch before running it with your team |
| **Observability** | "Monitoring Distributed Systems" | Google Tech Learning | Foundational talk on the difference between monitoring and observability |
| **Incident Response** | "Incident Management at Google" | SREcon Americas | How Google runs incidents — command structure, communication, handoffs |
| **Postmortems** | "Blameless Postmortems" | SREcon EMEA | The cultural prerequisites for effective postmortems |
| **Capacity Planning** | "Managing Load" | SREcon Americas | Google's approach to load shedding, rate limiting, and capacity |
| **AI/ML Reliability** | "Reliability for ML Systems" | SREcon 2023-2024 | The latest thinking on non-deterministic system reliability |
| **Security** | "Building Secure & Reliable Systems" talk | GOTO Chicago | Overview talk by the book's authors — great starting point |
| **Cultural** | "Why Heroism is Bad" | SREcon | The definitive version of this argument (see section 7) |

### Strategic Use

The video gallery is best used as a **just-in-time learning resource**. When preparing for a specific initiative (adopting SLOs, redesigning on-call, running a postmortem), watch the related talks 2-3 days before the event. The talks from 2022-2025 are most relevant for modern practice; the 2014-2018 talks are valuable for historical context but some specific advice (e.g., about specific tools) is outdated.

---

## 6. STPA — System Theoretic Process Analysis

**URL:** https://sre.google/resources/practices-and-processes/stpa/

STPA is a hazard analysis technique developed by Nancy Leveson at MIT. Google has adopted it for analyzing complex system failures. It is fundamentally different from the postmortem and root-cause analysis (RCA) approaches most SREs know.

### How STPA Differs from Traditional RCA

| Dimension | 5 Whys / RCA | STPA |
|-----------|-------------|------|
| Unit of analysis | Linear chain of events | Control loops and constraints |
| Causal model | Root cause (single failure) | Emergent from interactions |
| Scope | Individual incident | System design |
| Output | Corrective actions on specific components | Design changes to control structure |
| Best for | Simple, well-understood failures | Complex, socio-technical failures |

### Control Loops as the Unit of Analysis

STPA treats every system as a set of **control loops**: a controller (human or automated) that sends control actions to a process, and receives feedback from sensors. Failures occur when:

1. Control actions are missing (the controller doesn't act when it should)
2. Control actions are provided incorrectly (wrong action)
3. Control actions are provided at the wrong time (too early, too late, or out of order)
4. Control actions stop too soon or continue too long

This framing is powerful for SRE because it encompasses **both technical and human failures** in a single model. An on-call engineer who doesn't respond to an alert (a missing control action) can be analyzed using the same framework as a load balancer that fails to route traffic away from a degraded backend.

### When to Use STPA vs. 5 Whys vs. Fault Tree Analysis

| Scenario | Recommended Method | Why |
|----------|-------------------|-----|
| Simple, isolated failure (e.g., single server crash) | 5 Whys | Fast, cheap, sufficient |
| Known failure mode with clear path (e.g., disk full) | Fault Tree Analysis | Systematic, covers combinatorics |
| Complex socio-technical failure (e.g., cascading outage across teams) | STPA | Captures control loop dynamics |
| Safety-critical or ML system failure | STPA | Handles non-deterministic behavior |
| Postmortem after a major incident | 5 Whys + STPA | Whys for immediate actions, STPA for systemic redesign |

### Value for ML and Safety-Critical Systems

Traditional RCA assumes a **deterministic causal chain**: A caused B which caused C. This breaks down for ML systems where the same input can produce different outputs (model drift, non-deterministic inference) and for safety-critical systems where the cost of failure is so high that you need to prevent it before it occurs.

STPA is **proactive**: it analyzes the design of the control structure and identifies unsafe control actions before they cause failures. For ML systems, this means analyzing the feedback loop between training data quality, model deployment, and production monitoring — a control structure that traditional RCA is ill-equipped to model.

**Synthesis:** STPA is an advanced technique, not a replacement for postmortems. Every SRE should be proficient in 5 Whys. Senior SREs should add STPA to their analysis toolkit for complex, non-deterministic, and safety-critical systems.

---

## 7. Why Heroism is Bad

This is one of the most cited pieces of SRE cultural writing at Google. It exists as a talk (in the Video Gallery) and as a principle woven throughout the SRE book and Workbook.

### The Hero Culture Anti-Pattern

The "hero" is the engineer who stays up all night fighting a fire, who has access to systems no one else understands, who carries the pager for their entire team. In most organizations, heroism is rewarded — promotions, praise, the adrenaline of being "the one who saved the day."

**Why it's bad for reliability:**

| Hero Behavior | Systemic Cost |
|--------------|---------------|
| Fixing incidents solo without documentation | Knowledge silo — only one person can fix it next time |
| Working off-hours to "save" a service | Burnout, turnover, and expertise loss |
| Accumulating niche knowledge without sharing | Bus factor of 1 |
| Being the "go-to" person for every problem | Overload, bottleneck, no time for proactive work |
| Working around systemic problems | Masks the need for engineering investment |

### Connection to Blameless Culture and Toil Elimination

The heroism argument connects directly to two pillars of SRE:

1. **Blameless culture:** Hero narratives assign blame by inversion — the hero is celebrated, which implies everyone else (who didn't stay up all night) is at fault. This creates a culture where failures are hidden unless someone heroes their way through them.

2. **Toil elimination:** Heroic firefighting is the purest form of toil — manual, repetitive, non-durable work that could be automated. An organization that rewards heroism is incentivizing its engineers to *not* reduce toil, because reducing toil reduces opportunities for heroism.

**The antidote:** The SRE model argues that reliability should be achieved through engineering, not heroism. If a service fails the same way twice, the correct response is not to train better heroes — it's to build better systems. Error budgets, monitoring, SLOs, and blameless postmortems are all structural anti-heroism measures.

**Synthesis:** This is the most important cultural concept for new SRE teams to internalize. It's also the hardest to implement because it requires changing what "good work" looks like in an organization. Use the talk for team discussions; use the SRE book's toil framework for implementation.

---

## 8. SRE Fundamentals Course

A structured course that maps directly to the SRE book's chapter structure. It is Google's internal onboarding curriculum for new SREs, made public.

### How It Maps to the SRE Book

| Module | SRE Book Chapter(s) | Topic |
|--------|---------------------|-------|
| 1 | 1-2 | Introduction, The Environment (why SRE exists) |
| 2 | 3-4 | Embracing Risk, SLOs |
| 3 | 5-6 | Eliminating Toil, Monitoring |
| 4 | 7-8 | Automation, Release Engineering |
| 5 | 9-10 | Simplicity, Practical Maturation |
| 6 | 11-13 | On-Call, Incident Response, Postmortems |
| 7 | 14-15 | Managing Load, Capacity Planning |
| 8 | 16-18 | Emergency Response, Data Pipelines, Prod Meetings |
| 9 | 19-21 | SRE Org Models, CRE, the Future |

### Using It for Onboarding

The course provides:
- **Slides** that can be adapted for internal training
- **Discussion questions** for each module
- **Exercises** that test comprehension

This is the **ideal onboarding structure** for new SRE team members. A recommended 9-week cadence:

- Week N: Read the corresponding SRE book chapters
- Week N+1: Watch the Fundamentals course module
- Week N+1 follow-up: Team discussion of the discussion questions and exercises

This pairs reading (individual, deep) with discussion (team, social) — reinforcing learning through both modes.

The course does **not** cover the Workbook's practical templates or the organizational change content. It is purely an introduction to the concepts. Use it as the first pass, then layer on the Workbook for implementation.

---

## 9. Measuring Reliability

**URL:** https://sre.google/resources/practices-and-processes/measuring-reliability/

A focused resource that addresses the practical mechanics of defining and measuring reliability. It fills a gap between the SRE book's conceptual framework ("you need SLOs") and the actual work of implementing them.

### What It Covers

- **Service boundaries:** Where does "your service" end? Shared infrastructure, APIs, client-side code — what's included in your reliability measurement?
- **SLI selection:** Practical criteria for choosing which metrics to use as SLIs, with worked examples for different service types (CRUD APIs, batch processing, streaming, storage)
- **Measurement windows:** Rolling windows vs. calendar windows, sliding vs. fixed windows, and the implications for error budget interpretation
- **Counting failures:** How to count events (request-level, user-level, time-based) and the impact of each counting methodology on perceived reliability
- **Target setting:** How to set SLO targets that are achievable but meaningful, with guidance on distinguishing aspirational targets from contractual ones

### How It Complements the SLO/SLI Framework Reference

The SLO/SLI framework in the SRE book tells you *what* to build. Measuring Reliability tells you *how to build it correctly*. It covers edge cases the book glosses over:

- What happens when your measurement pipeline is itself unreliable? (The "who measures the measurer" problem)
- How to handle services with heterogeneous request types (reads vs. writes, small vs. large payloads)
- How to aggregate reliability across service tiers (critical vs. best-effort)

**Synthesis:** Read this *after* the SRE book's SLO chapters and *before* implementing your first SLO. It will save you from common measurement mistakes that produce misleading reliability numbers.

---

## 10. AI Engineering Reliable Operations

**Marked "New!" on sre.google**

This is Google's most recent SRE content, reflecting the growing importance of ML systems in production. It addresses the unique challenges of maintaining reliability in non-deterministic systems.

### Why ML Reliability is Different

| Dimension | Traditional Software | ML Systems |
|-----------|---------------------|------------|
| Correctness | Deterministic — same input, same output | Statistical — same input, different output |
| Failure mode | Crash, exception, error code | Silent degradation, model drift, concept drift |
| Debugging | Reproduce with fixed input | Reproduce in expectation (requires data distribution analysis) |
| Rollback | Revert to previous version | Revert model + data + features (all must be consistent) |
| Monitoring | Latency, error rate, throughput | Latency, error rate + prediction distribution, feature distribution, data quality |

### How This Extends SRE into Non-Deterministic Systems

The core SRE framework (SLOs, error budgets, monitoring, incident response) still applies to ML systems, but every technique needs adaptation:

- **SLIs for ML:** Must include data quality metrics (feature freshness, null rates, distribution shifts) alongside traditional performance metrics. A model may have 99.9% uptime but produce useless predictions because input features have drifted out of distribution.
- **Error budgets for ML:** Need to account for model staleness. An error budget that only tracks API failures misses the more common failure mode: a model that's still running but producing degraded predictions because it hasn't been retrained on recent data.
- **Monitoring for ML:** Requires multiple "verification layers": service health, pipeline health, model health, and business outcome health. Each layer can fail independently.
- **Incident response for ML:** Cannot just "restart" a model. Fixing a degraded model may require: rolling back training data, retraining with corrected labels, adjusting inference parameters, or even redesigning the feature pipeline.

**Synthesis:** This is essential reading for any SRE working with ML-powered systems. It should be paired with the STPA resource (section 6) — STPA provides the analysis methodology for understanding control loop failures in these complex, non-deterministic systems.

---

## 11. SRE Milestones & History

### Twentieth Anniversary (https://sre.google/20/)

20+ video testimonials from SREs across Google, from the founders of the discipline to current practitioners.

**What it reveals:**
- **The origin story:** Ben Treynor Sloss formalized SRE in 2003 when Google was already running at planetary scale. The early years were about survival; SRE emerged as a response to the failure of traditional ops models.
- **The inflection points:** The introduction of error budgets (c. 2010) was the theoretical breakthrough that made SRE scalable. Before error budgets, reliability was a theological argument; after, it became an engineering tradeoff.
- **The cultural evolution:** Early SRE was a startup within Google — small, elite, and somewhat adversarial with product teams. Modern SRE is more collaborative, with clearer engagement models and less "us vs. them" dynamics.

### Ask an SRE at Next '26 (https://sre.google/next26/)

A recent Q&A session (2025/2026) that reveals current SRE thinking at Google.

**Key takeaways:**
- **Platform engineering is SRE's evolution:** The line between SRE and platform engineering is blurring. SREs increasingly build internal platforms that product teams use to self-serve reliability.
- **AI as both tool and challenge:** SREs use AI for incident classification, anomaly detection, and runbook automation — but also struggle with the reliability of AI-powered systems themselves.
- **The diversity problem:** Google SRE remains less diverse than they'd like. The session includes honest discussion of what's being done to change this.

### Why History Matters

SRE is a young discipline (est. 2003). Its practices evolved in response to specific failures and constraints at Google. Understanding that history helps practitioners:

1. **Recognize which practices are universal** (error budgets, toil elimination, blameless culture) vs. **which are Google-specific** (Borg, specific monitoring architectures, engagement models that assume massive scale)
2. **Understand why practices are the way they are** — not just *what* to do, but *why* it worked at Google and why it might need adaptation elsewhere
3. **Avoid repeating the same mistakes** — most SRE practices are formalized scar tissue from specific failures

---

## 12. Mobaa — Museum of Borgmon Abstract Art

> *"Borgmon was a beautiful monster."* — Google SRE

Mobaa is a collection of **monitoring data visualizations turned into art**. Borgmon was Google's internal monitoring system — a powerful but famously idiosyncratic tool with its own query language, visualization format, and idiosyncrasies. As Borgmon was phased out (replaced by Monarch and other systems), some SREs created abstract art from old monitoring data.

### What Mobaa Reveals About SRE Culture

1. **Affection for the tool:** SREs form deep attachments to their tools. Borgmon was ugly, hard to configure, and opaque — but it *worked* at a scale no other tool could handle. The art is a form of appreciation.

2. **Operational folklore:** Monitoring data captures the history of a service's life — every spike is an outage, every dip is a deployment. The art preserves that history in a way that dashboards don't.

3. **Team-building through shared experience:** Mobaa started as an internal exhibit. SREs submitted pieces, voted on favorites, and held "gallery openings." This is SRE culture expressing itself through creativity rather than postmortems and runbooks.

4. **Healthy detachment from production:** Creating art from monitoring data transforms the relationship to production — from anxiety ("my pager might fire") to reflection ("this is what our service looked like"). This is psychologically healthy for engineers whose entire job is to worry about failure.

### Value for the SRE Profile

Mobaa is not a learning resource in the traditional sense. Its value is **cultural**: it demonstrates that SRE teams function best when they have outlets beyond pure operations. The best SRE teams are not the ones that never have outages — they're the ones that have rituals for processing and learning from outages. Mobaa is one such ritual.

**Synthesis:** Include Mobaa in the SRE profile's recommended "cultural immersion" materials alongside the Prodcast and the Milestones videos. It won't teach anyone to run better systems, but it will help them understand the community they're joining.

---

## 13. SRE Local Events

**Meetups in:** Munich, NYC, Pittsburgh, Sunnyvale

Google hosts regular SRE meetups in these cities. These are community events, not corporate communications.

### What They Offer

- **Presentations from local practitioners** (not just Googlers) sharing their SRE implementations
- **Workshop sessions** where attendees work through problems together
- **Networking** with other SREs facing similar challenges
- **Recruiting** for Google SRE roles (for those interested)

### Community Knowledge-Sharing Value

Local meetups serve a function that documentation cannot: they provide **context**. Every SRE implementation is different — understanding how other organizations adapted SRE practices to their specific constraints is invaluable. The meetup format also allows for questions and conversations that documentation and talks don't accommodate.

**For the SRE profile:** Include the meetup URLs and mention that these are the most reliable source of "SRE in the real world" case studies outside of formal case study publications. For teams without a local meetup, the YouTube recordings of past meetup talks are a good substitute.

---

## Cross-Reference: A Learning Path Matrix

The table below maps each resource to learning goals and experience levels.

| Resource | Deep Theory | Practical Application | Culture | Beginner | Intermediate | Advanced |
|----------|-------------|----------------------|---------|----------|-------------|----------|
| SRE Book | ★★★ | ★★ | ★★★ | ★ | ★★★ | ★★★ |
| SRE Workbook | ★★ | ★★★ | ★★ | ★★ | ★★★ | ★★★ |
| Building Secure & Reliable Systems | ★★★ | ★★ | ★★ | ★ | ★★ | ★★★ |
| SRE Classroom Workshops | ★★ | ★★★ | ★ | ★ | ★★★ | ★★★ |
| SRE Prodcast | ★ | ★ | ★★★ | ★★★ | ★★★ | ★★ |
| Video Gallery | ★★ | ★★ | ★★★ | ★★ | ★★★ | ★★★ |
| STPA | ★★★ | ★ | ★ | ★ | ★★ | ★★★ |
| Why Heroism is Bad | ★ | ★ | ★★★ | ★★★ | ★★★ | ★★ |
| Fundamentals Course | ★★ | ★ | ★★ | ★★★ | ★★ | ★ |
| Measuring Reliability | ★★ | ★★★ | ★ | ★★★ | ★★ | ★★ |
| AI Engineering | ★★★ | ★★ | ★ | ★ | ★★ | ★★★ |
| Milestones & History | ★ | ★ | ★★★ | ★★ | ★★★ | ★★ |
| Mobaa | ★ | ★ | ★★★ | ★ | ★★ | ★★★ |
| Local Events | ★ | ★★ | ★★★ | ★★ | ★★★ | ★★★ |

**Key:** ★ = Limited value, ★★ = Moderate value, ★★★ = High value

---

## Summary: The SRE Learning Ecosystem at a Glance

```
SRE ECOSYSTEM MAP
=================

                             [Foundations]
         ┌────────────────────────────────────────────┐
         │  SRE Book (original)                       │
         │  SRE Fundamentals Course (onboarding)      │
         │  Measuring Reliability (practical SLOs)    │
         └────────────────────────────────────────────┘
                              │
         ┌────────────────────┴────────────────────┐
         │                                         │
  [Implementation]                         [Advanced Topics]
  SRE Workbook                          Building Secure & Reliable
  - Templates, case studies             - Adversarial thinking
  - NALSD, engagement model             - Breakglass mechanisms
  - Org change management               - Design tradeoffs
  - Postmortem analysis                          │
         │                                         │
         ├──────────────────┬──────────────────────┤
         │                  │                      │
  [Hands-On Skills]   [Analysis Methods]    [New Frontiers]
  SRE Classroom       STPA                  AI Engineering
  - PubSub workshop   - Control loops       - ML reliability
  - Image server      - Safety-critical     - Non-deterministic
  - Art of SLOs       - ML systems          - Verification layers
         │                  │                      │
         └──────────────────┴──────────────────────┤
                                                   │
                              ┌────────────────────┘
                              │
                    [Culture & Community]
                    - SRE Prodcast (6 seasons)
                    - Video Gallery (90+ talks)
                    - Why Heroism is Bad
                    - Milestones & History
                    - Mobaa (monitoring art)
                    - Local Events (meetups)
```

---

## Recommended Reading Order by Role

### For a New SRE (first 90 days)
1. SRE Fundamentals Course (weeks 1-3)
2. SRE Book, Chapters 1-6 (weeks 1-2)
3. Measuring Reliability (week 3)
4. SRE Prodcast, Season 1 (weeks 2-4, for commuting)
5. Why Heroism is Bad (week 4 — cultural context)
6. SRE Workbook, Parts I-II (weeks 5-8)
7. SRE Classroom: Art of SLOs (week 8, run with team)
8. Milestones & History videos (week 9 — inspiration)

### For an Experienced SRE Expanding into Security
1. Building Secure & Reliable Systems, Chapters 1, 7, 9, 17 (read in order)
2. STPA overview (control loops as analysis tool)
3. Video Gallery: Security track talks
4. SRE Classroom: Distributed PubSub workshop (applies NALSD to a security-relevant system)

### For an SRE Team Adopting SLOs
1. SRE Book, Chapters 3-4 (Embracing Risk, SLOs)
2. Measuring Reliability (practical pitfalls)
3. SRE Workbook, Chapter on Implementing SLOs + appendices
4. SRE Classroom: Art of SLOs (team workshop)
5. SRE Prodcast, Season 2 (listening while implementing)
6. SRE Workbook, SLO Case Studies (Home Depot, New York Times)

### For an SRE Working with ML Systems
1. AI Engineering Reliable Operations (latest guidance)
2. STPA (control loop analysis for non-deterministic systems)
3. Building Secure & Reliable Systems, Chapter 14 (Testing)
4. Video Gallery: AI/ML track
5. SRE Prodcast, Season 6 (Modern SRE, ML episodes)

---

*This guide was synthesized from the resources at https://sre.google/ and related Google publications. It is a living document — the SRE ecosystem evolves as Google publishes new content. Revisit the source sites periodically for updates.*

# Third-Party Dependency Reliability

Use this reference when a service relies on a vendor, managed platform, external API, identity provider, payment processor, DNS, CDN, messaging system, or other dependency outside the team's direct control.

## Source anchors

Synthesized from *Seeking SRE*, “Working with Third Parties Shouldn’t Suck,” “SRE as a Success Culture,” “Engineering for Data Durability,” and “Introduction to Machine Learning for SRE,” with dependency and overload practices from *The Site Reliability Workbook*, “Managing Load” and “Data Processing Pipelines.”

## Dependency record

For each important dependency, record:

- user journeys and SLOs affected;
- owner on both sides and escalation path;
- vendor SLO, quotas, rate limits, maintenance model, and support boundaries;
- observed latency, errors, freshness, correctness, and availability from your service boundary;
- credential, quota, region, API-version, and account failure modes;
- fallback, cache, queue, degraded mode, or alternate-provider plan;
- data, privacy, and exit implications;
- last failure exercise and unresolved repairs.

A vendor dashboard is useful evidence but is not the service's user-visible boundary. Providers can report healthy while a regional route, credential, quota, integration version, or client-side timeout is failing.

## Buy, build, or adopt

Classify the dependency before debating implementation:

1. How critical is it to a user journey?
2. How likely and severe are its failure modes?
3. Can the team mitigate or substitute it?
4. What operational and security expertise does each option require?
5. What is the cost of exit, migration, or data recovery?

Criticality and substitutability determine how much redundancy, contract, testing, and internal expertise are justified. Do not build a replacement merely to avoid every external dependency, and do not accept a single provider as harmless because it is popular.

## Failure behavior

Design and test the client behavior explicitly:

- bounded timeouts aligned with the user journey;
- retries only for safe and retryable operations, with exponential backoff and a cap;
- circuit breaking or admission control to prevent retry storms;
- idempotency and deduplication for retried writes;
- queueing or asynchronous completion when delay is acceptable;
- cache or last-known-good behavior where correctness permits;
- clear degraded response and support communication;
- manual or alternate-provider path for critical operations.

If no fallback is feasible, say so. An undocumented dependency with no mitigation is a reliability risk, not an “external limitation.”

## Review and exercise

Review dependency health on a fixed cadence and after incidents. Run failure exercises for the most consequential paths, including provider outage, partial outage, elevated latency, quota exhaustion, expired credentials, malformed responses, and loss of the provider's status API. Verify that alarms, escalation, user messaging, data reconciliation, and recovery actually work.

## Agent procedure

Use this reference with `slo-sli-framework.md`, `reliability-design-and-change.md`, and `templates/reliability-design-review.md`. Report provider claims separately from measurements taken at the service boundary. Treat “no alternative” as an explicit risk requiring an owner and review date.

# Toil Elimination Reference

## Overview

Toil is the enemy of reliable systems and the single largest drain on Site Reliability Engineering productivity. This reference provides a comprehensive framework for identifying, measuring, prioritizing, and eliminating toil from operational workflows. Rooted in Google's SRE practices as documented in the Site Reliability Engineering books, this guide is designed for SRE agents and teams who need a structured approach to reducing operational burden.

---

## 1. What Is Toil?

Toil is a specific category of operational work that is fundamentally different from engineering work. It is not merely "hard work" or "a lot of work" — it is work that exhibits five defining characteristics.

### The Five Characteristics of Toil

**1. Manual**
The work requires human intervention. There is no automated process handling it. A human must SSH into a machine, click a button, run a command, or manually inspect output. If a system could handle the task without human involvement but currently doesn't, that work is toil.

**2. Repetitive**
The work is performed repeatedly. You handle the same kind of ticket, run the same command, or follow the same runbook more than once. A one-off task is not toil, even if it is manual and tedious. Repetition is the key differentiator — if you've done it three times, you will likely do it a hundred more.

**3. Automatable**
The work could be automated by engineering effort. If the task requires genuine human judgment that cannot be encoded in logic, it may not be toil — it may be craft. But if a program or script could perform the action with the same or better result, the work qualifies as automatable and therefore as toil.

**4. Tactical**
The work is reactive and interrupt-driven rather than strategic. It comes in as alerts, tickets, or pages and demands immediate attention. Tactical work displaces proactive engineering improvements because it consumes the time and cognitive energy needed for design and development.

**5. No Enduring Value**
The work does not make your systems permanently better. Once completed, the service is simply back to its previous state. Nothing was improved, no new capability was added, no debt was paid down. The value of the work decays to zero as soon as the immediate issue is resolved.

**6. Scales Linearly (The Multiplier Problem)**
As the service grows, the amount of toil grows proportionally. If you have 100 servers and each requires 10 minutes of manual patching per month, going to 1,000 servers means 100 minutes. Toil does not benefit from economies of scale — it tracks directly with infrastructure growth. Engineering work, by contrast, produces leverage: one automation script can handle 10 servers or 10,000 with negligible additional cost.

### What Toil Is NOT

- **Engineering work**: Designing, building, testing, and deploying new systems or features.
- **Craft**: Work requiring deep expertise, judgment, and non-repeatable problem-solving.
- **One-off efforts**: Even if painful, a task that will never repeat is not toil (though it may still warrant postmortem-driven changes).
- **On-call response**: Incident response is essential operational work. However, *excessive* on-call toil (page storms, false alarms, manual remediation steps) is a symptom of toil in the monitoring and alerting pipeline.

---

## 2. The 50% Engineering Time Mandate

### The Principle

Google SRE established a foundational rule: **an SRE team should spend no more than 50% of its time on operational work (toil + on-call overhead)**. The remaining 50% must be reserved for engineering projects that reduce toil, improve reliability, or add service capacity.

### Why 50%?

| Reason | Explanation |
|---|---|
| **Sustainability** | Without the 50% cap, toil expands to fill all available time. Teams burn out, attrition spikes, and the service enters a doom loop of increasing operational burden. |
| **Self-Improvement** | The engineering half is what makes tomorrow easier than today. Without it, the team can never escape the toil trap. |
| **Reliability** | Systems that never receive engineering investment become fragile piles of manual procedures. The more toil a team handles, the *less* reliable the service becomes, because improvements are not being made. |
| **Career Health** | Engineers who spend 80-100% of their time on toil stagnate. They learn nothing new, build nothing durable, and leave. The 50% mandate is a career preservation mechanism. |

### How to Measure It

Track engineering time using these categories:

| Category | Description | Counts Toward |
|---|---|---|
| **Pure Toil** | Manual, repetitive, automatable tasks | Operational (50% bucket) |
| **On-Call Overhead** | Incident response, pages, alert triage | Operational (50% bucket) |
| **Engineering Projects** | Building automation, tools, platforms, design | Engineering (50% bucket) |
| **Code Reviews & Design** | Peer review, architecture discussions, documentation of systems | Engineering (50% bucket) |
| **Meetings & Admin** | Stand-ups, planning, 1:1s, HR tasks | Neither (overhead; minimize) |
| **Learning & Growth** | Reading, training, conferences, spike experiments | Engineering (50% bucket) |

### The Breach Response

If operational work exceeds 50% for more than one quarter, the team must:
1. Freeze all new feature work and feature onboarding.
2. Dedicate the *entire* engineering half to toil reduction projects.
3. Escalate to management that the team is in "toil emergency" mode.
4. Re-evaluate the SLA/SLO — if the team cannot sustain the current reliability target without exceeding 50% toil, the SLO is likely wrong.

---

## 3. Toil Assessment Methodology

### Step 1: Tag and Track

Every task an SRE performs must be tagged with a toil classification. Use the following rubric to score each task on a 1-5 scale across each of the five characteristics.

#### Toil Assessment Rubric

| Characteristic | 1 (Not Toil) | 2 (Low) | 3 (Moderate) | 4 (High) | 5 (Pure Toil) |
|---|---|---|---|---|---|
| **Manual** | Fully automated, no human touch | One-click approval needed | CLI command with parameters | SSH + multi-step runbook | Physical data center visit |
| **Repetitive** | Never done before | Once per quarter | Once per week | Daily | Multiple times per day |
| **Automatable** | Requires human judgment (design work) | Would need ML to automate | Could be automated with moderate effort | Trivially scriptable | Already partially automated, needs finishing |
| **Tactical** | Proactive, planned project | Scheduled maintenance | On-call ticket during business hours | Page at 2 AM | Interrupts ongoing engineering work |
| **Enduring Value** | Permanently improves the system | Adds monitoring or documentation | Fixes a bug permanently | Restores service to previous state (only) | Actively makes the system worse (workaround debt) |
| **Scales Linearly** | Cost decreases per-unit with scale | Flat cost regardless of scale | Sub-linear growth | Linear growth | Super-linear growth (more users = more work per user) |

**Scoring**: Sum the six scores. A task scoring 18+ is overwhelmingly toil. A task scoring 6-10 may be legitimate operational work. A task scoring 11-17 is in the grey zone — investigate whether it can be automated or eliminated.

### Step 2: Time Sampling

For one sprint (two weeks) every quarter, have every team member log their time against a toil taxonomy. Use the following categories:

| Toil Category | Examples |
|---|---|
| **Incident Response** | Pages, triage, mitigation, postmortem |
| **Ticket Handling** | User requests, access grants, quota increases |
| **Release Engineering** | Manual deployments, rollbacks, cherry-picks |
| **Monitoring & Alerting** | Tuning thresholds, silencing noise, investigating false positives |
| **Capacity Planning** | Manual scaling, provisioning, resource juggling |
| **Data Operations** | Manual backups, restores, data migrations, log analysis |
| **Access & Permissions** | Granting access, rotating credentials, audit responses |
| **Incident Follow-Up** | Manual remediation of known issues, applying known fixes |

### Step 3: Calculate the Toil Ratio

```
Toil Ratio = (Hours of toil) / (Total engineering hours) × 100
```

**Target**: < 25% toil ratio (leaving room for on-call overhead within the 50% operational cap).

**Warning threshold**: > 35% toil ratio. Immediate intervention needed.

### Step 4: Trend Analysis

Track toil ratio over time. A decreasing trend confirms that toil reduction investments are paying off. An increasing trend means either:
- The service is growing faster than automation efforts
- New sources of toil have been introduced (new features, new integrations)
- Engineering time is being consumed elsewhere (meetings, bureaucratic overhead)

---

## 4. Automation Decision Tree

Not all toil should be automated. Some toil should be eliminated by redesign. Some should be accepted and budgeted for. Use this decision framework to determine the right response.

### Automation Decision Criteria Table

| Criteria | Automate Now | Automate Later | Do Not Automate |
|---|---|---|---|
| **Frequency** | Daily or more | Weekly to monthly | Quarterly or less |
| **Time per occurrence** | > 15 minutes | 5-15 minutes | < 5 minutes |
| **Error cost** | Human errors cause incidents | Human errors cause minor issues | Errors are trivially recoverable |
| **Stability of process** | Process is well-understood and stable | Process changes < 2x/year | Process changes frequently or is poorly understood |
| **Number of practitioners** | > 3 people do this task | 2-3 people do this task | 1 person or specialist only |
| **Business impact of delay** | Delays cause SLA breaches | Delays cause customer complaints | No time sensitivity |
| **Automation complexity** | Can be done in < 2 weeks of effort | 2-8 weeks of effort | > 8 weeks of effort or dependency on other teams |
| **Failure mode of automation** | Safe failure (script errors are recoverable) | Moderate risk | Automation failures could cause data loss or outages |

### Decision Tree Logic

```
Is the task toil? (scores 18+ on rubric)
    |
    +-- No --> Is it valuable engineering work?
    |           +-- Yes --> Keep doing it, document it
    |           +-- No  --> Eliminate or delegate it
    |
    +-- Yes --> Can we eliminate the need entirely?
                |
                +-- Yes --> Redesign the system to make the task irrelevant
                |
                +-- No --> Does it pass the "Automate Now" criteria?
                            |
                            +-- Yes (4+ criteria) --> Build automation
                            |
                            +-- No (0-3 criteria) --> Does it pass "Automate Later"?
                            |                           |
                            |                           +-- Yes --> Track in toil backlog, revisit quarterly
                            |                           |
                            |                           +-- No --> Accept as budgeted toil
                            |
                            +-- Mixed --> Apply cost-benefit analysis:

                    Cost = automation effort (hours)
                    Benefit = time saved per occurrence × frequency × 12 months

                    If Benefit >= 3 × Cost: automate
                    If Benefit < Cost: accept as budgeted toil
                    Otherwise: schedule for next planning cycle
```

### Common Automation Traps

| Trap | Why It's Dangerous | Better Approach |
|---|---|---|
| **Automating a bad process** | You just do the wrong thing faster | Fix the process first, then automate |
| **Gold-plating** | Building "enterprise-grade" automation for a task that happens twice | Use a simple script; upgrade only if frequency increases |
| **Automating instability** | Building automation on top of an unstable system | Stabilize the system first, then automate |
| **The half-automation** | Automating 80% but leaving 20% manual | The 20% still requires a human in the loop, defeating much of the benefit |
| **Automating before measuring** | You don't know how much time you'll actually save | Measure baseline toil time before and after automation |

---

## 5. Categories of Automation

Toil reduction automation falls into four tiers. Each tier represents increasing leverage and decreasing operational burden.

### Tier 1: Script-Based Automation

Single-purpose scripts that replace manual CLI commands or runbook steps.

| Example | Tooling | Effort | Impact |
|---|---|---|---|
| Log rotation script | Bash, Python, Go | Hours | Saves minutes per server per week |
| User access grant script | Python + LDAP library | Days | Saves 15 minutes per ticket |
| Database index rebuild | SQL script + cron | Hours | Eliminates manual DBA tasks |

**Best for**: High-frequency, well-defined, low-risk tasks. These are the "low-hanging fruit" of toil elimination.

### Tier 2: Zero-Touch Automation

Fully automated workflows that require no human initiation or approval. The system detects a condition and acts on it automatically.

| Example | Trigger | Action |
|---|---|---|
| Auto-scaling | CPU/memory threshold breach | Provision or de-provision instances |
| Auto-remediation | Specific error pattern in logs | Restart service, clear cache, rotate log |
| Automated certificate renewal | TTL approaching expiration | Request, validate, and install new certificate |
| Disk cleanup | Disk usage > 85% | Archive old logs, prune unused Docker images |

**Best for**: Well-understood failure modes with safe, reversible automated responses. Always include circuit breakers and kill switches.

### Tier 3: Self-Service Platforms

Web-based or API-driven interfaces that allow stakeholders (developers, product teams, internal users) to perform tasks themselves without SRE involvement.

| Example | Users | Toil Eliminated |
|---|---|---|
| Self-service deployment portal | Developers | "Can you deploy my branch?" requests |
| Self-service access dashboard | All employees | Access grant tickets |
| Service catalog / internal developer portal | All teams | Provisioning requests, environment requests |
| Self-service data export | Analysts, PMs | "Can you run this query for me?" requests |
| Incident dashboard | On-call engineers | Manual log digging, status check spreadsheets |

**Best for**: Tasks that are inherently manual because they require context from the requester. The platform shifts the burden from SRE to the requester, who has the context anyway.

### Tier 4: Design Elimination

The most powerful form of toil reduction: redesigning the system so the toil cannot exist. Rather than automating a bad process, you eliminate the process entirely.

| Example | Before | After |
|---|---|---|
| **Immutable infrastructure** | SSH into servers to patch and debug | Tear down and rebuild from golden images |
| **Blue-green deployments** | Manual rollback procedures | Traffic switch: flip a DNS record |
| **Database sharding** | Manual data migration scripts | Auto-sharding built into the data layer |
| **Config-driven systems** | Per-environment config changes in code | Centralized config service with RBAC |
| **Stateless services** | Session state management, drain procedures | Any instance can serve any request; no state to manage |

**Best for**: Systemic sources of toil that cannot be effectively scripted away. Requires architectural investment but pays the highest long-term dividend.

---

## 6. The Toil-to-Engineering Progression

Toil reduction is not a one-time project — it is a continuous progression. Teams move through distinct phases as they mature.

| Phase | Characteristics | Toil Ratio | Key Action |
|---|---|---|---|
| **1. Survival** | Everything is manual. Pages happen constantly. No automation exists. | 80-100% | Measure and triage. Automate the top 3 sources of pages. |
| **2. Awareness** | Some scripts exist, but they are fragile and unowned. Still mostly reactive. | 60-80% | Build a toil backlog. Assign owners for each automation project. |
| **3. Foundation** | Core automation exists. Runbooks are documented. Alerting is being tuned. | 40-60% | Implement the 50% engineering time mandate. Formalize toil tracking. |
| **4. Leverage** | Self-service platforms emerge. Auto-remediation handles common incidents. | 25-40% | Shift focus to design elimination. Build internal developer platform. |
| **5. Optimization** | Most operations are zero-touch. SRE time is predominantly engineering. | 10-25% | Tune remaining automation. Focus on reliability engineering and feature work. |
| **6. Autopilot** | Systems are largely self-healing. SREs write code and design systems full-time. | < 10% | Innovation mode. SRE is a product engineering function with reliability expertise. |

**The Trap at Each Phase**:

- **Survival → Awareness**: Teams can get stuck in survival mode if management does not grant engineering time. The 50% mandate must be enforced from the top.
- **Awareness → Foundation**: Teams build a dozen fragile scripts that no one maintains. Script rot becomes a new source of toil. Centralize and standardize.
- **Foundation → Leverage**: Teams automate everything *except* the things users ask for. The "last mile" of user requests remains a huge toil sink. Build self-service.
- **Leverage → Optimization**: Auto-remediation can mask underlying instability. Monitor for "silent failures" where automation hides a worsening system.

---

## 7. Common Sources of Toil (Ranked by Frequency)

Based on published SRE experience across multiple large-scale organizations, the following are the most common sources of toil, ranked by how frequently they appear in team assessments.

| Rank | Source | Typical Toil Ratio Contribution | Primary Automation Tier |
|---|---|---|---|
| 1 | **Alert noise / false positives** | 15-25% | Tier 2 (auto-remediation) + redesign alerting |
| 2 | **User / stakeholder requests** (access, deployments, queries, config changes) | 15-20% | Tier 3 (self-service platforms) |
| 3 | **Manual release / deployment processes** | 10-15% | Tier 1 (script) + Tier 4 (CI/CD pipelines) |
| 4 | **Incident response for known issues** | 10-15% | Tier 2 (auto-remediation) + Tier 4 (fix root cause) |
| 5 | **Monitoring and observability maintenance** (dashboards, log parsing, threshold tuning) | 8-12% | Tier 1 (script) + Tier 3 (self-service dashboards) |
| 6 | **Capacity management** (manual scaling, resource provisioning) | 5-10% | Tier 2 (auto-scaling) + Tier 4 (design elimination) |
| 7 | **Data operations** (manual backups, restores, ETL, data migrations) | 5-10% | Tier 1 (script) + Tier 4 (architectural changes) |
| 8 | **Access and permission management** | 5-8% | Tier 3 (self-service) + Tier 4 (PAM integration) |
| 9 | **Configuration management** (manual config changes across environments) | 5-8% | Tier 3 (self-service) + Tier 4 (GitOps) |
| 10 | **Documentation maintenance** | 3-5% | Tier 3 (self-service runbooks) + culture shift |
| 11 | **Postmortem and incident follow-up busywork** | 2-5% | Tier 1 (templating) + culture shift |
| 12 | **Compliance and audit evidence gathering** | 2-5% | Tier 1 (script) + Tier 3 (self-service audit trails) |

### The Pareto Principle Applied to Toil

In most organizations, **20% of toil sources generate 80% of the operational burden**. The top three sources (alert noise, user requests, manual deployments) typically account for 40-60% of all toil. Focusing automation efforts on these three categories first yields the highest return on engineering investment.

---

## 8. Toil Budget Tracking

### The Toil Budget Concept

Treat toil like a financial budget. Each team has a quarterly toil allowance (ideally < 25% of total engineering hours). Once the budget is consumed, no additional toil work is accepted — it must be automated, eliminated, or declined.

### How to Implement

**Step 1: Set the Budget**
- Target toil ratio: 25% (or less for mature teams)
- Hard cap: 35% (if exceeded, trigger a "toil freeze" — all non-critical work stops until toil is back under budget)

**Step 2: Track Consumption**
- Use a ticketing system where every task is tagged as `toil`, `engineering`, or `overhead`
- Weekly dashboard showing:
  - Current toil ratio
  - Toil remaining in budget (hours)
  - Projected date of budget exhaustion
  - Top 5 toil sources this week
- Monthly report to management with trend data

**Step 3: Budget Replenishment (Engineering Investment)**
- Every hour of toil eliminated frees up budget for the next quarter
- Track "toil debt" — the accumulated cost of not automating a task

| Metric | Formula | Target |
|---|---|---|
| **Toil Ratio** | Toil hours / Total hours | < 25% |
| **Toil Debt** | Sum of (time spent on automatable task × frequency) over past 12 months | Should be decreasing |
| **Automation ROI** | (Time saved per month × 12) / Engineering hours to build | > 3x |
| **Toil Velocity** | Change in toil ratio month-over-month | Negative (decreasing) |
| **Self-Service Adoption** | % of user requests handled by self-service vs. tickets | > 80% |

### Sample Toil Budget Card

| Item | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| Toil Budget (hours) | 200 | 200 | 200 | 200 |
| Toil Consumed | 250 | 220 | 180 | 150 |
| Over/Under Budget | -50 | -20 | +20 | +50 |
| Toil Ratio | 31% | 27% | 22% | 19% |
| Status | BREACH | WARNING | ON TRACK | ON TRACK |

### Automated Toil Tracking

Implement a bot or script that:
1. Scans all closed tickets weekly
2. Classifies them by toil/engineering using keyword matching + manual overrides
3. Posts a summary to the team's chat channel
4. Alerts when the budget is approaching exhaustion (at 75%, 90%, and 100% consumption)

---

## 9. How to Prioritize Toil Reduction

Not all toil elimination projects are equal. Use the following prioritization framework to decide what to tackle first.

### The Toil Reduction Priority Matrix

| | High Frequency | Low Frequency |
|---|---|---|
| **High Impact** | **CRITICAL** — Automate immediately | **MODERATE** — Schedule within next quarter |
| **Low Impact** | **MODERATE** — Quick wins, script in hours | **LOW** — Accept as budgeted toil or defer |

- **Frequency**: How often does this task occur?
- **Impact**: How much total engineering time is consumed? Include both direct task time and context-switching cost.

### Prioritization Factors (Weighted)

| Factor | Weight | Description |
|---|---|---|
| **Time Saved** | 40% | Total engineering hours recovered per month |
| **Error Potential** | 25% | How often does this task cause incidents or require rework? |
| **Morale Impact** | 15% | How much do team members hate this task? (Survey or vote) |
| **External Dependency** | 10% | Does automation require another team to change? (Lower = better) |
| **Visibility** | 10% | Will reducing this toil be visible to stakeholders? (Helps justify engineering time) |

### Scoring Method

For each toil source, calculate:

```
Priority Score = (Time Saved × 0.4) + (Error Potential × 0.25) + (Morale × 0.15) + (Dependency × 0.10) + (Visibility × 0.10)
```

Each factor scored 1-5 (1 = low, 5 = high). Sort by Priority Score descending. Tackle the top items first.

### The "Quick Win" Rule

Any toil item that can be automated in **one day or less** should be automated immediately, regardless of priority score. These quick wins build momentum and demonstrate the value of toil reduction to the team and management.

### Quarterly Toil Review Cadence

| Week | Activity |
|---|---|
| Week 1 | Review toil metrics from previous quarter. Update toil budget. |
| Week 2 | Run a team toil estimation session. Identify new toil sources. |
| Week 3 | Prioritize toil reduction projects for the quarter using the matrix. |
| Week 4+ | Execute. One day per sprint dedicated to toil reduction (hack day format). |

---

## 10. Organizational Consequences of Excessive Toil

Excessive toil is not just an operational problem — it is an organizational poison that erodes every aspect of team health.

### Career Stagnation

| Consequence | Mechanism |
|---|---|
| **No portfolio growth** | Engineers who spend 80%+ of their time on toil have no projects to show. Their resume stagnates. |
| **Skill atrophy** | Toil does not teach modern engineering practices, architecture, or design. Skills degrade over time. |
| **No mentorship** | Senior engineers buried in toil have no bandwidth to mentor juniors. The team's knowledge transfer pipeline breaks. |
| **Promotion dead-end** | Most organizations require demonstrated project impact for promotion. Toil produces no project artifacts. |

**Signs to watch for**: Engineers who have been with the company 2+ years but cannot point to a single system they built or improved.

### Low Morale

| Consequence | Mechanism |
|---|---|
| **Learned helplessness** | "Why bother automating? It'll just be torn down next quarter." Chronic toil breeds fatalism. |
| **Loss of ownership** | When every day is reactive firefighting, engineers stop feeling ownership of the systems they run. |
| **Boredom** | Toil is intellectually unfulfilling. Talented engineers need challenge and creativity. |
| **resentment** | "I'm just a ticket monkey" — engineers resent being used as human automation. |

**Signs to watch for**: Low participation in engineering discussions, disengagement in sprint planning, "whatever" responses to architecture questions.

### Attrition

| Consequence | Mechanism |
|---|---|
| **Talent flight** | The best engineers leave first. They have the most options and the lowest tolerance for toil. |
| **Brain drain** | When senior engineers leave, their undocumented tribal knowledge goes with them. Toil increases for remaining team members. |
| **Hiring difficulties** | Word spreads: "Don't join Team X — it's a firefighting nightmare." Recruitment becomes impossible. |
| **Replacement cost** | Each departed SRE costs 6-9 months of salary in recruiting, onboarding, and productivity loss. |

**Signs to watch for**: Voluntary turnover rate > 20% per year, multiple departures to the same competitor, roles open for 6+ months.

### The Toil Death Spiral

```
High toil → Low engineering investment → Poor system reliability
    ↑                                            ↓
More incidents → More manual remediation → More on-call burden
    ↑                                            ↓
Engineers leave ← Low morale ← No improvement projects ←
```

### Breaking the Spiral

The spiral can only be broken by an intentional, enforced decision to prioritize engineering time over operational work, even if that means temporarily accepting lower availability or slower response times.

| Intervention | Timeframe | Impact |
|---|---|---|
| Enforce 50% engineering time mandate | 1 quarter | Immediate protection of engineering time |
| Toil freeze | 1 sprint | Stop all work except P0 incidents and critical toil |
| Automation sprint | 1-2 sprints | Build automation for top 3 toil sources |
| Self-service platform investment | 1-2 quarters | Reduce user-request toil |
| System redesign (design elimination) | 2-4 quarters | Eliminate entire categories of toil |

### Measuring Organizational Health

| Metric | Healthy | Warning | Critical |
|---|---|---|---|
| Toil Ratio | < 25% | 25-35% | > 35% |
| Voluntary Turnover | < 10%/year | 10-20%/year | > 20%/year |
| Engineering Project Completion | 80%+ on time | 50-80% on time | < 50% on time |
| On-Call Satisfaction (survey) | > 4/5 | 3-4/5 | < 3/5 |
| Time to Onboard New SREs | < 3 months productive | 3-6 months | > 6 months |
| Number of Active Automation Projects | 5+ | 2-4 | 0-1 |

---

## 11. Practical Anti-Patterns and Pitfalls

### The "All Toil Is Bad" Fallacy

Some toil is acceptable and even necessary. A small amount of operational work keeps engineers connected to the reality of their systems. The goal is not zero toil, but *managed* toil — toil that is within budget, acknowledged, and stable or decreasing.

### The "Automate Everything" Trap

Automation is a tool, not a religion. Some things should not be automated:
- Tasks that change too frequently
- Tasks where the cost of automation failure is catastrophic
- Tasks that serve as a forcing function for system improvement (if automating a bad process, fix the process first)

### The "Set and Forget" Myth

Automation is not fire-and-forget. Every automated system requires:
- Monitoring (is the automation still working?)
- Maintenance (does it still match the current system state?)
- Circuit breakers (what happens when it fails?)
- Documentation (what does it do and how do we override it?)

Treat automation as a product, not a one-time script.

### The "Toil Tax" Trap

Teams that successfully reduce toil are often rewarded with *more* work — new services, more users, additional responsibilities. Without a formal toil budget that scales with responsibility, the team is punished for efficiency. Ensure that toil reduction gains are protected and reinvested, not consumed by scope growth.

---

## 12. Quick Reference: Toil Elimination Workflow

1. **Identify**: Tag every task with the toil rubric. Identify tasks scoring 18+.
2. **Quantify**: Measure time spent on each toil source. Calculate toil ratio.
3. **Categorize**: Is this script-based, zero-touch, self-service, or design elimination?
4. **Decide**: Use the automation decision tree. Should we automate, eliminate, or accept?
5. **Prioritize**: Score using the priority matrix. Quick wins first.
6. **Build**: One engineering day per sprint dedicated to toil reduction.
7. **Measure**: Recalculate toil ratio. Is it going down?
8. **Defend**: Protect engineering time. Escalate if toil ratio exceeds 35%.
9. **Repeat**: Quarterly toil review. Never stop.

---

## References and Further Reading

- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). *Site Reliability Engineering: How Google Runs Production Systems*. O'Reilly Media. — Chapter 6: "Eliminating Toil"
- Beyer, B., Murphy, N. R., Rensin, D. K., Kawahara, K., & Thorne, S. (2018). *The Site Reliability Workbook: Practical Ways to Implement SRE*. O'Reilly Media. — Chapter 5: "Toil Assessment"
- Google SRE Team. (2023). *SRE Fundamentals*. Google Cloud Training.
- Jones, C. (2020). *Toil: The SRE Killer*. USENIX SREcon.
- Adkins, H., Beyer, B., Blankinship, P., Lewandowski, P., Oprea, A., & Stubblefield, A. (2020). *Building Secure and Reliable Systems*. O'Reilly Media.
- Niall Murphy, et al. (2022). *Toil Reduction at Scale: Case Studies from Large Production Environments*. ACM Queue, 20(3).

# Troubleshooting Reference for Site Reliability Engineering

## Table of Contents

1. [Introduction](#introduction)
2. [The Hypothetico-Deductive Method](#the-hypothetico-deductive-method)
3. [Common Pitfalls](#common-pitfalls)
4. [Problem Report Format](#problem-report-format)
5. [Triage Prioritization](#triage-prioritization)
6. [Diagnostic Tools](#diagnostic-tools)
7. [Generic Debugging Techniques](#generic-debugging-techniques)
8. [Experimental Design](#experimental-design)
9. [The Value of Negative Results](#the-value-of-negative-results)
10. [Case Study: App Engine Latency](#case-study-app-engine-latency)
11. [Building for Observability](#building-for-observability)
12. [Troubleshooting Expertise: Systematic vs Experience-Based](#troubleshooting-expertise-systematic-vs-experience-based)
13. [References and Further Reading](#references-and-further-reading)

---

## Introduction

Troubleshooting is the art and science of diagnosing and resolving failures in complex systems. In Site Reliability Engineering (SRE), troubleshooting is not merely a reactive activity — it is a structured discipline that combines systematic reasoning, domain knowledge, empirical observation, and the careful design of experiments. Google's SRE teams, as documented in the SRE books, treat incidents as learning opportunities and invest heavily in the tools, practices, and culture that enable rapid diagnosis.

This reference distills the core troubleshooting methodology used by SRE practitioners into a practical guide. It covers the scientific method adapted for production systems, common cognitive traps, diagnostic tooling, experimental design, and the architectural patterns that make troubleshooting easier in the first place.

The target audience is an SRE agent operating on production systems: the methods described here should inform every investigation from a pager alert to a postmortem deep-dive.

---

## The Hypothetico-Deductive Method

At its core, troubleshooting is hypothesis-driven inquiry. The hypothetico-deductive method — borrowed from the scientific tradition — provides a rigorous framework:

### The Cycle

1. **Observe the failure.** Collect baseline data from monitoring, alerting, logs, and user reports. Establish what is actually happening, not what you assume is happening.

2. **Form one or more hypotheses.** A hypothesis is a testable explanation for the observed failure. It must be falsifiable — capable of being proven wrong by an experiment. Example: "The increased latency is caused by CPU throttling due to a co-located noisy neighbor on the same hypervisor."

3. **Design an experiment.** The experiment must produce a clear result that either supports or refutes the hypothesis. It should isolate one variable at a time. Experiments can be:
   - **Observational:** Examine metrics, logs, or traces for evidence that supports or contradicts the hypothesis. "Check if CPU throttling counters are elevated on the affected instances."
   - **Interventional:** Make a controlled change and observe the outcome. "Migrate one affected pod to a different node and measure whether latency returns to baseline."
   - **Reproduction:** Attempt to recreate the failure under controlled conditions. "Run the same request pattern against a staging environment with simulated CPU throttling."

4. **Execute the experiment and collect results.** Run the experiment cleanly — do not contaminate it with other simultaneous changes.

5. **Interpret results.** Did the outcome support or refute the hypothesis?
   - **Supported:** The hypothesis remains viable. Proceed to refine it, make predictions, and test further.
   - **Refuted:** The hypothesis is invalid. Discard it and form a new one.
   - **Inconclusive:** The experiment was poorly designed or the data is noisy. Redesign and retest.

6. **Iterate.** Repeat steps 2-5 with increasingly refined hypotheses until the root cause is identified.

7. **Confirm root cause.** Once you believe you have found the root cause, verify by:
   - Applying a fix and observing that the failure clears.
   - Reverting the fix and observing that the failure returns.
   - Ideally, reproducing the failure again from scratch.

### The Importance of Falsifiability

A non-falsifiable hypothesis is a liability. "The system is slow because of some networking issue" cannot be disproven by any experiment — it is therefore useless. A good hypothesis is specific enough to fail: "The system is slow because the database connection pool on host-42 is exhausted, causing request queuing at the middleware layer."

### Common Hypothesis Categories

When forming your initial hypotheses, draw from the **USE** method (Utilization, Saturation, Errors) adapted for system resources:

- **Utilization:** Is a resource being heavily used? CPU, memory, disk I/O, network bandwidth, file descriptors, connection pools.
- **Saturation:** Is a resource over-subscribed? Run queue length, memory pressure, disk queue depth, TCP backlog.
- **Errors:** Is a component producing errors? Application error rates, system call failures, dropped packets, timeouts.

For distributed systems, also consider:

- **Configuration drift:** Did a config change roll out that altered behavior?
- **Dependency failure:** Is an upstream or downstream service degrading?
- **Traffic pattern change:** Did the request mix shift (e.g., more expensive queries)?
- **Resource exhaustion:** A subtle depletion over time (e.g., connection leak, memory leak, disk filling).
- **Latency amplification:** A small slowdown in one service cascading due to retries or queuing.

---

## Common Pitfalls

Even experienced SREs fall into predictable traps. Awareness of these patterns reduces diagnostic time and prevents wasted effort.

### 1. Anchoring on Irrelevant Symptoms

**The trap:** The first observation you make becomes an anchor, and you interpret all subsequent data to confirm it.

**Example:** A service is returning 503 errors. You notice disk I/O is high and spend an hour investigating I/O patterns. The actual cause was a misconfigured load balancer that dropped backend connections — the disk I/O was a benign background process.

**Mitigation:** Generate at least three distinct hypotheses before running any experiment. Explicitly write down what evidence would refute your leading hypothesis.

### 2. Confusing Correlation with Causation

**The trap:** Two metrics move together; one is assumed to cause the other.

**Example:** Error rate rises at the same time memory usage increases. You investigate memory leaks for two hours. The actual cause was a new deployment that introduced a bug — the memory increase was due to the new code path allocating more objects, not the cause of the errors.

**Mitigation:** Correlation generates hypotheses, not conclusions. Design an experiment that isolates the putative cause. If you cannot manipulate the cause independently, you cannot prove causation.

### 3. Logical Fallacies

Common fallacies that derail troubleshooting:

- **Confirmation bias:** Seeking only evidence that confirms your hypothesis. Actively seek disconfirming evidence.
- **Base rate neglect:** Ignoring how common a failure mode is. The most common cause is often the simplest — check recent deploys, config changes, and dependency health first.
- **The Texas sharpshooter fallacy:** Drawing a target around where the arrows landed. "All five failed requests came from the East region, therefore the East region is the problem." But if you had 1000 requests, five failures in one region may be random.
- **False cause (post hoc ergo propter hoc):** "We restarted the database, and the error rate dropped, so the database was the problem." But the error rate may have dropped due to a traffic lull or a concurrent remediation.

### 4. Unsafe Testing in Production

**The trap:** Running experiments that risk customer impact or data loss.

**Examples:**
- Restarting a production database without understanding the cache warm-up cost.
- Killing a process to test auto-recovery in a stateful service.
- Changing firewall rules without a rollback plan.

**Mitigation:**
- Prefer observational experiments (metrics, traces, logs) over interventional ones when possible.
- When intervention is necessary, start with the smallest possible blast radius (one instance, one request type, one region).
- Always have a rollback plan before applying any change.
- Use canary analysis and gradual rollouts.

### 5. Premature Closure

**The trap:** Stopping the investigation once a plausible explanation is found, without verifying it is the actual root cause.

**Example:** High latency is "solved" by restarting a service that had memory pressure. The latency returns a week later because the actual root cause — a connection leak — was never fixed.

**Mitigation:** Always verify root cause by reproducing the failure from scratch or by confirming that the fix eliminates the symptom and reverting it brings the symptom back.

### 6. Fixing Symptoms Instead of Root Causes

**The trap:** Restarting a process, clearing a cache, or bumping a resource limit solves the immediate problem but does not address why the problem occurred.

**Mitigation:** Apply a tactical fix to stop the bleeding, then conduct a root cause analysis to find and fix the underlying issue. Distinguish between "mitigation" and "resolution."

### 7. The "It Worked Before" Fallacy

**The trap:** Assuming that because a change or configuration worked in the past, it cannot be the cause of a new failure.

**Example:** "We've been using the same database driver for two years, so it can't be the problem." But a new query pattern, new data volume, or new driver version may expose a latent bug.

**Mitigation:** No assumption is safe. Treat every component as a potential suspect, especially when the system state has changed elsewhere.

---

## Problem Report Format

Clear problem reporting is the foundation of effective troubleshooting. When an incident is reported — whether by a monitoring system, a user, or a fellow engineer — the report should contain three essential elements:

### 1. Expected Behavior

A precise statement of what the system should be doing. Without this, you cannot determine whether a deviation is an actual problem.

- Bad: "The site is down."
- Good: "The service should respond to HTTP GET /api/orders with a 200 status code and a JSON array containing the user's orders within 500ms p99 latency. Currently, 85% of requests are returning HTTP 503."

Include the source of the expectation: an SLO, a previous baseline, a documented specification, or a contract.

### 2. Actual Behavior

The observed deviation. Quantify everything possible.

- **Scope:** Does it affect all users? A subset? One region? One instance?
- **Severity:** What is the actual impact? Requests failing? Slow but succeeding? Data loss?
- **Timeline:** When did it start? Is it continuous or intermittent? Does it correlate with any known events?
- **Trend:** Is it staying constant, growing, or shrinking?

### 3. Reproduction Steps

How to observe the failure independently. This is critical for both verification and postmortem.

- "curl -w '%{http_code}\n' https://service.example.com/api/health returns 503."
- "Send 1000 requests of type X from region Y using our load-testing framework; observe p99 latency > 2s."
- "Check the error rate dashboard at [link]: service_A.errors.5xx shows a 12% rate since 14:32 UTC."

### Template

```
## Problem Report

### Expected Behavior
[What should happen, including SLOs or baselines]

### Actual Behavior
[What is happening instead, with quantified data]

### Scope
- Affected components:
- Affected regions/zones:
- User impact:
- Time window:

### Reproduction
1. [Step 1]
2. [Step 2]
3. [Step 3]

### Initial Diagnostics
[Any first-pass data already collected: dashboards checked, logs reviewed, etc.]
```

---

## Triage Prioritization

When an incident arrives, the first priority is **containment** — not root cause analysis. The triage order is:

### Stop the Bleeding First

1. **Assess impact and severity.** What is the blast radius? Are users affected? Is data at risk? Use the priority matrix:
   - **P0:** Critical — service down, data loss, security breach. Immediate response.
   - **P1:** Major — degraded service for many users. Respond within minutes.
   - **P2:** Minor — degraded service for a subset of users, or cosmetic issues. Respond within hours.
   - **P3:** Cosmetic — low-impact issues. Respond when convenient.

2. **Apply immediate mitigation.** The goal is to restore service, not to find the root cause. Mitigation options (in roughly this order):
   - Roll back a recent deployment.
   - Fail traffic over to a healthy region or replica.
   - Scale up capacity (add instances, increase resource limits).
   - Restart a service (with appropriate caution for stateful services).
   - Block malicious or problematic traffic.
   - Enable a feature flag or circuit breaker to disable a suspect feature.

3. **Verify mitigation.** Confirm that the mitigation actually restored the service. Monitor the recovery trend, not just the current value.

4. **Tag the incident for follow-up.** Create a tracking issue or incident record. Note what mitigation was applied and what data was collected before the mitigation (pre-mitigation state is invaluable for root cause analysis).

5. **Conduct root cause analysis.** Only after the immediate crisis is contained should you switch to thorough investigation. Use the hypothetico-deductive method described above.

### The "5 Whys" in Triage

During triage, shallow versions of the "5 Whys" can be useful — but only to find a mitigation, not to declare a root cause.

- Why are we paging? > Error rate is high.
- Why is error rate high? > Database queries are timing out.
- Why are they timing out? > Connection pool is exhausted.
- Why is it exhausted? > Connections are not being returned to the pool after queries.

Mitigation: Restart the service to clear the pool. Root cause investigation follows.

### When NOT to Mitigate Immediately

There are cases where immediate mitigation is harmful:

- The mitigation itself is risky (e.g., restarting a database with slow recovery).
- The failure is self-correcting and the mitigation would destroy forensic evidence.
- The scope is narrow and the cost of investigation is lower than the cost of mitigation.

Use judgment: if 100 users are seeing an intermittent error and you can collect traces while they do, the investigation may be more valuable than a blind restart.

---

## Diagnostic Tools

Modern production systems generate vast amounts of telemetry. The SRE must know what tools are available and, more importantly, when to use each one.

### 1. Monitoring and Alerting (The First Look)

- **Dashboards:** High-level service health. Look for anomaly patterns: spikes, step-changes, flatlines, slow drifts.
- **Alerting:** Rules that fire when SLO burn rates exceed thresholds. Alerts should tell you what is wrong, not just that something is wrong.
- **Red/Green metrics:** At a glance, is the service red (failing) or green (healthy)? Instrument four golden signals: latency, traffic, errors, saturation.

**When to use:** Always — this is the starting point for any investigation.

### 2. Logging (The Narrative)

- **Structured logs:** JSON-formatted log entries with consistent fields: severity, service, trace_id, request_id, user_id, latency, status_code.
- **Log aggregation:** Centralized logging systems (ELK, Loki, BigQuery). Query by trace_id to reconstruct a single request's journey.
- **Log levels:** DEBUG, INFO, WARN, ERROR, FATAL. During active troubleshooting, temporarily increase log verbosity on affected instances (via runtime config, not redeployment).

**When to use:** When you need to see the detailed behavior of individual requests or components. Essential for understanding error paths.

### 3. Distributed Tracing (The Flow)

- **Trace spans:** Each unit of work (RPC, DB query, cache lookup) produces a span with start time, duration, and metadata.
- **Trace context:** Propagated via HTTP headers (traceparent, x-cloud-trace-context). Must be passed across service boundaries.
- **Sampling:** Head-based (consistent) or tail-based (dynamic). For debugging specific failures, use a higher sampling rate on error traces.

**When to use:** When the failure involves multiple services or when you need to understand latency breakdown across hops. Invaluable for "needle in a haystack" problems.

### 4. State Endpoints and Debug Interfaces

- **Health checks:** `/healthz`, `/readyz` — basic liveness and readiness.
- **Metrics endpoints:** `/metrics` — Prometheus-format metrics exposed by the application.
- **Debug endpoints:** `/debug/pprof/`, `/debug/vars` — runtime profiling data (goroutines, heap, mutex contention).
- **State dumps:** `/debug/state`, `/admin/dump` — internal state for manual inspection.
- **Configuration endpoints:** `/debug/flags`, `/admin/config` — current runtime configuration.
- **Live profiling:** Continuous profiling (e.g., Google's profiling agent, Pyroscope) captures CPU, heap, and mutex profiles without requiring a request to a debug endpoint.

**When to use:** When you need to inspect the internal state of a running process. Critical for diagnosing resource leaks, deadlocks, and configuration issues.

### 5. Infrastructure Tools

- **Process-level:** `top`, `htop`, `ps`, `lsof`, `strace`, `perf`, `bpftrace`.
- **Network:** `netstat`, `ss`, `tcpdump`, `curl`, `mtr`, `dig`, `iftop`, `tcptraceroute`.
- **Disk:** `iostat`, `iotop`, `df`, `du`, `dstat`.
- **Kernel:** `vmstat`, `sar`, `/proc/*`, `sysdig`.
- **Container/Kubernetes:** `kubectl describe`, `kubectl logs`, `kubectl exec`, `crictl`, `ctr`.

**When to use:** When the failure appears to be at the infrastructure layer rather than the application layer — high system load, network issues, disk failures.

### Choosing the Right Tool

The order of tool use often follows a narrowing funnel:

1. **Monitoring dashboards** — identify what is wrong and when it started.
2. **Logs** — find specific error messages or request patterns.
3. **Tracing** — understand which service or call is the bottleneck.
4. **State endpoints** — inspect the offending component's internals.
5. **Infrastructure tools** — examine the host if resource exhaustion is suspected.

---

## Generic Debugging Techniques

These techniques apply across virtually all systems and failure modes. They are the "universal tools" of the SRE troubleshooting toolkit.

### Divide and Conquer (Binary Search)

The most powerful debugging technique. Narrow the search space by splitting the system in half and determining which half contains the fault.

**Application to request paths:**
- A request goes through: LB -> Auth -> API Gateway -> Service A -> Cache -> Service B -> Database.
- Check if the failure occurs before or after Service A. If after, the fault is in Service B, Cache, or DB.
- Check if it occurs before or after the Cache. If after, suspect Service B or DB.
- Continue halving until the culprit is isolated.

**Application to time ranges:**
- The failure started sometime in the last 6 hours.
- Check monitoring data from 3 hours ago. If the failure was present, divide the first 3 hours. If not, divide the last 3 hours.

**Application to instance sets:**
- 10 instances: 5 are failing, 5 are healthy. What is different between the two groups? Compare config, deployment version, resource allocation, node location.

### Bisection (for Root Cause in Code)

When the root cause is suspected to be a code change, use git bisect to find the exact commit that introduced the regression.

```
git bisect start
git bisect bad  # current version is broken
git bisect good <last-known-good-commit>
# git checks out the midpoint; test it
# git bisect good / git bisect bad
# repeat until the offending commit is found
```

### Correlate Changes

Most production failures are caused by change. Correlate the failure timeline with:

- **Deployments:** What changed in the last N minutes/hours? Check deployment history for ALL services, not just the one failing.
- **Configuration changes:** Feature flags, routing rules, quota limits, firewall rules.
- **Infrastructure changes:** Scaling events, node upgrades, network changes, DNS updates.
- **Dependency changes:** External API version bumps, database schema migrations, certificate rotations.
- **Traffic changes:** Sudden traffic spikes, shifts in request composition, new clients connecting.

### The "What Changed?" Checklist

When investigating any incident, ask these questions first:

1. Was there a deployment to any service in the call path in the window before the incident?
2. Was there a configuration change?
3. Was there an infrastructure change (scaling, node pool update, network rule)?
4. Did a dependency change (database schema, external API, certificate)?
5. Did traffic patterns change (volume, request mix, client distribution)?
6. Did any resource cross a threshold (disk 90% full, connection pool exhausted, rate limit hit)?

### The "Breadcrumb Trail" Technique

Follow the breadcrumbs from the observable symptom backward through the system:

- Symptom: User sees "500 Internal Server Error."
- Breadcrumb 1: The load balancer logged the 500 with upstream response time > 30s.
- Breadcrumb 2: The application log shows the request timed out waiting for a database query.
- Breadcrumb 3: The database slow query log shows the query took 25s.
- Breadcrumb 4: The query plan shows a full table scan on a 100M-row table.
- Breadcrumb 5: The index used by the query was dropped by a recent migration.

Each breadcrumb is a piece of evidence that narrows the search space.

### Compare to a Known Good Baseline

If you have a stale but known-good version of the system, compare its behavior:

- Compare metrics from a healthy period to the current period: what changed?
- Compare a live healthy instance to a failing instance: config, process list, resource usage.
- Compare request traces from healthy requests to failing requests: which span is different?

### Work Backward from the Error

Error messages often contain the key to diagnosis:

- HTTP 503: The service is unavailable. Check capacity, health checks, load balancer configuration.
- HTTP 504: A gateway timeout. Something upstream is slow. Check tracing.
- HTTP 429: Rate limiting. Check quota usage for the client.
- TCP connection refused: The port is not listening. Check if the process is running.
- TCP connection reset: The process died or the connection was closed abruptly. Check for crashes, OOM kills, health check timeouts.

---

## Experimental Design

Good experiments are the difference between guessing and knowing. Every experiment in an incident investigation should meet the following criteria.

### Test in Order of Likelihood

Prioritize hypotheses by:

1. **How likely they are** given prior incidents and common failure modes. Deployments and config changes are the most common causes of production failures.
2. **How easy they are to test.** A log query that takes 10 seconds should come before a database restart that takes 10 minutes.
3. **How dangerous the test is.** Observational tests (checking metrics) come before interventional tests (killing a process).

A reasonable order for most incidents:

1. Check deployment history and config changes (2 minutes).
2. Review dashboards for the four golden signals (2 minutes).
3. Check dependency health (external services, databases, caches) (2 minutes).
4. Examine error logs and traces for the affected requests (5 minutes).
5. Check resource utilization and saturation on affected instances (5 minutes).
6. Attempt reproduction in a non-production environment (10-30 minutes).
7. Interventional tests with controlled blast radius (10-60 minutes).

### Isolate One Variable at a Time

The cardinal rule of experimental design: **change one thing at a time.** If you change two things and the problem goes away, you do not know which change fixed it. If you change two things and the problem remains, you do not know whether either change had an effect.

- Bad: "We restarted the service and increased the connection pool size." Did the restart or the pool size fix it?
- Good: "We increased the connection pool size on one instance and observed no improvement. We restarted that instance and the problem resolved."

### Account for Confounding Factors

A confounding factor is a variable that influences both the independent variable and the outcome, creating a spurious association.

**Examples of confounders in production debugging:**

- **Time of day:** "We restarted the service at 3:00 AM and latency dropped." But 3:00 AM is naturally low-traffic — the latency drop may be due to reduced load, not the restart.
- **Coincident changes:** "We rolled back the deployment and the error rate dropped." But another team also deployed a fix for an upstream dependency at the same time.
- **Hawthorne effect:** "We started monitoring query performance and latency improved." Simply observing a system can change behavior (engineers writing better queries because they know they are being measured).

**Mitigation:** Before declaring a causal relationship, ask: "What else changed at the same time?" Correlate timelines from all sources.

### Control Groups and Canaries

Whenever possible, run experiments with a control group:

- Route 1% of traffic to a canary with the proposed fix, leave 99% on the current version. Compare outcomes.
- Apply a change to one instance but not its peers. Compare metrics.
- Use feature flags to enable a change for a subset of requests.

This is why progressive rollouts (e.g., 1% -> 5% -> 25% -> 100%) are standard SRE practice — they inherently provide a control group.

### Reproducibility

A failure that cannot be reproduced cannot be diagnosed with certainty. If you cannot reproduce the failure in production, try:

- Replaying the exact request that failed (from logs or trace data).
- Creating an identical environment (same configuration, same dependencies, same data).
- Simulating the conditions under which the failure occurs (load, traffic pattern, resource constraints).
- Using chaos engineering tools to inject the suspected failure mode.

If a failure is truly non-reproducible, it may be a transient issue (network blip, CPU steal, cosmic ray). Document it and move on, but ensure monitoring is in place to detect recurrence.

---

## The Value of Negative Results

In troubleshooting, negative results — experiments that refute a hypothesis — are as valuable as positive ones. Each refuted hypothesis narrows the search space.

### Why Negative Results Matter

1. **They eliminate possibilities.** Every hypothesis you can confidently reject brings you closer to the true cause.
2. **They prevent wasted effort.** Proving that "it is not the database" prevents future investigations from revisiting the database.
3. **They build confidence.** A thorough investigation that tested and eliminated several plausible hypotheses builds more confidence in the final root cause than one that tested only the winning hypothesis.
4. **They are archival.** Documenting negative results in incident reports or runbooks prevents future engineers from retreading the same dead ends.

### How to Document Negative Results

Every experiment should be recorded with:

- **Hypothesis tested:** "Hypothesis: High latency is due to database connection pool exhaustion."
- **Experiment:** "Checked pg_stat_activity and max_connections on the primary database instance."
- **Result:** "Connection count is 45 of 200, well below the limit. All connections show idle time under 100ms. Hypothesis refuted."
- **Next:** "Next hypothesis: High latency is due to CPU throttling on the application instances."

### Avoiding "Happy Path" Thinking

When an incident is resolved, there is a natural tendency to only document what fixed it. Resist this. The dead ends and wrong turns are valuable learning material. A postmortem that says "Restarted the service" without mentioning the 45 minutes spent investigating the wrong hypothesis is a postmortem that has wasted a future engineer's time.

---

## Case Study: App Engine Latency

This case study, drawn from Google's SRE literature, illustrates the hypothetico-deductive method in action.

### The Symptom

A team running on Google App Engine noticed that request latency had been gradually increasing over the course of several weeks. The increase was subtle — p50 latency went from 100ms to 130ms, p99 from 500ms to 800ms — but it was persistent and growing.

### Initial Observations

- The increase was present across all versions of the application (ruling out any single release).
- The increase correlated with the number of active versions deployed on the same App Engine instance.
- The increase was more pronounced at p99 than at p50.
- CPU utilization on the App Engine instances was flat. Memory was flat.

### Hypothesis 1: Competing Versions Cause Resource Contention

**Hypothesis:** Each deployed version of the application consumes memory for its compiled code and runtime state. As more versions accumulate, the working set exceeds memory capacity, causing the garbage collector to run more frequently and increasing latency.

**Experiment:** Measure GC pause time and frequency as a function of the number of deployed versions. Deploy a new version and observe whether GC metrics change.

**Result:** GC pause time did increase slightly, but the increase was too small to account for the observed latency shift. The correlation between version count and latency remained even after accounting for GC time.

**Conclusion:** Hypothesis partially supported (some GC impact) but not sufficient to explain the full latency increase.

### Hypothesis 2: Instance Scrubbing Overhead

**Hypothesis:** App Engine instances periodically "scrub" old versions from disk. With many versions, this scrubbing consumes I/O bandwidth, causing latency spikes for requests hitting the same disk.

**Experiment:** Monitor disk I/O latency and scrub activity (via logs) as version count increases. Compare I/O latency distributions for instances with few vs. many versions.

**Result:** Disk scrub activity was indeed higher on instances with more versions. However, the disk I/O latency distribution did not differ significantly between the two groups.

**Conclusion:** Hypothesis refuted.

### Hypothesis 3: Connection Pool Fragmentation

**Hypothesis:** Each version maintains its own connection pool to the database. With more versions, the total number of connections grows, overwhelming the database's connection pool. The database itself becomes the bottleneck.

**Experiment:** Monitor database-side connection counts and query latency. Correlate with the number of App Engine versions deployed.

**Result:** Database connection count did not increase significantly. Database-side query latency was flat.

**Conclusion:** Hypothesis refuted.

### Hypothesis 4: Request Routing Overhead

**Hypothesis:** The App Engine frontend (the routing layer that dispatches requests to instances) must maintain a routing table for all versions. With many versions, the routing table grows, and routing decisions take longer. The effect is most pronounced at the tails (p99) because of occasional routing table churn.

**Experiment:** Measure time spent in the App Engine frontend (from request arrival at frontend to arrival at the application) as a function of version count. This requires trace-level visibility into the frontend layer.

**Result:** The frontend routing time increased by several hundred milliseconds at p99 for instances with many versions. The routing layer was computing version-to-instance mappings on every request for versions that had been deployed for a long time, rather than caching the mapping.

**Conclusion:** Hypothesis supported. Root cause identified.

### Resolution

- Remove old, unused versions from App Engine instances.
- Implement a caching layer in the frontend routing logic.
- Apply a limit on the number of concurrently deployed versions.

### Key Takeaways from the Case Study

1. **Multiple hypotheses were tested and refuted.** Each negative result was valuable because it narrowed the search space and eliminated plausible alternative explanations.
2. **Observational experiments were used first.** The team did not start by blindly restarting services or migrating instances; they measured.
3. **The correlation-versus-causation trap was avoided.** The correlation between version count and latency was noted, but the team did not stop there — they tested specific causal mechanisms.
4. **The tail (p99) was more informative than the median.** The failure mode was most visible at the tail, which pointed toward a sporadic overhead rather than a constant one.
5. **The root cause was an emergent property.** The latency was not caused by a bug in the application code or a single misconfiguration, but by an emergent effect of many small factors combining — exactly the kind of failure that systematic troubleshooting excels at catching.

---

## Building for Observability

The best troubleshooting is the troubleshooting that never needs to happen — because the system makes its internal state visible. Observability is the property of a system that allows you to understand its behavior from the outside, without having to deploy new code or restart it.

### White-Box Metrics

Black-box metrics (is the service up? is it responding?) tell you that something is wrong. White-box metrics tell you what it is.

**Examples of white-box metrics every service should expose:**

- Request counts, error counts, and latency distributions (histograms) by: endpoint, status code, caller, and error type.
- Queue depths: request queue, work queue, message queue.
- Connection pool state: active, idle, waiting, max size.
- Cache hit/miss ratios per cache.
- Rate limiter state: tokens available, requests throttled.
- Resource pool sizes and utilization (threads, goroutines, file descriptors).
- GC or allocation metrics: pause time, allocation rate, heap size.

### Structured Logging

Unstructured log lines are nearly impossible to query at scale. Every log entry should be a structured event with a consistent schema.

**Minimum required fields:**
- `timestamp` — RFC 3339 with microsecond precision.
- `severity` — DEBUG, INFO, WARN, ERROR, FATAL.
- `service` — Name of the service producing the log.
- `trace_id` — The distributed trace ID for correlation across services.
- `request_id` — A unique identifier for the request (may match trace_id).
- `message` — Human-readable description.
- `error` — If an error: error type, error message, stack trace (first N frames).

**Additional useful fields:**
- `user_id` or `customer_id` for user-specific debugging.
- `instance_id` or `pod_name` for instance-level debugging.
- `version` — Application version or commit SHA.
- `latency_ms` — Time spent in the current operation.
- `http_method`, `http_path`, `http_status` — For HTTP services.

### Consistent Request IDs

A single request may traverse dozens of services. Without a consistent request ID propagated through all of them, reconstructing the request's path is impossible.

**Best practices:**
- Generate a unique request ID at the first entry point (load balancer, API gateway, or ingress).
- Propagate it via HTTP headers (`X-Request-Id`, `traceparent`) to all downstream services.
- Include it in every log entry, every trace span, and every error report.
- Return it to the client in the response headers so that users can reference it in support tickets.
- Use W3C Trace Context (`traceparent`/`tracestate`) for compatibility with open-source tracing systems.

### The Three Pillars of Observability

1. **Metrics** — Aggregated, time-series data about the system. Best for alerting and dashboards. Low cardinality.
2. **Logs** — Discrete events with structured payloads. Best for detailed investigation of specific requests or errors. High cardinality.
3. **Traces** — End-to-end views of a single request across service boundaries. Best for understanding latency bottlenecks and service dependencies.

All three pillars are necessary. Metrics tell you "something is broken." Logs tell you "here is what went wrong for this request." Traces tell you "here is exactly where it went wrong in the request path."

### Instrumentation as Investment

Observability is an investment with compound returns. Every metric, log field, and trace span you add today will pay dividends in every future incident. The marginal cost of adding one more counter or one more log field is near zero; the marginal benefit during a P0 incident is enormous.

A practical rule: **instrument every dependency call.** Every RPC, every database query, every cache lookup, every queue enqueue/dequeue should produce at minimum:
- A latency metric (histogram or summary).
- A success/failure counter.
- A log line on error with the error details.
- A trace span with the dependency name and timing.

---

## Troubleshooting Expertise: Systematic vs Experience-Based

Not all troubleshooting expertise is the same. Understanding the two major types — systematic and experience-based — helps you know when to rely on each and how to cultivate both.

### Systematic Troubleshooting

Systematic troubleshooters apply the hypothetico-deductive method rigorously. They follow a process regardless of the domain:

- They generate multiple hypotheses before testing any.
- They design experiments with controls and confounding factors in mind.
- They document each hypothesis and result.
- They can work effectively on any system, even ones they have never seen before.
- They are slower on familiar problems but more reliable on unfamiliar ones.

**Strengths:**
- Works for novel or unprecedented failures.
- Produces reproducible, documented results.
- Teaches well and scales across teams.
- Resistant to cognitive biases.

**Weaknesses:**
- Slower for routine or well-known problems.
- Can feel mechanical or overly academic.
- Requires discipline to maintain when under time pressure.

### Experience-Based Troubleshooting

Experienced troubleshooters rely on pattern matching and intuition built from years of solving similar problems:

- "This looks like the DNS propagation issue we had last month."
- "I've seen this exact error before — it's the connection pool leak."
- "That latency pattern is classic noisy neighbor on the hypervisor."

**Strengths:**
- Extremely fast for familiar failure modes.
- Intuition can sometimes leap past intermediate steps.
- Pattern recognition improves with every incident.

**Weaknesses:**
- Brittle when faced with novel failures.
- Prone to confirmation bias (seeing the same pattern everywhere).
- Hard to transfer or teach ("it's just a feeling").
- Can lead to premature closure.

### The Hybrid Approach

The best SREs combine both approaches:

1. **Start with pattern recognition.** "This looks like X — let me check these three specific metrics quickly." This is fast and often correct.
2. **If pattern matching fails (within 5-10 minutes), switch to systematic.** "The familiar patterns don't fit. Let me step back, generate hypotheses, and design experiments."
3. **Document both.** When you find the root cause, note whether it matched a known pattern or was novel. Feed novel findings into your team's knowledge base.

### Building Experience Systematically

Systematic troubleshooting is the foundation because it can be taught, practiced, and improved. Experience is then built on top of it:

- Every incident adds a new pattern to your mental library.
- Every postmortem connects the pattern to the root cause mechanism.
- Every runbook turns a pattern into a playbook for the next occurrence.
- Every blameless culture encourages sharing both successes and failures.

In short: **systematic method is the engine; experience is the fuel.** You need both, but the engine must come first because without it, experience is just a collection of anecdotes with no way to distinguish correlation from causation.

---

## References and Further Reading

- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). *Site Reliability Engineering: How Google Runs Production Systems*. O'Reilly Media.
- Beyer, B., Murphy, N. R., Rensin, D., Kawahara, K., & Thorne, S. (2018). *The Site Reliability Workbook: Practical Ways to Implement SRE*. O'Reilly Media.
- Adkins, H., Beyer, B., & Thurnherr, V. (2020). *Building Secure and Reliable Systems*. O'Reilly Media.
- Driscoll, M. (2016). *Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems*. AMACOM.
- Agans, D. J. (2006). *Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems*. AMACOM.
- Petoff, J. (2015). "Troubleshooting Distributed Systems: The Hypothethico-Deductive Method." *USENIX SREcon*.
- Tufte, E. R. (1997). *Visual Explanations: Images and Quantities, Evidence and Narrative*. Graphics Press.
- Kahneman, D. (2011). *Thinking, Fast and Slow*. Farrar, Straus and Giroux.
- Kleppmann, M. (2017). *Designing Data-Intensive Applications*. O'Reilly Media.
- Google SRE Team. "The 'What To Do When You Break Production' Guide." Internal Google SRE Documentation.
- Google SRE Team. "App Engine Latency Case Study." In *Site Reliability Engineering*, Chapter 14.

# Twenty Years of SRE Lessons Learned & Prodverbs: A Synthesized Analysis

> **Source material:**
> - [Lessons Learned from Twenty Years of Site Reliability Engineering](https://sre.google/resources/practices-and-processes/twenty-years-of-sre-lessons-learned/) — Google SRE team
> - [Prodverbs](https://sre.google/prodverbs/) — Google SRE team
>
> This document synthesizes, analyzes, and cross-references both resources against the canonical Google SRE book. It is not a copy — it distills incident context, practical application, and gaps the original SRE book left open.

---

## Part 1: Eleven Lessons — Synthesized Analysis

Each lesson below follows a consistent structure: **Incident Context** → **The Lesson** → **What It Means in Practice** → **How to Apply It**.

---

### 1. Risk/Mitigation Proportionality

| Aspect | Detail |
|---|---|
| **Incident** | YouTube caching outage |
| **Key insight** | A mitigation carries its own risk; a failed mitigation can prolong an outage |
| **Original SRE book gap** | The book discusses error budgets but not *mitigation risk budgeting* |

**Incident Context:** During a YouTube outage caused by a caching layer failure, the SRE team attempted a complex mitigation. When that mitigation failed, it made matters worse — extending the outage significantly. The team had chosen a high-risk, high-reward mitigation without considering what happened if the mitigation itself failed.

**The Lesson:**
> "Monitor the severity of the incident and choose your mitigation risk level accordingly." — Google SRE

**In Practice:** Every mitigation action has a probability of success and a probability of making things worse. These form a 2x2 matrix:

| | Low-severity incident | High-severity incident |
|---|---|---|
| **Low-risk mitigation** | Safe choice, standard playbook | May be too slow |
| **High-risk mitigation** | Unnecessary risk | Potentially justifiable |

**How to Apply:**
- Classify incident severity within the first 60 seconds using a tier system (S1–S4)
- Predefine what "risk budget" your team has for each severity tier
- Maintain a "mitigation risk register" — track which mitigations have historically backfired
- During an incident, pause before executing a risky mitigation and ask: "If this fails, are we in a worse position than before?"

> **Counterintuitive insight:** Sometimes doing nothing is the correct response to a moderate-severity incident, especially when all available mitigations carry non-trivial failure risk.

---

### 2. Test Recovery Mechanisms Before They're Needed

| Aspect | Detail |
|---|---|
| **Incident** | Generalized — synthesis across multiple Google outages |
| **Key insight** | "Don't try a ladder during a fire" — untested recovery procedures fail under pressure |

**Incident Context:** Multiple Google outages revealed that recovery procedures documented in runbooks had not been tested. When SREs attempted them during real incidents, they discovered missing permissions, incorrect scripts, stale database snapshots, or steps that assumed a different state of the system.

**The Lesson:**
> "Test recovery mechanisms during calm periods, not during emergencies." — Google SRE

**In Practice:** Recovery procedures are code. They rot, they have bugs, and they depend on system state that changes over time. Treating them as documentation rather than executable tests is a recipe for failure.

**How to Apply:**
- Run quarterly "fire drills" where on-call engineers execute the full recovery procedure in a staging environment
- Automate recovery procedure testing in CI/CD pipelines — if a deploy changes a system, re-validate the recovery path
- Use Chaos Engineering to verify that recovery actually works under degraded conditions
- Track "recovery test coverage" as a metric: what percentage of your failure modes have had their recovery path exercised within the last N months?

---

### 3. Canary All Changes

| Aspect | Detail |
|---|---|
| **Incident** | YouTube config change — 13-minute global outage |
| **Key insight** | A single misconfigured flag brought down a global service; canarying would have caught it |

**Incident Context:** A configuration change to YouTube's serving stack was pushed globally without a staged rollout. The change triggered a cascading failure that caused a 13-minute global outage. A canary deployment (to 1–2% of traffic) would have revealed the problem before it reached all users.

**The Lesson:**
> "A small canary deployment is infinitely better than a global outage." — Google SRE

**In Practice:** The canary strategy is formally described in Google's paper *"Introducing the Canary Strategy"* but the lesson from YouTube is that even teams who *know* about canarying skip it. The failure mode is not technical ignorance but process discipline failure.

**How to Apply:**
- Enforce canary gating at the infrastructure level — not as a human process step
- Define a "canary policy" for every type of change: config changes, code deploys, infrastructure changes, DNS changes
- Canary duration should be based on monitoring signal confidence, not a fixed timer
- Automate rollback when canary metrics breach SLO thresholds

| Change Type | Canary Size | Canary Duration | Metric Gate |
|---|---|---|---|
| Code deploy | 1% → 5% | 5 min | Error rate + latency P99 |
| Config change | 2% | 10 min | Request success rate |
| DNS change | 5% of regions | 30 min | Query resolution latency |
| Infrastructure | 1 cell | Until all health checks pass (min 15 min) | CPU/memory/error rate |

---

### 4. The Big Red Button

| Aspect | Detail |
|---|---|
| **Incident** | Google Calendar near-miss (engineer unplugging their desktop prevented propagation) |
| **Key insight** | Simple, pre-tested, single-action emergency responses save the day when complex thinking fails |

**Incident Context:** A Google Calendar incident was approaching critical mass. An engineer, realizing a dangerous rollout was in progress, physically unplugged their desktop from the network. This simple action prevented the bad change from propagating further. The lesson: sometimes the most effective emergency response is a single, well-understood action — not a multi-step procedure.

**The Lesson:**
> "A Big Red Button — a single, pre-tested action to stop or roll back a change — is essential." — Google SRE

**In Practice:** Under incident pressure, cognitive load is high. Engineers make mistakes executing multi-step procedures. A "Big Red Button" (BRB) is a single action (button click, single command, physical action) that:
- Immediately halts change propagation
- Rolls back the last N minutes of changes
- Notifies the relevant team

**How to Apply:**
- Identify the most common "oh shit" scenarios for your service
- For each, design a single-action mitigation (one kubectl command, one API call, one button)
- Test the BRB monthly — it must always work
- Guard the BRB against accidental triggering (confirm dialogs, multi-person approval for very destructive actions)
- Log every BRB activation for post-incident review

> **Design principle:** A BRB should be safe to press. If pressing the BRB could cause its own outage, it's not a BRB — it's another dangerous mitigation.

---

### 5. Unit Tests Are Insufficient — Integration Testing Is Essential

| Aspect | Detail |
|---|---|
| **Incident** | Google Calendar outage |
| **Key insight** | Unit tests passed; the system failed because real-world interaction paths weren't tested |

**Incident Context:** A Calendar change passed all unit tests with 100% code coverage but caused an outage because the interaction between components — the *integration* — behaved differently than any component in isolation.

**The Lesson:**
> "Unit tests tell you that components work in isolation. Integration tests tell you the system works. You need both." — Google SRE

**In Practice:** Unit tests verify that each function/class/module behaves correctly in isolation. But production failures increasingly come from *interactions* between components: race conditions, protocol mismatches, timeout interactions, data format assumptions, and non-deterministic behavior that only emerges under load.

**How to Apply:**
- Use the "testing trophy" model (with integration tests as the bulk of your test suite), not the testing pyramid
- Every critical user journey must have an integration test that exercises the full path through the system
- Run integration tests against a realistic environment, not mocks
- Include "dark launch" testing — run new code paths in production without affecting users to observe behavior

---

### 6. Backup Communication Channels

| Aspect | Detail |
|---|---|
| **Incident** | OAuth token incident — 350M users logged out |
| **Key insight** | When your service goes down, the tools you rely on for incident response may also go down |

**Incident Context:** A misconfiguration in Google's OAuth system caused approximately 350 million users to be logged out. Crucially, Google's internal incident response tools (Hangouts, Meet, internal dashboards) depended on the same authentication system that had failed. SREs could not communicate via the usual channels to coordinate the response.

**The Lesson:**
> "Design backup communication channels that do not depend on your primary service." — Google SRE

**In Practice:** This lesson reveals a deep architectural vulnerability: *dependency coupling at the tooling layer*. If your incident response tools share the same auth system, the same network, or the same infrastructure as your production service, a failure of that shared dependency blinds you at the exact moment you need visibility.

**How to Apply:**
- Maintain at least one out-of-band communication channel (e.g., a separate Slack workspace on a different provider, a PagerDuty conference bridge on non-Google infrastructure, satellite phones for critical infrastructure)
- Test the backup channel quarterly — just having it isn't enough
- Ensure runbooks are accessible via the backup channel (e.g., printed copies, a static S3 bucket on a different cloud provider)
- Audit your incident response toolchain for shared dependencies with the production service

| Primary Channel | Backup Channel | Dependency Risk |
|---|---|---|
| Google Meet/Hangouts | Twilio conference bridge + Slack on different workspace | Shared OAuth |
| Internal status dashboard | Static site on different provider | Shared authentication |
| Email (Gmail/G Suite) | SMS/pager alerts | Shared identity system |

> **Key insight from the incident:** "We couldn't use our own tools to fix our own tools."

---

### 7. Intentionally Degrade Performance Modes

| Aspect | Detail |
|---|---|
| **Incident** | Generalized — cumulative experience across Google services |
| **Key insight** | Binary up/down thinking ignores a whole spectrum of useful degraded states |

**Incident Context:** Google SREs observed that many services were designed as either "fully working" or "completely broken." In reality, most failure modes could be partially mitigated by intentionally degrading performance — serving stale data, disabling non-critical features, reducing recommendation quality, or showing a simpler UI.

**The Lesson:**
> "Move beyond binary up/down thinking. Build graceful degradation into your service." — Google SRE

**In Practice:** Every feature can be classified along a degradation continuum. The goal is to define, for each feature, what "degraded mode" looks like and how to trigger it.

**How to Apply:**
- For each feature, define 3–4 degradation levels (full → reduced → minimal → disabled)
- Implement feature flags for every degraded mode
- Build a "degradation dashboard" showing which systems are operating below full capacity
- Test degraded modes in production — your users should know what a degraded experience looks like (it's better than an error page)
- SLOs should have different targets for degraded mode (e.g., normal: 99.9% → degraded: 99.0%)

> **Practical framework:** If your service serves personalized recommendations and the ML pipeline fails, can you serve generic popular items? If your database is overloaded, can you serve stale cache? If your auth system is slow, can you extend session lifetimes?

---

### 8. Test for Disaster Resilience

| Aspect | Detail |
|---|---|
| **Incident** | Generalized — multiple Google disaster scenarios |
| **Key insight** | Recovery testing ≠ resilience testing |

**Incident Context:** Google found that teams who had tested *recovery* procedures (e.g., "restore database from backup") were still caught off guard by *resilience* failures (e.g., "can the system survive a simultaneous region outage and a DNS failure?"). The two types of testing uncover fundamentally different failure modes.

**The Lesson:**
> "Recovery testing verifies you can fix a problem. Resilience testing verifies you can survive it." — Google SRE

**In Practice:** Recovery testing is about speed — can you get back to healthy fast enough? Resilience testing is about redundancy — does the system stay healthy when parts fail? Both are necessary.

**How to Apply:**
- Run **tabletop exercises** monthly: gather the team, describe a disaster scenario, and trace through what happens. No code changes — just reasoning about failure modes.
- Use **chaos engineering** (e.g., Chaos Monkey, Litmus) to introduce real failures in staging/production
- Distinguish between:
  - **Recovery Time Objective (RTO):** How fast can you recover?
  - **Resilience Duration:** How long can the system operate in degraded mode before becoming unrecoverable?
- Run a "game day" at least quarterly where a specific disaster scenario is simulated

| Test Type | Example | Frequency |
|---|---|---|
| Tabletop exercise | "What if us-central1 drops off the internet?" | Monthly |
| Chaos experiment | Kill 3 random pods during peak traffic | Bi-weekly |
| Game day | Full simulated region failure + on-call rotation | Quarterly |

---

### 9. Automate Mitigations

| Aspect | Detail |
|---|---|
| **Incident** | 6-day networking outage (March 2023) |
| **Key insight** | Manual mitigations are too slow; automate the response if the signal is clear |

**Incident Context:** A Google networking incident in March 2023 stretched for six days. The root cause was complex and took days to fully understand, but the *signal* that something was wrong was clear within minutes. The team realized that automated mitigations could have ended user impact in minutes rather than days.

**The Lesson:**
> "If you have a clear signal and a known safe action, automate it. Save root-cause analysis for after user impact is resolved." — Google SRE

**In Practice:** This inverts the traditional priority: most teams prioritize understanding the root cause before acting. The lesson is that *mitigation first, diagnosis second* is the correct order during an incident. If you can write a rule that reliably detects the failure condition and performs a safe action, automate it.

**How to Apply:**
- Identify "clear signal + safe action" pairs in your post-incident reviews
- Implement automated mitigations as runbook actions triggered by monitoring alerts
- Use a graduated response: notify (1 min) → suggest action (30 sec) → automated action (immediate)
- Build a "mitigation automation register" — document all automated mitigations, their triggers, their actions, and their safety constraints

> **Counterintuitive insight:** The 6-day outage could have been a 6-minute outage had the mitigation been automated. The root cause took days to find anyway — there was no time saved by not automating.

---

### 10. Reduce Time Between Rollouts

| Aspect | Detail |
|---|---|
| **Incident** | Pokémon GO outage — database field removal |
| **Key insight** | Slow rollout cadence makes it impossible to reason about the safety of any single change |

**Incident Context:** When Pokémon GO launched on Google infrastructure, it experienced a major outage caused by removing a database field. The root cause was that the rollout cadence was so slow (weeks between releases) that each change accumulated significant delta from the previous state. When the field removal was rolled out, it was bundled with many other changes — making it impossible to reason about the safety of that specific change in isolation.

**The Lesson:**
> "Release frequently. Small deltas are safe deltas." — Google SRE

**In Practice:** The relationship between release frequency and change safety is counterintuitive: *more frequent releases lead to fewer outages*. This is because:
- Each change is smaller and easier to review
- Rollback is faster and has lower blast radius
- The time window for correlated failures to accumulate is shorter
- Each release has lower uncertainty about what it changes

**How to Apply:**
- Target deployment frequency measured in hours, not weeks
- Each release should change a single logical unit
- If a release takes more than 15 minutes to review, it's too large
- Track "time from commit to production" as a key SRE metric

| Rollout Cadence | Risk Profile | Typical Failure Mode |
|---|---|---|
| Weekly | Low | Small blast radius, easy rollback |
| Monthly | Medium | Moderate change accumulation |
| Quarterly | High | Changes bundle, hard to reason about safety |
| Bi-annual | Very high | Essentially a new system, very high risk |

---

### 11. Single Global Hardware Version Is a SPOF

| Aspect | Detail |
|---|---|
| **Incident** | Networking zero-day (March 2020) |
| **Key insight** | When all your hardware is the same, one vulnerability takes everything down |

**Incident Context:** In March 2020, a zero-day vulnerability was discovered in a specific networking chipset that Google used broadly. If Google had standardized on a single networking hardware version globally, a single vulnerability could have taken down the entire network. The incident was mitigated because Google maintained multiple network backbones with diverse hardware.

**The Lesson:**
> "Infrastructure diversity is a resilience strategy. Homogeneity is a single point of failure." — Google SRE

**In Practice:** This extends beyond hardware. Any single shared dependency — whether hardware, software library, cloud provider, or vendor — is a potential single point of failure. Diversity at the infrastructure layer provides isolation from supply chain attacks, zero-day vulnerabilities, and manufacturing defects.

**How to Apply:**
- Maintain at least two independent network backbones with different hardware
- Use diversity at every infrastructure layer: compute (different CPU architectures), storage (different storage vendors), networking (different switching/routing hardware)
- For critical software dependencies, have a "second implementation" strategy
- Test failover between diverse infrastructure regularly

> **Key insight:** "Every single point of failure fails eventually" (also a Prodverb — see below). Global hardware uniformity is a hidden SPOF that most teams don't consider.

---

### Lessons Summary Table

| # | Lesson | Novel to 2024? | Gap in Original SRE Book |
|---|---|---|---|
| 1 | Risk/mitigation proportionality | Yes | Book covers error budgets but not mitigation risk |
| 2 | Test recovery mechanisms | Partial | Covered briefly, not emphasized with case study |
| 3 | Canary all changes | No | Well covered in the book |
| 4 | Big Red Button | Yes | Not addressed |
| 5 | Integration testing > unit tests | Partial | Mentioned but not with Calendar case study |
| 6 | Backup communication channels | Yes | Not addressed |
| 7 | Intentional degraded performance | Yes | Not addressed — book is binary up/down focused |
| 8 | Disaster resilience testing | Partial | Tabletop exercises mentioned briefly |
| 9 | Automate mitigations | Partial | Automation covered, but 6-day case study is new |
| 10 | Reduce time between rollouts | Yes | Book assumes fast rollouts; Pokémon GO case is novel |
| 11 | Hardware version diversity | Yes | Not addressed in the original book |

---

## Part 2: Prodverbs — Synthesized Analysis

The [Prodverbs page](https://sre.google/prodverbs/) collects pithy, experience-hardened maxims from Google's SRE history. Below, each is analyzed with its full meaning, the scenarios it addresses, and practical application.

---

### "Cascading failures happen, so guard against them."

**Full Meaning:** A failure in one component can trigger failures in others, which propagate back to the original component in a feedback loop. These are the most dangerous class of failures because they can take down an entire system from a single trigger.

**Scenario:** A single database replica slows down. The load balancer retries failed queries, increasing load on remaining replicas, which slows them down, triggering more retries. The entire database tier collapses.

**Application:**
- Implement circuit breakers (fail fast rather than retry into death)
- Use bulkheading — isolate components into failure domains
- Set connection pools with hard limits
- Monitor for "retry storms" — exponential backoff is not optional

---

### "Production is never homogenous."

**Full Meaning:** Even if you design for uniformity, production will evolve heterogeneity. Different versions of binaries run simultaneously, configs drift, machines age differently, and traffic patterns vary.

**Scenario:** A team assumes all production servers are identical and deploys a change that only works on one kernel version. Two servers crash; the rest are fine — but nobody knows which ones.

**Application:**
- Never assume uniformity — build for heterogeneity
- Inventory your production environment regularly and track drift
- Design deployment strategies that work across version skew
- Test against the oldest and newest versions in your fleet

---

### "An empty config doesn't mean you should delete everything."

**Full Meaning:** A configuration management tool reporting "no config defined" could mean (a) there truly is no config, or (b) the tool can't reach the config source and returned a default empty value. Acting on the latter assumption is catastrophic.

**Incident Context:** This prodverb is widely attributed to a real Google incident where a configuration management tool lost connectivity to its source of truth, reported "empty config" for a critical service, and an automated cleanup script deleted the service's infrastructure.

**Application:**
- Before acting on "empty" results, verify the tool's connection to its source of truth
- Implement "delete protection" on production resources
- Use "config validation" — require a signature or checksum on config data
- If a system returns empty, treat it as an alert, not an instruction

> **Practical pattern:** Never trust a tool that returns "empty" when it can't reach its data source. Always distinguish between "known empty" and "unknown state."

---

### "Every single point of failure fails eventually."

**Full Meaning:** This is the central theorem of reliability engineering. Given enough time, every component, every dependency, every assumption will fail. The question is not *if* but *when*.

**Application:**
- Maintain a "SPOF register" — document every single point of failure in your architecture
- Assign a maximum acceptable risk level to each SPOF
- Prioritize elimination based on impact and likelihood
- Accept that some SPOFs are unavoidable (e.g., power grid) — but have a recovery plan

| SPOF Type | Example | Mitigation |
|---|---|---|
| Hardware | Single network switch | Redundant switches |
| Software | Single cloud provider | Multi-cloud or hot standby |
| Human | Single person with knowledge | Cross-training, runbooks |
| Process | Single deployment pipeline | Backup pipeline, manual fallback |

---

### "First mitigate, then diagnose, then fix."

**Full Meaning:** The correct order of operations during an incident is: (1) stop the bleeding, (2) understand why the bleeding happened, (3) fix the underlying cause. This is the operational version of triage in medicine.

**Common Violation:** Engineers are naturally curious. The temptation during an incident is to start diagnosing *before* mitigating. This prolongs user impact.

**Application:**
- During an incident, the first action should *always* be mitigation (rollback, stop traffic, restart)
- Set a timer: 2 minutes for initial mitigation, then diagnose
- Post-incident, distinguish between "mitigation action" and "fix action" in your timeline
- Train on-call engineers to suppress their curiosity during the mitigation phase

---

### "If two systems must agree for them to work, someday they will inevitably disagree."

**Full Meaning:** Any system that requires two (or more) subsystems to maintain consistent state will eventually experience a state divergence. This is a fundamental property of distributed systems.

**Scenario:** A primary database and a read replica. The replica falls behind, the primary fails over, the replica is promoted — but it's missing the last 500 writes. Data loss ensues.

**Application:**
- Design for state divergence — assume consistency will be violated
- Use consensus protocols (Paxos, Raft) where strong consistency is required
- Implement reconciliation processes that detect and repair divergence
- Monitor for divergence metrics (replica lag, clock skew, transaction conflicts)

> **Practical insight:** "Eventually consistent" is a promise about the future, not a guarantee about the present.

---

### "Decrease variance, increase mean."

**Full Meaning:** Reducing the variance in your system's behavior is often more valuable than improving the average performance. A system with consistent, predictable behavior is easier to operate, debug, and trust.

**Application:**
- When optimizing, target P99/P999 latency, not P50
- Eliminate "noisy neighbors" through resource isolation
- Standardize deployment patterns, config formats, and monitoring schemas
- Track "operational variance" as a metric: how much does behavior change day-to-day?

| Metric | High Variance | Low Variance |
|---|---|---|
| Deployment time | 5 min – 2 hours | 8–12 min |
| Error rate | 0.01% – 5% | 0.01% – 0.05% |
| P99 latency | 50 ms – 2 seconds | 100–150 ms |

---

### "Backups are only as good as the last restore."

**Full Meaning:** Having backups is not enough. The only test that matters is whether you can *restore* from those backups. Untested backups are not backups — they're wishful thinking.

**Application:**
- Schedule regular restore drills for every backup
- Automate restore verification — don't rely on humans to run it
- Measure "Restore Success Rate" as an SRE metric
- Test restores of the *most recent* backup, not just a known good one from last month

> **Corollary:** "A backup that has never been restored is a backup that has never been tested."

---

### "If you have no SLOs, toil is your job."

**Full Meaning:** Service Level Objectives (SLOs) define what "good enough" means. Without them, every minor issue is potentially critical, every degradation requires immediate investigation, and your team spends all its time firefighting. SLOs are the mechanism that transforms toil into engineering.

**Application:**
- Define SLOs for every service in terms of user-facing metrics
- Use the error budget to decide what deserves immediate attention vs. what can wait
- When SLOs are being met, accept that sub-perfect behavior is acceptable
- Review and adjust SLOs quarterly

---

### "Hope is not a strategy."

**Full Meaning:** Hoping that a failure won't happen, that a system will recover on its own, that an alert is a false positive — these are all operational failures disguised as optimism. Reliability must be engineered, not wished into existence.

**Application:**
- Replace "hope" with "plan" for every identified risk
- When you hear "I hope that doesn't happen," ask "What's our plan if it does?"
- Use pre-mortems: assume a future failure, work backward to prevent it

---

### "Scale maintenance sublinearly with the growth of the service."

**Full Meaning:** As your service grows, operational workload should grow slower than the service itself. If toil grows linearly (or worse, superlinearly) with service size, your team will be overwhelmed. The goal is sublinear scaling of maintenance effort.

**Application:**
- Automate every operational task that grows with service size
- Invest in tooling that reduces per-server or per-request maintenance overhead
- Track "toil per unit of service" as a metric
- Target: 50% of SRE time on engineering, 50% or less on toil

---

### "May all your incidents be novel."

**Full Meaning:** This is a benediction from one SRE to another. Novel incidents teach you something new about your system. Repeated incidents (the same failure mode recurring) indicate that your post-incident actions were insufficient.

**Application:**
- Track incident types and look for repeats
- If the same root cause causes multiple incidents, the corrective actions from the first PIR were inadequate
- Novel incidents are opportunities to learn; repeated incidents are failures of process

---

### Prodverbs Reference Table

| Prodverb | Core Category | Practical Application |
|---|---|---|
| Cascading failures happen | Failure modes | Circuit breakers, bulkheading |
| Production is never homogenous | Operations | Build for heterogeneity |
| Empty config ≠ delete everything | Automation peril | Verify before acting on "empty" |
| Every SPOF fails eventually | Design principle | SPOF register, elimination |
| Mitigate → diagnose → fix | Incident response | Priority ordering during incidents |
| Two agreeing systems will disagree | Distributed systems | Design for state divergence |
| Decrease variance, increase mean | Metrics strategy | Target P99 before P50 |
| Backups = last restore | Data integrity | Regular restore drills |
| No SLOs → toil is your job | SLO importance | Define SLOs for every service |
| Hope is not a strategy | Operations culture | Plans, not prayers |
| Scale maintenance sublinearly | Automation | Toil tracking, automation ROI |
| May all incidents be novel | Learning culture | Track incident repetition |

---

## Part 3: Synthesis — How These Lessons and Prodverbs Complement the Core SRE Book

The original Google SRE book (Beyer et al., 2016) established the foundational framework: SLOs, error budgets, monitoring, toil elimination, and the SRE model itself. The "Twenty Years" lessons and Prodverbs extend this framework in important ways.

### What the Original SRE Book Covers Well

The book thoroughly addresses:
- The SRE model and its relationship to traditional ops
- Service Level Objectives, Indicators, and Agreements
- Error budgets as a decision-making mechanism
- Monitoring and alerting (the "four golden signals")
- Automation and the elimination of toil
- Capacity planning and load shedding
- Post-incident review culture

### What the New Lessons Add

#### Novel Additions (Not in the Original Book)

**1. Risk/Mitigation Proportionality (Lesson 1):** The book discusses accepting risk via error budgets, but doesn't address the *risk of mitigations themselves*. This is a critical operational insight: mitigations have their own probability of failure, and choosing the wrong mitigation for the severity level can make things worse.

**4. Big Red Button (Lesson 4):** The concept of a pre-designed, single-action emergency response is entirely absent from the original book. It addresses runbooks and procedures but not the cognitive-load insight that complex procedures fail during emergencies.

**6. Backup Communication Channels (Lesson 6):** The original book doesn't consider that your incident response tools might fail *along with* your service. This is a cultural blind spot in the original work — it assumes the tooling layer is always available.

**7. Intentional Degraded Performance (Lesson 7):** The original book frames reliability in binary terms (available/unavailable). The degraded-mode concept — deliberately serving a reduced experience — is not addressed.

**10. Reduce Time Between Rollouts (Lesson 10):** While the book assumes fast rollouts, the Pokémon GO case study demonstrates the *causal mechanism*: slow rollouts produce large deltas, which makes safety reasoning impossible. This is a more precise diagnosis than the book provides.

**11. Hardware Version Diversity (Lesson 11):** Infrastructure diversity is not discussed in the original book. It assumes homogeneous infrastructure is the goal. The March 2020 zero-day demonstrates that homogeneity is itself a vulnerability.

#### Reinforcements and Extensions

**Lesson 2 (Test Recovery):** Reinforces the SRE book's emphasis on automation but adds the specific insight that untested recovery procedures are a form of technical debt.

**Lesson 3 (Canary All Changes):** Extends the book's canary discussion with a concrete failure case (YouTube 13-minute outage) and the insight that the failure mode is often process discipline, not technical knowledge.

**Lesson 5 (Integration Testing):** Extends the book's discussion of testing with the Calendar outage case study, which demonstrates that even 100% unit test coverage can miss interaction failures.

**Lesson 8 (Disaster Resilience):** Extends the book's brief mention of tabletop exercises into a full framework distinguishing recovery testing from resilience testing.

**Lesson 9 (Automate Mitigations):** Extends the book's automation discussion with the 6-day networking outage case study, which provides the "clear signal → safe action" framework for prioritizing automated mitigation over root cause analysis.

### How Prodverbs Fill Gaps

The Prodverbs provide operational wisdom that the book's formal framework doesn't capture:

| Prodverb | Gap in Original Book | How It Fills It |
|---|---|---|
| "Empty config ≠ delete everything" | Book assumes automation is safe; doesn't address automation failure modes | Identifies a specific class of automation-induced disasters |
| "Production is never homogenous" | Book assumes you can enforce uniformity | Acknowledges that production will always drift, and you must design for that |
| "May all your incidents be novel" | Book focuses on PIR process; doesn't address incident *repetition* | Introduces the concept that repeated incidents indicate process failure |
| "Decrease variance, increase mean" | Book focuses on mean-based metrics | Shifts focus to variance reduction as a primary reliability strategy |
| "If two systems agree, they will disagree" | Book assumes eventual consistency is adequate | Acknowledges that state divergence is not a corner case but a certainty |

### Practical Framework for Teams

To integrate these lessons with the original SRE book:

1. **Start with the SRE book foundation** — error budgets, SLOs, monitoring, toil elimination
2. **Adopt the "Big Red Button"** as an operational stopgap (Lesson 4)
3. **Audit your communication channels** for shared dependencies (Lesson 6)
4. **Design degraded modes** for every critical feature (Lesson 7)
5. **Run quarterly tabletop exercises** for disaster scenarios (Lesson 8)
6. **Automate at least one mitigation** per quarter (Lesson 9)
7. **Track deployment frequency** and push toward hourly rollouts (Lesson 10)
8. **Diversify infrastructure** at the hardware and vendor layers (Lesson 11)
9. **Internalize the Prodverbs** — they encode decades of hard-won experience in single sentences
10. **Review incident types** quarterly and flag repeated failures (Prodverb: "May all your incidents be novel")

### Closing Thought

> "The SRE book gave us the theory. These twenty years of lessons gave us the scars. The Prodverbs gave us the wisdom." — Synthesized from Google SRE community discussions

The original SRE book is a *design document* — it describes how to build reliable systems. The "Twenty Years" lessons are an *after-action report* — they describe what still goes wrong despite good design. The Prodverbs are an *operational manual* — they encode the instincts that keep you out of trouble between the formal processes.

All three are essential. None alone is sufficient.

#!/usr/bin/env python3
"""
SLO Burn Rate Calculator

Calculates error budget burn rate from SLI data.

Usage:
    python3 slo-burn-rate.py --slo-target 99.9 --window 1h --good-events 99500 --total-events 100000
    python3 slo-burn-rate.py --slo-target 99.99 --window 7d --error-rate 0.001
    python3 slo-burn-rate.py --slo-target 99.95 --window 30d --good-events 998500 --total-events 1000000
    cat sli_data.json | python3 slo-burn-rate.py --slo-target 99.9 --window 1h
"""

import argparse
import json
import sys
import math
from datetime import timedelta

# ---------------------------------------------------------------------------
# Window parsing
# ---------------------------------------------------------------------------

WINDOW_SECONDS = {
    "1h": 3600,
    "6h": 21600,
    "24h": 86400,
    "7d": 604800,
    "30d": 2592000,
}


def parse_window(window_str: str) -> int:
    """Convert a window string like '1h', '24h', '7d', '30d' to seconds."""
    window_str = window_str.strip().lower()
    if window_str in WINDOW_SECONDS:
        return WINDOW_SECONDS[window_str]

    # Fallback: try to parse programmatically
    if window_str.endswith("h"):
        hours = int(window_str[:-1])
        return hours * 3600
    elif window_str.endswith("d"):
        days = int(window_str[:-1])
        return days * 86400
    elif window_str.endswith("m"):
        minutes = int(window_str[:-1])
        return minutes * 60
    elif window_str.endswith("s"):
        return int(window_str[:-1])
    else:
        raise ValueError(
            f"Unrecognized window format: '{window_str}'. "
            f"Use e.g. 1h, 6h, 24h, 7d, 30d."
        )


def format_duration(seconds: int) -> str:
    """Format a duration in seconds to a human-readable string."""
    if seconds < 60:
        return f"{seconds}s"
    elif seconds < 3600:
        return f"{seconds // 60}m"
    elif seconds < 86400:
        hours = seconds // 3600
        remainder = seconds % 3600
        if remainder == 0:
            return f"{hours}h"
        return f"{hours}h{remainder // 60}m"
    else:
        days = seconds // 86400
        remainder = seconds % 86400
        if remainder == 0:
            return f"{days}d"
        return f"{days}d{remainder // 3600}h"


# ---------------------------------------------------------------------------
# Core calculation
# ---------------------------------------------------------------------------


def calculate_burn_rate(
    slo_target: float,
    window_seconds: int,
    good_events: int | None = None,
    total_events: int | None = None,
    error_rate: float | None = None,
) -> dict:
    """
    Calculate error budget burn rate and related metrics.

    Parameters
    ----------
    slo_target : float
        Target availability percentage (e.g. 99.9 for 99.9%).
    window_seconds : int
        Analysis window in seconds.
    good_events : int, optional
        Number of successful events.
    total_events : int, optional
        Total number of events.
    error_rate : float, optional
        Direct error rate (e.g. 0.001 for 0.1% error rate).

    Returns
    -------
    dict with keys:
        sli_target, window, window_seconds, good_events, total_events,
        error_rate, measured_availability, burn_rate,
        budget_remaining_pct, time_to_exhaustion, status
    """
    slo_threshold = slo_target / 100.0  # e.g. 99.9 -> 0.999
    allowed_error_rate = 1.0 - slo_threshold  # e.g. 0.001

    if error_rate is not None:
        # Direct error rate provided
        measured_error_rate = error_rate
        measured_availability = 1.0 - measured_error_rate
        if good_events is None and total_events is None:
            good_events = None
            total_events = None
        elif good_events is not None and total_events is not None:
            pass  # keep both
        else:
            # Can't derive the missing one
            good_events = None
            total_events = None
    elif good_events is not None and total_events is not None:
        if total_events == 0:
            return {
                "error": "total_events must be greater than 0",
                "slo_target": slo_target,
                "window": format_duration(window_seconds),
            }
        measured_availability = good_events / total_events
        measured_error_rate = 1.0 - measured_availability
    else:
        return {
            "error": (
                "Provide either --good-events + --total-events, "
                "or --error-rate directly."
            ),
            "slo_target": slo_target,
            "window": format_duration(window_seconds),
        }

    # Burn rate
    if allowed_error_rate == 0:
        burn_rate = float("inf") if measured_error_rate > 0 else 0.0
    else:
        burn_rate = measured_error_rate / allowed_error_rate

    # Budget remaining
    if total_events is not None and total_events > 0:
        total_budget = allowed_error_rate * total_events
        consumed = total_events - good_events if good_events is not None else measured_error_rate * total_events
        budget_remaining = max(0.0, total_budget - consumed)
        budget_remaining_pct = (
            (budget_remaining / total_budget * 100.0) if total_budget > 0 else 0.0
        )
    else:
        # Without event counts, use the error rate as a proxy
        # Budget remaining = (allowed_error_rate - measured_error_rate) / allowed_error_rate
        if allowed_error_rate > 0:
            budget_remaining_pct = max(
                0.0,
                ((allowed_error_rate - measured_error_rate) / allowed_error_rate) * 100.0,
            )
        else:
            budget_remaining_pct = 0.0 if measured_error_rate > 0 else 100.0

    # Time to exhaustion
    if burn_rate > 0:
        # How long until the remaining budget burns at the current rate?
        window_error_budget = allowed_error_rate  # per-unit budget for the window
        remaining_fraction = max(0.0, 1.0 - (measured_error_rate / allowed_error_rate)) if allowed_error_rate > 0 else 0.0

        if burn_rate > 0 and remaining_fraction > 0:
            time_to_exhaustion_secs = (remaining_fraction / burn_rate) * window_seconds
        elif remaining_fraction <= 0:
            time_to_exhaustion_secs = 0
        else:
            time_to_exhaustion_secs = float("inf")
    else:
        time_to_exhaustion_secs = float("inf")

    # Status determination
    if burn_rate == 0 and measured_error_rate == 0:
        status = "healthy"
    elif budget_remaining_pct <= 0 or measured_error_rate >= allowed_error_rate:
        status = "exhausted"
    elif budget_remaining_pct < 30 and burn_rate > 3:
        status = "burning"
    elif budget_remaining_pct < 70 and burn_rate > 1:
        status = "warning"
    elif burn_rate > 1:
        # Elevated but budget still healthy
        status = "warning"
    else:
        status = "healthy"

    if budget_remaining_pct < 5:
        status = "exhausted"
    elif budget_remaining_pct < 30:
        # Re-check; might be "burning" if rate is really high
        if burn_rate >= 10:
            status = "burning"
        elif status != "burning":
            status = "warning"
    elif budget_remaining_pct < 70:
        if burn_rate >= 3:
            status = "burning"
        elif burn_rate > 1:
            status = "warning"

    # Format time to exhaustion
    if time_to_exhaustion_secs == float("inf"):
        formatted_ttl = "infinite"
    elif time_to_exhaustion_secs <= 0:
        formatted_ttl = "exhausted"
    else:
        formatted_ttl = format_duration(int(time_to_exhaustion_secs))

    return {
        "slo_target": slo_target,
        "window": format_duration(window_seconds),
        "window_seconds": window_seconds,
        "good_events": good_events,
        "total_events": total_events,
        "error_rate": round(measured_error_rate, 6),
        "measured_availability": round(measured_availability * 100.0, 4),
        "burn_rate": round(burn_rate, 4),
        "budget_remaining_pct": round(budget_remaining_pct, 2),
        "budget_remaining": round(budget_remaining_pct, 2),
        "time_to_exhaustion": formatted_ttl,
        "time_to_exhaustion_seconds": (
            int(time_to_exhaustion_secs)
            if time_to_exhaustion_secs not in (float("inf"), float("-inf"))
            else None
        ),
        "status": status,
    }


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------


def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Calculate SLO error budget burn rate.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=__doc__,
    )
    parser.add_argument(
        "--slo-target",
        type=float,
        required=True,
        help="SLO target as a percentage (e.g. 99.9 for 99.9%%).",
    )
    parser.add_argument(
        "--window",
        type=str,
        default="1h",
        help=(
            "Analysis window. Supports: 1h, 6h, 24h, 7d, 30d "
            "or custom like 90m, 2h30m, 14d. Default: 1h."
        ),
    )
    parser.add_argument(
        "--good-events",
        type=int,
        default=None,
        help="Number of successful (good) events.",
    )
    parser.add_argument(
        "--total-events",
        type=int,
        default=None,
        help="Total number of events.",
    )
    parser.add_argument(
        "--error-rate",
        type=float,
        default=None,
        help="Direct error rate (e.g. 0.001 for 0.1%% error rate).",
    )
    parser.add_argument(
        "--pretty",
        action="store_true",
        default=False,
        help="Pretty-print JSON output with indentation.",
    )
    parser.add_argument(
        "--color",
        action="store_true",
        default=False,
        help="Add terminal color codes to the JSON output for readability.",
    )
    return parser.parse_args(argv)


def _color_status(status: str) -> str:
    colors = {
        "healthy": "\033[32m",    # green
        "warning": "\033[33m",    # yellow
        "burning": "\033[31m",    # red
        "exhausted": "\033[1;31m",  # bold red
    }
    reset = "\033[0m"
    return f"{colors.get(status, '')}{status}{reset}"


def main() -> None:
    args = parse_args()

    # If reading from stdin, look for JSON data
    if not sys.stdin.isatty():
        try:
            stdin_data = sys.stdin.read().strip()
            if stdin_data:
                input_data = json.loads(stdin_data)
                # Merge stdin data with CLI args, CLI takes precedence
                if args.good_events is None and "good_events" in input_data:
                    args.good_events = input_data["good_events"]
                if args.total_events is None and "total_events" in input_data:
                    args.total_events = input_data["total_events"]
                if args.error_rate is None and "error_rate" in input_data:
                    args.error_rate = input_data["error_rate"]
                if args.window == "1h" and "window" in input_data:
                    # Only override if default hasn't been changed
                    pass  # keep explicit CLI
        except (json.JSONDecodeError, Exception):
            pass  # If stdin isn't valid JSON, just use CLI args

    try:
        window_seconds = parse_window(args.window)
    except ValueError as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)

    result = calculate_burn_rate(
        slo_target=args.slo_target,
        window_seconds=window_seconds,
        good_events=args.good_events,
        total_events=args.total_events,
        error_rate=args.error_rate,
    )

    indent = 2 if args.pretty else None

    if args.color:
        # Color only the status field for terminal readability
        status_colorized = _color_status(result.get("status", ""))
        json_str = json.dumps(result, indent=indent)
        # Replace the plain status string with colorized version in the JSON
        plain_status = result.get("status", "")
        safe_status = json.dumps(plain_status)
        color_status_with_quotes = json.dumps(plain_status.replace(plain_status, status_colorized))
        # Simple approach: just replace the status value
        # But this can be fragile. Let's just print JSON clean and add color on top.
        print(json_str, file=sys.stderr)
        print(f"Status: {_color_status(result.get('status', ''))}")
    else:
        print(json.dumps(result, indent=indent))

    # Exit code indicates health
    status = result.get("status", "")
    if status == "healthy":
        sys.exit(0)
    elif status in ("warning", "burning"):
        sys.exit(1)
    elif status == "exhausted":
        sys.exit(2)
    elif "error" in result:
        sys.exit(3)


if __name__ == "__main__":
    main()

# Error Budget Policy

| Metadata | Value |
|---|---|
| **Document Owner** | [SRE Team Lead] |
| **Version** | 1.0 |
| **Last Reviewed** | [YYYY-MM-DD] |
| **Review Cadence** | [Quarterly / Semi-Annual] |
| **Approved By** | [Engineering Director / VP of Infrastructure] |

---

## 1. Purpose

This document defines the error budget policy for all services covered under the Site Reliability Engineering (SRE) framework. The error budget is the primary mechanism for balancing feature velocity against service reliability. It quantifies how much "unreliability" is acceptable over a given window and governs when releases must be halted to protect the user experience.

---

## 2. Scope

This policy applies to the following services:

| Service | SLO Target | Measurement Method | Owner Team |
|---|---|---|---|
| [Service Name] | [e.g. 99.9%] | [e.g. Request success rate] | [Team] |
| [Service Name] | [e.g. 99.99%] | [e.g. Latency P99 < 200ms] | [Team] |
| [Service Name] | [e.g. 99.95%] | [e.g. Request success rate] | [Team] |

All services with an SLO defined in [link to SLO documentation] are subject to this policy. Services without an explicit SLO are considered best-effort and are not governed by error budget rules.

---

## 3. Error Budget Mechanics

### 3.1 Accrual

Error budgets are calculated over a **rolling [30]-day window**. The total budget available at the start of any window is:

```
total_error_budget = (1 - slo_target) * total_requests
```

For a service with an SLO of 99.9% and 10,000,000 requests over 30 days:

```
total_error_budget = (1 - 0.999) * 10,000,000 = 10,000 errors
```

### 3.2 Consumption

Each request that does **not** meet the SLO criteria (e.g., 5xx status code, latency exceeding P99 threshold) consumes one unit of error budget. Consumption is calculated as:

```
consumed_budget = bad_events_count
budget_remaining = total_error_budget - consumed_budget
```

### 3.3 Burn Rate

The **burn rate** measures how fast the error budget is being consumed relative to the target:

```
burn_rate = (1 - measured_availability) / (1 - slo_target)
```

A burn rate of:
- **1.0** means errors are occurring exactly at the SLO boundary.
- **> 1.0** means errors are exceeding the allowed rate (budget is being consumed faster than it accrues).
- **< 1.0** means the service is outperforming its SLO (budget is accumulating).

---

## 4. Consumption Thresholds & Actions

The error budget is divided into four color-coded zones. Each zone triggers specific operational and release-gating actions.

| Zone | Budget Remaining | Burn Rate (over 1h) | Burn Rate (over 6h) | Actions |
|---|---|---|---|---|
| **Green** | > 70% | < 1 | < 1 | Normal operations. Releases permitted. No action required. |
| **Yellow** | 30% – 70% | 1 – 3 | 1 – 2 | Tag incident. Create postmortem for any SLO-violating event. Notify team lead. Releases proceed with caution. |
| **Red** | 5% – 30% | 3 – 10 | 2 – 5 | **Releases halted.** On-call escalation. Engineering leadership notified. Daily standup review. Blameless postmortem required. |
| **Exhausted** | < 5% | > 10 | > 5 | **All non-critical releases frozen.** Emergency incident response activated. Executive briefing required. Auto-scaling and resilience measures enforced. RCAs due within 48 hours. |

### Detailed Actions by Threshold

#### Green (Normal)
- All release types permitted.
- Standard monitoring and alerting.
- Regular on-call shift rotation.

#### Yellow (Elevated Consumption)
1. Tag all recent deployments and changes in the [Deployment Tracker].
2. Open a [Low-Severity Incident] in the incident management system.
3. Review recent changes with the on-call engineer.
4. Notify the service team lead.
5. **Release Gate**: Critical/hotfix releases only with team lead approval.

#### Red (Critical Consumption)
1. **Immediately halt all non-critical releases.**
2. Escalate to the [Secondary On-Call Engineer].
3. Notify [Engineering Manager] and [SRE Manager].
4. Initiate a [War Room] if burn rate exceeds [8].
5. Create a blameless postmortem draft.
6. Review dashboards, logs, and metrics to identify root cause.
7. **Release Gate**: Only emergency security patches and P0 incident fixes permitted.

#### Exhausted (Budget Depleted)
1. **Freeze all releases** except patches for CVSS >= [7.0] vulnerabilities.
2. Activate [Major Incident Response Protocol].
3. Brief [VP of Engineering / CTO] within [1 hour].
4. Compile draft Root Cause Analysis (RCA) within [24 hours], final within [48 hours].
5. Implement mandatory resilience improvements (rate limiting, circuit breakers, auto-scaling).
6. **Release Gate**: Only incident remediation code allowed. Requires SRE Director approval.
7. Conduct a reliability postmortem with all stakeholders.

---

## 5. Release Gating Rules

| Release Type | Green | Yellow | Red | Exhausted |
|---|---|---|---|---|
| Standard feature releases | Allowed | Halted | Halted | Halted |
| Bug fixes (non-critical) | Allowed | Allowed | Halted | Halted |
| Critical bug fixes / hotfixes | Allowed | Team lead approval | SRE lead approval | SRE Director approval |
| Security patches (CVSS < 7) | Allowed | Allowed | SRE lead approval | SRE Director approval |
| Security patches (CVSS >= 7) | Allowed | Allowed | Allowed | Allowed |
| Infrastructure changes | Allowed | Caution required | Halted | Halted |
| Config changes | Allowed | Peer review required | SRE lead approval | Halted |

Releases may resume when the error budget returns to **Yellow** or above and the [Release Review Board] approves.

---

## 6. Exceptions & Overrides

| Exception | Process | Approver |
|---|---|---|
| Emergency release during freeze | File [Exception Request] with justification | [VP of Engineering] |
| SLO adjustment | Submit [SLO Change Proposal] with data | [SRE Steering Committee] |
| Budget over-allocation (new feature launch) | Request via [Governance Board] | [CTO / VP Eng] |
| One-time budget extension | Submit [Budget Waiver Request] | [SRE Director] |
| Experimental / Beta services | Opt-out of error budget policy | [Product Manager + SRE Lead] |

---

## 7. Escalation Paths

### When Budget is Exhausted

```
Level 0: On-Call Engineer (Primary)
    ↓ (if unresolved in 15 min)
Level 1: On-Call Engineer (Secondary) + Service Team Lead
    ↓ (if unresolved in 30 min)
Level 2: SRE Manager + Engineering Manager
    ↓ (if unresolved in 60 min)
Level 3: Director of Engineering / VP of Infrastructure
    ↓ (if unresolved in 120 min)
Level 4: Incident Commander + Executive Team
```

### Communication Chain

1. **Internal**: [#oncall-alerts Slack channel] — automated alerts from monitoring.
2. **Engineering**: [#sre-eng channel] — incident coordination.
3. **Leadership**: Email to [sre-leadership@example.com] for Red zone or above.
4. **Executive**: Email + Slack DM to [VP Engineering] for Exhausted zone.

---

## 8. Review Cadence

| Review Type | Frequency | Participants |
|---|---|---|
| Error budget report review | Weekly | Service team + SRE |
| Policy effectiveness review | Monthly | SRE team |
| SLO and budget adjustment | Quarterly | SRE + Product + Engineering leadership |
| Full policy audit | Annually | SRE team + Legal + Compliance |

### Required Artifacts

- Weekly error budget report (auto-generated by [Monitoring System] dashboard).
- Monthly consumption trends analysis.
- Quarterly SLO attainment review with [Stakeholder Group].
- Annual policy effectiveness report.

---

## 9. Appendices

### A. Example Budget Calculation

| Service | SLO | Total Requests | Error Budget | Consumed | Remaining | % Remaining | Status |
|---|---|---|---|---|---|---|---|
| api-gateway | 99.9% | 50,000,000 | 50,000 | 5,000 | 45,000 | 90% | Green |
| user-service | 99.95% | 10,000,000 | 5,000 | 3,500 | 1,500 | 30% | Yellow |
| search-service | 99.99% | 100,000,000 | 10,000 | 9,200 | 800 | 8% | Red |
| payment-service | 99.99% | 5,000,000 | 500 | 490 | 10 | 2% | Exhausted |

### B. Related Documents

- [Service Level Objectives (SLO) Definition]
- [Incident Response Playbook]
- [Release Management Policy]
- [Postmortem Template]
- [On-Call Rotation Schedule]
- [SLO Burn Rate Script / Tooling]

### C. Glossary

| Term | Definition |
|---|---|
| **Error Budget** | The maximum number of failures allowed in a service over a given time window before violating the SLO. |
| **Burn Rate** | The rate at which the error budget is being consumed relative to the target. |
| **SLO** | Service Level Objective — the target level of reliability for a service. |
| **SLI** | Service Level Indicator — the actual measured value of reliability. |
| **SLA** | Service Level Agreement — a contractual commitment to reliability. |
| **Exhausted** | State where the error budget is depleted (less than 5% remaining). |

# Incident Command Checklist

**Role:** Incident Commander (IC)
**Purpose:** Structured runbook for managing an active incident from page to postmortem.
**Usage:** Check off items as completed. Print or keep open in a dedicated window.

---

## Severity Reference Table

| Sev | Title | Description | SLO Impact | Example | Response Time |
|:---:|-------|-------------|:----------:|---------|:-------------:|
| **SEV1** | Critical Outage | Complete or near-complete service unavailability affecting all users | Critical drop | Site down, core API unreachable | < 5 min |
| **SEV2** | Major Degradation | Significant feature impairment or partial outage affecting a subset of users | Major drop | Payment failures, high latency on major endpoint | < 15 min |
| **SEV3** | Minor Issue | Isolated feature bug or minor degradation with workaround available | Minor drop | UI glitch on non-critical page, slow but functional | < 60 min |
| **SEV4** | Informational | Non-urgent observation; no user-facing impact | No SLO impact | Cosmetic bug, log noise, low-severity alert | Next business day |

> **Escalation rule:** If uncertainty exists between two severity levels, declare at the **higher** severity and re-evaluate during triage.

---

## Phase 1: Incident Recognition ⚠️

Goal: Acknowledge the alert and formally declare an incident.

- [ ] **Acknowledge the page/alert**
  - Respond to the monitoring alert, ticket, or manual report within the SLO response time for the suspected severity.
  - _Note the timestamp — this is `t_incident_ack` for your timeline._

- [ ] **Confirm severity level**
  - Use the Severity Reference Table above. Check user-facing impact, SLO burn rate, and affected services.
  - _Unsure? Declare SEV1 and triage down._

- [ ] **Open the incident in the tracking system**
  - Create incident record in PagerDuty/OpsGenie/incident management tool.
  - _Record the incident ID / internal ticket number._

- [ ] **Declare the incident**
  - Broadcast a clear declaration to the #incidents channel or equivalent: "I am declaring a SEV[X] incident for [service] — [brief symptom]."
  - Formats: `!ic declare --severity SEV1 --service api-gateway --summary "503s on all routes"`
  - _This is `t_incident_declared`._

- [ ] **Set a 5-minute timer for initial triage**
  - Start a timer. Initial triage must begin before it expires.

---

## Phase 2: Initial Triage 🔍

Goal: Understand what's broken, who needs to be involved, and start organizing.

- [ ] **Assess blast radius and user impact**
  - Check dashboards (latency, error rate, throughput, saturation).
  - Identify affected services, user segments (e.g. free-tier vs. paid), and geographic regions.
  - Verify vs. the last known good state (LKGS) — when was the last deploy/change?

- [ ] **Select the response team**
  - Identify the right subject-matter experts (SMEs) by service ownership.
  - Call in on-call engineers from affected teams.
  - _Do not over-invite — too many cooks slows response._

- [ ] **Assign core roles**
  - **Incident Commander (IC):** YOU — owns coordination, decision-making, and timeline.
  - **Scribe:** Responsible for real-time timeline/tick-tock documentation. _Assign this immediately — it is the most commonly forgotten role._
  - **Communications Lead (Comms):** Owns external/internal status updates.
  - **Subject Matter Experts (SMEs):** Engineers actively debugging/mitigating.
  - **Sub-ICs (if needed):** Delegate management of isolated subsystems (e.g. database team, frontend team).

- [ ] **Escalate if needed**
  - If the severity is beyond current team scope or requires exec awareness, escalate through the on-call chain.
  - Notify management / NOC / designated stakeholders per your org escalation policy.

- [ ] **Update incident status**
  - Set status to **TRIAGING** in the incident tracking system.
  - Confirm the severity one more time with the assembled team.

---

## Phase 3: Response / Mitigation 🛠️

Goal: Drive toward mitigation while maintaining clear communication and documentation.

### Communications

- [ ] **Establish the primary communications channel**
  - Dedicated Slack/Discord channel (e.g. `#incident-servicedown-YYYYMMDD`).
  - Pin the incident ID, current severity, and list of IC/Comms/Scribe/SMEs.
  - _All incident-related discussion happens here — no DMs about the incident._

- [ ] **Set up the incident timeline document**
  - Google Doc / Notion / HackMD shared with the entire response team.
  - Template includes: timestamp, event, action, owner.
  - Scribe begins recording every event immediately.

- [ ] **Draft first external communication (if applicable)**
  - Status page update (e.g. Statuspage.io).
  - Comms Lead owns this; IC approves before publishing.
  - Format: "We are investigating reports of [symptom] affecting [scope]. Next update in [X] minutes."

### Mitigation

- [ ] **Assemble hypotheses**
  - SMEs brainstorm root cause candidates.
  - List on the timeline doc with owners for each.
  - _Log everything, even dead ends — they prevent re-tracing._

- [ ] **Drive parallel investigation**
  - SMEs work independently on different hypotheses.
  - IC removes blockers: access, permissions, credentials, config changes.

- [ ] **Evaluate mitigation options**
  - Possible mitigations: rollback, feature flag disable, traffic shift, scale-up, config revert.
  - Risk/reward each option. _Speed is important; correctness is more important for SEV1._
  - IC makes the final call on which mitigation to pursue.

- [ ] **Pass the pre-mutation gate**
  - Confirm current human authorization for the specific action and scope, independently attributable to the human authority. The acting agent cannot self-assign the IC role or authorize its own action.
  - Record the target, affected population, maximum blast radius, success and abort/rollback criteria, rollback path, and stopping authority.
  - If the gate cannot be satisfied, stop and hand off or escalate rather than mutating production.

- [ ] **Apply mitigation**
  - Execute the chosen action. _Scribe logs who did what and when._
  - _This is the target we want to reach: `t_mitigation_applied`._

- [ ] **Issue progress updates**
  - Comms Lead sends regular status updates (every 15 min for SEV1, every 30 min for SEV2).
  - IC reviews and approves each update.
  - Even "no new information" is a valid update.

---

## Phase 4: Resolution ✅

Goal: Confirm the incident is truly over and stabilize the system.

> Use the [R-01 closure evidence sequence](runbook-template.md#r-01-post-incident-steps) before declaring the incident resolved. Green monitoring and health checks are partial evidence; if any required evidence is missing, do not mark the incident **RESOLVED**, record the unverified boundary, and hand off or escalate while it remains in its active response state.

- [ ] **Verify the fix in production**
  - SMEs confirm the mitigation resolved the symptoms:
    - Error rate returned to baseline.
    - Latency normalized.
    - All affected endpoints returning correct responses.
    - Relevant dependency health is verified.
    - Data/state correctness and secondary effects are checked.
  - _Scribe logs `t_verification_complete`._

- [ ] **Monitor stability window**
  - Observe the system for at least one full monitoring cycle (recommended: 15 min for SEV1, 5 min for SEV2) with stable metrics.
  - Watch for secondary effects (cascading failures, degraded dependent services).

- [ ] **Run a smoke test of critical user journeys**
  - Execute the team's standard post-incident smoke test suite (or manually verify: login, search, checkout, key API calls).
  - _If smoke tests fail, return to Phase 3._

- [ ] **Confirm impact bounds**
  - Data loss? Corrupted state? Need for manual recovery (e.g. replay queue, re-sync replicas)?
  - Document known edge cases that might still be affected.

- [ ] **Declare the incident resolved**
  - Only after the R-01 closure evidence sequence, completed stability window, and independent human confirmation pass: "This incident is now RESOLVED. The mitigation was [summary]. Monitoring continues."
  - Update incident tracking system status to **RESOLVED**.
  - _This is `t_resolved`._

- [ ] **Update external status page**
  - Set status to "Resolved — no further issues expected."
  - Include a brief summary and, if available, an ETA for the postmortem.

---

## Phase 5: Recovery 🔄

Goal: Return to normal operations, capture follow-up work, and schedule the learning event.

### Immediate Follow-Up

- [ ] **Restore normal operations**
  - Close the incident channel — leave it archived for reference, not active.
  - Remove on-call overrides, restore normal rotation if it was modified.
  - Comms Lead sends a final summary to wider team/stakeholders.

- [ ] **Evaluate data / state recovery needs**
  - Any manual repair needed? Queue replays? Database repairs?
  - Create tracking tickets for each remediation task.

- [ ] **Triage and document follow-up action items**
  - Capture every "we should fix this" that came up during the incident.
  - Categorize as:
    - **P0 / Must fix:** Directly contributed to the incident or would have reduced severity.
    - **P1 / Should fix:** Would improve detection or response time.
    - **P2 / Nice to have:** Would improve general robustness.
  - File tickets in your issue tracker with labels like `post-incident`, `incident-YYYYMMDD`.

### Postmortem

- [ ] **Schedule the postmortem meeting**
  - Within 48 hours for SEV1, within 5 business days for SEV2.
  - Required attendees: IC, Scribe, Comms Lead, SMEs. Optional: stakeholders.
  - Duration: 60 min for SEV1, 30 min for SEV2.

- [ ] **Assign the postmortem owner**
  - Usually the IC or a designated engineering manager.
  - Owner is responsible for drafting the doc and tracking action items to closure.

- [ ] **Create the postmortem document**
  - Use the organization's standard postmortem template (or create one with sections for):
    1.  **Summary** — What happened, in 2-3 sentences.
    2.  **Timeline** — Curated from the scribe's tick-tock log.
    3.  **Impact** — Users affected, duration, data loss (if any).
    4.  **Root Cause** — What actually caused it.
    5.  **Trigger** — What started it (deploy, config change, external dependency).
    6.  **Detection** — How we learned about it. How quickly?
    7.  **Response** — What went well, what didn't.
    8.  **Action Items** — The P0/P1/P2 items with owners and due dates.
  - _Blameless postmortem culture applies — focus on systems, not people._

- [ ] **Send postmortem for review**
  - Share with the response team before publishing widely.
  - Address factual corrections before finalizing.

---

## Quick Reference: IC Do's and Don'ts

| Do | Don't |
|----|-------|
| Delegate debugging to SMEs | Jump into debugging yourself |
| Keep the timeline updated | Trust memory — log everything, even small things |
| Make decisions and own them | Wait for consensus on time-sensitive calls |
| Clear blockers for the team | Micromanage engineers |
| Keep comms structured and regular | Go silent for extended periods |
| Escalate when out of depth | Hero-mode through a crisis alone |
| Hand off cleanly if rotating out | Drop the IC role without a full handoff |

---

## Key Timestamps to Record

| Event | Variable | Description |
|-------|----------|-------------|
| Page received | `t_ack` | When the IC first acknowledged the alert |
| Incident declared | `t_declared` | Formal declaration timestamp |
| Team assembled | `t_assembled` | Full response team in the channel |
| Mitigation applied | `t_mitigated` | When the fix was deployed |
| Verification complete | `t_verified` | Symptoms confirmed gone |
| Incident resolved | `t_resolved` | Formal resolution declaration |
| Time to Mitigation (TTM) | `t_mitigated - t_declared` | Key metric |
| Time to Resolve (TTR) | `t_resolved - t_declared` | Key metric |

---

*Template version: 1.0 — Last updated: 2025-06-05*
*Maintainer: Site Reliability Engineering*

# Incident Communication Templates

Copy-paste ready message templates for communicating during every phase of an incident. Replace `[bracket]` placeholders with actual information.

---

## 1. Incident Detected — Initial Notification

Use when an incident is first confirmed. Send to the incident response channel and on-call contacts.

```
[SUMMARY]: [BRIEF SUMMARY OF INCIDENT]

Severity: [SEV1 / SEV2 / SEV3 / SEV4]
Status: DETECTED — Investigating
Affected Services: [service-a, service-b, service-c]
Impact Description: [Describe what is broken or degraded and who is affected]
Time Started: [YYYY-MM-DD HH:MM UTC]
Last Updated: [YYYY-MM-DD HH:MM UTC]
Timeline Doc: [link to timeline doc]

Actions Taken:
  - [action taken so far]
  - [action taken so far]

Next Steps:
  - [next step]
  - [next step]

Responder(s): [@oncall-person]
```

---

## 2. Investigating — Status Update

Use when the team is actively investigating but has not yet identified the root cause or started mitigation.

```
[SEVERITY UPDATE / NO CHANGE]

Severity: [SEV1 / SEV2 / SEV3 / SEV4]
Status: INVESTIGATING
Affected Services: [service-a, service-b, service-c]
Impact Description: [current description of impact]
Time Started: [YYYY-MM-DD HH:MM UTC]
Last Updated: [YYYY-MM-DD HH:MM UTC]
Timeline Doc: [link to timeline doc]

Current Findings:
  - [finding — e.g. elevated error rate on endpoint X]
  - [finding — e.g. correlation with deployment Y at Z time]

Actions Taken:
  - [action]
  - [action]

Next Steps:
  - [next step]
  - [next step]

ETA: [if known, otherwise "TBD — still investigating"]
```

---

## 3. Mitigation In Progress / Monitoring

Use when a fix or workaround is being deployed or monitored. Richer detail than the initial notification. Set `Status` to `MITIGATING` while the action is in progress and `MONITORING` while closure evidence or the stability window is still in progress.

```
[SUMMARY]: [BRIEF SUMMARY]

Severity: [SEV1 / SEV2 / SEV3 / SEV4]
Status: [MITIGATING / MONITORING]
Affected Services: [service-a, service-b, service-c]
Impact Description: [current state of impact — may be improving]
Time Started: [YYYY-MM-DD HH:MM UTC]
Last Updated: [YYYY-MM-DD HH:MM UTC]
Timeline Doc: [link to timeline doc]

Root Cause (if known): [root cause summary or "still under investigation"]

Mitigation Actions Taken:
  - [rollback of deployment X]
  - [scaled up resource Y]
  - [disabled feature flag Z]
  - [other action]

Expected Outcome: [what the mitigation should achieve — e.g. "error rate should drop below 1% within 5 min"]

Risks / Trade-offs: [any known side effects of mitigation]

Next Steps:
  - Monitor metrics for [N] minutes
  - Prepare roll-forward fix
  - Stakeholder comms at [time of next update]

Closure Evidence In Progress:
  - User-facing SLOs and critical journeys: [pending / observed evidence or link]
  - Dependency health: [pending / observed evidence or link]
  - Data/state correctness: [pending / observed evidence or link]
  - Secondary effects and backlog recovery: [pending / observed evidence or link]
  - Stability window: [defined window and progress]
  - Unverified boundary: [what remains untested]
  - Next check: [time or criterion]

ETA to Full Resolution: [time or "TBD"]
```

---

## 4. Resolved

Use only after the [R-01 closure evidence sequence](runbook-template.md#r-01-post-incident-steps) and the defined stability window pass. Healthy services, green dashboards, and a cleared alert are partial evidence, not a resolution verdict. If evidence is incomplete, use a `MONITORING` update instead and retain `MITIGATING` or `MONITORING` status.

```
[SUMMARY]: [INCIDENT RESOLVED]

Severity: [SEV1 / SEV2 / SEV3 / SEV4]
Status: RESOLVED
Affected Services: [service-a, service-b, service-c]
Impact Description: [summary of what happened and what was impacted]
Time Started: [YYYY-MM-DD HH:MM UTC]
Time Resolved: [YYYY-MM-DD HH:MM UTC]
Duration: [X hours Y minutes]
Timeline Doc: [link to timeline doc]

Root Cause: [root cause summary — e.g. "memory leak in service-b triggered by payload spike"]

Resolution Actions:
  - [action that fixed the issue]
  - [follow-up action taken]

Closure Evidence:
  - User-facing SLOs and critical journeys: [evidence or link]
  - Dependency health: [evidence or link]
  - Data/state correctness: [evidence or link]
  - Secondary effects and backlog recovery: [evidence or link]
  - Stability window: [defined window and observed result]
  - Evidence boundary: [what was actually exercised]
  - Independent human confirmation: [name/role and approval record or timestamp]

Post-Incident Items Opened:
  - [link to Jira/Asana bug or task]
  - [link to Jira/Asana bug or task]
```

---

## 5. Post-Incident Summary

Use after the post-mortem has been completed (usually 24-72 hours after resolution).

```
[INCIDENT SUMMARY]

Title: [incident title]
Severity: [SEV1 / SEV2 / SEV3 / SEV4]
Date: [YYYY-MM-DD]
Duration: [X hours Y minutes]

What Happened:
[3-5 sentence narrative of the incident timeline]

Business Impact:
  - [metric affected and value — e.g. "5 min of complete checkout downtime"]
  - [revenue / user / SLA impact]
  - [number of affected customers or requests]

Root Cause:
[1-2 sentence root cause description]

Trigger:
[what caused the incident to start — e.g. deployment, config change, external dependency failure]

Detection:
[How was it detected — monitoring alert, customer report, manual observation? Time to detect.]

Timeline (Key Events):
  - [HH:MM UTC] — [event]
  - [HH:MM UTC] — [event]
  - [HH:MM UTC] — [event]
  - [HH:MM UTC] — [event]

Action Items (from post-mortem):
| # | Action Item | Owner | Priority | Due Date | Tracking Link |
|---|-------------|-------|----------|----------|---------------|
| 1 | [action]    | [name]| [P0/P1]  | [date]   | [link]        |
| 2 | [action]    | [name]| [P0/P1]  | [date]   | [link]        |
| 3 | [action]    | [name]| [P0/P1]  | [date]   | [link]        |

Lessons Learned:
  - What went well: [observation]
  - What went wrong: [observation]
  - What to improve: [observation]

Timeline Doc: [link to timeline doc]
Post-Mortem Doc: [link to post-mortem doc]
```

---

## 6. Executive Summary Template

Use for C-level / VP communication during or immediately after an incident.

Do not report **RESOLVED** or **Full resolution** until the [R-01 closure evidence sequence](runbook-template.md#r-01-post-incident-steps) and the defined stability window pass. If evidence is incomplete, report **MONITORING** or **MITIGATING**, name the unverified boundary, and hand off or escalate.

```
[EXECUTIVE SUMMARY — INCIDENT]

Subject: [SEV X] Incident Summary — [service / area]

Brief Summary:
[2-3 sentence summary of what happened, in business terms]

Business Impact:
  - Duration of impact: [X hours Y minutes]
  - Customers affected: [number or percentage]
  - Revenue impact: [estimate or "none"]
  - SLA implications: [yes/no + details]
  - Reputation / trust considerations: [if any]

Timeline (Executive View):
  - [HH:MM UTC] — Incident detected
  - [HH:MM UTC] — Mitigation started
  - [HH:MM UTC] — Partial recovery
  - [HH:MM UTC] — Full resolution verified after closure evidence and stability window

Root Cause (if known): [1-2 sentence root cause, in business-readable language — e.g. "A recent configuration change caused our payment gateway to reject valid transactions."]

Current Status: [MONITORING / MITIGATING / RESOLVED]
Resolution Gate: [For RESOLVED, R-01 closure evidence, stability window, and human confirmation are complete; otherwise retain MONITORING or MITIGATING.]
Unverified Boundary: [none after full resolution, or what remains untested while MONITORING/MITIGATING]

Key Action Items:
  - [short-term fix deployed or planned]
  - [preventative measure being implemented]
  - [post-mortem scheduled for: date/time]

Communications:
  - Internal: [channel / who was notified]
  - External (if applicable): [customer-facing comms status]

Next Update: [time or "post-mortem report will follow within 48h"]

Point of Contact for Questions: [name / title / contact info]
```

---

## Quick Reference Card

| Phase               | When to Use                                       | Key Fields                                       |
|---------------------|---------------------------------------------------|--------------------------------------------------|
| DETECTED            | Incident first confirmed                          | Severity, affected services, impact, time started |
| INVESTIGATING       | Active diagnosis, no mitigation yet               | Findings, actions taken, ETA (or TBD)            |
| MITIGATING          | Deploying a fix or workaround                     | Mitigation actions, expected outcome, risks      |
| MONITORING          | Mitigation applied; closure evidence or stability window in progress | Evidence collected, unverified boundary, next check |
| RESOLVED            | R-01 closure evidence, stability window, and independent human confirmation complete | Root cause, resolution, closure evidence         |
| POST-INCIDENT       | After post-mortem (24-72h later)                  | Full timeline, action items, lessons learned     |
| EXECUTIVE SUMMARY   | C-level / VP comms during or right after incident | Business impact, SLA, revenue, clear language    |

# On-Call Rotation Schedule Template

| Metadata | Value |
|---|---|
| **Team** | [Team Name / SRE] |
| **Schedule Owner** | [SRE Manager / Rotation Coordinator] |
| **Version** | 1.0 |
| **Last Updated** | [YYYY-MM-DD] |

---

## 1. Calendar Pattern

### 1.1 Rotation Structure

The on-call rotation follows a [weekly / biweekly / daily] shift pattern with **three tiers**:

| Tier | Role | Coverage | Response Time SLA |
|---|---|---|---|
| **Primary** | First responder | [24/7 / Business hours / Follow-the-sun] | [X minutes] |
| **Secondary** | Escalation / Backup | Overlapping with Primary | [Y minutes] |
| **Escalation** | Engineering management / SMEs | 24/7 as needed | [Z minutes] |

### 1.2 Rotation Schedule

```
Rotation Group A → Rotation Group B → Rotation Group C → Rotation Group A ...
```

Each rotation group consists of:
- **[1]** Primary Engineer(s)
- **[1]** Secondary Engineer(s)
- **[1]** Escalation Contact (Manager / Senior Engineer)

| Group | Primary | Secondary | Escalation | Rotation Period |
|---|---|---|---|---|
| Group A | [Name] | [Name] | [Name] | [Start Date] – [End Date] |
| Group B | [Name] | [Name] | [Name] | [Start Date] – [End Date] |
| Group C | [Name] | [Name] | [Name] | [Start Date] – [End Date] |

### 1.3 Time Zone Coverage

- **Business Hours (Local):** [e.g., 09:00 – 17:00 UTC / 09:00 – 17:00 ET]
- **After-Hours:** [e.g., 17:00 – 09:00 UTC]
- **Weekend Coverage:** [e.g., Full 48-hour shift / Rotating weekend coverage]
- **Follow-the-Sun Handoff:** [Primary shifts between US/EU/APAC regions]

### 1.4 Calendar Management

- All rotation assignments are managed in [Calendar System, e.g., Google Calendar / PagerDuty / OpsGenie].
- Calendar invites are sent [7 / 14] days before the shift begins.
- Swaps must be reflected in the calendar and the scheduling tool.

---

## 2. Shift Handoff Protocol

### 2.1 Handoff Timing

| Type | Time | Duration |
|---|---|---|
| Daily handoff (business hours) | [Time, e.g., 16:00 local] | [15 min] |
| Weekly handoff (rotation change) | [Day and Time, e.g., Monday 10:00 UTC] | [30 min] |
| Exception / Escalation handoff | As needed | [15 min] |

### 2.2 Handoff Communication Checklist

The outgoing engineer **must** communicate the following during handoff:

- [ ] **Active incidents** — List all open incidents with status, severity, and current actions.
- [ ] **Resolved incidents** — Any incidents that occurred during the shift and their resolution.
- [ ] **Ongoing investigations** — Items being actively investigated that did not result in an incident ticket.
- [ ] **Pending alerts** — Any alert conditions that are active but not yet triaged.
- [ ] **Recent changes** — Deployments, config changes, or infrastructure changes during the shift.
- [ ] **Known issues** — Non-urgent issues, tech debt, or known behavior that may cause future alerts.
- [ ] **Maintenance windows** — Any planned maintenance during the incoming shift.
- [ ] **Documentation updates** — Any new runbooks, playbooks, or wiki pages created or updated.
- [ ] **Tools / access issues** — Any problems with monitoring, alerting, or access that need attention.
- [ ] **General notes** — Anything else the incoming engineer should know.

### 2.3 Handoff Meeting Format

1. Outgoing engineer reviews the handoff log.
2. Walk through open incidents and alerts on the [Monitoring Dashboard].
3. Review any changes deployed or pending.
4. Transfer ownership of any communication threads.
5. Update the on-call status in [Status System, e.g., Slack status / PagerDuty].
6. Confirm that the [On-Call Phone / Pager] has been transferred.
7. Outgoing engineer remains available for [30 minutes] after handoff for questions.

### 2.4 Handoff Log

Maintain a handoff log at [Link to Handoff Document or Wiki Page] that captures:

```
Date: [YYYY-MM-DD]
Outgoing Engineer: [Name]
Incoming Engineer: [Name]
Active Incidents:
  - INC-######: [Description, Severity, Status]
Resolved Incidents:
  - INC-######: [Description, Resolution]
Pending Alerts: [List]
Changes Deployed: [List]
Known Issues: [List]
Action Items: [List]
Handoff Verified By: [Name]
```

---

## 3. Escalation Chain

### 3.1 Standard Escalation Path

```
  Primary On-Call (Tier 1)
      │ Response within [X] minutes
      ↓
  Secondary On-Call (Tier 2)
      │ Response within [Y] minutes
      ↓
  Engineering Manager / SRE Lead (Tier 3)
      │ Response within [Z] minutes
      ↓
  Director of Engineering / VP (Tier 4)
      │ Response within [W] minutes
      ↓
  Incident Commander / Executive Team (Tier 5)
```

### 3.2 Escalation Contacts

| Tier | Role | Contact Name | Phone | Email / Slack |
|---|---|---|---|---|
| Tier 1 | Primary On-Call | [Name / Role] | [+1-555-...] | [@slack-handle] |
| Tier 2 | Secondary On-Call | [Name / Role] | [+1-555-...] | [@slack-handle] |
| Tier 3 | SRE Manager | [Name] | [+1-555-...] | [email / @slack] |
| Tier 4 | Engineering Director | [Name] | [+1-555-...] | [email / @slack] |
| Tier 5 | VP of Engineering | [Name] | [+1-555-...] | [email / @slack] |

### 3.3 Non-Response Escalation

If the Primary On-Call does not acknowledge an alert within [X] minutes:
1. Alert re-fires to Secondary On-Call.
2. If Secondary does not acknowledge within [Y] minutes, alert goes to Tier 3.
3. Each subsequent tier follows the same escalation timer.
4. If all tiers are unresponsive for [Z] total minutes, the [Major Incident Protocol] is triggered.

---

## 4. On-Call Responsibilities

### 4.1 During the Shift

- [ ] Respond to all alerts within the defined SLA.
- [ ] Triage and classify incidents (severity, impact, urgency).
- [ ] Mitigate or resolve incidents per established runbooks.
- [ ] Log all actions in the incident management system.
- [ ] Maintain the handoff log.
- [ ] Update on-call status (Slack status, PagerDuty, etc.).
- [ ] Perform proactive monitoring checks at least every [N] hours.
- [ ] Attend daily standup / handoff meetings.
- [ ] Escalate when appropriate — do not hesitate.

### 4.2 Not During the Shift

- [ ] No deployment or change duties (unless explicitly coordinating).
- [ ] No major project work — focus on incident readiness.
- [ ] Respond to critical pages within [X] minutes, even during off-hours.
- [ ] Maintain situational awareness of ongoing incidents and changes.

### 4.3 Post-Shift Responsibilities

- [ ] Complete the handoff for the incoming engineer.
- [ ] Follow up on any incomplete investigations or incident tasks.
- [ ] Ensure incident tickets are properly documented and closed.
- [ ] Submit any runbook or documentation updates discovered during the shift.
- [ ] Participate in postmortem reviews for incidents that occurred during the shift.

---

## 5. Incident Response SLAs

| Severity | Description | Acknowledge Time | Response Time | Update Frequency | Resolution Target |
|---|---|---|---|---|---|
| **P0 — Critical** | Service down, data loss, security breach | [2 min] | [5 min] | Every [15 min] | [2 hours] |
| **P1 — Major** | Feature impairment, degraded performance | [5 min] | [15 min] | Every [30 min] | [4 hours] |
| **P2 — Minor** | Partial impairment, cosmetic issues | [15 min] | [1 hour] | Every [2 hours] | [24 hours] |
| **P3 — Low** | Non-critical, informational | [30 min] | [4 hours] | Daily | [72 hours] |
| **P4 — Trivial** | Questions, feature requests | [1 hour] | [8 hours] | Per shift | [Next sprint] |

### SLA Definitions

- **Acknowledge Time**: Time from alert firing to engineer acknowledging the page.
- **Response Time**: Time from acknowledgment to engineer beginning work on the issue.
- **Update Frequency**: How often status updates must be posted to the incident ticket.
- **Resolution Target**: Target time to mitigate or resolve (may not include root-cause fix).

---

## 6. Communication During Rotation

### 6.1 Channels

| Purpose | Channel | Participants |
|---|---|---|
| Alert notifications | [#oncall-alerts] | On-call engineers + all SRE |
| Incident coordination | [#oncall-incidents] | On-call engineers + involved teams |
| Status updates | [#sre-standup] | SRE team |
| Escalation paging | [PagerDuty / OpsGenie] | On-call chain |
| Emergency communication | Phone / SMS | On-call chain |

### 6.2 Status Reporting

- **Shift Start**: Post in [#oncall-status]: "Primary: [Name], Secondary: [Name], Start: [time]"
- **Active Incidents**: Post severity, impact, and current status in [#oncall-incidents].
- **Shift End**: Post summary of incidents handled and handoff status.
- **Escalation Events**: Notify manager immediately when escalation path is triggered.

### 6.3 Silence Periods

- [ ] Do not page the on-call engineer for non-urgent communications (e.g., bug reports, feature questions, meeting invites) — use [async channel, e.g., #sre-questions].
- [ ] Do not expect responses during documented out-of-office hours unless a critical page is sent.
- [ ] Schedule non-urgent maintenance windows during [designated low-traffic period].

---

## 7. Tools and Access Needed

### 7.1 Required Tools

| Tool | Purpose | Access URL | Account Type |
|---|---|---|---|
| [Monitoring System, e.g., Datadog / Grafana] | Observability and alerting | [URL] | [Role-based] |
| [Incident Management, e.g., PagerDuty / OpsGenie] | Alert routing and paging | [URL] | [Engineer account] |
| [Logging System, e.g., Splunk / ELK] | Log analysis | [URL] | [Read-write] |
| [Ticketing System, e.g., Jira / ServiceNow] | Incident tracking | [URL] | [Agent access] |
| [Runbook / Documentation] | Runbooks and playbooks | [URL] | [Read-write] |
| [Chat / Comms, e.g., Slack] | Communication | [URL] | [Standard account] |
| [Infrastructure Dashboard] | System health overview | [URL] | [Read-only] |
| [Deployment System, e.g., Spinnaker / ArgoCD] | Deployment visibility | [URL] | [Read-only] |
| [Cloud Console, e.g., AWS / GCP / Azure] | Infrastructure management | [URL] | [Read-only / Limited write] |

### 7.2 Required Access

- [ ] [Monitoring System] — Alert configuration and dashboard access.
- [ ] [Incident Management] — Ability to acknowledge, resolve, and re-route incidents.
- [ ] [Logging System] — Full log search and export capabilities.
- [ ] [Ticketing System] — Create, update, and close incident tickets.
- [ ] [Chat / Comms] — All relevant Slack channels.
- [ ] [SSH / Bastion] — Access to production instances (read-only or limited write as defined).
- [ ] [Kubernetes / Orchestration] — `kubectl` access with read-only / troubleshooting permissions.
- [ ] [Database Query Tool] — Read-only query access for investigation.
- [ ] [Runbook / Documentation Wiki] — Edit access to update runbooks.

### 7.3 Hardware Requirements

- [ ] Company laptop with VPN access.
- [ ] Stable internet connection (backup recommended).
- [ ] [On-Call Phone / Device] — if applicable.
- [ ] Access to [Corporate SSO / 2FA].

---

## 8. Pre-Flight Checklist for Joining Rotation

> Complete this checklist at least [2 days] before your on-call shift begins.

### 8.1 Communication Setup

- [ ] Added to all on-call Slack channels: [#channel1], [#channel2].
- [ ] Slack status configured to show "On-Call" during the shift.
- [ ] Phone number verified in [PagerDuty / OpsGenie].
- [ ] Notification preferences configured (push, SMS, phone call).
- [ ] Scheduled handoff call with outgoing engineer.
- [ ] Out-of-office / reduced availability communicated to team.

### 8.2 Tooling & Access Verification

- [ ] Log in to [Monitoring System] and verify access to all relevant dashboards.
- [ ] Log in to [Incident Management] and verify ability to acknowledge alerts.
- [ ] Log in to [Logging System] and verify search capabilities.
- [ ] Log in to [Cloud Console] and verify access.
- [ ] Test [VPN / Bastion] access to production environment.
- [ ] Verify [Kubernetes / kubectl] access with a test command.
- [ ] Verify [SSH key] is loaded and working.
- [ ] Verify [Database query tool] access with a read-only test query.
- [ ] Test [Chat / Comms] channels are unmuted and notifications enabled.
- [ ] Confirm [Phone / SMS] delivery works for the on-call number.

### 8.3 Documentation Review

- [ ] Read the following runbooks:
  - [Incident Response Runbook]
  - [Service Restart / Recovery Runbook]
  - [Database Failover Runbook]
  - [Deployment Rollback Runbook]
- [ ] Review recent incident postmortems (last [30] days).
- [ ] Review current known issues and active incident tickets.
- [ ] Familiarize yourself with service architecture diagram at [Link].
- [ ] Review the Error Budget Policy at [Link].
- [ ] Confirm location of escalation contacts list.

### 8.4 Knowledge Transfer

- [ ] Attend handoff meeting with outgoing engineer.
- [ ] Review handoff log from prior shifts (last [7] days).
- [ ] Get walkthrough of any active incidents or pending investigations.
- [ ] Understand current release / deployment status.
- [ ] Identify any known upcoming maintenance windows.
- [ ] Review recently deployed changes (past [72] hours).

### 8.5 Practical Drills (Optional but Recommended)

- [ ] Walk through a simulated P0 incident using [Game Day / Chaos Engineering] platform.
- [ ] Practice using the [War Room] process.
- [ ] Verify ability to escalate to all tiers in the escalation chain.
- [ ] Time yourself on acknowledging a test alert.

### 8.6 Final Confirmation

> **I confirm that I have completed the pre-flight checklist and am ready to assume on-call responsibilities.**

Signature: _________________________________
Date: _______________________________________

---

## 9. Appendix

### A. Rotation Calendar Template

```
Week [N] ([Start Date] – [End Date]):
  Primary:   [Name] ([Time Zone])
  Secondary: [Name] ([Time Zone])
  Escalation: [Name]

Week [N+1] ([Start Date] – [End Date]):
  Primary:   [Name] ([Time Zone])
  Secondary: [Name] ([Time Zone])
  Escalation: [Name]
```

### B. Related Documents

- [Incident Response Playbook]
- [Error Budget Policy]
- [Postmortem Template]
- [Runbook Index]
- [Service Architecture Diagram]
- [Escalation Contacts List]

---
title: "Operational Overload Recovery: [Team / Service]"
doc_id: OOR-[TEAM]-[YYYYMMDD]
status: active | recovering | stable | escalated
opened: [YYYY-MM-DD]
owner: "[Manager / Team Lead]"
---

# Operational Overload Recovery — [Team / Service]

## Trigger and evidence

- **Overload threshold:** [e.g., operational work > 50% for two consecutive weeks]
- **Observed period:** [ ]
- **Evidence:** [pages, tickets, incidents, interruptions, toil sample]
- **User or service risk:** [ ]
- **What is unknown:** [ ]

## Immediate protection

- [ ] Declare the overload state to stakeholders.
- [ ] Protect [X]% of engineering time for load reduction.
- [ ] Freeze or renegotiate nonessential commitments.
- [ ] Route, suppress, or retire non-actionable alerts.
- [ ] Set an escalation path for unsafe fatigue or unstaffed coverage.
- [ ] Identify the service scope that will be reduced if capacity is exceeded.

## Load inventory

| Work source | Count / hours | User impact | Repetition / trigger | Proposed action | Owner |
|---|---:|---|---|---|---|
| Pages | [ ] | [ ] | [ ] | [tune / automate / remove] | [ ] |
| Manual changes | [ ] | [ ] | [ ] | [ ] | [ ] |
| Support / tickets | [ ] | [ ] | [ ] | [ ] | [ ] |
| Incident follow-up | [ ] | [ ] | [ ] | [ ] | [ ] |
| Other interruptions | [ ] | [ ] | [ ] | [ ] | [ ] |

## Recovery queue

Prioritize the smallest interventions that remove recurring load or reduce blast radius.

| Item | Failure or toil condition | Expected reduction | Verification criterion | Owner | Due |
|---|---|---:|---|---|---|
| [ ] | [ ] | [ ] | [ ] | [ ] | [ ] |

## Scope and authority decisions

- **Work deferred:** [ ]
- **Work stopped:** [ ]
- **Service scope renegotiated:** [ ]
- **Staffing or leadership escalation:** [ ]
- **Temporary exception and expiry:** [ ]

## Weekly review

| Week | Operational share | Pages / engineer | Engineering hours protected | Reliability / user signal | Decision |
|---|---:|---:|---:|---|---|
| [ ] | [ ] | [ ] | [ ] | [ ] | [ ] |

## Exit criteria

- [ ] Operational work is below the stated threshold for [N] review periods.
- [ ] The top recurring load sources have owners and verified reductions.
- [ ] On-call coverage is sustainable without unsafe fatigue or coercion.
- [ ] Deferred work and temporary exceptions have been reconciled.
- [ ] The service's SLO, alerts, runbooks, and escalation paths are current.
- [ ] A follow-up review is scheduled to detect relapse.

## Closeout

- **Closed by:** [ ]
- **Date:** [ ]
- **What changed:** [ ]
- **What remains risky:** [ ]

# Postmortem: [Incident Title]

> **Blameless Culture Reminder:** This postmortem is a blameless analysis of what happened, why it happened, and how we prevent it from happening again. The goal is to learn and improve our systems and processes — not to assign blame or punishment. All participants are expected to contribute honestly and constructively.

---

## Header Metadata

| Field          | Value |
|----------------|-------|
| **Date**       | [YYYY-MM-DD] |
| **Title**      | [Short, descriptive title of the incident] |
| **Severity**   | [SEV1 / SEV2 / SEV3 / SEV4] |
| **Duration**   | [Start time] – [End time] ([Total duration in minutes/hours]) |
| **Date(s)**    | [Date range of the incident] |
| **Reported by**| [Name / Team] |
| **Participants** | [Name / Team], [Name / Team], [Name / Team] |

### Severity Definitions

| Severity | Description |
|----------|-------------|
| **SEV1** | Complete service outage or critical data loss affecting all users |
| **SEV2** | Major feature degradation or partial outage affecting a significant subset of users |
| **SEV3** | Minor degradation, non-critical feature impact, or single-user issue |
| **SEV4** | Cosmetic issue, internal tooling problem, or question/inquiry |

---

## Incident Summary

[1-2 paragraphs providing a high-level overview of the incident. Describe what happened, which systems were involved, the overall impact on users and business, and how the incident was ultimately resolved. This section should be understandable by someone outside the immediate team — executives, customer support, other engineering teams.]

**Example:** "On [DATE] between [START TIME] and [END TIME] UTC, the [SERVICE NAME] experienced a [DESCRIPTION OF FAILURE]. This caused [IMPACT] for [NUMBER] users. Root cause was [BRIEF ROOT CAUSE]. The incident was resolved by [RESOLUTION ACTION]. Total time to resolve was [DURATION]."

---

## Timeline

All times in [UTC / LOCAL TZ].

| Timestamp (UTC) | Event | Who |
|-----------------|-------|-----|
| [YYYY-MM-DD HH:MM] | [Incident begins — first failure symptom observed] | [System / Person] |
| [YYYY-MM-DD HH:MM] | [Alert triggered / page sent] | [Monitoring system] |
| [YYYY-MM-DD HH:MM] | [First responder acknowledged] | [Name] |
| [YYYY-MM-DD HH:MM] | [Initial investigation — what was checked] | [Name] |
| [YYYY-MM-DD HH:MM] | [Escalation to additional team members] | [Name] |
| [YYYY-MM-DD HH:MM] | [Root cause identified] | [Name] |
| [YYYY-MM-DD HH:MM] | [Mitigation action taken] | [Name] |
| [YYYY-MM-DD HH:MM] | [Service restored / incident resolved] | [Name] |
| [YYYY-MM-DD HH:MM] | [Monitoring confirmed healthy / all-clear] | [System / Person] |
| [YYYY-MM-DD HH:MM] | [Postmortem meeting scheduled] | [Name] |

### Key Duration Metrics

| Metric | Duration |
|--------|----------|
| Time to detection (TTD) | [MM minutes] |
| Time to response (TTR) | [MM minutes] |
| Time to mitigation (TTM) | [MM minutes] |
| Time to resolution (TTR) | [MM minutes] |
| Total incident duration | [HH:MM] |

---

## Impact

### Affected Services

- [Service name] — [Description of how it was affected]
- [Service name] — [Description of how it was affected]
- [Service name] — [Description of how it was affected]

### User Impact

- **[Number]** users were affected
- **[Number]** requests failed / timed out ( **[X]%** error rate )
- **[Number]** support tickets filed related to this incident
- **[Description of user-facing symptoms]**

### Business Impact

- **[Amount]** in estimated revenue loss
- **[Number]** failed transactions / orders
- **[Description of SLA/SLO breach, if applicable]**
- **[Other business metrics affected]**

### Metrics at Time of Incident

| Metric | Baseline | During Incident | Post-Recovery |
|--------|----------|-----------------|---------------|
| Error rate | [X]% | [X]% | [X]% |
| Latency p50 | [X]ms | [X]ms | [X]ms |
| Latency p99 | [X]ms | [X]ms | [X]ms |
| CPU utilization | [X]% | [X]% | [X]% |
| Memory usage | [X]% | [X]% | [X]% |
| [Other metric] | [Value] | [Value] | [Value] |

---

## Root Cause

### Summary

[1-2 sentences describing the root cause at a high level.]

### 5 Whys Analysis

| Why? | Answer |
|------|--------|
| **1.** Why did [symptom] happen? | [Answer] |
| **2.** Why did [cause from #1] happen? | [Answer] |
| **3.** Why did [cause from #2] happen? | [Answer] |
| **4.** Why did [cause from #3] happen? | [Answer] |
| **5.** Why did [cause from #4] happen? | **[Root cause — the fundamental system or process issue]** |

### Root Cause Diagram

```
[SYMPTOM]
    |
    v
[Why #1]
    |
    v
[Why #2]
    |
    v
[Why #3]
    |
    v
[Why #4]
    |
    v
[ROOT CAUSE]
```

---

## Contributing Factors

List factors that contributed to the incident's severity, duration, or impact beyond the root cause.

| Factor | Description | Category |
|--------|-------------|----------|
| [Factor 1] | [Description of how this factor contributed] | [Process / People / Technology / External] |
| [Factor 2] | [Description of how this factor contributed] | [Process / People / Technology / External] |
| [Factor 3] | [Description of how this factor contributed] | [Process / People / Technology / External] |

---

## Detection

- **How was the incident first detected?** [Alert / Customer report / Manual observation / Scheduled health check / Other]
- **Time to detection:** [Time between first symptom and first alert/notification]
- **Detection mechanism:** [Describe the monitoring, alert, or human process that surfaced the issue]
- **Did existing monitoring cover the failure mode?** [Yes / No / Partially]
- **If no, what monitoring gap existed?** [Description of gap]
- **Was there a faster way this could have been detected?** [Yes / No — explain]

---

## Response

- **Time to respond:** [Time between first alert and first person taking action]
- **Who responded?** [Names / Teams]
- **What worked well during the response?**
  - [Thing that worked well and why]
  - [Thing that worked well and why]
- **What didn't work well during the response?**
  - [Thing that didn't work well and why]
  - [Thing that didn't work well and why]
- **Were runbooks available and accurate?** [Yes / No — describe]
- **Were the right people reachable?** [Yes / No — describe]
- **Communication channels used:** [Slack / PagerDuty / Zoom / Email / Other]
- **Communication effectiveness:** [Rating 1-5 with comments on clarity, speed, audience]

### Response Timeline Gaps

| Gap | Description | Improvement |
|-----|-------------|-------------|
| [Gap 1] | [What went wrong] | [How to fix] |
| [Gap 2] | [What went wrong] | [How to fix] |

---

## Action Items

Action items derived from this postmortem. Each item has a reference ID that can be linked from epics or tickets.

| ID | Description | Owner | Deadline | Type | Epic/Ticket |
|----|-------------|-------|----------|------|-------------|
| ACT-[001] | [Detailed description of the action to take] | [Name] | [YYYY-MM-DD] | [Prevent / Detect / Mitigate / Process] | [LINK-001] |
| ACT-[002] | [Detailed description of the action to take] | [Name] | [YYYY-MM-DD] | [Prevent / Detect / Mitigate / Process] | [LINK-002] |
| ACT-[003] | [Detailed description of the action to take] | [Name] | [YYYY-MM-DD] | [Prevent / Detect / Mitigate / Process] | [LINK-003] |
| ACT-[004] | [Detailed description of the action to take] | [Name] | [YYYY-MM-DD] | [Prevent / Detect / Mitigate / Process] | [LINK-004] |
| ACT-[005] | [Detailed description of the action to take] | [Name] | [YYYY-MM-DD] | [Prevent / Detect / Mitigate / Process] | [LINK-005] |

### Action Type Definitions

| Type | Description |
|------|-------------|
| **Prevent** | Changes that prevent the incident from recurring |
| **Detect** | Improvements to monitoring, alerting, and observability |
| **Mitigate** | Changes that reduce blast radius or speed recovery if it happens again |
| **Process** | Changes to runbooks, documentation, communication, or training |

### Epic / Ticket References

**LINK-001:** [Epic/Ticket system and ID, e.g., JIRA SRE-1234] — [Brief title]
**LINK-002:** [Epic/Ticket system and ID, e.g., JIRA SRE-1235] — [Brief title]
**LINK-003:** [Epic/Ticket system and ID, e.g., JIRA SRE-1236] — [Brief title]
**LINK-004:** [Epic/Ticket system and ID, e.g., JIRA SRE-1237] — [Brief title]
**LINK-005:** [Epic/Ticket system and ID, e.g., JIRA SRE-1238] — [Brief title]

---

## Lessons Learned

### What Went Well

- [Specific positive observation]
- [Specific positive observation]
- [Specific positive observation]

### What Went Wrong

- [Specific negative observation]
- [Specific negative observation]
- [Specific negative observation]

### What We Were Lucky About

- [Factor outside our control that worked in our favor]
- [Near-miss that could have made things worse]

### Surprises

- [Unexpected behavior or outcome encountered during the incident]
- [Something that was thought to be in place but was not]

---

## Follow-Up Plan

### Immediate Actions (Next 7 Days)

- [ ] [Action item] — Owner: [Name]
- [ ] [Action item] — Owner: [Name]

### Short-Term Actions (Next 30 Days)

- [ ] [Action item] — Owner: [Name]
- [ ] [Action item] — Owner: [Name]

### Long-Term Actions (Next 90 Days)

- [ ] [Action item] — Owner: [Name]
- [ ] [Action item] — Owner: [Name]

### Review Schedule

| Review | Date | Participants |
|--------|------|-------------|
| Action item check-in #1 | [YYYY-MM-DD] | [Names] |
| Action item check-in #2 | [YYYY-MM-DD] | [Names] |
| Follow-up postmortem review | [YYYY-MM-DD] | [Names] |

---

## Appendix

### Supporting Data

- [Link to dashboard / Grafana / Datadog screenshots]
- [Link to relevant logs (Splunk / ELK / CloudWatch)]
- [Link to chat transcript / incident Slack channel]
- [Link to incident ticket / PagerDuty timeline]
- [Link to zoom recording / meeting notes]

### Related Postmortems

- [Link to related postmortem #1]
- [Link to related postmortem #2]

### Changes Since Last Review

[List any relevant changes made to systems/processes since the last postmortem review that are pertinent to this incident.]

---

## Sign-Off

| Role | Name | Date |
|------|------|------|
| Incident Commander | [Name] | [YYYY-MM-DD] |
| Technical Lead | [Name] | [YYYY-MM-DD] |
| SRE Lead | [Name] | [YYYY-MM-DD] |
| Engineering Manager | [Name] | [YYYY-MM-DD] |
| Product Manager (if applicable) | [Name] | [YYYY-MM-DD] |

---

*This postmortem was created on [YYYY-MM-DD] and last updated on [YYYY-MM-DD]. Template version 1.0.*

---
title: "Reliability Design Review: [System or Change]"
doc_id: RDR-[SERVICE]-[VERSION]
status: draft | reviewed | approved | rejected
created: [YYYY-MM-DD]
owner: "[Name / Team]"
reviewers: "[Names / Teams]"
---

# Reliability Design Review — [System or Change]

## Decision

- **Decision:** [approve / approve with conditions / defer / reject]
- **Decision owner:** [Name / role]
- **Review date:** [YYYY-MM-DD]
- **Next review trigger:** [date, SLO breach, traffic threshold, architecture change]

## 1. User and business context

- **Critical user journeys:** [What users need to accomplish]
- **User impact if this fails:** [availability, latency, correctness, freshness, privacy, data loss]
- **Existing or proposed SLOs:** [Link or concise statement]
- **Error-budget trade-off:** [What budget is consumed or protected?]

## 2. System and dependency boundaries

- **Request path:** [Describe or link diagram]
- **Async/data paths:** [Queues, pipelines, batch jobs, replay]
- **Dependencies:** [Service, owner, failure mode, fallback]
- **Shared resources:** [Databases, quotas, clusters, control planes]
- **Failure domains:** [What can fail together?]

## 3. Concrete assumptions

| Dimension | Normal | Expected peak | Worst credible case | Evidence / owner |
|---|---:|---:|---:|---|
| Requests or events per second | [ ] | [ ] | [ ] | [ ] |
| Payload or storage growth | [ ] | [ ] | [ ] | [ ] |
| Latency budget | [ ] | [ ] | [ ] | [ ] |
| Concurrent work / queue depth | [ ] | [ ] | [ ] | [ ] |
| Recovery point / recovery time | [ ] | [ ] | [ ] | [ ] |

## 4. Failure and degraded modes

| Failure or saturation condition | User-visible effect | Detection | Mitigation / shed / defer | Recovery verification | Owner |
|---|---|---|---|---|---|
| [ ] | [ ] | [ ] | [ ] | [ ] | [ ] |

Include retry amplification, timeout behavior, dependency failure, quota exhaustion, bad configuration, partial rollout, data corruption, and loss of observability where relevant.

## 5. Change and rollout safety

- **Change units:** [small, reversible steps]
- **Canary population and control:** [ ]
- **Observation window and sample size:** [ ]
- **Success metrics:** [SLO, user journey, saturation, dependency]
- **Abort thresholds and authority:** [ ]
- **Rollback or roll-forward:** [ ]
- **Configuration validation and audit path:** [ ]

## 6. Data, privacy, and security reliability

- **Correctness and completeness checks:** [ ]
- **Durability, backup, restore, and replay evidence:** [ ]
- **Privacy or security failure modes:** [ ]
- **Access and break-glass controls:** [ ]

## 7. Operations and human work

- **Dashboards and alerts:** [Links; each page has an action]
- **Runbook:** [Link]
- **On-call readiness:** [Access, training, escalation, handoff]
- **Expected toil:** [Estimate and reduction plan]
- **Cognitive-load risks:** [Ambiguous signals, hidden state, complex procedures]

## 8. Verification plan

- [ ] Unit and integration behavior verified
- [ ] Load or capacity assumptions tested
- [ ] Failure and degraded modes exercised
- [ ] Canary and rollback exercised
- [ ] Restore or replay verified where applicable
- [ ] User-visible SLO and telemetry verified in the live boundary

## 9. Open risks and conditions

| Risk / unknown | Evidence needed | Owner | Due / trigger | Decision if unresolved |
|---|---|---|---|---|
| [ ] | [ ] | [ ] | [ ] | [ ] |

## 10. Approval record

- **Service owner:** [Name / date]
- **Reliability reviewer:** [Name / date]
- **Security / privacy reviewer (if applicable):** [Name / date]
- **Product or business owner:** [Name / date]

---
title: "Reliability Ownership Charter: [Service]"
doc_id: ROC-[SERVICE]-[VERSION]
status: draft | active | superseded
created: [YYYY-MM-DD]
owner: "[Service owner]"
review_date: [YYYY-MM-DD]
---

# Reliability Ownership Charter — [Service]

## Service and user journeys

- **Service:** [ ]
- **Critical user journeys:** [ ]
- **SLOs and dashboards:** [ ]
- **Supported scope:** [ ]
- **Explicitly unsupported scope:** [ ]

## Ownership boundaries

| Responsibility | Accountable owner | Contributing teams | Escalation path |
|---|---|---|---|
| Product and user priority | [ ] | [ ] | [ ] |
| Application code and deploy | [ ] | [ ] | [ ] |
| Runtime and infrastructure | [ ] | [ ] | [ ] |
| Monitoring and alert quality | [ ] | [ ] | [ ] |
| Runbook and training | [ ] | [ ] | [ ] |
| Dependency relationship | [ ] | [ ] | [ ] |
| Backup, restore, and recovery | [ ] | [ ] | [ ] |
| Incident command and communications | [ ] | [ ] | [ ] |

## Pager and maintenance model

- **Primary and secondary rotation:** [ ]
- **Who must page and who may be consulted:** [ ]
- **Expected maintenance / engineering allocation:** [ ]
- **Escalation and handoff rules:** [ ]
- **Conditions that make the rotation unsafe:** [ ]

## Engagement agreement

- **Engagement mode:** [service-owned / embedded / platform enablement / working group / hybrid]
- **Reliability goals for this period:** [ ]
- **Work the enabling team will do:** [ ]
- **Work the service team must own:** [ ]
- **Exit or transition criteria:** [ ]
- **What happens if the service cannot operate within agreed SLO and toil limits:** [ ]

## Evidence and review

- **Baseline incident and toil evidence:** [ ]
- **Success measures:** [ ]
- **Review cadence:** [ ]
- **Open risks and owners:** [ ]

## Sign-off

- Service owner: [name / date]
- Reliability or enabling owner: [name / date]
- Product owner: [name / date]
- Platform / security / privacy owners as applicable: [name / date]

# Runbook: [Service Name]

> **Version:** [1.0.0]
> **Last Updated:** [YYYY-MM-DD]
> **Owner:** [Team Name / Individual]
> **Review Cadence:** [Quarterly / Bi-annual / Annual]

---

## Table of Contents

1. [Service Overview](#service-overview)
2. [SLOs & Error Budget](#slos--error-budget)
3. [On-Call Quick Reference](#on-call-quick-reference)
4. [Monitoring & Alerting](#monitoring--alerting)
5. [Common Failure Modes](#common-failure-modes)
6. [Detailed Troubleshooting Procedures](#detailed-troubleshooting-procedures)
7. [Escalation Paths](#escalation-paths)
8. [Recovery Procedures](#recovery-procedures)

---

## Service Overview

### Description

[Briefly describe what this service does, its purpose, and the critical function it serves in the broader system architecture.]

### Architecture

[High-level description of the service architecture — key components, dependencies, data flow, upstream/downstream services.]

### Owner

| Field | Value |
|-------|-------|
| **Engineering Team** | [Team Name] |
| **Team Channel** | [#slack-channel] |
| **Primary DRI** | [Name / Role] |
| **Service Catalog URL** | [https://link-to-service-catalog] |
| **Code Repository** | [https://github.com/org/repo] |

### Links

| Resource | URL |
|----------|-----|
| **Grafana Dashboard** | [https://grafana.example.com/d/...] |
| **Datadog Dashboard** | [https://app.datadoghq.com/dashboard/...] |
| **CloudWatch Dashboard** | [https://console.aws.amazon.com/cloudwatch/...] |
| **Kibana / Logs** | [https://kibana.example.com/app/discover#...] |
| **Jaeger / Tempo Traces** | [https://tracing.example.com/...] |
| **PagerDuty Schedule** | [https://pagerduty.com/schedules/...] |
| **Status Page** | [https://status.example.com/] |
| **Runbook (this doc)** | [link] |

### Dependencies

| Dependency | Criticality | Notes |
|------------|-------------|-------|
| [Database e.g. PostgreSQL] | Critical | [connection details, failover info] |
| [Cache e.g. Redis] | High | [cluster info, eviction policy] |
| [Queue e.g. Kafka] | High | [topic names, consumer groups] |
| [External API] | Medium | [rate limits, quota info] |
| [Auth provider] | Critical | [token expiry, rotation schedule] |

### Key Metrics

| Metric | Target | Description |
|--------|--------|-------------|
| `[p99_latency_ms]` | < [500ms] | End-to-end request latency |
| `[requests_per_second]` | [N] | Throughput |
| `[error_rate]` | < [0.1%] | Ratio of 5xx responses |
| `[cpu_utilization]` | < [80%] | Instance CPU usage |
| `[memory_utilization]` | < [80%] | Instance memory usage |

---

## SLOs & Error Budget

### Service Level Objectives

| SLO | Target | Window | Measurement Method |
|-----|--------|--------|--------------------|
| **Latency** | [99% of requests < 500ms] | [28 days] | [Histogram buckets / Prometheus] |
| **Availability** | [99.9%] | [28 days] | [Ratio of successful requests] |
| **Throughput** | [Handle N req/s] | [1 hour] | [Max RPS measured] |
| **Freshness** | [Data < 5min old] | [1 hour] | [Lag monitoring] |

### Error Budget

| Period | Budget | Remaining | Burn Rate |
|--------|--------|-----------|-----------|
| **28 days** | [0.1% = 43m 12s] | [XX.X% remaining] | [Alert if > 2x target] |

### Burn Rate Alerts

| Severity | Burn Rate | Duration | Action |
|----------|-----------|----------|--------|
| **Warning** | [2x] | [1 hour] | Investigate |
| **Critical** | [10x] | [6 minutes] | Page on-call |
| **Critical** | [2x] | [6 hours] | Page on-call |

---

## On-Call Quick Reference

> The commands below include read-only and mutating examples. Before any command that mutates production, apply the operational closure gate: verify current human authorization for the specific action and scope, record the target, affected population, maximum blast radius, success and abort/rollback criteria, rollback path, and stopping authority. If the gate cannot be satisfied, stop and hand off or escalate.

### How to Access

```bash
# SSH to production instances
ssh [user]@[bastion-host]
ssh [instance-name].[region].internal

# Kubernetes access
kubectl config use-context [cluster-name]
kubectl get pods -n [namespace]

# Database access
psql -h [host] -U [user] -d [database]
```

### How to Restart

```bash
# Restart application service
sudo systemctl restart [service-name]

# Roll restart Kubernetes deployment
kubectl rollout restart deployment/[deployment-name] -n [namespace]
kubectl rollout status deployment/[deployment-name] -n [namespace]

# Safe restart with traffic drain
./scripts/safe-restart.sh [service-name]
```

### Common Commands

```bash
# Check service health
curl -s http://localhost:[port]/health | jq .

# View recent logs
journalctl -u [service-name] --since "1 hour ago" -n 100

# Check current version
[service-name] --version
curl -s http://localhost:[port]/version | jq .

# Check active connections
netstat -anp | grep [port] | wc -l

# Check disk space
df -h /data
```

### Common Issues at a Glance

| Symptom | Try First |
|---------|-----------|
| Service returning 5xx | Check `/health` endpoint, restart service |
| High latency on p99 | Check CPU/memory, database query times |
| Alerts firing after deploy | Rollback to last known good version |
| Database connection errors | Check connection pool, restart app |
| Out of memory | Increase resources, rollback recent change |
| TLS / certificate errors | Check cert expiry, restart with reload |

---

## Monitoring & Alerting

### Key Dashboards

1. **[Service Overview Dashboard](https://grafana.example.com/d/service-overview)** — Primary dashboard for latency, error rate, throughput, saturation (the Four Golden Signals).
2. **[Infrastructure Dashboard](https://grafana.example.com/d/infra)** — CPU, memory, disk, network I/O per instance/container.
3. **[Database Dashboard](https://grafana.example.com/d/db)** — Connections, query latency, replication lag, cache hit ratio.
4. **[Dependency Dashboard](https://grafana.example.com/d/deps)** — Upstream/downstream health, queue depths, API latency.
5. **[Business Metrics Dashboard](https://grafana.example.com/d/biz)** — User-facing metrics: signups, active users, conversion.

### Logging

- **Log Aggregator:** [Kibana / Loki / CloudWatch Logs]
- **Log Level:** [INFO in production, DEBUG on-demand]
- **Structured Log Format:** [JSON]
- **Log Retention:** [30 days hot, 90 days cold]
- **Log Query:** [`{service="[service-name]"} | json`]

**Useful Log Queries:**

```kql
# All errors in last hour
{service="[service-name]"} | json | level = "error"

# Requests for a specific user
{service="[service-name]"} | json | user_id = "[user-id]"

# Trace a single request ID
{service="[service-name]"} | json | trace_id = "[trace-id]"
```

### Tracing

- **Tracing Backend:** [Jaeger / Tempo / X-Ray]
- **Sampling Rate:** [1% head-based, 100% for errors]
- **Trace Query by Service:** [`service.name="[service-name]"`]

### Alert Rules

| Alert Name | Condition | Severity | Auto-Close |
|------------|-----------|----------|------------|
| `[HighErrorRate]` | error_rate > [1%] for [5min] | Critical | [15min after recovery] |
| `[HighLatency]` | p99_latency > [1s] for [5min] | Warning | [30min after recovery] |
| `[LowDiskSpace]` | disk_usage > [90%] | Warning | [Disabled] |
| `[ServiceDown]` | up{job="[service]"} == 0 for [1min] | Critical | [10min after recovery] |

---

## Common Failure Modes

> Each failure mode is self-contained through mitigation. After any mitigation or recovery action, follow R-01 before declaring the incident resolved.
>
> Before executing any **Resolution** command, apply the operational closure gate: verify human authorization for the specific action and scope, record the target, affected population, maximum blast radius, success and abort/rollback criteria, rollback path, and stopping authority. These rows describe mitigation options, not permission to execute them. If the gate cannot be satisfied, stop and hand off or escalate.

### FM-01: Service Unreachable / High Error Rate

| Field | Value |
|-------|-------|
| **Symptom** | `5xx` responses > [N]%, health check failing, pager alert firing |
| **Likely Cause** | Recent deployment, resource exhaustion, upstream dependency failure |
| **Diagnosis** | 1. Check `/health` and `/metrics` endpoints<br>2. Review recent deployments (`kubectl rollout history` or equivalent)<br>3. Check CPU/memory on instance<br>4. Check upstream dependencies (database, cache, external APIs)<br>5. Review recent logs for panic/OOM/panic |
| **Resolution** | 1. **If caused by recent deploy:** Rollback immediately: `kubectl rollout undo deployment/[deploy]`<br>2. **If resource exhaustion:** Scale up: `kubectl scale deployment/[deploy] --replicas=[N]`<br>3. **If upstream failure:** Check dependency runbook, pager dependency owner<br>4. **Last resort:** Restart the service<br>5. If none of the above work, [escalate](#escalation-paths)<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-02: High Latency

| Field | Value |
|-------|-------|
| **Symptom** | p99/p95 latency exceeds SLO threshold, users report slowness |
| **Likely Cause** | Traffic spike, database query degradation, slow upstream, GC pressure |
| **Diagnosis** | 1. Check traffic volume vs baseline (is this a spike?)<br>2. Check database slow query log — `SELECT * FROM pg_stat_activity WHERE state = 'active'`<br>3. Check GC metrics (if JVM: `jstat -gcutil`, if Go: `go_memstats_gc_cpu_fraction`)<br>4. Check upstream dependency latencies<br>5. Review tracing dashboard for slow spans |
| **Resolution** | 1. **Traffic spike:** Auto-scale groups should handle; manually increase replicas if needed<br>2. **Slow queries:** Kill runaway queries: `SELECT pg_terminate_backend(pid) WHERE ...`; add missing index<br>3. **GC pressure:** Increase heap/memory, tune GC parameters<br>4. **Upstream slow:** Circuit-breaker should trip; verify upstream health<br>5. **Temporary fix:** Rate-limit or shed non-critical traffic<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-03: Out of Memory / OOM Killed

| Field | Value |
|-------|-------|
| **Symptom** | Container/process killed, OOM in kernel logs, instance becomes unresponsive |
| **Likely Cause** | Memory leak in code, traffic surge, insufficient resource limits |
| **Diagnosis** | 1. Check `dmesg | grep -i oom` for kernel OOM killer messages<br>2. Check memory metrics via dashboard (heap vs non-heap if JVM)<br>3. Review recent code changes for potential memory leaks<br>4. Check if memory limit was recently reduced<br>5. Heap dump analysis (if available): `jmap -dump:live,format=b,file=heap.hprof <pid>` |
| **Resolution** | 1. **Immediate:** Restart the service to reclaim memory<br>2. **Increase limits:** Edit resource `limits.memory` for the container/Pod<br>3. If caused by code change: rollback the release<br>4. Schedule memory leak investigation with engineering team<br>5. Consider enabling memory request-based autoscaling<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-04: Database Connection Pool Exhaustion

| Field | Value |
|-------|-------|
| **Symptom** | Application logs show `connection refused`, `too many connections`, or `connection timeout` |
| **Likely Cause** | Connection leak in application, insufficient pool size, DB restart |
| **Diagnosis** | 1. Check active connections on DB: `SELECT count(*) FROM pg_stat_activity`<br>2. Check max connections: `SHOW max_connections`<br>3. Identify connections by application: `SELECT application_name, count(*) FROM pg_stat_activity GROUP BY 1`<br>4. Check if connections are idle-in-transaction: `SELECT * FROM pg_stat_activity WHERE state = 'idle in transaction'`<br>5. Review application connection pool metrics (hikariCP, etc.) |
| **Resolution** | 1. **Emergency:** Kill idle connections: `SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle'`<br>2. **Kill idle-in-transaction:** Same as above with `state = 'idle in transaction'`<br>3. Restart the application to reset its connection pool<br>4. If leak persists, increase `max_connections` temporarily on DB<br>5. Schedule fix for connection leak (usually unclosed `Connection` / `Session` objects)<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-05: Disk Space Full

| Field | Value |
|-------|-------|
| **Symptom** | Disk usage alert firing, application unable to write logs/data, `no space left on device` |
| **Likely Cause** | Logs not rotated, data not cleaned up, unexpected large files (core dumps, heap dumps) |
| **Diagnosis** | 1. `df -h` to identify full partition<br>2. `du -sh /* 2>/dev/null` to find large directories<br>3. `du -sh /var/log/* | sort -rh | head -10` for log sizes<br>4. `find / -type f -size +1G -exec ls -lh {} \;` for large files<br>5. Check logrotate status: `logrotate -d /etc/logrotate.d/[app]` |
| **Resolution** | 1. **Immediate:** `sudo journalctl --vacuum-size=500M` or `truncate -s0 /var/log/[app].log`<br>2. Clean old logs: `find /var/log -name "*.log.*" -mtime +7 -delete`<br>3. Clean temp files: `sudo rm -rf /tmp/*`<br>4. Verify logrotate is working: `sudo logrotate -f /etc/logrotate.conf`<br>5. If persistent, add disk monitoring alarm at 80%<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-06: TLS / Certificate Expiry

| Field | Value |
|-------|-------|
| **Symptom** | Clients receive `certificate expired` or `x509: certificate has expired` errors |
| **Likely Cause** | Auto-renewal failed, cert-manager/ACME issues, manual cert not replaced |
| **Diagnosis** | 1. Check cert expiry: `echo | openssl s_client -servername [domain] -connect [domain]:443 2>/dev/null | openssl x509 -noout -dates`<br>2. Check cert-manager logs (if Kubernetes): `kubectl logs -n cert-manager -l app=cert-manager`<br>3. Check certificate resource status: `kubectl get certificate -A`<br>4. Verify DNS resolution for ACME challenge domain |
| **Resolution** | 1. **Manual renewal:** If cert-manager: `kubectl delete certificate [name]` to trigger re-issue, or fix the ACME challenge<br>2. **Manual cert replacement:** Upload new cert to LB/Ingress<br>3. Restart ingress controller / LB after cert update<br>4. If auto-renewal is broken, create a ticket for the platform team<br>5. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-07: Upstream API Degraded / Down

| Field | Value |
|-------|-------|
| **Symptom** | Our service returns errors for operations that depend on [upstream], upstream latency spikes |
| **Likely Cause** | Upstream outage, rate limiting, network partition |
| **Diagnosis** | 1. Check upstream status page<br>2. Test upstream directly: `curl -v https://upstream.example.com/health`<br>3. Check our circuit breaker metrics<br>4. Check for recent upstream API changes<br>5. Check network connectivity: `ping`, `traceroute`, `nslookup` |
| **Resolution** | 1. **If circuit breaker open:** Wait for half-open / reset, or manually reset if safe<br>2. **If rate limited:** Throttle requests, request quota increase<br>3. **If upstream outage:** Enable fallback/graceful degradation (serve stale data, queue requests)<br>4. Page upstream PagerDuty escalation<br>5. Consider feature flags to disable upstream-dependent features temporarily<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-08: Slow Consumer / Queue Backlog

| Field | Value |
|-------|-------|
| **Symptom** | Queue depth growing, consumer lag increasing, messages not processed in time |
| **Likely Cause** | Consumer crashed or stuck, processing logic regression, queue partition imbalance |
| **Diagnosis** | 1. Check consumer group lag (Kafka: `kafka-consumer-groups --bootstrap-server ... --group [group] --describe`)<br>2. Check consumer process health and logs<br>3. Check if messages are stuck on poison-pill messages (deserialization errors)<br>4. Check partition assignment and rebalance events<br>5. Check downstream that the consumer writes to |
| **Resolution** | 1. **Restart consumers:** `kubectl rollout restart deployment/[consumer]`<br>2. **Skip poison-pill messages:** Seek consumer offset past bad message<br>3. **Scale consumers:** Increase partitions + consumer replicas<br>4. **If DB is bottleneck:** Investigate and resolve DB performance first<br>5. If backlog is critical, consider replaying messages from an earlier offset after fix<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-09: DNS Resolution Failures

| Field | Value |
|-------|-------|
| **Symptom** | `lookup [hostname]` failures, `connection refused`, intermittent timeouts |
| **Likely Cause** | DNS resolver outage, cached stale records, DNS propagation delay, /etc/resolv.conf misconfiguration |
| **Diagnosis** | 1. Test resolution: `dig [hostname] @[dns-server]`<br>2. Check `/etc/resolv.conf`<br>3. Check `nslookup [hostname]` and `host [hostname]`<br>4. Check DNS server reachability: `nc -zv [dns-server] 53`<br>5. Compare results across different resolvers (e.g., `8.8.8.8`) |
| **Resolution** | 1. **Flush DNS cache:** `sudo systemd-resolve --flush-caches` or restart `nscd`<br>2. **Update resolv.conf:** Ensure valid nameservers listed<br>3. **Restart DNS-sidecar** (if Kubernetes with CoreDNS/node-local-dns)<br>4. **Override in /etc/hosts** as temporary measure<br>5. If using Kubernetes, check CoreDNS pods and service: `kubectl -n kube-system get pods -l k8s-app=kube-dns`<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

### FM-10: Pod CrashLoopBackOff (Kubernetes)

| Field | Value |
|-------|-------|
| **Symptom** | Pod repeatedly restarting, `CrashLoopBackOff` status, zero ready replicas |
| **Likely Cause** | Startup failure (config error, missing secret, dependency unavailable), OOM, liveness probe failing |
| **Diagnosis** | 1. Describe pod: `kubectl describe pod [pod-name] -n [namespace]`<br>2. Check pod logs: `kubectl logs [pod-name] -n [namespace] --previous`<br>3. Check events: `kubectl get events -n [namespace] --sort-by='.lastTimestamp'`<br>4. Verify configmaps/secrets are mounted: `kubectl exec -it [pod] -- cat /path/to/config`<br>5. Check liveness/readiness probe configuration |
| **Resolution** | 1. **Fix config/secrets:** Correct the ConfigMap or Secret and re-deploy<br>2. **Fix missing dependency:** Start the dependency or fix the connection string<br>3. **Increase resources:** If OOM-killed, increase `limits.memory`<br>4. **Fix probe:** Correct the probe endpoint/timeout/period<br>5. Rollback to last known good version if config doesn't help<br>6. After mitigation, follow **R-01** before declaring resolved; retain **MITIGATING** or **MONITORING** and escalate if evidence is missing. |

---

## Detailed Troubleshooting Procedures

> Before executing any command in these procedures that mutates production, apply the operational closure gate: verify current human authorization for the specific action and scope, record the target, affected population, maximum blast radius, success and abort/rollback criteria, rollback path, and stopping authority. These commands are examples, not permission to execute them. If the gate cannot be satisfied, stop and hand off or escalate.

### T-01: Initial Incident Triage

```
                   ┌──────────────────────┐
                   │  Alert Fires / User   │
                   │  Reports Issue        │
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  1. ACKNOWLEDGE      │
                   │  Ack the alert in    │
                   │  PagerDuty/OpsGenie  │
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  2. TRIAGE           │
                   │  - What's affected?  │
                   │  - How many users?   │
                   │  - Is it critical?   │
                   │  - Create incident   │
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  3. INITIAL COMMS    │
                   │  - Post in #incidents│
                   │  - Ping team channel │
                   │  - Update status page│
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  4. DIAGNOSE         │
                   │  Check dashboards,   │
                   │  logs, traces, use   │
                   │  Common Failure Modes│
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  5. MITIGATE         │
                   │  Rollback, restart,  │
                   │  scale up, etc.      │
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  6. VERIFY           │
                   │  Run R-01 closure    │
                   │  evidence checks     │
                   └──────────┬───────────┘
                              │
                   ┌──────────▼───────────┐
                   │  7. RESOLVE          │
                   │  Only after R-01     │
                   │  and stability       │
                   └──────────────────────┘
```

### T-02: Reading Health Check Endpoints

```bash
# Standard health check
curl -s http://localhost:[port]/health | jq .

# Verbose health check (includes dependency status)
curl -s http://localhost:[port]/healthz | jq .

# Readiness check (Kubernetes)
curl -s http://localhost:[port]/readyz | jq .

# Expected output for healthy service:
# {"status":"ok","version":"1.2.3","uptime":"72h14m","dependencies":{"database":"ok","cache":"ok","queue":"degraded"}}
```

### T-03: Capturing Diagnostic Data

```bash
# Collect a diagnostics bundle
./scripts/diagnostics.sh > /tmp/diag-$(date +%s).txt

# Capture thread dump (Java)
jstack <pid> > /tmp/thread-dump-$(date +%s).txt

# Capture heap histogram (Java)
jmap -histo:live <pid> > /tmp/heap-hist-$(date +%s).txt

# Capture goroutine dump (Go)
curl -s http://localhost:[debug-port]/debug/pprof/goroutine?debug=2 > /tmp/goroutines.txt

# Capture metrics snapshot
curl -s http://localhost:[port]/metrics > /tmp/metrics-$(date +%s).txt
```

### T-04: Safe Rollback Procedure

```bash
# Step 1: Identify last known good version
kubectl rollout history deployment/[deployment] -n [namespace]

# Step 2: Rollback to previous revision
kubectl rollout undo deployment/[deployment] -n [namespace]

# Step 3: Monitor rollout status
kubectl rollout status deployment/[deployment] -n [namespace]

# Step 4: Start R-01 closure verification
curl -s http://[service-url]/health | jq .
# A health endpoint is partial evidence. Complete R-01: verify user-facing SLOs
# and critical journeys, dependencies, data/state, secondary effects, and the
# defined stability window and independent human confirmation before declaring the incident resolved.

# Step 5: Keep the incident Mitigating or Monitoring if evidence is missing;
# record the unverified boundary and hand off or escalate.

# Step 6: After R-01 closure evidence and independent human confirmation are
# complete, the human incident owner sends the resolution announcement.
```

---

## Escalation Paths

### Primary Escalation

| Level | Contact | Method | Response Time |
|-------|---------|--------|---------------|
| **L1** | On-call Engineer | PagerDuty / Phone | [15min] |
| **L2** | [Team Name] Senior Engineer | PagerDuty / Slack | [30min] |
| **L3** | [Team Name] Tech Lead | Phone / Slack @mention | [1hr] |
| **L4** | Engineering Manager | Phone / Slack @mention | [2hr] |

### Dependency Escalation

| Dependency | Team | PagerDuty Schedule |
|------------|------|--------------------|
| [Database Cluster] | [Data Platfom Team] | [link] |
| [Kubernetes Cluster] | [Infra Team] | [link] |
| [Network / DNS] | [Networking Team] | [link] |
| [External API] | [Vendor Support] | [vendor-ticket-link] |

### Escalation Procedure

1. **5 minutes:** If not resolved, page L1.
2. **15 minutes:** If L1 cannot resolve, escalate to L2 (Senior Engineer).
3. **30 minutes:** If L2 needs additional context, involve L3 (Tech Lead).
4. **60 minutes:** If incident is customer-facing or SEV-1, notify L4 (Engineering Manager).
5. **SEV-1 criteria:** Service down for > 5min, data loss, security breach, revenue impact.
6. **Declare SEV-1:** Post in #severe-incidents, create incident channel, invite relevant teams.

### Communication Template

```
INCIDENT: #[incident-number]
STATUS: [Investigating / Mitigating / Resolved]
SERVICE: [service-name]
IMPACT: [Describe user/business impact]
TIMELINE:
  [HH:MM UTC] - Alert fired
  [HH:MM UTC] - On-call acknowledged
  [HH:MM UTC] - Rollback initiated
NEXT STEPS: [What's being done]
```

---

## Recovery Procedures

### R-01: Post-Incident Steps

1. **Verify full recovery at the user boundary** — Confirm user-facing SLOs and critical user journeys, relevant dependency health, data/state correctness, and secondary effects such as backlog recovery. Health checks, baseline error/latency, and a cleared alert are partial evidence, not a resolution verdict.
2. **Observe a stability window** — Monitor for the defined window and confirm that recovery holds without cascading or delayed effects. Record the evidence and the boundary actually exercised.
3. **Human-confirm the evidence** — A human other than the acting automation must review and confirm the complete evidence set, with the confirmation independently attributable to that person. An agent-assigned IC role, automation-authored incident record, green alert, or self-reported health check is not sufficient.
4. **Keep unresolved incidents visible** — If any required evidence or human confirmation is missing, retain the incident in **MITIGATING** or **Monitoring**, record the unverified boundary, and hand off or escalate rather than marking it resolved.
5. **Update status page** — Mark the incident as resolved if used, but only after full recovery evidence and human confirmation are complete.
6. **Resolve alert** — Close PagerDuty / OpsGenie alert after the resolution decision, not merely because the alert condition cleared.
7. **Tag, annotate, and notify** — Add incident severity, team, and service tags, then send the verified summary to the team channel and affected users.

### R-02: Data / State Recovery

> **Warning:** Data recovery procedures should only be attempted by engineers with database admin access.
>
> Before any production restore, verify current human authorization for the specific restore and scope, record the target, affected population, maximum blast radius, success and abort/rollback criteria, rollback path, and stopping authority. Restore to staging and verify integrity first; staging evidence does not authorize the production restore. If the gate cannot be satisfied, stop and hand off or escalate rather than restoring production.

```bash
# Step 1: Identify the recovery point (RPO)
# Step 2: Restore from backup to a staging environment first
# Step 3: Verify data integrity in staging
# Step 4: Schedule maintenance window
# Step 5: Perform database restore in production
# Step 6: Verify application works against restored data
```

**Backup Locations:**

| Resource | Backup Frequency | Retention | Restore Process |
|----------|-----------------|-----------|-----------------|
| [Database] | [Hourly WAL + Daily full] | [30 days] | [link to restore doc] |
| [Object storage] | [Cross-region replication] | [N/A] | [link to restore doc] |
| [Configuration] | [Git-controlled] | [Permanent] | Terraform apply |

### R-03: Incident Report Template

```markdown
## Post-Mortem: [Title]

**Date:** [YYYY-MM-DD]
**Duration:** [Start] — [End] ([X] minutes)
**Severity:** [SEV-1 / SEV-2 / SEV-3]
**Team:** [Team Name]

### Summary
[2-3 sentence executive summary]

### Timeline
- [HH:MM UTC] — [Event]
- [HH:MM UTC] — [Action taken]
- [HH:MM UTC] — [Recovery verified]

### Root Cause
[What actually caused the incident]

### Impact
- [N] requests failed
- [N] users affected
- [N] minutes of downtime

### Action Items
| Action | Owner | Ticket |
|--------|-------|--------|
| [Fix root cause] | [Name] | [Jira/GH link] |
| [Improve monitoring] | [Name] | [Jira/GH link] |
| [Update runbook] | [Name] | [PR link] |

### Lessons Learned
- What went well:
- What went wrong:
- What we'll do differently:
```

### R-04: Runbook Update Checklist

After any incident, update this runbook:

- [ ] Add any new failure modes discovered
- [ ] Update diagnosis steps that were incorrect or incomplete
- [ ] Update resolution steps that worked
- [ ] Update escalation paths if contacts have changed
- [ ] Update links that were broken
- [ ] Update SLOs / error budget if thresholds have changed
- [ ] Increment version number
- [ ] Update "Last Updated" date

---

## Appendix

### A — Useful Scripts & Aliases

```bash
# Quick health check
alias svc-health='curl -s http://localhost:[port]/health | jq .'

# Check recent deploys
alias svc-history='kubectl rollout history deployment/[deployment] -n [namespace]'

# Tail logs
alias svc-logs='kubectl logs -f deployment/[deployment] -n [namespace]'

# Shell into a running pod
alias svc-shell='kubectl exec -it deployment/[deployment] -n [namespace] -- /bin/bash'
```

### B — Environment Information

| Environment | URL / Access | Replicas | Instance Type |
|-------------|-------------|----------|---------------|
| **Production** | [prod-url] | [N] | [e.g. m5.xlarge] |
| **Staging** | [staging-url] | [N] | [e.g. t3.large] |
| **Development** | [dev-url] | [N] | [e.g. t3.medium] |

### C — Configuration Reference

| Config Key | Description | Default | Production Value |
|------------|-------------|---------|------------------|
| `[DATABASE_URL]` | Primary DB connection string | — | [redacted] |
| `[REDIS_URL]` | Redis connection string | — | [redacted] |
| `[LOG_LEVEL]` | Logging verbosity | `info` | `info` |
| `[MAX_CONNECTIONS]` | DB pool size | `10` | `[N]` |
| `[REQUEST_TIMEOUT]` | Upstream request timeout | `30s` | `10s` |

### D — Related Runbooks

| Runbook | Service | Link |
|---------|---------|------|
| [Database Runbook] | [Database] | [link] |
| [Cache Runbook] | [Redis/Memcached] | [link] |
| [Infrastructure Runbook] | [Kubernetes/AWS] | [link] |
| [Auth Runbook] | [Auth Service] | [link] |

---

> **Document Status:** This runbook is a living document. If you find errors, omissions, or out-of-date information during an incident, fix it immediately and submit a PR.
>
> **Template Attribution:** Based on the [Site Reliability Engineering](https://sre.google/) principles and Google's SRE books.

# Pre-Launch Reliability Review Checklist

> **Service:** \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> **Owner:** \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> **Review Date:** \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> **Reviewer(s):** \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> **Target Launch:** \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

---

## Instructions

Check each item as **Pass**, **Fail**, **N/A**, or **Warn** (requires documented exception). All **Fail** and **Warn** items must have an action item and owner listed in the Action Items section at the end. A service may not launch with any unchecked **Fail** items.

| Status | Meaning |
| ------ | ------- |
| [P]    | Pass — item verified and meets standards |
| [F]    | Fail — blocker; must be resolved before launch |
| [W]    | Warn — acceptable with documented exception/risk |
| [-]    | N/A — not applicable to this service |

---

## 1. Service Design

### 1.1 Architecture Review

- [ ] [\_] Architecture diagram is documented and up to date (components, data flow, network boundaries)
- [ ] [\_] Architecture has been reviewed by a senior engineer or SRE
- [ ] [\_] Single points of failure (SPOF) have been identified and documented
- [ ] [\_] Service is stateless or state is externalized (database, cache, object store)
- [ ] [\_] Stateful components have a clear durability and replication strategy
- [ ] [\_] Database schema includes indexes for query patterns; no full-table scans on critical paths
- [ ] [\_] External integration points are abstracted behind interfaces (no hard coupling)
- [ ] [\_] Protocol/API versioning strategy is defined (backward-compatible by default)

### 1.2 Failure Modes

- [ ] [\_] A failure modes analysis (FMEA / fault tree) has been performed
- [ ] [\_] Each failure mode has a documented blast radius and mitigation
- [ ] [\_] Crash-only / fail-fast behavior is verified — no silent corruption on partial failure
- [ ] [\_] Partial failure of downstream dependencies does not cascade to the whole service
- [ ] [\_] Startup/shutdown ordering is defined for multi-component services

### 1.3 Dependency Map

- [ ] [\_] All internal and external dependencies are inventoried (services, libraries, APIs, data stores)
- [ ] [\_] Each dependency is classified by criticality: critical, important, or nice-to-have
- [ ] [\_] Critical dependencies have a defined degradation mode when unavailable
- [ ] [\_] Network latency / bandwidth requirements for each dependency are known
- [ ] [\_] Dependency version pinning and update cadence is defined

---

## 2. Resilience

### 2.1 Retry & Backoff

- [ ] [\_] Transient-failure retry logic is implemented with **exponential backoff and jitter**
- [ ] [\_] Retry budgets / caps are defined (max retries, total time window)
- [ ] [\_] Retries do not amplify load on degraded downstream services (client-side shedding)
- [ ] [\_] Idempotency keys are used for mutating operations across retries
- [ ] [\_] Retries on network errors are separated from retries on application-level errors

### 2.2 Circuit Breakers

- [ ] [\_] Circuit breaker pattern is implemented for all remote / downstream calls
- [ ] [\_] Thresholds (error rate, latency, concurrency) are tuned and documented
- [ ] [\_] Half-open recovery probes are configured (test a single request before closing)
- [ ] [\_] Circuit breaker state transitions are observable (metrics, events, logs)

### 2.3 Bulkheads

- [ ] [\_] Resource pools (thread pools, connection pools, memory arenas) are partitioned by workload
- [ ] [\_] No single tenant / user / request can exhaust shared resources
- [ ] [\_] Per-customer or per-feature resource quotas are enforced
- [ ] [\_] Connection pool sizes are tuned and documented per dependency

### 2.4 Graceful Degradation

- [ ] [\_] Fallback behavior is defined for each non-critical dependency failure
- [ ] [\_] Degraded mode is observable (logged, metered, and surfaced in status endpoints)
- [ ] [\_] Static / cached responses are served when live data is unavailable (if acceptable)
- [ ] [\_] Feature flags allow disabling non-critical functionality at runtime
- [ ] [\_] Rate limiting is implemented at the appropriate layer (ingress, per-tenant, per-endpoint)

---

## 3. Observability

### 3.1 Metrics

- [ ] [\_] RED metrics (Rate, Errors, Duration) are collected for every endpoint / operation
- [ ] [\_] USE metrics (Utilization, Saturation, Errors) are collected for every resource
- [ ] [\_] Business / application-level metrics are defined (throughput, conversions, queue depth)
- [ ] [\_] Metrics are tagged with meaningful dimensions (service, version, region, tenant)
- [ ] [\_] Custom metrics are within the cardinality budget for the metrics backend
- [ ] [\_] Health check endpoints (`/healthz`, `/readyz`, `/livez`) are implemented and distinct

### 3.2 Logging

- [ ] [\_] Structured logging (JSON) is used — machine-parseable with consistent schema
- [ ] [\_] Log levels (debug, info, warn, error) are used consistently and without noise
- [ ] [\_] Request IDs / trace IDs are propagated through all components and logs
- [ ] [\_] Sensitive data (PII, credentials, tokens) is never logged
- [ ] [\_] Error logs include stack traces and contextual metadata
- [ ] [\_] Log retention and archival policy is defined

### 3.3 Distributed Tracing

- [ ] [\_] Traces are emitted for all service-to-service calls
- [ ] [\_] Trace sampling strategy is defined (head-based, tail-based, or rate-based)
- [ ] [\_] Critical user journeys are identifiable by trace attributes
- [ ] [\_] gRPC / HTTP middleware for trace propagation is instrumented

### 3.4 Dashboards

- [ ] [\_] **Service overview dashboard** exists (RED metrics, saturation, error budget)
- [ ] [\_] **Dependency health dashboard** exists (upstream/downstream latency and errors)
- [ ] [\_] **Resource dashboard** exists (CPU, memory, disk, network, connections)
- [ ] [\_] Dashboards have meaningful time ranges, annotations, and thresholds
- [ ] [\_] Dashboards are shared with the owning team and documented in the runbook

### 3.5 Alert Rules

- [ ] [\_] Alerts exist for: error budget burn rate, high latency, high error rate, saturation
- [ ] [\_] Alerts use multi-window / multi-condition evaluation to reduce flapping
- [ ] [\_] Page-worthy alerts are well-separated from low-severity warnings
- [ ] [\_] Alert thresholds are calibrated to generate actionable notifications (not noise)
- [ ] [\_] Each alert has a documented runbook link and a clear remediation step

---

## 4. Capacity

- [ ] [\_] Traffic estimates are documented (peak QPS, data volume, concurrent connections)
- [ ] [\_] Scaling plan is defined: horizontal (stateless tier) and vertical (stateful tier)
- [ ] [\_] Auto-scaling policies are configured with min/max bounds and cooldown periods
- [ ] [\_] Resource limits (CPU, memory, disk, connections) are set per container / process
- [ ] [\_] Load testing results demonstrate capacity for 2×-3× peak estimated traffic
- [ ] [\_] Database connection pooling and max connections are configured
- [ ] [\_] Downstream dependency rate limits are known and not exceeded at peak
- [ ] [\_] Storage growth rate is projected; capacity alerts exist for 80% / 90% thresholds
- [ ] [\_] Cold-start / warm-up time is measured; prewarming strategy is defined if needed

---

## 5. Release

### 5.1 CI/CD Pipeline

- [ ] [\_] CI pipeline runs linting, unit tests, integration tests, and security scans
- [ ] [\_] CD pipeline is fully automated (no manual steps for standard releases)
- [ ] [\_] Artifacts are immutable (versioned, checksummed, stored in artifact registry)
- [ ] [\_] Infrastructure changes are codified (IaC — Terraform, Pulumi, CloudFormation, etc.)
- [ ] [\_] Database migrations are automated, reversible, and tested in CI

### 5.2 Rollback Plan

- [ ] [\_] Rollback procedure is documented and tested
- [ ] [\_] Rollback target is a known-good previous version (not a rebuild from scratch)
- [ ] [\_] Database schema changes are backward-compatible for at least one release cycle
- [ ] [\_] Rollback success criteria are defined (e.g., error rate returns to baseline)
- [ ] [\_] Rollback time target is documented (e.g., < 15 minutes from decision)

### 5.3 Canary Strategy

- [ ] [\_] Canary / progressive delivery process is defined
- [ ] [\_] Canary metrics and success criteria are defined (error rate, latency, error budget)
- [ ] [\_] Automatic rollback on canary failure is configured
- [ ] [\_] Traffic shifting is gradual (e.g., 1% → 5% → 25% → 100%)
- [ ] [\_] Feature flags allow toggling new functionality without redeployment

---

## 6. Security

- [ ] [\_] Authentication boundaries are defined (service-to-service, user-to-service)
- [ ] [\_] Service-to-service communication uses mutual TLS (mTLS) or equivalent
- [ ] [\_] Secrets (API keys, database passwords, certificates) are managed via a vault solution
- [ ] [\_] No secrets are hardcoded in code, configuration files, or environment variables
- [ ] [\_] Secrets are rotated on a defined schedule and after any compromise event
- [ ] [\_] Data classification labels are applied (public, internal, confidential, restricted)
- [ ] [\_] Encryption in transit (TLS ≥ 1.2) is enforced for all network communication
- [ ] [\_] Encryption at rest is enabled for all data stores
- [ ] [\_] Input validation and output encoding are applied (no injection vulnerabilities)
- [ ] [\_] Least-privilege principle is applied to service accounts / IAM roles
- [ ] [\_] Dependency vulnerabilities are scanned in CI (SCA / SAST)
- [ ] [\_] A penetration test or security review has been completed (or is scheduled)

---

## 7. Incident Response

- [ ] [\_] Runbook exists and covers: common failure scenarios, diagnostic steps, and remediation
- [ ] [\_] Runbook includes: service owner, escalation contacts, dashboard links, and alert links
- [ ] [\_] Runbook is stored alongside the service code or in a known wiki location
- [ ] [\_] On-call rotation is established and the team is trained on the service
- [ ] [\_] On-call engineers have production access and can deploy fixes
- [ ] [\_] Escalation path is documented (primary → secondary → engineering manager → director)
- [ ] [\_] Incident severity levels are defined (SEV1–SEV4) with corresponding response SLAs
- [ ] [\_] Postmortem process is defined (blameless, within 5 business days)
- [ ] [\_] A communication template exists for incident status updates
- [ ] [\_] Service is registered with the incident management platform (PagerDuty, Opsgenie, etc.)

---

## 8. Service Level Objectives (SLOs)

- [ ] [\_] Service Level Indicators (SLIs) are defined for availability, latency, and durability
- [ ] [\_] SLO targets are defined for each SLI (e.g., 99.9% availability, p99 < 200ms)
- [ ] [\_] SLIs are measured and instrumented (not just theoretical)
- [ ] [\_] Error budget is calculated (1 − SLO) and tracked over a defined window (e.g., 30 days)
- [ ] [\_] Error budget burn rate alerts are configured (fast, medium, slow burn)
- [ ] [\_] SLO attainment is visible on a dashboard shared with the team
- [ ] [\_] Consequences for exhausting the error budget are defined (freeze features, rollback)
- [ ] [\_] SLOs are documented and agreed upon with product / business stakeholders

---

## 9. Compliance

- [ ] [\_] Applicable regulatory frameworks are identified (SOC 2, HIPAA, PCI-DSS, GDPR, FedRAMP, etc.)
- [ ] [\_] Data residency requirements are documented and enforced (which regions data may reside in)
- [ ] [\_] Data retention and deletion policies are implemented
- [ ] [\_] Audit logs are captured for all administrative and data-access actions
- [ ] [\_] Audit log retention period meets regulatory requirements
- [ ] [\_] Access controls satisfy compliance requirements (RBAC, least privilege, separation of duties)
- [ ] [\_] Privacy impact assessment (PIA) or data protection review has been completed (if applicable)
- [ ] [\_] Licensing compliance for all dependencies is verified
- [ ] [\_] Third-party vendor security assessments are on file for external dependencies

---

## Action Items (Fails & Warns)

| # | Category | Item | Status | Owner | Due Date | Notes |
| - | -------- | ---- | ------ | ----- | -------- | ----- |
|   |          |      |        |       |          |       |
|   |          |      |        |       |          |       |
|   |          |      |        |       |          |       |
|   |          |      |        |       |          |       |
|   |          |      |        |       |          |       |

---

## Scoring Rubric

### Overall Review Result

| Score | Range | Meaning | Launch Decision |
| ----- | ----- | ------- | --------------- |
| **PASS** | 0 Fails, < 5 Warns | All critical items pass; minor or no exceptions | Authorized to launch |
| **CONDITIONAL PASS** | 0 Fails, ≥ 5 Warns | No blockers, but significant risks documented | Launch authorized with conditions; all Warn items must have remediation plans |
| **FAIL** | ≥ 1 Fail | At least one blocker remains | Launch blocked; must resolve all Fails before re-review |

### Category Health Score

Count how many categories have **zero Fails** versus **one or more Fails**:

- **9/9** categories clear — Excellent
- **7–8/9** categories clear — Good; address outstanding Fails
- **5–6/9** categories clear — Needs improvement; significant work required
- **< 5/9** categories clear — Unacceptable; launch should not proceed

### Sign-off

| Role | Name | Signature | Date |
| ---- | ---- | --------- | ---- |
| Service Owner | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_ |
| SRE Reviewer | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_ |
| Engineering Manager | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_ |
| Security Reviewer (if applicable) | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_ |
| Compliance Reviewer (if applicable) | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_ | \_\_\_\_\_\_\_\_\_\_ |

---
title: "SLO Declaration: [Service Name]"
doc_id: SLO-[SERVICE-CODE]-[VERSION]
status: draft | reviewed | approved | superseded
created: [YYYY-MM-DD]
last_modified: [YYYY-MM-DD]
owner: "[Owner Name / Team]"
approver: "[Approver Name]"
classification: internal | customer-facing | critical-infrastructure
---

# SLO Declaration — [Service Name]

## 1. Service Overview

| Field | Value |
|---|---|
| **Service Name** | [Service Name] |
| **Service ID** | [SVC-XXXXX] |
| **Description** | [Brief description of service purpose and functionality] |
| **Owner** | [Team Name / Individual] |
| **Escalation Path** | [#slack-channel](https://...), [@oncall-handoff] |
| **Tier Classification** | **Tier [1/2/3/4]** — [Critical / High / Medium / Low] |
| **Service Dependencies** | [List of upstream/downstream services] |
| **Documentation Links** | [Runbook], [Architecture Doc], [Dashboards] |

### Tier Definitions

| Tier | Definition | Acceptable Uptime |
|---|---|---|
| 1 | Customer-facing critical path; revenue-impacting | ≥ 99.99% |
| 2 | Important internal service; customer-facing non-critical | ≥ 99.9% |
| 3 | Internal tooling; best-effort availability | ≥ 99.5% |
| 4 | Experimental / dev-only; no formal commitment | No commitment |

---

## 2. SLO Window

| Parameter | Value |
|---|---|
| **Measurement Window** | [e.g., 28-day rolling / 30-day calendar / quarterly] |
| **Window Start** | [YYYY-MM-DD] |
| **Window End** | [YYYY-MM-DD or "rolling"] |
| **Evaluation Cadence** | [e.g., daily / weekly / upon window close] |
| **Reporting Channels** | [e.g., Grafana dashboard, PagerDuty, weekly SRE review] |

---

## 3. SLI Specifications

Each SLI is defined with an explicit metric name, data source, and measurement method.

### 3.1. SLI Definition Table

| SLI ID | Indicator Name | Metric Name | Data Source | Measurement Method | Good Criteria | Aggregation |
|---|---|---|---|---|---|---|
| SLI-001 | Request Latency (p99) | `[service]_http_request_duration_ms` | [Prometheus / Datadog / CloudWatch / custom] | Histogram quantile over 1m buckets; sampled every [15s] | ≤ [200]ms at p99 | Rolling window average |
| SLI-002 | Error Rate | `[service]_http_requests_total{status=~"5.."}` | [Prometheus / Datadog / CloudWatch / custom] | Count of 5xx / total requests; measured in [1m] bins | Ratio < [0.001] (0.1%) | Rolling window sum |
| SLI-003 | Availability | `[service]_probe_success` | [Synthetic probes / Blackbox exporter / Health endpoint] | Synthetic check every [60s] from [3] regions | Probe returns HTTP 200 in ≥ [2]/[3] regions | Rolling window % |
| SLI-004 | Throughput | `[service]_requests_per_second` | [Prometheus / Data source] | Rate() over [5m] intervals | Within [baseline ± 20%] of expected throughput | Rolling window avg |
| SLI-005 | [Custom SLI Name] | `[metric_name]` | [Data source] | [Measurement method] | [Good criteria] | [Aggregation] |

### 3.2. SLI Details (Optional — for complex SLIs)

**SLI-001: Request Latency (p99)**
- **Metric**: `[service]_http_request_duration_ms`
- **Source**: Prometheus (datasource: `[prometheus-datasource-name]`)
- **Query**: `histogram_quantile(0.99, sum(rate([service]_http_request_duration_seconds_bucket[1m])) by (le))`
- **Exclusions**: Requests to `/healthz`, `/metrics`; requests from synthetic probes
- **Measurement Notes**: [e.g., Measured at load balancer edge, excludes client-side latency]

**SLI-002: Error Rate**
- **Metric**: `[service]_http_requests_total`
- **Source**: [Data source]
- **Query**: `sum(rate([service]_http_requests_total{status=~"5.."}[1m])) / sum(rate([service]_http_requests_total[1m]))`
- **Exclusions**: Expected errors (rate-limited requests, known bad actors); 503s from deliberate traffic-shedding
- **Measurement Notes**: [e.g., Only counts requests that reached application servers]

---

## 4. SLO Targets & Error Budgets

| SLI ID | SLO Target | Measurement Window | Error Budget | Budget per Window | Consumed So Far |
|---|---|---|---|---|---|
| SLI-001 | p99 latency ≤ 200ms for ≥ 99.9% of requests | 28-day rolling | 0.1% of requests (43.2 minutes of bad requests per 30d) | [43.2] min | [X.X]% |
| SLI-002 | Error rate < 0.1% for ≥ 99.95% of time | 28-day rolling | 0.05% of time (≈ 21.6 minutes of excessive errors per 30d) | [21.6] min | [X.X]% |
| SLI-003 | Availability ≥ 99.99% | 28-day rolling | 0.01% (≈ 4.3 minutes of downtime per 30d) | [4.3] min | [X.X]% |
| SLI-004 | Throughput within baseline ±20% for ≥ 99.5% of time | 28-day rolling | 0.5% of time (≈ 3.6 hours per 30d) | [3.6] hr | [X.X]% |
| SLI-005 | [Custom target] | [Window] | [Error budget calculation] | [Amount] | [X.X]% |

### Error Budget Policy

- **Consumption ≤ 50%**: Normal operations; feature releases permitted.
- **Consumption 50–75%**: Increased monitoring, reduce deployment velocity, review changes.
- **Consumption 75–100%**: **Error budget at risk.** Feature freezes in effect; only P0/P1 fixes and rollbacks permitted. SRE review required.
- **Consumption > 100%**: **Budget exhausted.** Service classified as "out of SLO." Immediate incident response triggered. Exhaustion requires a postmortem with root cause analysis.

---

## 5. Burn Rate Alert Thresholds

| Severity | Burn Rate | SLO Consumption | Example (30d window) | Alert Action |
|---|---|---|---|---|
| **Page (P0)** | ≥ 1440× (1h budget in 1h) | Consumes entire budget in ≤ 1 hour | 0.1% of 30d ≈ 43m → burning budget in 1h | Immediate page; incident response; auto-rollback |
| **Page (P1)** | ≥ 144× (10h budget in 1h) | Consumes 10% of budget in 1 hour | Consumes 4.3m of budget per hour | Page on-call; escalation if sustained > 2h |
| **Warning** | ≥ 14.4× (10h budget in 10h) | Consumes 10% of budget in 10 hours | Consumes 4.3m over 10h window | Slack notification; dashboard alert |
| **Watch** | ≥ 1.44× (10h budget in 100h) | Consumes 10% of budget in 100 hours | Slow drift detection | Informational; included in weekly report |

### Multi-Window, Multi-Burn-Rate (MWMBR) Configuration

| Alert | Short Window | Long Window | Short Budget Consumption | Long Budget Consumption |
|---|---|---|---|---|
| Critical Page | 1m | 1h | ≥ [X]% | ≥ [Y]% |
| Warning | 5m | 6h | ≥ [X]% | ≥ [Y]% |

---

## 6. Release Gating Rules

| Condition | Gate Action | Override Authority |
|---|---|---|
| Error budget consumption > 75% | Block new production deployments | VP of Engineering + SRE Lead |
| Error budget consumption > 100% | Block all deployments (including roll-forward) | CTO / VP Eng; rollback still permitted |
| Any SLI breaching SLO for > 2 consecutive evaluation windows | Block canary → full rollout | SRE Lead; requires approved mitigation plan |
| Critical burn-rate alert active (P0/P1) | Block any deployment until alert clears | N/A (auto-gate) |
| Release window blackout ([specify dates]) | Block production releases | Release Engineering |
| [Custom rule] | [Gate action] | [Override authority] |

### Release Exception Process

1. Engineer submits exception request via [link to form/tool].
2. SRE Lead reviews impact analysis and mitigation plan.
3. If approved by both SRE Lead and Service Owner, deploy proceeds with monitoring overlay.
4. Exception automatically expires after [X] hours.

---

## 7. Exception Handling

### 7.1. Planned Exceptions

| Exception ID | Description | SLI(s) Affected | Start Date | End Date | Approved By | Notes |
|---|---|---|---|---|---|---|
| EXC-001 | [e.g., Database migration] | SLI-001, SLI-002 | [YYYY-MM-DD] | [YYYY-MM-DD] | [Name] | Expected latency increase of ~50ms during window |
| EXC-002 | [Description] | [SLI IDs] | [Start] | [End] | [Name] | [Notes] |

### 7.2. Exclusion Rules

The following events are **excluded** from SLO measurement:

- **Scheduled maintenance** with [X] days' advance notice and approved change window.
- **Known client throttling** where the service intentionally returns 429/503 for traffic management.
- **External dependency failures** that the service cannot mitigate (requires dependency downtime documentation).
- **Pre-release environments** (staging, dev, canary deployments).
- **Graceful degradation modes** explicitly documented and approved by SRE.

### 7.3. Exceptional Events

Events not covered by the above exclusions that require SLO relief must follow the exception process in Section 6.1.

---

## 8. Version History

| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | [YYYY-MM-DD] | [Author Name] | Initial SLO declaration |
| 1.1 | [YYYY-MM-DD] | [Author Name] | [Summary of changes] |
| 1.2 | [YYYY-MM-DD] | [Author Name] | [Summary of changes] |

---

## 9. Review Schedule & Sign-off

| Item | Detail |
|---|---|
| **Review Cadence** | Quarterly (or triggered by architectural / capacity changes) |
| **Next Review Date** | [YYYY-MM-DD] |
| **Last Reviewed** | [YYYY-MM-DD] |

### Sign-off

| Role | Name | Date | Signature |
|---|---|---|---|
| **Service Owner** | [Name] | [YYYY-MM-DD] | |
| **SRE Lead** | [Name] | [YYYY-MM-DD] | |
| **VP of Engineering** | [Name] | [YYYY-MM-DD] | *(Tier 1 only)* |

---

*This SLO declaration should be reviewed and updated whenever the service architecture, capacity, or business requirements change materially. Keep this document in version control alongside service configuration.*