# Worked example 1: OOM cascade in `payments-api`

A realistic OOM incident in a payment processing service, triggered by a deploy that increased per-request memory footprint. Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). The fixtures and replay test under `../fixtures/01-oom-cascade/` and `../tests/replay_01_oom_cascade.py` exercise this example end-to-end.

## Scenario

- **Service**: `payments-api` (HTTP service, 8 pods, behind `api-gateway`).
- **Deploy at 2026-03-12 14:18 UTC**: introduced webhook payload buffering. Per-request memory footprint went from ~80 MB to ~210 MB. Pod memory limit unchanged at 512 MB.
- **Alert fires at 14:32 UTC**: `payments_api_error_rate > 5%`.
- **Cascade**: at 14:30 the pods hit the memory limit, the kubelet starts `OOMKill`ing them, the Deployment's ReplicaSet restarts them, in-flight requests fail with `502`, the `api-gateway` retries failed requests (3 retries, 500 ms backoff), the retry storm doubles the incoming request rate, the still-recovering pods OOM faster, error rate climbs from 0.4% to 47% in 90 seconds.

## Step 1: anchor the window

Earliest unambiguous symptom is the alert at `2026-03-12T14:32:00Z`. No customer reports filed yet. T0 set there.

- **T0**: `2026-03-12T14:32:00Z`
- **Tnow**: `2026-03-12T14:36:00Z` (investigation triggered four minutes after the alert)
- **Window**: `[14:17:00Z, 14:36:00Z]`

The 15-minute lead-in is what catches the 14:18 deploy in step 2. Without it, the bisection in step 2 would return empty.

## Step 2: bisect the change surface

Changes overlapping the window:

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | Deploy `payments-api@v4.18.0` | `14:18:14Z` | Merge commit `9f3a2c1`. Diff touches `internal/webhook/buffer.go` (new payload-buffering path). |
| Cluster / HPA | (none) | | No node-pool / HPA / resource-limit changes in window. |
| RBAC | (none) | | No Role / RoleBinding changes in window. |
| ConfigMap / flags | (none) | | No flips in window. |
| CronJob | (none) | | No batch jobs in window. |

One change in window: the `payments-api` deploy 14 minutes before T0. Strong candidate for the deploy-correlator path. Need to check the OOM path in parallel (both paths can be true; OOM is often *triggered* by a deploy).

## Step 3: classify against the four reference paths

Match against the four reference paths:

- **OOM**: pod restart count climbed from 0 to 12 in the window. Pod / kubelet events show `OOMKilled` on 6 of 8 pods. Memory metric shows RSS at 510 MB (limit 512 MB) immediately before each OOMKill. Strong match.
- **DNS**: no `NXDOMAIN` / `SERVFAIL` in logs, no `getaddrinfo` errors, outbound latency unchanged. No match.
- **Cascading-failure**: retry count at `api-gateway` spiked 3.2x in the window. Latency on downstream `ledger-svc` calls unchanged. Cascade signature is present but is *downstream of* the OOM, not the root.
- **Deploy-correlator**: 14:18 deploy is the only change in window, diff touches the failing surface (`internal/webhook/buffer.go` is in the request path), and the new code allocates ~130 MB more per concurrent request. Strong match.

Classification: **OOM**, triggered by the deploy-correlator path. The cascading-failure signature is a second-order effect (retry storm amplifying OOM pressure), not an independent path.

## Step 4: confirm with three independent signals

Three independent signals supporting the OOM-via-deploy hypothesis:

1. **Pod / kubelet events** (`fixtures/01-oom-cascade/pod_events.jsonl`): `OOMKilled` on 6 of 8 pods between 14:30 and 14:35.
2. **Metrics** (`fixtures/01-oom-cascade/metrics.json`): RSS at 510 MB at 14:30:12Z, immediately before the first OOMKill.
3. **Deploy diff** (`fixtures/01-oom-cascade/deploys.json`): commit `9f3a2c1` adds a `WebhookBuffer` struct that holds full payload bodies in memory; the per-request memory increase aligns with the observed RSS jump.

A fourth signal corroborates the cascade as second-order: **traces** (`fixtures/01-oom-cascade/traces.jsonl`) show retry-storm spans from `api-gateway` only after the first OOMKill, not before.

Hypothesis confidence is high. Three signals, three independent sources.

## Step 5: quantify blast radius

- **Users affected**: 100% of payment requests fail or hit retry latency. Read-only endpoints (`GET /payments/:id`) are also affected because they share the same pods.
- **Surfaces affected**: all payment endpoints (`POST /charge`, `POST /refund`, `GET /payments/:id`, webhook callbacks).
- **Business impact**: payment processing offline. SLO burn at 12x normal rate. In a real incident, the agent should also pull the revenue-per-minute figure from the finance dashboard the user has on hand.

## Step 6: propose mitigation before root cause

Ordered recommendation:

1. **`kubectl rollout undo`** `payments-api` to `v4.17.4` (the previous successful ReplicaSet). The deploy is the clearly implicated change, the rollback is one `kubectl rollout undo deployment/payments-api`, and the OOM behavior should clear within one pod-restart cycle (~30 s).
2. **ConfigMap-flag off** the webhook buffering path. Not applicable: the change shipped without a flag, which is itself a process gap to flag in the postmortem.
3. **Raise `resources.limits.memory`** from 512 MB to 1 GB as a stopgap if the rollback is delayed. Acknowledged that this does not address cause and will roughly double per-pod cost.
4. **Traffic-shift** to a previous region: not applicable, single-region service.
5. **Manual intervention** (`kubectl delete pod` to force fresh restarts): only if rollback and the limit bump are both blocked. Does not address cause; pods will OOM again within minutes.

Recommended action: **`kubectl rollout undo` to `v4.17.4`**.

## Step 7: hand off

Handoff payload for `postmortem-author`:

```json
{
  "timeline": [
    {"t": "2026-03-12T14:18:14Z", "event": "Deploy payments-api@v4.18.0 (commit 9f3a2c1)"},
    {"t": "2026-03-12T14:30:12Z", "event": "First OOMKill on pod payments-api-7d9c-x4k2"},
    {"t": "2026-03-12T14:32:00Z", "event": "T0: alert payments_api_error_rate > 5% fires"},
    {"t": "2026-03-12T14:33:45Z", "event": "api-gateway retry count 3.2x baseline"},
    {"t": "2026-03-12T14:36:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "OOM (deploy-triggered)",
      "confidence": "high",
      "evidence": ["OOMKilled events on 6/8 pods", "RSS at memory limit", "deploy diff adds ~130MB/req via WebhookBuffer"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert payments-api to v4.17.4",
  "open_questions": [
    "Why did the new WebhookBuffer path ship without a memory-footprint review?",
    "Why is there no per-request memory guardrail in the payment service load tests?"
  ]
}
```

## Why this is the OOM reference path

This example is the reference because it shows all four of these in one incident:

- A clear OOM signature at the pod level.
- A clear deploy-correlator signature (one change in window, on the failing surface).
- A clear cascading-failure signature as a second-order effect (api-gateway retry storm amplifying the primary failure).
- A safe mitigation (revert) that comes before root cause.

The methodology handles the layering: classify the primary path, note the cascade as second-order, recommend the revert. Skipping any of steps 1 through 4 produces a wrong answer (most commonly: classifying as cascading-failure and recommending circuit-breaker tuning, which does nothing to fix the OOM).

# Worked example 2: DNS resolution failure in `inventory-svc`

A realistic DNS incident in an internal HTTP service caused by a typo in a CoreDNS `Corefile` ConfigMap update. Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). Fixtures and replay test under `../fixtures/02-dns-resolution-failure/` and `../tests/replay_02_dns.py`.

## Scenario

- **Service**: `inventory-svc` (HTTP service, calls upstream `catalog-svc.internal:8080`).
- **Infra change at 2026-04-08 09:32 UTC**: `Corefile` ConfigMap updated to add a new `.internal` forward. The change introduced a typo in the upstream resolver IP, breaking resolution for `*.internal` zones intermittently (about 30% of queries returned `SERVFAIL`).
- **Alert fires at 09:47 UTC**: `inventory_svc_error_rate > 3%`.
- **No code deploy in window**. The change surface in step 2 contains only the ConfigMap update.

## Step 1: anchor the window

- **T0**: `2026-04-08T09:47:00Z` (alert fire).
- **Tnow**: `2026-04-08T09:53:00Z` (investigation triggered).
- **Window**: `[09:32:00Z, 09:53:00Z]`.

The 15-minute lead-in catches the 09:32 ConfigMap update. Without it, the change surface would return empty and the agent would risk misclassifying as a flaky-network external failure.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | No `inventory-svc` Deployment rollouts in window. |
| Cluster / HPA | `kube-system/coredns` Corefile ConfigMap updated | `09:32:18Z` | Added `forward .internal 10.100.0.53` line; typo: should be `10.100.0.5`. |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

One change in window: a cluster-level change to CoreDNS configuration. The fact that *zero* Deployment rollouts touched `inventory-svc` is itself a signal pushing away from the deploy-correlator path and toward an environmental cause.

## Step 3: classify against the four reference paths

- **OOM**: no `OOMKilled` events, memory metrics flat at baseline. No match.
- **DNS**: 47 `SERVFAIL` log lines for `catalog-svc.internal` in window, `getaddrinfo: Name or service not known` errors in client pod logs, recent change to CoreDNS configuration. Strong match.
- **Cascading-failure**: retry counts elevated but pattern is intermittent (correlates with the ~30% SERVFAIL rate), not the saturating-thread-pool shape of a true cascade. Match is weak; signature is downstream of DNS.
- **Deploy-correlator**: no `inventory-svc` Deployment rollouts in window. No match against `inventory-svc`. The CoreDNS configuration *change* is structurally a "change in window touching the failing surface", flagged as a secondary explanation.

Classification: **DNS**, triggered by a cluster-level change to CoreDNS. The agent does not need to invent a separate "config-correlator" path because the DNS path already encodes "recent change to the CoreDNS `Corefile` ConfigMap, kube-dns, `ndots`/search-domain, or a NetworkPolicy" as a confirming signal.

## Step 4: confirm with three independent signals

1. **Logs** (`fixtures/02-dns-resolution-failure/logs.jsonl`): 47 `SERVFAIL` log lines from `inventory-svc` calling `catalog-svc.internal`, plus 8 `getaddrinfo` errors in `payments-api` (a second consumer of the same zone).
2. **Metrics** (`fixtures/02-dns-resolution-failure/metrics.json`): DNS resolver error counter elevated 12x baseline. Application error rate at `inventory-svc` climbed from 0.2% to 3.4%. Memory and CPU flat.
3. **Cluster change** (`fixtures/02-dns-resolution-failure/deploys.json`): `kube-system/coredns` ConfigMap updated 15 minutes before T0, diff touches the `.internal` zone forward configuration.

A fourth signal: **traces** (`fixtures/02-dns-resolution-failure/traces.jsonl`) show failing spans with `dns.error=SERVFAIL` attribute, isolated to outbound calls to `*.internal` hostnames. Other outbound calls (e.g. to `api.stripe.com`) are unaffected. Asymmetric pattern is consistent with DNS scoped to one zone.

## Step 5: quantify blast radius

- **Users affected**: ~30% of requests that depend on `catalog-svc` fail or hit retry latency. Read-only catalog browsing degraded; checkout still works because the order-creation path does not call `catalog-svc` synchronously.
- **Surfaces affected**: all `inventory-svc` endpoints that hit `catalog-svc`, plus any other consumer of `*.internal` zones (at least `payments-api`).
- **Business impact**: catalog browsing degraded across the platform. SLO burn at 8x normal rate for read endpoints.

## Step 6: propose mitigation before root cause

1. **Revert** the `kube-system/coredns` ConfigMap to the version from `09:30:00Z`. The change is the clearly implicated single change in window, and reverting it takes one `kubectl apply` of the previous ConfigMap.
2. **Feature-flag off**: not applicable. No ConfigMap flag controls DNS resolution.
3. **Scale up**: not applicable. The failure is not capacity-bound (no `kubectl scale` / HPA ceiling / `resources.limits` change would help).
4. **Traffic-shift**: not applicable in this single-cluster topology. If the cluster had a sibling cluster on a different DNS configuration, shifting traffic there would be a viable stopgap.
5. **Manual intervention**: restarting CoreDNS pods does not help because they reload from the same broken ConfigMap. Restarting client pods does not help because they will hit the same resolver. Skip.

Recommended action: **revert the `kube-system/coredns` ConfigMap**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-04-08T09:32:18Z", "event": "kube-system/coredns ConfigMap updated (typo in .internal forward IP)"},
    {"t": "2026-04-08T09:34:00Z", "event": "First SERVFAIL log line from inventory-svc"},
    {"t": "2026-04-08T09:47:00Z", "event": "T0: alert inventory_svc_error_rate > 3% fires"},
    {"t": "2026-04-08T09:53:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "DNS (infra-change-triggered)",
      "confidence": "high",
      "evidence": ["47 SERVFAIL log lines in window", "DNS resolver error counter 12x baseline", "CoreDNS ConfigMap change 15min before T0 touches .internal zone", "outbound calls to non-.internal hosts unaffected"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert kube-system/coredns ConfigMap to the 09:30:00Z version",
  "open_questions": [
    "Why did the ConfigMap change ship without a syntax-check or a staged rollout?",
    "Why is there no end-to-end DNS resolution health check on the cluster's internal zones?"
  ]
}
```

## Why this is the DNS reference path

- The failure shape is intermittent (not 100%), which is the signature DNS issues often present and which the OOM and deploy-correlator paths cannot explain.
- The change surface contains an infrastructure change rather than a code deploy, which forces the agent to widen its mental model beyond "what code shipped".
- The signal asymmetry (outbound calls to `*.internal` fail, calls to `*.com` work) is diagnostic for DNS scoped to a zone, and would be invisible without the trace-level attribute.

Skipping step 4 (three independent signals) is the most likely failure mode here: an agent that classifies on logs alone may over-attribute the failure to the consumer service rather than the resolver.

# Worked example 3: cascading failure from upstream slowdown

A realistic cascading-failure incident: an upstream dependency (`ledger-svc`) slows down due to a DB query plan flip, the caller (`payments-api`) saturates its thread pool waiting for it, and the API gateway retry budget amplifies the failure into a wider error rate. **No code deploy, no DNS, no OOM, no infra change in window.** Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). Fixtures and replay test under `../fixtures/03-cascading-failure-retry-storm/` and `../tests/replay_03_cascade.py`.

## Scenario

- **Service**: `payments-api` (depends synchronously on `ledger-svc`).
- **Upstream**: `ledger-svc`'s P99 latency drifts from ~50 ms to ~180 ms over the morning; at 11:00 UTC it crosses a knee in the query-planner cost model and jumps to ~620 ms. The underlying cause is a DB table that grew past the index-only threshold, not a code change. No `ledger-svc` deploy in days.
- **Cascade**: `payments-api`'s thread pool (50 workers, ~50 ms baseline per request) saturates once requests start taking ~600 ms. Queue depth grows. New requests sit in queue then fail with `503 service unavailable`. The `api-gateway` retries (up to 3, 500 ms backoff). Retries hit the still-saturated pool, amplifying load.
- **Alert at 11:08 UTC**: `payments_api_latency_p99 > 1000ms`.
- The methodology must classify this as **cascading-failure** and recommend **circuit-breaker / traffic-shift**, not revert. There is no change to revert.

## Step 1: anchor the window

- **T0**: `2026-03-20T11:08:00Z` (alert fire).
- **Tnow**: `2026-03-20T11:15:00Z`.
- **Window**: `[10:53:00Z, 11:15:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | No rollouts in window. Most recent `payments-api` deploy was 9 days ago. |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

**Zero changes in window.** Per the methodology, this is itself a strong signal: the failure is likely external (capacity, dependency, upstream provider) rather than a self-inflicted regression. The agent should *not* hunt for a phantom deploy and *should* widen the dependency-health check.

## Step 3: classify against the four reference paths

- **OOM**: `payments-api` RSS p95 flat at ~210 MB against a 512 MB limit. No `OOMKilled` events. No match.
- **DNS**: no `SERVFAIL` / `getaddrinfo` errors in logs. Outbound calls resolve normally. No match.
- **Cascading-failure**: `payments-api` thread-pool saturation warnings in logs, queue depth growing, P99 latency tripled, `api-gateway` retry rate spiked from baseline to 8x. Upstream `ledger-svc` P99 latency jumped from ~50 ms to ~620 ms inside the window. **Strong match.**
- **Deploy-correlator**: no deploy in window. No match.

Classification: **cascading-failure**. The root degradation is upstream (`ledger-svc`); the failure propagates through `payments-api` thread-pool saturation and is amplified by gateway retries.

## Step 4: confirm with three independent signals

1. **Metrics** (`fixtures/03-cascading-failure-retry-storm/metrics.json`): `upstream_latency_p99_ms` (the latency `payments-api` observes calling `ledger-svc`) climbed from 52 ms at 10:53 to 624 ms at 11:08. `payments_api_latency_p99` followed it up to 1170 ms. `gateway_retry_rate_rps` spiked from 6 baseline to 48 at T0 (8x).
2. **Application logs** (`fixtures/03-cascading-failure-retry-storm/logs.jsonl`): repeated `thread pool saturated, queue depth 142` warnings from `payments-api` starting at 11:03; `ledger client request timeout` errors from 11:04 onward; gateway 503 + retry attempt logs.
3. **Traces** (`fixtures/03-cascading-failure-retry-storm/traces.jsonl`): the slow span is consistently the `ledger-svc` hop. Pre-window baseline traces show `ledger-svc` at ~50 ms; in-window traces show 580 to 920 ms on the same span, with `payments-api` accumulating queued requests behind it.

Hypothesis confidence is high. Three independent signal sources, plus a fourth from the change-audit channel: "no changes in window" is itself a positive signal that the failure is environmental, not regression-driven.

## Step 5: quantify blast radius

- **Users affected**: P99 latency crossed 1 s for all payment requests. Around 12% of requests time out at the gateway after exhausting retry budget. Roughly 88% complete, but slowly.
- **Surfaces affected**: every endpoint on `payments-api` that touches the `ledger-svc` synchronous path (which is most of them). Downstream consumers of `payments-api` see slower responses; some upstream callers degrade silently.
- **Business impact**: latency-degraded payments. SLO burn on `payments-api` P99 latency SLO at ~6x normal. Revenue impact softer than the OOM example because most requests still complete.

## Step 6: propose mitigation before root cause

There is no change to revert. The mitigation list is therefore different from a deploy-triggered incident.

1. **Open the circuit breaker** on the `payments-api` → `ledger-svc` call path. Fail fast (return cached / degraded responses where possible) instead of letting requests queue and time out. Stops the retry storm from amplifying the upstream degradation. Does not fix `ledger-svc` itself.
2. **Shed traffic** from `payments-api`: route non-critical payment operations (e.g. async batch jobs that hit `ledger-svc`) away from the production pool, or have the gateway prefer a healthy replica if one exists. Buys time while `ledger-svc` recovers.
3. **Scale `ledger-svc` reads** (if the workload is read-heavy and the slowdown is on read paths) as a secondary stopgap. Acknowledged that this does not fix the underlying query-plan issue; it only adds capacity to absorb the slower per-request cost.
4. **Revert**: not applicable, no change in window.
5. **Manual intervention** (`kubectl rollout restart deployment/payments-api` to drain the saturated queue): only if circuit-breaker and traffic-shift are blocked. Does not address cause; the pool will saturate again within minutes if `ledger-svc` is still slow.

Recommended action: **open the circuit breaker on the `ledger-svc` call path**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-03-20T10:53:00Z", "event": "ledger-svc upstream latency p99 ~52ms (baseline)"},
    {"t": "2026-03-20T11:00:00Z", "event": "ledger-svc latency knee, p99 jumps from 180ms to 480ms"},
    {"t": "2026-03-20T11:03:30Z", "event": "First 'thread pool saturated' warning from payments-api"},
    {"t": "2026-03-20T11:04:18Z", "event": "First 'ledger client request timeout' error"},
    {"t": "2026-03-20T11:08:00Z", "event": "T0: alert payments_api_latency_p99 > 1000ms fires"},
    {"t": "2026-03-20T11:15:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Cascading-failure (upstream slowdown)",
      "confidence": "high",
      "evidence": ["upstream ledger-svc p99 climbed 52ms -> 624ms", "payments-api thread pool saturation in logs", "gateway retry rate 8x baseline", "no changes in window"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Open circuit breaker on payments-api -> ledger-svc call path",
  "open_questions": [
    "What caused the ledger-svc query-plan flip at ~11:00? Index bloat? Stats staleness? Table growth past planner threshold?",
    "Why did payments-api degrade rather than fail open? Should circuit-breaker default-state be reconsidered?",
    "Is there a tested traffic-shift path for payments-api, or is this the first time it's needed?"
  ]
}
```

## Why this is the cascading-failure reference path

- The "no change in window" signal is dispositive. An agent that biases toward "find the deploy that broke things" will hunt for a phantom deploy and waste time.
- The cascade is detectable via *upstream latency growth* even when retry rate isn't dramatic, which is the second signal pattern the methodology's `_has_cascade_signature` looks for.
- The mitigation ordering is genuinely different (circuit-breaker first, not revert), which the example proves the methodology produces correctly.

Skipping step 2 here is the most likely failure mode: an agent that doesn't enumerate "zero changes" as a signal will misclassify the incident as outside-reference-paths and escalate when in fact the cascading-failure path is the clean fit.

# Worked example 4: pure deploy-correlator (serialization regression)

A realistic deploy-correlator incident: a deploy ships a serialization change that returns binary-encoded responses on an endpoint whose downstream consumers expect JSON. **Not OOM, not DNS, not a cascade.** A clean deploy-correlator signature: one change in window, on the failing surface, with the deploy diff explaining the failure mode. Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). Fixtures and replay test under `../fixtures/04-deploy-correlator-serialization/` and `../tests/replay_04_deploy_correlator.py`.

## Scenario

- **Service**: `checkout-api`. The `GET /cart/:id` endpoint returns the user's current cart contents to multiple downstream consumers (web frontend, mobile apps, partner widgets).
- **Deploy at 2026-02-15 13:12 UTC**: shipped a "performance" change that switched the `/cart/:id` response from JSON to a binary Protobuf encoding behind the same `Content-Type: application/json` header. The change was wrapped in a feature flag during development but the flag was removed in the final PR. None of the downstream consumers can decode Protobuf, so they fail to parse the response and surface errors back to users.
- **Alert at 13:25 UTC**: `checkout_api_error_rate > 2%`.
- **What the methodology must produce**: classify as **deploy-correlator** (not OOM, not DNS, not cascade), recommend reverting the deploy. The deploy diff explicitly touches the failing surface (`/cart/:id` serialization), which is the M4 confirmation-bias guard satisfied.

## Step 1: anchor the window

- **T0**: `2026-02-15T13:25:00Z` (alert fire).
- **Tnow**: `2026-02-15T13:30:00Z`.
- **Window**: `[13:10:00Z, 13:30:00Z]`.

The 15-minute lead-in catches the 13:12 deploy, 13 minutes before T0.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | Deploy `checkout-api@v6.4.0` | `13:12:08Z` | Commit `d1c4e22`. Diff: `internal/serializer/cart.go` switches response encoding from JSON to Protobuf for the `/cart/:id` endpoint. |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | The PR description mentions a flag was removed before merge. |
| CronJob | (none) | | |

One change in window. The diff explicitly touches the endpoint the consumers are calling, which is the M4 (deploy-correlator confirmation bias) guard: deploy-correlator is only the right classification when the deploy diff actually intersects the failing surface, not just because the timing lines up.

## Step 3: classify against the four reference paths

- **OOM**: RSS p95 flat at ~140 MB against a 512 MB limit. No `OOMKilled` events. No match.
- **DNS**: no `SERVFAIL` / `getaddrinfo` errors. Outbound calls resolve normally. No match.
- **Cascading-failure**: gateway retry rate goes from 3 rps baseline to ~5 rps after the deploy. That's a 1.7x increase, well below the cascade threshold (3x). Upstream latency unchanged (no slow dependency). No match.
- **Deploy-correlator**: 13:12 deploy is the only change in window; diff touches `/cart/:id` serialization; downstream consumers report `unexpected EOF` / `failed to parse response` errors in the window. **Strong match.**

Classification: **deploy-correlator**.

## Step 4: confirm with three independent signals

1. **Metrics** (`fixtures/04-deploy-correlator-serialization/metrics.json`): `error_rate_pct` jumps from 0.3% baseline to 4.8% at T0; `request_rate_rps` flat (the failure is not amplifying traffic). `gateway_retry_rate_rps` only 1.7x baseline (no cascade signature). RSS flat (no OOM signature).
2. **Logs** (`fixtures/04-deploy-correlator-serialization/logs.jsonl`): downstream consumer services log `unexpected EOF parsing JSON` and `failed to parse application/json response from checkout-api` starting 90 s after the deploy.
3. **Deploy diff** (`fixtures/04-deploy-correlator-serialization/deploys.json`): commit `d1c4e22` switches the `/cart/:id` response encoding from JSON to Protobuf while keeping the `Content-Type: application/json` header.
4. **Traces** (`fixtures/04-deploy-correlator-serialization/traces.jsonl`): `checkout-api` spans complete successfully (the service returns 200s); the failure is at the *consumer* span, which decodes the response and errors out. This is diagnostic: a server-side OOM or upstream-cascade incident would have failing spans on the `checkout-api` hop.

Four independent signal sources. Confidence high.

## Step 5: quantify blast radius

- **Users affected**: every user whose client calls `GET /cart/:id` and parses the response as JSON. Conservatively 100% of cart fetches, but the failure is silent for some clients that accept arbitrary bytes and only surface when the cart UI tries to render.
- **Surfaces affected**: `/cart/:id` endpoint, plus any code path that depends on the cart payload being JSON-decodable. Web frontend cart page, mobile app cart screen, partner-widget cart integration. The endpoint itself responds 200.
- **Business impact**: checkout funnel partially blocked (users cannot see their cart contents). SLO impact on the checkout success-rate SLO, not on the `checkout-api` availability SLO.

## Step 6: propose mitigation before root cause

1. **`kubectl rollout undo`** `checkout-api` to `v6.3.7` (the previous successful ReplicaSet, commit `b09d318`). The deploy is the clearly implicated change, the rollback is one `kubectl rollout undo deployment/checkout-api`, and downstream consumers will start decoding successfully on the next request.
2. **ConfigMap-flag off** the new serialization path. Not applicable: the PR description confirms the flag was removed before merge. Adding the flag back would itself be a code change with its own rollout cycle and is slower than rolling back.
3. **`kubectl scale` / HPA**: not applicable. The failure is not capacity-bound.
4. **Traffic-shift**: not applicable. No canary / blue-green / regional split is configured for this service.
5. **Manual intervention** (kill request mid-flight): does not address cause. Skip.

Recommended action: **`kubectl rollout undo` to `v6.3.7`**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-02-15T13:12:08Z", "event": "Deploy checkout-api@v6.4.0 (commit d1c4e22)"},
    {"t": "2026-02-15T13:13:42Z", "event": "First 'failed to parse application/json response' from web frontend"},
    {"t": "2026-02-15T13:25:00Z", "event": "T0: alert checkout_api_error_rate > 2% fires"},
    {"t": "2026-02-15T13:30:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Deploy-correlator (serialization regression)",
      "confidence": "high",
      "evidence": ["deploy v6.4.0 in window touches /cart/:id encoding", "consumer-side parse errors in logs", "checkout-api spans return 200 (failure is consumer-side)", "no OOM / DNS / cascade signature"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert checkout-api to v6.3.7",
  "open_questions": [
    "Why was the feature flag removed in the final PR? Was it reviewed?",
    "Why didn't the contract test between checkout-api and its consumers catch the encoding switch?",
    "Are there other Content-Type-honest serialization paths in the codebase that could regress the same way?"
  ]
}
```

## Why this is the deploy-correlator reference path

- It is a *clean* deploy-correlator: no OOM, no DNS, no cascade. The methodology must reach deploy-correlator without being tempted into OOM-via-deploy (which is what example 01 tests).
- It exercises the M4 confirmation-bias guard: the deploy isn't just temporally correlated; the diff explicitly touches the failing surface, and the failure shape (consumer-side parse errors with successful 200s upstream) matches the diff's stated change. An agent that classified on timing alone would be vulnerable.
- It exercises the case where mitigation is unambiguously revert (one change, isolated, reversible), in contrast to example 07 where a bundled deploy makes revert blast radius unsafe.

# Worked example 5: outside reference paths (third-party rate limit)

A failure that doesn't fit any of the four reference paths: a third-party payment provider rate-limits the platform's account. No OOM, no DNS, no internal cascade, no deploy. The methodology must **classify as outside-reference-paths and escalate** rather than force-fit one of the four canonical classifications. Exercises FAILURE_MODES.md rule M1. Mirrors the seven methodology steps in [`../SKILL.md`](../SKILL.md). Fixtures and replay test under `../fixtures/05-outside-reference-paths-third-party-rate-limit/` and `../tests/replay_05_outside_paths.py`.

## Scenario

- **Service**: `payments-api` (a Kubernetes Deployment in your cluster) calls the external `api.stripe.com` for charge processing.
- **Upstream**: at 2026-05-04 16:38 UTC, the payment provider's rate limiter starts returning `HTTP 429 Too Many Requests` on a fraction of charge requests. Their public status page shows no incident; account-level rate limits have changed silently. Around 35% of `POST /v1/charges` calls fail with 429.
- **Alert at 16:44 UTC**: `payments_api_error_rate > 2%`.
- **Methodology must produce**: classification `outside-reference-paths`, escalation flagged with reason M1 ("failure classified outside the four reference paths"). No mitigation recommendation beyond traffic-shift / feature-flag without a human approver.

## Step 1: anchor the window

- **T0**: `2026-05-04T16:44:00Z`.
- **Tnow**: `2026-05-04T16:50:00Z`.
- **Window**: `[16:29:00Z, 16:50:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | No `payments-api` Deployment rollouts in window. Most recent `payments-api` rollout was 3 days ago. |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

Zero changes in window. Per the methodology, this is itself a signal that the failure is likely external (upstream provider, certificate expiry, capacity drift) rather than self-inflicted. The agent should hunt for an *external* root cause.

## Step 3: classify against the four reference paths

- **OOM**: `payments-api` pod RSS p95 flat at ~190 MB against a 512 MB `resources.limits.memory`. No `OOMKilled` events. No match.
- **DNS**: `api.stripe.com` resolves successfully (verified by the trace `dns_target` attribute on the failing spans). No `SERVFAIL` / `getaddrinfo` errors. No match.
- **Cascading-failure**: gateway retry rate doubled (3 to 6 rps) but is below the 3x cascade threshold; the failure is at the upstream-provider hop, not in the internal dependency graph. No match for the cascade signature the methodology defines.
- **Deploy-correlator**: no deploy in window. No match.

Classification: **outside-reference-paths**. The signals support a third-party rate-limit hypothesis (HTTP 429 responses from `api.stripe.com` visible in logs and traces), but the methodology's four reference paths do not cover external-provider rate limiting.

## Step 4: confirm with three independent signals

1. **Logs** (`fixtures/05-outside-reference-paths-third-party-rate-limit/logs.jsonl`): `upstream returned 429` errors from `payments-api` against `api.stripe.com`, with the `X-Rate-Limit-Remaining: 0` and `Retry-After` headers captured in the log.
2. **Metrics** (`fixtures/05-outside-reference-paths-third-party-rate-limit/metrics.json`): `error_rate_pct` climbs from 0.3 to 4.2 across the window; internal `rss_bytes_p95` flat; `dns_resolver_errors_rps` flat at baseline.
3. **Traces** (`fixtures/05-outside-reference-paths-third-party-rate-limit/traces.jsonl`): failing spans terminate at the `stripe` hop with `http.status_code=429` attribute. Other outbound calls (e.g. to internal services) succeed normally.

Three independent sources. Hypothesis: external rate limit on the payment provider's API. Confidence is high *on the hypothesis*; the methodology is honest that the classification does not fit the four reference paths.

## Step 5: quantify blast radius

- **Users affected**: roughly 35% of payment requests fail with the user-visible "could not process payment, please try again" error. Reads / other endpoints unaffected.
- **Surfaces affected**: `POST /v1/charges` path only (the rate limit appears scoped to charges). Refunds and lookups still work.
- **Business impact**: payment throughput cut by ~35%. SLO burn on `payments-api` error-rate SLO. Direct revenue impact for the duration.

## Step 6: propose mitigation before root cause

Because the classification is outside the four reference paths, the methodology constrains the mitigation set: traffic-shift and feature-flag are allowed unilaterally; anything broader requires a human approver per the M1 escalation rule.

1. **Revert**: not applicable, no change in window to revert.
2. **Feature-flag off** the charge flow and surface a maintenance message to users. Allowed without escalation per M1, with the caveat that this is degraded service, not mitigation.
3. **Traffic-shift**: not applicable unless a secondary payment provider is wired up. If one exists, route charges to it.
4. **Scale**: not applicable. `kubectl scale` / raising the HPA ceiling / raising `resources.limits` would not help — the limit is external, not internal capacity.
5. **Manual intervention**: not applicable.

Anything beyond the above (e.g. contacting the provider, lifting the rate limit, retrying with backoff at higher concurrency) requires a human in the loop because the methodology cannot rule out making the situation worse (more requests against a rate limiter compounds the problem).

Recommended next action: **escalate to a human** with the rate-limit hypothesis and the partial-mitigation options.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-05-04T16:38:14Z", "event": "First HTTP 429 from api.stripe.com on POST /v1/charges"},
    {"t": "2026-05-04T16:41:02Z", "event": "Error rate crosses 1%"},
    {"t": "2026-05-04T16:44:00Z", "event": "T0: alert payments_api_error_rate > 2% fires"},
    {"t": "2026-05-04T16:50:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Outside reference paths (third-party rate limit)",
      "confidence": "high (on hypothesis), classified as outside-reference-paths",
      "evidence": ["HTTP 429 from api.stripe.com in logs and traces", "X-Rate-Limit-Remaining: 0 header", "no changes in window", "internal telemetry healthy (no OOM, no DNS, no cascade signature)"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Escalate to a human. Partial options: feature-flag charges off with maintenance message; contact provider; investigate retry/backoff strategy under human oversight.",
  "escalate_to_human": true,
  "escalation_reasons": ["M1: failure classified outside the four reference paths"],
  "open_questions": [
    "Did the provider change rate limits on our account silently, or did we cross a threshold via organic growth?",
    "Is there a secondary payment provider we can shift charges to?",
    "What does the provider's status page actually show, and is there a support channel response time we can quote?"
  ]
}
```

## Why this is the outside-reference-paths reference example

- It exercises the M1 escalation path explicitly. The methodology must refuse to force-fit the failure into OOM/DNS/cascade/deploy-correlator just because the agent is *capable* of producing a confident-looking answer.
- It distinguishes between *hypothesis confidence* (high: 429s in logs + traces + headers) and *classification confidence* (low: outside the four reference paths). The handoff format makes that distinction explicit so the postmortem-author can carry it forward.
- It models the partial-mitigation set: feature-flag and traffic-shift are allowed unilaterally; anything else requires a human. This is the methodology's safety net for incidents it does not fully understand.

# Worked example 6: ambiguous T0 (slow-burn memory leak)

A failure where T0 is genuinely unclear: the operator says "the service has been slow all morning", error rate has been creeping up for hours, no single alert fire-time pinpoints the start. The methodology must **flag T0 as ambiguous, classify with reduced confidence, and escalate with M2** so a human re-runs the investigation against a widened window. Exercises FAILURE_MODES.md rule M2. Fixtures and replay test under `../fixtures/06-ambiguous-t0-slow-burn/` and `../tests/replay_06_ambiguous_t0.py`.

## Scenario

- **Service**: `recommendations-api`. Operator reports at 12:15 that "recs have been slow all morning, finally got around to looking".
- **Underlying issue**: a memory leak introduced by a Deployment rollout three days ago. Pod RSS has been climbing steadily for ~72 hours, finally crossing the threshold where GC pauses started showing up around 09:30 today. Error rate ticked up from 0.3% to 1.4% over the morning, well below the 2% alert threshold.
- **No alert fired**. The investigation was triggered manually by the operator.
- **Methodology must produce**: classification (likely OOM, given the signature), but with the **t0_ambiguous flag set**, escalation reasons including **M2**, and the recommended mitigation must include a "re-check after widening window" step before any irreversible action.

## Step 1: anchor the window

The operator-reported symptom ("slow all morning") does not pinpoint a single T0. Per SKILL.md step 1:

> If T0 is ambiguous (e.g. "slow all morning"), pick the earliest unambiguous signal and note the ambiguity in the timeline. Do not silently round.

Earliest unambiguous signal: the first GC pause warning at `2026-04-19T09:32:14Z`. T0 set there with `t0_ambiguous = true`.

- **T0**: `2026-04-19T09:32:14Z` (earliest unambiguous signal, ambiguity flagged).
- **Tnow**: `2026-04-19T12:15:00Z`.
- **Window**: `[09:17:14Z, 12:15:00Z]`.

The window is nearly 3 hours wide instead of the typical 20 to 30 minutes. The 15-minute lead-in is structurally insufficient: the actual triggering deploy is 3 days outside the window. The methodology must flag this.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none in window) | | Most recent `recommendations-api` rollout: `v3.8.0` at `2026-04-16T15:22:00Z`, ~70 hours before T0. **Outside the window.** |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none in window) | | |

Zero changes in the window. **This is a misleading "zero changes" signal**: it suggests an environmental cause when the actual cause is a 3-day-old rollout whose effect built up slowly. The methodology's standard "no changes → external cause" pivot would be wrong here, which is exactly why M2 escalation matters.

## Step 3: classify against the four reference paths

- **OOM**: pod RSS p95 climbs from baseline ~120 MB to ~485 MB over the window (`resources.limits.memory` 512 MB). GC pause durations grow from ~20 ms to ~180 ms. No `OOMKilled` events yet, but the trajectory points there within an hour. **Strong match for OOM signature** (RSS >= 90% of limit).
- **DNS**: no `SERVFAIL` / `getaddrinfo` errors. No match.
- **Cascading-failure**: no cascade signature. Upstream latency flat. No match.
- **Deploy-correlator**: no Deployment rollout in window. No match against the windowed surface (but the actual cause is a rollout 3 days outside the window).

Classification: **OOM**. The signature is strong even with the ambiguous T0.

## Step 4: confirm with three independent signals

1. **Metrics**: `rss_bytes_p95` 119 MB at 09:17 climbs to 485 MB at 12:15 (95% of the 512 MB `resources.limits.memory`). GC pause p99 from 22 ms to 184 ms across the same window.
2. **Logs**: `GC pause exceeded soft threshold` warnings starting 09:32:14Z, increasing in frequency through the window.
3. **Traces**: latency tail on `recommendations-api` spans grows from p99 ~80 ms at 09:30 to p99 ~310 ms at 12:00, consistent with GC pauses stealing serving time.

Three signals, three sources. But all of them are consistent with a *slow* OOM trajectory, not a sudden one. The M2 escalation matters because the change that *caused* the leak is invisible to this investigation.

## Step 5: quantify blast radius

- **Users affected**: error rate 1.4% at Tnow, slowly climbing. Latency degradation visible to all users of `recommendations-api`.
- **Surfaces affected**: every endpoint on `recommendations-api`. Recommendation cards on home page, related items on product pages, email recommendations pipeline.
- **Business impact**: degraded recommendations quality, lower click-through rate. No hard outage yet but trajectory points to OOMKills within the hour.

## Step 6: propose mitigation before root cause

Because T0 is ambiguous and the actual triggering change is outside the window, the methodology's mitigation list has a critical addition: **re-check with a widened window before executing**.

1. **Re-check with a widened window first.** Re-run the investigation with `T0 = T0 - 4h` or wider, until the change-surface bisection returns a meaningful candidate. The 3-day-old deploy will appear. *This must precede any irreversible mitigation.*
2. **Revert the implicated change** with `kubectl rollout undo` once the widened-window investigation identifies it. The current investigation has no change to revert.
3. **Scale up** by raising `resources.limits.memory` from 512 MB to 1 GB as a short-term stopgap to delay the OOMKill trajectory while step 1 runs. Acknowledged this does not address cause.
4. **Manual intervention** (rolling restart of `recommendations-api` pods to reset RSS): only if step 3 is blocked. The leak will resume; this is bridge time, not mitigation.

Recommended action: **re-run with widened window, then `kubectl rollout undo` the identified change**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-04-19T09:17:14Z", "event": "Window start (15 min before earliest unambiguous signal)"},
    {"t": "2026-04-19T09:32:14Z", "event": "T0 (ambiguous): first GC pause exceeded soft threshold"},
    {"t": "2026-04-19T10:45:00Z", "event": "Error rate crosses 1%"},
    {"t": "2026-04-19T12:15:00Z", "event": "Tnow: operator-triggered investigation"}
  ],
  "ranked_hypotheses": [
    {
      "path": "OOM (slow trajectory, cause outside window)",
      "confidence": "medium (T0 ambiguous; widened-window investigation required to identify causal change)",
      "evidence": ["RSS p95 reached 95% of limit", "GC pause p99 8x baseline", "latency tail growing"]
    }
  ],
  "t0_ambiguous": true,
  "escalate_to_human": true,
  "escalation_reasons": ["M2: T0 is ambiguous; re-run with a widened window before acting on the recommended mitigation"],
  "mitigation_taken": null,
  "mitigation_recommended": "Re-run investigation with widened window (T0 - 4h or more) to identify the causal change, then revert that change.",
  "open_questions": [
    "When did the leak actually start? Likely deploy v3.8.0 at 2026-04-16T15:22:00Z, but confirm with widened-window run.",
    "Is there a per-request memory allocation chart that would distinguish a leak from organic growth?",
    "Why did no alert fire in the 3-day climb? Is the alert threshold (2% error rate) the wrong shape for a slow leak?"
  ]
}
```

## Why this is the ambiguous-T0 reference example

- It's the case where the methodology's *default* T0-picking heuristic ("first user-visible symptom") produces a window that misses the actual cause. Without M2 escalation, the methodology would confidently recommend "no change to revert, must be environmental" and be wrong.
- It exercises the slow-trajectory vs. sudden-failure distinction. The OOM signature is present but evolving over hours, not minutes; the standard 15-minute lead-in is insufficient.
- It models the gap between *what the operator can see* and *what the agent can act on*. Escalation is the safety net: the agent surfaces the partial picture, flags the gap, recommends a re-run.

# Worked example 7: blast-radius asymmetric revert (deploy bundle)

A deploy bundle that shipped six unrelated changes in one release train. Only one of them is causing the incident, but reverting the bundle reverts all six. The methodology must **classify the path correctly and rank revert as the top mitigation**, but then **escalate with M3** because the revert's blast radius exceeds the incident's blast radius. Exercises FAILURE_MODES.md rule M3. Fixtures and replay test under `../fixtures/07-blast-radius-asymmetric-revert/` and `../tests/replay_07_blast_radius.py`.

## Scenario

- **Service**: `notifications-svc` (a Kubernetes Deployment). Sends transactional emails, SMS, and push notifications; SMS goes through the external Twilio provider.
- **Rollout at 2026-06-02 09:48 UTC**: a weekly release train rolled out a new ReplicaSet bundling six unrelated PRs (commit `8b3f4d1`, version `v8.12.0`):
  1. SMS provider integration update (Twilio API v2 migration).
  2. Email template refactor.
  3. Push notification batching optimization.
  4. Removal of deprecated SES region fallback.
  5. New /healthz endpoint.
  6. Dependency upgrades (express 4.x → 5.x).
- **The actual failure**: the SMS integration update (PR 1 of 6) shipped with an invalid sender ID format that the new Twilio API rejects. SMS delivery is now failing 100% of the time. The other five changes are fine.
- **Alert at 10:03 UTC**: `notifications_sms_error_rate > 50%`.
- **Methodology must produce**: classification deploy-correlator (or close), `kubectl rollout undo` as the top mitigation, **escalation with M3** because rolling back `v8.12.0` reverts the other five working changes too.

## Step 1: anchor the window

- **T0**: `2026-06-02T10:03:00Z`.
- **Tnow**: `2026-06-02T10:10:00Z`.
- **Window**: `[09:48:00Z, 10:10:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | Rollout `notifications-svc@v8.12.0` | `09:48:23Z` | Commit `8b3f4d1`. **Bundle of 6 PRs.** Diff summary lists Twilio v2 migration, email template refactor, push batching, SES fallback removal, /healthz, express 5.x upgrade. |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

One change in window, but the change is a bundle of six. This is the M3 trigger.

## Step 3: classify against the four reference paths

- **OOM**: pod RSS p95 flat at ~210 MB against the 512 MB `resources.limits.memory`. No `OOMKilled` events. No match.
- **DNS**: no `SERVFAIL` / `getaddrinfo` errors. Outbound to Twilio resolves fine. No match.
- **Cascading-failure**: no retry storm, no upstream-latency growth. The failure is at the external SMS provider hop with HTTP 400, not slowness. No match.
- **Deploy-correlator**: rollout in window; diff explicitly touches the SMS path (Twilio API v2 migration line in the bundle); SMS error rate jumped from 0 to 100% within 90 seconds of the rollout. **Match.**

Classification: **deploy-correlator**, specifically the Twilio API v2 migration PR within the bundle. The other five PRs are not implicated by any signal.

## Step 4: confirm with three independent signals

1. **Logs** (`fixtures/07-blast-radius-asymmetric-revert/logs.jsonl`): `Twilio API rejected request: 400 invalid 'from' field, expected E.164 format` errors starting 09:49:51Z.
2. **Metrics** (`fixtures/07-blast-radius-asymmetric-revert/metrics.json`): `sms_error_rate_pct` jumps from 0.0% baseline to 100% at 09:50; `email_error_rate_pct` and `push_error_rate_pct` flat (proving the failure is SMS-scoped, not service-wide).
3. **Deploy diff** (`fixtures/07-blast-radius-asymmetric-revert/deploys.json`): bundle includes Twilio v2 migration line that matches the failing surface.
4. **Traces** (`fixtures/07-blast-radius-asymmetric-revert/traces.jsonl`): SMS-sending spans terminate at the `twilio` hop with `http.status_code=400`; email-sending and push-sending spans complete successfully.

Four independent signal sources. Confidence high.

## Step 5: quantify blast radius

- **Users affected**: every user who would receive an SMS notification. Email and push channels are unaffected.
- **Surfaces affected**: SMS sending only. Other notification channels healthy.
- **Business impact**: SMS-dependent flows broken (2FA SMS, OTP, delivery alerts). Email and push fallbacks compensate partially for some flows but not for 2FA.

**Revert blast radius**: `kubectl rollout undo` to `v8.11.4` would also roll back five working changes (email template refactor, push batching, SES fallback removal, /healthz, express 5.x). Two of those (the SES removal and express 5.x upgrade) have downstream consumers already depending on the new state. The rollback is asymmetric: it removes a 100%-broken feature (SMS) at the cost of regressing five healthy features.

## Step 6: propose mitigation before root cause

1. **Revert** `notifications-svc` to `v8.11.4` with `kubectl rollout undo`. This is the methodology's top recommendation, BUT it carries the M3 caveat: blast radius exceeds the incident's blast radius. **Requires a human approver.**
2. **Feature-flag off**: not applicable; the Twilio v2 migration shipped without a ConfigMap flag.
3. **Scale**: not applicable.
4. **Traffic-shift**: not applicable.
5. **Manual intervention**: not applicable.

A human-approved alternative the methodology surfaces but does not unilaterally recommend: **forward-fix** by shipping a targeted patch to the SMS sender ID format (a one-line fix), rather than rolling back the bundle. Forward-fix avoids the regression of the five healthy changes but introduces deployment-pipeline risk (the bundle's other changes have been live for ~15 min; rolling back and re-shipping the five working changes is operationally noisier).

Recommended action: **`kubectl rollout undo` + escalate to human for blast-radius approval, with forward-fix surfaced as the cleaner alternative**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-06-02T09:48:23Z", "event": "Deploy notifications-svc@v8.12.0 (bundle of 6 PRs, commit 8b3f4d1)"},
    {"t": "2026-06-02T09:49:51Z", "event": "First Twilio 400 invalid 'from' field error"},
    {"t": "2026-06-02T10:03:00Z", "event": "T0: alert notifications_sms_error_rate > 50% fires"},
    {"t": "2026-06-02T10:10:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Deploy-correlator (Twilio v2 migration within bundle)",
      "confidence": "high",
      "evidence": ["bundle diff includes Twilio v2 migration line", "sms-only error pattern, email/push unaffected", "Twilio 400 invalid 'from' field errors in logs", "traces fail at twilio hop"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Revert notifications-svc to v8.11.4 (M3 escalation flagged: bundle of 6 changes, revert reverts all six)",
  "alternative_mitigation": "Forward-fix: ship a targeted patch to the SMS sender ID format. Preserves the five healthy changes in the bundle but takes longer than revert.",
  "escalate_to_human": true,
  "escalation_reasons": ["M3: recommended revert affects 6 bundled changes, broader blast radius than the incident; requires a human approver before executing"],
  "open_questions": [
    "Why did the SMS sender ID format change land without a contract test against the Twilio v2 sandbox?",
    "Should the release train be split into one-PR-per-deploy for high-risk integration changes like SMS provider migrations?",
    "Is there a way to revert a single commit out of a bundle, or does the deploy pipeline require atomic version rollback?"
  ]
}
```

## Why this is the asymmetric-revert reference example

- It models a real-world operational tension: revert is often correct *at the methodology level* but unsafe *at the operational level* when the change being reverted is bundled. The methodology must surface this conflict, not gloss it.
- It exercises M3 explicitly, with `bundle_size = 6` in the deploy fixture and a methodology that escalates when bundle size > 1 on a revert recommendation.
- It models the *alternative-mitigation* slot in the handoff: when the top recommendation is conflicted, the methodology should surface a cleaner alternative (forward-fix) rather than only flagging the conflict and stopping.

# Worked example 8: deploy-correlator confirmation bias (M4)

A Deployment rollout and an RBAC change land in the same window. The rollout is innocent. The RBAC change is the actual cause. **The methodology must NOT classify as `deploy-correlator` from timing alone**: the rollout diff does not touch the failing surface, so it should not satisfy the M4 guard, and the classification falls through to `outside-reference-paths` with escalation. Exercises FAILURE_MODES.md rule M4. Fixtures and replay test under `../fixtures/08-deploy-correlator-confirmation-bias/` and `../tests/replay_08_confirmation_bias.py`.

## Scenario

- **Service**: `users-api`, a Kubernetes Deployment in namespace `users`. Handles authentication via `/login` and profile management via `/profile`.
- **Change 1, rollout**: `users-api@v9.1.0` at 2026-07-11 14:22 UTC. Diff: refactors the `/profile` page's avatar rendering to use a new CDN URL format. **Does not touch authentication or secret-retrieval code.**
- **Change 2, RBAC**: at 2026-07-11 14:25 UTC, a platform-team change deleted the RoleBinding `users-api-secrets-reader`, which granted the `users-api` ServiceAccount `get` on Secrets in namespace `users`. The change was intended for a sibling cluster (operator error). Deleting the RoleBinding broke `users-api`'s ability to read its DB-password Secret from the Kubernetes API at startup.
- **Failure surface**: `/login` (auth requires DB lookups which require the Secret) starts failing with `auth_failed: cannot read Secret` errors as pods cycle and the new pods can't read the Secret from the API.
- **Alert at 14:33 UTC**: `users_api_error_rate > 2%`.
- **Methodology must produce**: classification **NOT** `deploy-correlator` (the rollout diff doesn't touch the auth surface). Either `outside-reference-paths` with M1 escalation, surfacing the RBAC change as a high-confidence hypothesis.

## Step 1: anchor the window

- **T0**: `2026-07-11T14:33:00Z`.
- **Tnow**: `2026-07-11T14:38:00Z`.
- **Window**: `[14:18:00Z, 14:38:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | Deploy `users-api@v9.1.0` | `14:22:14Z` | Commit `4e2c891`. Diff: `internal/profile/avatar.go` switches avatar CDN URL format. **No auth or secret code touched.** |
| RBAC | RoleBinding deletion | `14:25:42Z` | Deleted RoleBinding `users-api-secrets-reader` granting the `users-api` ServiceAccount `get` on Secrets in namespace `users`. Operator note in audit log: "intended for sibling cluster". |
| Cluster / HPA | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

**Two changes in window.** The rollout is closer to T0 in time, so an agent classifying on timing alone would over-weight it. The RBAC change is further from T0 but explicitly touches Secret access, which is in the auth path.

## Step 3: classify against the four reference paths

The failing surface is authentication (`/login`). Failing-surface hints for this investigation: `secret`, `auth`, `rbac`.

- **OOM**: RSS p95 flat. No `OOMKilled` events. No match.
- **DNS**: no `SERVFAIL` errors. No match.
- **Cascading-failure**: no cascade signature. No match.
- **Deploy-correlator**: the rollout diff (`internal/profile/avatar.go`) does NOT match the failing-surface hints (`secret`, `auth`, `rbac`). The diff touches `/profile` rendering, not the failing surface. **M4 guard: classification fails despite timing correlation, because the methodology requires diff-touches-failing-surface evidence, not just temporal coincidence.**

Classification: **outside-reference-paths**. The four reference paths do not cover RBAC-change-induced auth failures directly. The methodology surfaces the RBAC change as a high-confidence hypothesis but escalates rather than auto-recommending an RBAC revert.

## Step 4: confirm with three independent signals

1. **Logs**: `auth_failed: cannot read Secret: secrets "users-api-db-password" is forbidden: User "system:serviceaccount:users:users-api" cannot get resource "secrets" in API group "" in the namespace "users" (403 Forbidden)` errors starting 14:26:08Z, immediately after the RBAC change.
2. **Change audit**: the RBAC change at 14:25:42Z explicitly deleted the RoleBinding `users-api-secrets-reader` granting the `users-api` ServiceAccount `get` on Secrets; the audit log note "intended for sibling cluster" indicates operator error.
3. **Traces**: failing spans terminate at the `kube-apiserver` hop with `http.status_code=403` / `error=Forbidden` when reading the Secret. `/profile` traces complete successfully (the rollout is healthy).
4. **Metrics**: `error_rate_pct` jumps from baseline to ~3.4%; `auth_failure_count_rps` spikes from 0 to ~7 rps.

Four independent signals. The hypothesis (the RBAC change broke Secret access) is high-confidence. The classification is outside-paths because the methodology does not have a dedicated RBAC-correlator reference path.

## Step 5: quantify blast radius

- **Users affected**: every new login attempt fails. Existing sessions continue working until pods cycle.
- **Surfaces affected**: `/login` and any path requiring fresh DB credentials. `/profile` (the surface the rollout touched) is healthy.
- **Business impact**: new logins blocked. Customer-facing impact grows as existing sessions expire.

## Step 6: propose mitigation before root cause

Because the classification is outside-paths, the methodology surfaces hypothesis-driven mitigation candidates but does not auto-recommend any of them:

1. **Likely action** (requires human approval per M1): restore the deleted RoleBinding `users-api-secrets-reader` to grant the `users-api` ServiceAccount `get` on Secrets again. The audit log note ("intended for sibling cluster") confirms the change was misapplied; restoring it in this cluster is safe.
2. **`kubectl rollout undo`**: NOT recommended. The rollout diff does not match the failing surface. Reverting it would be the deploy-correlator confirmation bias the M4 guard exists to prevent.
3. **Feature-flag off**: not applicable (no flag).
4. **Traffic-shift**: not applicable (namespace-wide RBAC scope).
5. **Manual intervention**: not applicable.

Recommended next action: **escalate to a human with the RBAC-restore hypothesis and the explicit note that the v9.1.0 rollout is NOT implicated despite the timing**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-07-11T14:22:14Z", "event": "Deploy users-api@v9.1.0 (commit 4e2c891, /profile avatar render change)"},
    {"t": "2026-07-11T14:25:42Z", "event": "RBAC change: RoleBinding users-api-secrets-reader deleted (get on Secrets in ns users; operator note: 'intended for sibling cluster')"},
    {"t": "2026-07-11T14:26:08Z", "event": "First 'auth_failed: cannot read Secret ... is forbidden (403 Forbidden)' error"},
    {"t": "2026-07-11T14:33:00Z", "event": "T0: alert users_api_error_rate > 2% fires"},
    {"t": "2026-07-11T14:38:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "RBAC-change-induced auth failure (outside reference paths)",
      "confidence": "high (on hypothesis), classified as outside-reference-paths",
      "evidence": ["RBAC change explicitly deleted RoleBinding users-api-secrets-reader (get on Secrets)", "auth_failed '403 Forbidden' Secret-read errors in logs", "/profile (rollout surface) traces healthy", "audit log note flags operator error"]
    },
    {
      "path": "Deploy-correlator (v9.1.0) - RULED OUT",
      "confidence": "low - rejected on M4 guard",
      "evidence_against": ["rollout diff touches /profile avatar rendering, not auth or Secret access", "/profile traces complete successfully", "failing surface is /login, not /profile"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Escalate. Likely action: restore the deleted RoleBinding users-api-secrets-reader (per audit log note, the change was misapplied). Do NOT run kubectl rollout undo: the rollout is not implicated.",
  "escalate_to_human": true,
  "escalation_reasons": ["M1: failure classified outside the four reference paths"],
  "open_questions": [
    "Why did the RBAC change land in this cluster instead of the sibling cluster? Process gap in the change-management tooling?",
    "Should the agent get a dedicated 'rbac-correlator' reference path? It is recurring enough to be reference-quality.",
    "Are there other workloads using the same RoleBinding that are about to fail when their pods cycle?"
  ]
}
```

## Why this is the confirmation-bias reference example

- It exercises M4 structurally: the agent has a rollout in window AND a failure to explain, but the rollout diff does not touch the failing surface. The methodology must resist the obvious-but-wrong classification.
- It models the *ruled-out hypothesis* slot in the handoff: not just what the methodology recommends, but what it explicitly considered and rejected. This is what protects against an operator accidentally running `kubectl rollout undo` in haste.
- It surfaces a gap the methodology has on purpose: there is no dedicated `rbac-correlator` reference path. The methodology is honest about this by classifying as outside-paths rather than force-fitting one of the four. Adding `rbac-correlator` is a methodology evolution question, not a bug.

# Worked example 9: zero changes in window (external TLS certificate expiry)

A failure where the change-surface bisection returns empty and the actual cause is external: a partner's TLS certificate expired. Tests the methodology's pivot from "find the change" to "consider external causes" when step 2 returns zero. Fixtures and replay test under `../fixtures/09-zero-changes-external-cert-expiry/` and `../tests/replay_09_cert_expiry.py`.

## Scenario

- **Service**: `webhook-receiver`, a Kubernetes Deployment whose pods call `partner.example.com` to register webhook subscriptions.
- **Failure**: at 2026-08-20 03:00 UTC, the partner's TLS certificate expired. Outbound calls now fail with `tls: certificate has expired`. The partner has not yet rotated the cert.
- **Alert at 03:18 UTC**: `webhook_receiver_error_rate > 4%`.
- **No changes anywhere on our side**. The cert expiry happens at midnight UTC for whatever timezone the partner is in, with no announcement.
- **Methodology must produce**: classification `outside-reference-paths`, the change-surface bisection should return zero, and the methodology should surface the "zero changes → external cause" pivot in its reasoning. Escalation with M1.

## Step 1: anchor the window

- **T0**: `2026-08-20T03:18:00Z` (alert fire).
- **Tnow**: `2026-08-20T03:25:00Z`.
- **Window**: `[03:03:00Z, 03:25:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

**Zero changes in window.** Per SKILL.md step 2: "treat as a strong signal in itself: the failure is likely external (upstream provider, certificate expiry, DNS, capacity drift) rather than a self-inflicted regression."

The methodology must explicitly enumerate this and pivot to external-cause hypotheses for step 3.

## Step 3: classify against the four reference paths

- **OOM**: RSS p95 flat. No `OOMKilled`. No match.
- **DNS**: `partner.example.com` resolves successfully (`dns_target` attribute on traces shows valid IPs). The handshake fails *after* DNS, not at DNS. No DNS-classification match.
- **Cascading-failure**: no internal upstream-latency growth, no retry-storm signature beyond a modest retry-budget increase. No match.
- **Deploy-correlator**: no deploy. No match.

Classification: **outside-reference-paths**. The failure shape (TLS handshake error, `certificate has expired` in error message) points to a third-party cert issue.

## Step 4: confirm with three independent signals

1. **Logs**: `tls: certificate has expired` errors from `webhook-receiver` calling `partner.example.com`, starting 03:01:14Z (just after midnight UTC).
2. **Traces**: failing spans terminate at the `webhook-receiver` → `partner` hop with `tls.error=certificate_expired`. The `dns_target` attribute resolves successfully; the failure is post-DNS.
3. **Metrics**: error rate climbs from 0.2% baseline to 6.8% as queued webhook deliveries fail; internal RSS flat; DNS resolver error counter flat.

Three independent signal sources. Hypothesis (external partner cert expiry) is high-confidence on the evidence even though the classification is outside-paths.

## Step 5: quantify blast radius

- **Users affected**: every webhook subscription update to this partner. New subscriptions queue; existing connections to other partners work fine.
- **Surfaces affected**: only the partner-specific webhook path.
- **Business impact**: integration with this partner degraded. Other partners unaffected.

## Step 6: propose mitigation before root cause

Outside-paths classification means the methodology constrains the mitigation set:

1. **`kubectl rollout undo`**: not applicable; no rollout or change to revert.
2. **Feature-flag off** the partner integration if a flag exists, surfacing a "partner integration paused" message to users. Allowed without escalation.
3. **Traffic-shift** to a redundant partner endpoint if one is configured. Verify the endpoint actually bypasses the expired cert.
4. **Scale**: not applicable.
5. **Manual intervention**: not applicable.

The actual remediation requires the *partner* to rotate the cert. We can only escalate and pause the integration on our side.

Recommended next action: **escalate to a human with the cert-expiry hypothesis and the partner's incident contact details**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-08-20T03:01:14Z", "event": "First tls: certificate has expired error from partner.example.com"},
    {"t": "2026-08-20T03:18:00Z", "event": "T0: alert webhook_receiver_error_rate > 4% fires"},
    {"t": "2026-08-20T03:25:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "External: partner TLS certificate expired (outside reference paths)",
      "confidence": "high (on hypothesis), classified as outside-reference-paths",
      "evidence": ["tls: certificate has expired in logs", "TLS error post-DNS in traces", "zero changes on our side in window", "partner.example.com is the consistent failing target"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Escalate. Pause partner integration via feature flag if available. Contact partner's incident channel for cert rotation ETA.",
  "escalate_to_human": true,
  "escalation_reasons": ["M1: failure classified outside the four reference paths"],
  "open_questions": [
    "Does the partner have a cert-monitoring SLA they violated?",
    "Should we monitor partner cert expiry proactively (out-of-band check that warns 14 days before expiry)?",
    "Do we have a redundant partner endpoint with an independent cert chain?"
  ]
}
```

## Why this is the zero-changes reference example

- It validates SKILL.md step 2's claim that "zero changes is itself a strong signal." Without that pivot, an agent that always hunts for an internal cause would waste investigation time and possibly invent a phantom hypothesis.
- It separates *DNS path* from *TLS issue*: both involve failing outbound calls, but the diagnostic shape is different (DNS fails at resolution; TLS fails after resolution). The example exercises the methodology's discipline on this distinction.
- It models the case where remediation is fundamentally *out of our control*. The methodology's job in that case is to surface a clear hypothesis and stop, not to invent action.

# Worked example 10: multi-region asymmetry (config drift)

The same image tag runs in two regional Kubernetes clusters. One cluster fails, the other does not. The methodology must **surface the regional asymmetry as a first-class signal** instead of force-fitting OOM/DNS/cascade on aggregate metrics that hide the asymmetry. Escalates with a regional-asymmetry reason and a config-drift hypothesis. Fixtures and replay test under `../fixtures/10-multi-region-asymmetry/` and `../tests/replay_10_multi_region.py`.

## Scenario

- **Service**: `image-svc`. The same image tag (same code) runs in two regional clusters, `us-east-1` and `us-west-2`.
- **Failure**: `us-east-1` error rate climbs from 0.3% to 25%. `us-west-2` stays at baseline (~0.3%). The image tags are identical, so it cannot be a code regression.
- **Underlying cause** (which the methodology surfaces as a hypothesis, not as a confirmed root cause): per-cluster config drift. The `us-east-1` cluster's ConfigMap/GitOps overlay was never reconciled — an Argo CD sync failed silently and was never retried — leaving `image-svc` in that cluster pointing at a stale in-cluster object-store Service (`image-store`) that has no ready endpoints, while `us-west-2` points at the correct Service. The failure only surfaced today because of cache invalidation upstream.
- **Alert at 11:42 UTC**: aggregate `image_svc_error_rate > 5%` (the aggregate is dragged up by `us-east-1`'s share of traffic).
- **Methodology must produce**: detection of the per-region asymmetry, classification `outside-reference-paths` (no single reference path explains it), escalation with the regional-asymmetry reason, and a config-drift hypothesis in the handoff.

## Step 1: anchor the window

- **T0**: `2026-09-14T11:42:00Z`.
- **Tnow**: `2026-09-14T11:50:00Z`.
- **Window**: `[11:27:00Z, 11:50:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | |
| Cluster / HPA | (none in window) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

Zero changes in window. As with example 09, this points to an external or environmental cause. The regional asymmetry, surfaced in step 4, narrows the hypothesis space further.

## Step 3: classify against the four reference paths

Run the classifier on aggregate metrics first:

- **OOM**: aggregate RSS p95 flat. No `OOMKilled` events. No match.
- **DNS**: the `image-store.storage.svc.cluster.local` Service name resolves; the failure is `no endpoints available`, not a DNS `SERVFAIL`. No match.
- **Cascading-failure**: no cascade signature. No match.
- **Deploy-correlator**: no rollout. No match.

Classification: **outside-reference-paths**. The four reference paths do not have a `regional-asymmetry` shape.

## Step 4: confirm with three independent signals

1. **Logs**: `image-svc` in `us-east-1` logs `image fetch failed: Get "http://image-store.storage.svc.cluster.local/o/raw": no endpoints available for service "image-store"`. `us-west-2` logs no such errors.
2. **Metrics (regional)**: per-region samples (`fixtures/10-multi-region-asymmetry/metrics.json`) show `us-east-1` error rate 25% vs `us-west-2` error rate 0.3% over the window. **Regional asymmetry detector trips** with a 50x+ ratio.
3. **Traces**: failing spans only originate from pods in the `us-east-1` cluster, terminating at the in-cluster `image-store` Service hop with `http.status_code=503` / `error=NoEndpoints`. `us-west-2` pods serve identical requests successfully.

Three independent signal sources. The methodology's regional-asymmetry detector surfaces this as an additional structured signal in the handoff.

## Step 5: quantify blast radius

- **Users affected**: roughly the share of traffic served by the `us-east-1` cluster (typically ~60% in this topology). Users routed to `us-west-2` are unaffected.
- **Surfaces affected**: image processing endpoints in the `us-east-1` cluster.
- **Business impact**: ~60% of image uploads fail. Read-only image fetches degraded.

## Step 6: propose mitigation before root cause

Because the classification is outside-paths and a regional asymmetry is detected, the methodology's mitigation options shift:

1. **Traffic-shift** all `image-svc` traffic to the `us-west-2` cluster while `us-east-1` is investigated. Allowed without escalation (it is the canonical "traffic-shift away from failing region" move).
2. **Investigate config drift** in the `us-east-1` cluster. Compare the `image-svc` ConfigMap/GitOps overlay against `us-west-2` to identify the stale `image-store` Service reference, and check Argo CD sync status for the silently failed reconcile. This requires a human in the loop.
3. **`kubectl rollout undo`**: not applicable (no rollout or change in window).
4. **Scale**: not applicable.

Recommended next action: **traffic-shift to the `us-west-2` cluster immediately**, then escalate the config-drift investigation to a human.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-09-14T11:32:00Z", "event": "First 'no endpoints available for service image-store' error in us-east-1"},
    {"t": "2026-09-14T11:42:00Z", "event": "T0: aggregate image_svc_error_rate > 5% fires"},
    {"t": "2026-09-14T11:50:00Z", "event": "Tnow: investigation triggered"}
  ],
  "ranked_hypotheses": [
    {
      "path": "ConfigMap/GitOps Service drift between us-east-1 and us-west-2 clusters (outside reference paths)",
      "confidence": "high (on hypothesis); classification outside-reference-paths",
      "evidence": ["per-region error rate 25% (us-east-1) vs 0.3% (us-west-2)", "'no endpoints available for service image-store' errors only from us-east-1 pods", "same image tag running in both clusters", "zero changes in window"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Traffic-shift to us-west-2 cluster while ConfigMap/GitOps Service-drift investigation runs.",
  "regional_asymmetry": {
    "detected": true,
    "per_region_peak_error_rate_pct": {"us-east-1": 25.0, "us-west-2": 0.3},
    "asymmetry_ratio": 83.3
  },
  "escalate_to_human": true,
  "escalation_reasons": [
    "M1: failure classified outside the four reference paths",
    "regional-asymmetry detected with 83x ratio"
  ],
  "open_questions": [
    "Why did the Argo CD sync for the us-east-1 cluster fail silently three weeks ago? Where is the missing sync-failed alert?",
    "Are other workloads in the us-east-1 cluster pointing at the same stale image-store Service, or otherwise unreconciled?",
    "Should config drift checks run as a daily cross-cluster GitOps/ConfigMap diff job?"
  ]
}
```

## Why this is the multi-region-asymmetry reference example

- It exposes the failure mode where *aggregate metrics hide the truth*. An agent that classifies on aggregate signals would see a moderate error rate and miss the diagnostic that only one cluster is affected.
- It exercises the regional-asymmetry detector, which is the methodology's mechanism for surfacing per-region patterns as a first-class signal in the handoff.
- It models a class of operational reality (ConfigMap/GitOps drift across clusters) that the four reference paths do not directly cover, requiring the methodology to escalate honestly rather than guess.

# Worked example 11: capacity-bound organic growth

A failure with no change, no provider issue, no internal cascade. The service has been growing organically for weeks and finally crossed its capacity threshold. The methodology should **recommend scaling, not revert**, because there is no change to revert and the dominant signal is request-rate growth. Exercises the "scale_resource as primary mitigation" branch for outside-paths classifications with growing traffic. Fixtures and replay test under `../fixtures/11-capacity-bound-organic-growth/` and `../tests/replay_11_capacity_bound.py`.

## Scenario

- **Service**: `search-svc` (search index queries).
- **Pattern**: request rate has been climbing organically for ~3 weeks as a marketing campaign drove user growth. Capacity was sized for ~400 rps; today's morning peak hit 720 rps. The service's query queue starts rejecting requests when it overflows. No deploy, no DNS, no internal cascade.
- **Alert at 10:55 UTC**: `search_svc_error_rate > 3%`.
- **Methodology must produce**: classification `outside-reference-paths`, blast-radius detector flags `request_rate_growing=true`, and the mitigation list leads with **scale_resource** instead of revert (no change to revert).

## Step 1: anchor the window

- **T0**: `2026-10-08T10:55:00Z`.
- **Tnow**: `2026-10-08T11:05:00Z`.
- **Window**: `[10:40:00Z, 11:05:00Z]`.

## Step 2: bisect the change surface

| Source | Event | Time | Detail |
|---|---|---|---|
| Rollout | (none) | | |
| Cluster / HPA | (none) | | |
| RBAC | (none) | | |
| ConfigMap / flags | (none) | | |
| CronJob | (none) | | |

Zero changes in window. Same pattern as examples 09 and 10: hunt for an external or environmental cause.

## Step 3: classify against the four reference paths

- **OOM**: RSS p95 flat at ~290 MB against 512 MB limit (~57%). No `OOMKilled`. No match.
- **DNS**: no resolution errors. No match.
- **Cascading-failure**: no upstream-latency growth. No retry-storm pattern (the rejections are from the service itself, not from cascading retries). No match.
- **Deploy-correlator**: no deploy. No match.

Classification: **outside-reference-paths**. The signature is *request-rate growth* + *internal queue saturation*, which isn't one of the four reference paths.

## Step 4: confirm with three independent signals

1. **Metrics**: `request_rate_rps` climbed from 412 at 10:40 to 718 at T0 (1.7x in 15 minutes); `error_rate_pct` from 0.3% to 4.6%; `queue_depth_p99` from 12 to 184 (15x); `rss_bytes_p95` flat at ~290 MB.
2. **Logs**: `query queue full, rejecting request` warnings from `search-svc`, climbing in frequency through the window.
3. **Traces**: failing spans show `error=queue_full` at the `search-svc` hop. The failing requests never reach the search index; they are rejected at admission.

Three independent sources. The diagnostic is clean: the service is dropping load it cannot serve.

## Step 5: quantify blast radius

- **Users affected**: ~4.6% of searches at T0, climbing as traffic climbs. Successful searches still complete normally (no latency degradation on accepted requests).
- **Surfaces affected**: search query endpoint. All other endpoints healthy.
- **Business impact**: degraded search quality (some users see "search temporarily unavailable"). No silent corruption or data loss.

The `request_rate_growing` flag is `true` in the blast radius output (rate grew >= 1.5x across the window), which triggers the scaling mitigation branch.

## Step 6: propose mitigation before root cause

Because the classification is outside-paths AND `request_rate_growing` is true, the methodology's mitigation list leads differently from a change-induced incident:

1. **`kubectl scale` / raise the HPA ceiling** for `search-svc`. Request rate has climbed past current capacity headroom; the immediate fix is more replicas (`kubectl scale deployment/search-svc --replicas=N`, or raise the HPA `maxReplicas`). Acknowledged that capacity planning (forecasting, HPA tuning) is the root cause; scaling addresses the symptom.
2. **`kubectl rollout undo`**: not applicable; no change to revert.
3. **ConfigMap-flag off**: only as a temporary degradation play if scaling is delayed. Pausing low-value search features can free queue slots.
4. **Traffic-shift**: only if a secondary search cluster exists. Same caveat as 09 and 10.
5. **Manual intervention**: not applicable.

Recommended action: **raise the HPA ceiling / `kubectl scale` `search-svc` now; queue capacity planning review as a follow-up**.

## Step 7: hand off

```json
{
  "timeline": [
    {"t": "2026-10-08T10:40:00Z", "event": "request rate 412 rps (sustainable)"},
    {"t": "2026-10-08T10:48:00Z", "event": "request rate crosses 600 rps; first 'queue full' warning"},
    {"t": "2026-10-08T10:55:00Z", "event": "T0: alert search_svc_error_rate > 3% fires; request rate 692 rps"},
    {"t": "2026-10-08T11:05:00Z", "event": "Tnow: investigation triggered; request rate 718 rps, error rate 4.6%"}
  ],
  "ranked_hypotheses": [
    {
      "path": "Capacity-bound organic growth (outside reference paths)",
      "confidence": "high (on hypothesis), classified as outside-reference-paths",
      "evidence": ["request rate grew 412 -> 718 rps (1.7x) across window", "queue_depth_p99 grew 12 -> 184 (15x)", "query queue full warnings in logs", "no changes in window", "RSS / cpu / DNS / cascade signatures absent"]
    }
  ],
  "mitigation_taken": null,
  "mitigation_recommended": "Scale search-svc horizontally to absorb the new traffic level. Capacity planning review as a follow-up.",
  "escalate_to_human": true,
  "escalation_reasons": ["M1: failure classified outside the four reference paths"],
  "open_questions": [
    "Why did autoscaling not kick in? Are the autoscale thresholds tuned to absolute capacity instead of queue saturation?",
    "Is the 3-week request-rate climb visible in a forecast dashboard? If not, where would it have been caught?",
    "What is the upper bound on this service's horizontal scaling? Are downstream dependencies (search index shards) also at risk?"
  ]
}
```

## Why this is the capacity-bound reference example

- It models the case where the methodology's *default mitigation* (revert) is structurally wrong. There is no change to revert; recommending one would be a confident-looking error.
- It exercises the `request_rate_growing` blast-radius signal, which is the methodology's mechanism for detecting saturation in the absence of a discrete event.
- It surfaces a class of incidents where the underlying cause is operational (capacity planning) rather than software-defect. The methodology surfaces this honestly via outside-paths classification and scaling mitigation, instead of inventing a software hypothesis to explain the error rate.

# Replay tests for `kubectl-investigator`

Stdlib-only Python tests that exercise the methodology in [`../SKILL.md`](../SKILL.md) against committed fixtures. No external credentials required.

## Running the tests

From the skill directory (`skills/kubectl-investigator/`):

```bash
for t in tests/replay_*.py; do python "$t" || exit 1; done
```

Each test prints `PASS` or `FAIL` and exits with the appropriate code. The current suite has 11 tests covering the four reference paths, the FAILURE_MODES escalation rules (M1, M2, M3, M4), and edge cases (zero changes in window, multi-region asymmetry, capacity saturation), totalling 99 assertions. Wire them into CI as plain `python` invocations.

## What the tests assert

Each replay test loads the fixtures for one worked example, runs the reference methodology (`_methodology.py`) end-to-end against them, and asserts:

- The investigation window is anchored correctly (step 1).
- The change surface contains the expected change(s) (step 2).
- The failure is classified into the right reference path (step 3).
- At least three independent signal sources are confirmed (step 4).
- The blast-radius numbers fall in the expected range (step 5).
- The recommended mitigation is correct, with the right action ordering (step 6).
- The handoff payload is well-formed and the escalation flag matches expectations (step 7).

A test fails when the methodology regresses on any of these. Treat a failed replay test as a regression in `SKILL.md` or in the reference implementation, not a test bug.

## Fixture schema

Each example has its own fixture directory under `../fixtures/<example-slug>/`. Files are committed JSON / JSONL with no external dependencies.

| File | Format | Purpose |
|---|---|---|
| `deploys.json` | JSON | Single object with `deploys`, `infra_changes` (cluster / node-pool / HPA / manifest edits), `rbac_changes` (Role / RoleBinding / ServiceAccount edits), `feature_flags`, `scheduled_jobs` (CronJobs) arrays. Each event has a timestamp field (`deployed_at`, `changed_at`, `flipped_at`, `ran_at`) and a `diff_summary` string. |
| `pod_events.jsonl` | JSONL (optional) | Kubernetes pod / kubelet events. One JSON object per line with `t`, `pod`, `reason` (e.g. `OOMKilled`, `BackOff`, `Unhealthy`), `message`. Skip if the example is not pod-level. |
| `metrics.json` | JSON | Time-series snapshot. Fields: `service`, `memory_limit_bytes`, `samples[]`. Each sample has `t`, `rss_bytes_p95`, `error_rate_pct`, `request_rate_rps`, `gateway_retry_rate_rps`, and any path-specific counter (e.g. `dns_resolver_errors_rps`). |
| `logs.jsonl` | JSONL (optional) | Application or system logs. One JSON object per line with `t`, `level`, `service`, `msg`, and any path-specific fields. |
| `traces.jsonl` | JSONL (optional) | Distributed traces. One JSON object per line with `t`, `trace_id`, `request_id`, `spans[]`, `outcome`, `retries`. |

The reference implementation (`_methodology.py`) gracefully handles missing optional files (returns empty lists).

## Adding a new replay test

When you contribute a new worked example to the skill:

1. Drop fixtures under `../fixtures/<example-slug>/` following the schema above.
2. Add `replay_NN_<example-slug>.py` in this directory, modeled on the existing two.
3. Pick assertions that cover the seven methodology steps. Existing tests range from 6 to 13 assertions. New tests typically want 7 to 15.
4. Run locally, commit, and reference the test in the example's markdown narrative.

A new test that does not exercise at least one independent signal source the existing tests do not exercise will fail review. The point of the replay corpus is breadth.

## Why stdlib only

Skills get adopted when they run anywhere with zero setup. A `pip install` is an adoption tax. The reference implementation uses only `datetime`, `json`, `pathlib`, `dataclasses`, and `typing`. If a future test requires a third-party dependency (e.g. `pytest`), that's a signal the methodology is leaking implementation detail.

## Future: agent-shaped tests

The reference implementation in `_methodology.py` is a deterministic stand-in. A natural follow-up is to run the same fixtures through an actual LLM agent loaded with `SKILL.md` and assert the agent produces the same classification + mitigation. That work is out of scope for the first reference example; contributions welcome.

"""
Replay test for examples/01-oom-cascade.md.

Runs the reference methodology in `_methodology.py` against the OOM cascade
fixtures and asserts the methodology produces the expected classification,
mitigation, and handoff payload.

Stdlib only. Run with: `python tests/replay_01_oom_cascade.py`.
Exits 0 on success, 1 on any assertion failure.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "01-oom-cascade"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-03-12T14:32:00Z",
        tnow_iso="2026-03-12T14:36:00Z",
        failing_surface_hints=["webhook", "buffer"],
    )

    assertions = [
        # Step 1: window anchored correctly with the 15-minute lead-in.
        (inv.window[0].isoformat() == "2026-03-12T14:17:00+00:00", f"window start expected 14:17:00Z, got {inv.window[0].isoformat()}"),

        # Step 2: change surface contains the 14:18 deploy and nothing else.
        (len(inv.change_surface) == 1, f"expected 1 change in window, got {len(inv.change_surface)}"),
        (inv.change_surface[0]["kind"] == "deploy", f"expected deploy, got {inv.change_surface[0]['kind']}"),
        (inv.change_surface[0]["version"] == "v4.18.0", f"expected v4.18.0, got {inv.change_surface[0]['version']}"),

        # Step 3: classified as OOM (not cascading-failure, not deploy-correlator).
        (inv.classified_path == "OOM", f"expected OOM, got {inv.classified_path}"),
        (any("OOMKilled" in e for e in inv.classification_evidence), "expected OOMKilled in classification evidence"),
        (any("webhook" in e.lower() or "buffer" in e.lower() for e in inv.classification_evidence), "expected webhook/buffer surface match in evidence"),

        # Step 4: at least three independent signal sources.
        (len({s["source"] for s in inv.confirming_signals}) >= 3, f"expected >=3 independent signal sources, got {len({s['source'] for s in inv.confirming_signals})}"),

        # Step 5: blast radius reflects payment-traffic failure (>10% error rate at peak).
        (inv.blast_radius["error_rate_peak_pct"] >= 10, f"expected error_rate_peak_pct >= 10, got {inv.blast_radius['error_rate_peak_pct']}"),

        # Step 6: mitigation ranked, revert first.
        (inv.mitigation_ranked[0]["action"] == "revert", f"expected revert as first mitigation, got {inv.mitigation_ranked[0]['action']}"),
        (any("scale_resource" == m["action"] for m in inv.mitigation_ranked), "expected scale_resource as a fallback mitigation"),

        # Step 7: handoff payload includes the right classified path and is not escalated.
        (inv.handoff["classified_path"] == "OOM", "handoff payload should carry OOM classification"),
        (inv.escalate_to_human is False, f"OOM-with-3-signals should not escalate, got escalation_reasons={inv.escalation_reasons}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_01_oom_cascade")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_01_oom_cascade ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  recommended:     {inv.mitigation_ranked[0]['action']} -> {inv.mitigation_ranked[0]['target']}")
    print(f"  signals:         {len(inv.confirming_signals)} from {len({s['source'] for s in inv.confirming_signals})} sources")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/02-dns-resolution-failure.md.

Runs the reference methodology against the DNS fixtures and asserts the
methodology produces the expected classification, mitigation, and handoff.

Stdlib only. Run with: `python tests/replay_02_dns.py`.
Exits 0 on success, 1 on any assertion failure.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "02-dns-resolution-failure"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-04-08T09:47:00Z",
        tnow_iso="2026-04-08T09:53:00Z",
        failing_surface_hints=["coredns", ".internal"],
    )

    assertions = [
        # Step 1: window anchored correctly.
        (inv.window[0].isoformat() == "2026-04-08T09:32:00+00:00", f"window start expected 09:32:00Z, got {inv.window[0].isoformat()}"),

        # Step 2: change surface contains the CoreDNS ConfigMap change and no code deploys.
        (len(inv.change_surface) == 1, f"expected 1 change in window, got {len(inv.change_surface)}"),
        (inv.change_surface[0]["kind"] == "infra", f"expected infra change, got {inv.change_surface[0]['kind']}"),
        (inv.change_surface[0].get("resource") == "kube-system/coredns", f"expected coredns ConfigMap, got {inv.change_surface[0].get('resource')}"),

        # Step 3: classified as DNS (not OOM, not deploy-correlator).
        (inv.classified_path == "DNS", f"expected DNS, got {inv.classified_path}"),
        (any("DNS error log lines" in e for e in inv.classification_evidence), f"expected DNS log evidence, got {inv.classification_evidence}"),

        # Step 4: at least three independent signal sources.
        (len({s["source"] for s in inv.confirming_signals}) >= 3, f"expected >=3 independent signal sources, got {len({s['source'] for s in inv.confirming_signals})}"),

        # Step 5: blast radius reflects elevated but partial error rate (intermittent DNS).
        (1 <= inv.blast_radius["error_rate_peak_pct"] <= 10, f"expected partial error rate (1-10%), got {inv.blast_radius['error_rate_peak_pct']}"),

        # Step 6: mitigation ranked, revert first (ConfigMap revert is the path).
        (inv.mitigation_ranked[0]["action"] == "revert", f"expected revert as first mitigation, got {inv.mitigation_ranked[0]['action']}"),

        # Step 7: handoff includes DNS classification, no escalation needed.
        (inv.handoff["classified_path"] == "DNS", "handoff payload should carry DNS classification"),
        (inv.escalate_to_human is False, f"DNS-with-3-signals should not escalate, got escalation_reasons={inv.escalation_reasons}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_02_dns")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_02_dns ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  recommended:     {inv.mitigation_ranked[0]['action']} -> {inv.mitigation_ranked[0]['target']}")
    print(f"  signals:         {len(inv.confirming_signals)} from {len({s['source'] for s in inv.confirming_signals})} sources")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/03-cascading-failure-retry-storm.md.

Stdlib only. Run with: `python tests/replay_03_cascade.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "03-cascading-failure-retry-storm"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-03-20T11:08:00Z",
        tnow_iso="2026-03-20T11:15:00Z",
    )

    assertions = [
        (inv.window[0].isoformat() == "2026-03-20T10:53:00+00:00", f"window start expected 10:53:00Z, got {inv.window[0].isoformat()}"),

        # Step 2: zero changes in window.
        (len(inv.change_surface) == 0, f"expected 0 changes in window, got {len(inv.change_surface)}"),

        # Step 3: cascading-failure classification, triggered by upstream latency growth.
        (inv.classified_path == "cascading-failure", f"expected cascading-failure, got {inv.classified_path}"),
        (any("upstream" in e.lower() or "retry rate" in e.lower() for e in inv.classification_evidence), f"expected upstream-latency or retry-rate evidence, got {inv.classification_evidence}"),

        # Step 4: at least 3 independent signal sources (no change_audit possible since no changes).
        (len({s["source"] for s in inv.confirming_signals}) >= 3, f"expected >=3 independent signal sources, got {len({s['source'] for s in inv.confirming_signals})}"),

        # Step 5: blast radius reflects partial failure with high gateway retry pressure.
        (5 <= inv.blast_radius["error_rate_peak_pct"] <= 20, f"expected partial error rate (5-20%), got {inv.blast_radius['error_rate_peak_pct']}"),

        # Step 6: mitigation should NOT lead with revert (no change in window); circuit-breaker first.
        (inv.mitigation_ranked[0]["action"] == "circuit_breaker", f"expected circuit_breaker as first mitigation, got {inv.mitigation_ranked[0]['action']}"),
        (any(m["action"] == "traffic_shift" for m in inv.mitigation_ranked), "expected traffic_shift as a follow-up mitigation"),
        (not any(m["action"] == "revert" for m in inv.mitigation_ranked), "should NOT recommend revert when no change in window"),

        # Step 7: handoff payload reflects the classification, no escalation needed (3 signals confirmed).
        (inv.handoff["classified_path"] == "cascading-failure", "handoff payload should carry cascading-failure classification"),
        (inv.escalate_to_human is False, f"cascade-with-3-signals should not escalate, got escalation_reasons={inv.escalation_reasons}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_03_cascade")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_03_cascade ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  recommended:     {inv.mitigation_ranked[0]['action']} -> {inv.mitigation_ranked[0]['target']}")
    print(f"  signals:         {len(inv.confirming_signals)} from {len({s['source'] for s in inv.confirming_signals})} sources")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/04-deploy-correlator-serialization.md.

Stdlib only. Run with: `python tests/replay_04_deploy_correlator.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "04-deploy-correlator-serialization"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-02-15T13:25:00Z",
        tnow_iso="2026-02-15T13:30:00Z",
        failing_surface_hints=["cart", "serializer"],
    )

    assertions = [
        (inv.window[0].isoformat() == "2026-02-15T13:10:00+00:00", f"window start expected 13:10:00Z, got {inv.window[0].isoformat()}"),

        # Step 2: one change in window, the v6.4.0 deploy.
        (len(inv.change_surface) == 1, f"expected 1 change in window, got {len(inv.change_surface)}"),
        (inv.change_surface[0]["version"] == "v6.4.0", f"expected v6.4.0, got {inv.change_surface[0].get('version')}"),

        # Step 3: classified as deploy-correlator (NOT OOM, NOT cascade, NOT DNS).
        (inv.classified_path == "deploy-correlator", f"expected deploy-correlator, got {inv.classified_path}"),
        (any("cart" in e.lower() or "serializer" in e.lower() for e in inv.classification_evidence), f"expected cart/serializer surface match, got {inv.classification_evidence}"),

        # Step 4: at least 3 independent signal sources.
        (len({s["source"] for s in inv.confirming_signals}) >= 3, f"expected >=3 signal sources, got {len({s['source'] for s in inv.confirming_signals})}"),

        # Step 5: blast radius is partial, low single-digit error rate.
        (2 <= inv.blast_radius["error_rate_peak_pct"] <= 10, f"expected partial error rate (2-10%), got {inv.blast_radius['error_rate_peak_pct']}"),

        # Step 6: mitigation leads with revert.
        (inv.mitigation_ranked[0]["action"] == "revert", f"expected revert as first mitigation, got {inv.mitigation_ranked[0]['action']}"),
        (inv.mitigation_ranked[0].get("target", "").startswith("checkout-api"), f"expected revert target to be checkout-api, got {inv.mitigation_ranked[0].get('target')}"),
        # Should NOT propose circuit_breaker (that's cascade-specific).
        (not any(m["action"] == "circuit_breaker" for m in inv.mitigation_ranked), "should NOT recommend circuit_breaker for deploy-correlator"),

        # Step 7: handoff payload + no escalation.
        (inv.handoff["classified_path"] == "deploy-correlator", "handoff payload should carry deploy-correlator"),
        (inv.escalate_to_human is False, f"deploy-correlator with surface match should not escalate, got {inv.escalation_reasons}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_04_deploy_correlator")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_04_deploy_correlator ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  recommended:     {inv.mitigation_ranked[0]['action']} -> {inv.mitigation_ranked[0]['target']}")
    print(f"  signals:         {len(inv.confirming_signals)} from {len({s['source'] for s in inv.confirming_signals})} sources")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/05-outside-reference-paths-third-party-rate-limit.md.

Stdlib only. Run with: `python tests/replay_05_outside_paths.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "05-outside-reference-paths-third-party-rate-limit"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-05-04T16:44:00Z",
        tnow_iso="2026-05-04T16:50:00Z",
    )

    assertions = [
        (inv.window[0].isoformat() == "2026-05-04T16:29:00+00:00", f"window start expected 16:29:00Z, got {inv.window[0].isoformat()}"),

        # Step 2: zero changes in window.
        (len(inv.change_surface) == 0, f"expected 0 changes in window, got {len(inv.change_surface)}"),

        # Step 3: outside reference paths (no OOM, no DNS, no cascade signature, no deploy).
        (inv.classified_path == "outside-reference-paths", f"expected outside-reference-paths, got {inv.classified_path}"),

        # Step 6: no revert proposed (no change to revert).
        (not any(m["action"] == "revert" for m in inv.mitigation_ranked), "should NOT recommend revert when no change in window"),

        # Step 7: ESCALATE per M1.
        (inv.escalate_to_human is True, "outside-reference-paths must escalate per M1"),
        (any("M1" in r for r in inv.escalation_reasons), f"expected M1 escalation reason, got {inv.escalation_reasons}"),

        # Handoff should mark escalation explicitly.
        (inv.handoff["escalate_to_human"] is True, "handoff payload must carry escalate_to_human=True"),
        (inv.handoff["classified_path"] == "outside-reference-paths", "handoff classification must be outside-reference-paths"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_05_outside_paths")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_05_outside_paths ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  escalate:        {inv.escalate_to_human} ({', '.join(inv.escalation_reasons)})")
    print(f"  signals:         {len(inv.confirming_signals)} from {len({s['source'] for s in inv.confirming_signals})} sources")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/06-ambiguous-t0-slow-burn.md.

Stdlib only. Run with: `python tests/replay_06_ambiguous_t0.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "06-ambiguous-t0-slow-burn"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-04-19T09:32:14Z",
        tnow_iso="2026-04-19T12:15:00Z",
        t0_ambiguous=True,
    )

    assertions = [
        # Step 1: window spans ~3 hours (T0 + 15min lead-in).
        ((inv.tnow - inv.window[0]).total_seconds() >= 9000, f"expected widened window >=150min, got {(inv.tnow - inv.window[0]).total_seconds() / 60:.0f}min"),

        # Step 2: zero changes in window (the actual causal deploy is 3 days outside).
        (len(inv.change_surface) == 0, f"expected 0 changes in window, got {len(inv.change_surface)}"),

        # Step 3: OOM classification on signature (slow trajectory).
        (inv.classified_path == "OOM", f"expected OOM (slow), got {inv.classified_path}"),

        # Step 7: t0_ambiguous propagated, escalation triggered with M2.
        (inv.t0_ambiguous is True, "expected t0_ambiguous=True on Investigation"),
        (inv.escalate_to_human is True, "expected escalation when T0 is ambiguous"),
        (any("M2" in r for r in inv.escalation_reasons), f"expected M2 escalation reason, got {inv.escalation_reasons}"),

        # Handoff payload carries t0_ambiguous flag.
        (inv.handoff["t0_ambiguous"] is True, "handoff must carry t0_ambiguous=True"),
        (inv.handoff["escalate_to_human"] is True, "handoff must carry escalate_to_human=True"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_06_ambiguous_t0")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_06_ambiguous_t0 ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  t0_ambiguous:    {inv.t0_ambiguous}")
    print(f"  escalate:        {inv.escalate_to_human} ({', '.join(inv.escalation_reasons)})")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/07-blast-radius-asymmetric-revert.md.

Stdlib only. Run with: `python tests/replay_07_blast_radius.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "07-blast-radius-asymmetric-revert"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-06-02T10:03:00Z",
        tnow_iso="2026-06-02T10:10:00Z",
        failing_surface_hints=["twilio", "sms"],
    )

    assertions = [
        # Step 2: one change in window, with bundle_size > 1.
        (len(inv.change_surface) == 1, f"expected 1 change in window, got {len(inv.change_surface)}"),
        (inv.change_surface[0].get("bundle_size") == 6, f"expected bundle_size=6, got {inv.change_surface[0].get('bundle_size')}"),

        # Step 3: classified as deploy-correlator.
        (inv.classified_path == "deploy-correlator", f"expected deploy-correlator, got {inv.classified_path}"),

        # Step 6: revert is still the top mitigation (methodology surfaces it).
        (inv.mitigation_ranked[0]["action"] == "revert", f"expected revert as top mitigation, got {inv.mitigation_ranked[0]['action']}"),

        # Step 7: ESCALATE because of M3 (bundle_size > 1).
        (inv.escalate_to_human is True, "must escalate when bundle_size > 1 on a revert-recommended path"),
        (any("M3" in r for r in inv.escalation_reasons), f"expected M3 escalation reason, got {inv.escalation_reasons}"),
        (any("6" in r for r in inv.escalation_reasons), f"expected M3 reason to mention 6 changes, got {inv.escalation_reasons}"),

        # Handoff carries the escalation explicitly.
        (inv.handoff["escalate_to_human"] is True, "handoff must carry escalate_to_human=True"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_07_blast_radius")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_07_blast_radius ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  recommended:     {inv.mitigation_ranked[0]['action']} (escalated: {inv.escalate_to_human})")
    print(f"  escalation:      {', '.join(inv.escalation_reasons)}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/08-deploy-correlator-confirmation-bias.md.

Stdlib only. Run with: `python tests/replay_08_confirmation_bias.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "08-deploy-correlator-confirmation-bias"


def main() -> int:
    # Failing surface hints point at auth / secret retrieval. The deploy diff does NOT
    # touch this surface; the RBAC change does. M4 guard should ensure deploy-correlator
    # is rejected.
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-07-11T14:33:00Z",
        tnow_iso="2026-07-11T14:38:00Z",
        failing_surface_hints=["secret", "auth", "rbac"],
    )

    assertions = [
        # Step 2: both changes in window (deploy + RBAC).
        (len(inv.change_surface) == 2, f"expected 2 changes in window, got {len(inv.change_surface)}"),

        # Step 3: NOT classified as deploy-correlator (M4 guard).
        (inv.classified_path != "deploy-correlator", f"M4 guard failed: classified as deploy-correlator despite mismatched diff surface"),

        # Should classify as outside-reference-paths (no reference path covers RBAC-correlator).
        (inv.classified_path == "outside-reference-paths", f"expected outside-reference-paths, got {inv.classified_path}"),

        # Step 6: NO revert of the deploy should be in mitigation list.
        (not any(m["action"] == "revert" and "users-api" in m.get("target", "") for m in inv.mitigation_ranked), "must NOT recommend reverting the (innocent) deploy"),

        # Step 7: ESCALATE with M1 (outside paths).
        (inv.escalate_to_human is True, "must escalate per M1"),
        (any("M1" in r for r in inv.escalation_reasons), f"expected M1 in reasons, got {inv.escalation_reasons}"),

        # Confirming signals should include change_audit for the RBAC change.
        (any(s["source"] == "change_audit" for s in inv.confirming_signals), f"expected change_audit signal for the RBAC change, got sources {[s['source'] for s in inv.confirming_signals]}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_08_confirmation_bias")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_08_confirmation_bias ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  escalate:        {inv.escalate_to_human} ({', '.join(inv.escalation_reasons)})")
    print(f"  M4 guard:        deploy correctly NOT classified as the cause despite timing")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/09-zero-changes-external-cert-expiry.md.

Stdlib only. Run with: `python tests/replay_09_cert_expiry.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "09-zero-changes-external-cert-expiry"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-08-20T03:18:00Z",
        tnow_iso="2026-08-20T03:25:00Z",
    )

    assertions = [
        # Step 2: zero changes in window (this is the key signal for this scenario).
        (len(inv.change_surface) == 0, f"expected 0 changes in window, got {len(inv.change_surface)}"),

        # Step 3: outside-reference-paths (no internal classification fits).
        (inv.classified_path == "outside-reference-paths", f"expected outside-reference-paths, got {inv.classified_path}"),

        # Should NOT be classified as DNS just because there are TLS errors on outbound calls.
        # (TLS != DNS: the resolver works, the handshake fails.)
        (inv.classified_path != "DNS", "must distinguish TLS handshake failure from DNS resolution failure"),

        # Step 6: no revert (no change in window).
        (not any(m["action"] == "revert" for m in inv.mitigation_ranked), "must not recommend revert with zero changes"),

        # Step 7: ESCALATE per M1.
        (inv.escalate_to_human is True, "must escalate per M1"),
        (any("M1" in r for r in inv.escalation_reasons), f"expected M1, got {inv.escalation_reasons}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_09_cert_expiry")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_09_cert_expiry ({len(assertions)} assertions)")
    print(f"  classified_path: {inv.classified_path}")
    print(f"  escalate:        {inv.escalate_to_human}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/10-multi-region-asymmetry.md.

Stdlib only. Run with: `python tests/replay_10_multi_region.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "10-multi-region-asymmetry"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-09-14T11:42:00Z",
        tnow_iso="2026-09-14T11:50:00Z",
    )

    assertions = [
        (len(inv.change_surface) == 0, f"expected 0 changes in window, got {len(inv.change_surface)}"),

        # Step 3: outside-reference-paths (no reference path for regional config drift).
        (inv.classified_path == "outside-reference-paths", f"expected outside-reference-paths, got {inv.classified_path}"),

        # Regional asymmetry detected by the detector.
        (inv.regional_asymmetry.get("detected") is True, f"expected regional_asymmetry.detected=True, got {inv.regional_asymmetry}"),
        ("us-east-1" in inv.regional_asymmetry.get("per_region_peak_error_rate_pct", {}), "expected per-region breakdown to include us-east-1"),
        ("us-west-2" in inv.regional_asymmetry.get("per_region_peak_error_rate_pct", {}), "expected per-region breakdown to include us-west-2"),

        # ESCALATE with M1 + regional-asymmetry.
        (inv.escalate_to_human is True, "must escalate per M1 + regional asymmetry"),
        (any("regional" in r.lower() for r in inv.escalation_reasons), f"expected regional-asymmetry escalation, got {inv.escalation_reasons}"),

        # Handoff payload carries the regional asymmetry data.
        (inv.handoff["regional_asymmetry"].get("detected") is True, "handoff must include regional_asymmetry"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_10_multi_region")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_10_multi_region ({len(assertions)} assertions)")
    print(f"  classified_path:    {inv.classified_path}")
    print(f"  regional_asymmetry: {inv.regional_asymmetry.get('per_region_peak_error_rate_pct')}")
    print(f"  escalate:           {inv.escalate_to_human} ({', '.join(inv.escalation_reasons)})")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Replay test for examples/11-capacity-bound-organic-growth.md.

Stdlib only. Run with: `python tests/replay_11_capacity_bound.py`.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
from _methodology import run_investigation  # noqa: E402

FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "11-capacity-bound-organic-growth"


def main() -> int:
    inv = run_investigation(
        fixture_dir=FIXTURE_DIR,
        t0_iso="2026-10-08T10:55:00Z",
        tnow_iso="2026-10-08T11:05:00Z",
    )

    assertions = [
        (len(inv.change_surface) == 0, f"expected 0 changes in window, got {len(inv.change_surface)}"),

        # Step 3: outside-reference-paths (no reference path for capacity saturation).
        (inv.classified_path == "outside-reference-paths", f"expected outside-reference-paths, got {inv.classified_path}"),

        # Step 5: request rate growth detected (>=1.5x baseline across window).
        (inv.blast_radius.get("request_rate_growing") is True, f"expected request_rate_growing=True, got {inv.blast_radius.get('request_rate_growing')}"),

        # Step 6: scale_resource is in the recommendation list (the key behavior for this scenario).
        (any(m["action"] == "scale_resource" for m in inv.mitigation_ranked), f"expected scale_resource in mitigations, got {[m['action'] for m in inv.mitigation_ranked]}"),

        # No revert recommendation (no change to revert).
        (not any(m["action"] == "revert" for m in inv.mitigation_ranked), "must not recommend revert with zero changes"),

        # Step 7: ESCALATE per M1.
        (inv.escalate_to_human is True, "must escalate per M1"),
        (any("M1" in r for r in inv.escalation_reasons), f"expected M1, got {inv.escalation_reasons}"),
    ]

    failed = [msg for ok, msg in assertions if not ok]
    if failed:
        print("FAIL: replay_11_capacity_bound")
        for msg in failed:
            print(f"  - {msg}")
        return 1

    print(f"PASS: replay_11_capacity_bound ({len(assertions)} assertions)")
    print(f"  classified_path:       {inv.classified_path}")
    print(f"  request_rate_growing:  {inv.blast_radius.get('request_rate_growing')}")
    print(f"  recommended actions:   {', '.join(m['action'] for m in inv.mitigation_ranked)}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""
Reference implementation of the kubectl-investigator methodology.

This module is a deterministic stand-in for what an AI agent does when it
follows SKILL.md. It exists so replay tests can assert that the methodology,
applied to known fixtures, produces the expected classification, mitigation,
and handoff payload.

Stdlib only. No external dependencies. No external credentials. Runs anywhere
Python 3.10+ runs.
"""

from __future__ import annotations

import json
from dataclasses import dataclass, field
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any

WINDOW_LEAD_IN = timedelta(minutes=15)


def _parse_ts(s: str) -> datetime:
    return datetime.fromisoformat(s.replace("Z", "+00:00"))


def _in_window(ts_iso: str, window: tuple[datetime, datetime]) -> bool:
    ts = _parse_ts(ts_iso)
    return window[0] <= ts <= window[1]


@dataclass
class Investigation:
    """Structured output of the methodology, one per incident."""

    t0: datetime
    tnow: datetime
    window: tuple[datetime, datetime]
    change_surface: list[dict[str, Any]] = field(default_factory=list)
    classified_path: str = ""
    classification_evidence: list[str] = field(default_factory=list)
    confirming_signals: list[dict[str, str]] = field(default_factory=list)
    blast_radius: dict[str, Any] = field(default_factory=dict)
    mitigation_ranked: list[dict[str, str]] = field(default_factory=list)
    handoff: dict[str, Any] = field(default_factory=dict)
    escalate_to_human: bool = False
    escalation_reasons: list[str] = field(default_factory=list)
    t0_ambiguous: bool = False
    regional_asymmetry: dict[str, Any] = field(default_factory=dict)


def load_fixture(fixture_dir: Path, name: str) -> Any:
    """Load a fixture file. .json returns parsed JSON; .jsonl returns a list."""
    path = fixture_dir / name
    if name.endswith(".jsonl"):
        with path.open() as f:
            return [json.loads(line) for line in f if line.strip()]
    with path.open() as f:
        return json.load(f)


def anchor_window(t0_iso: str, tnow_iso: str) -> tuple[datetime, datetime, tuple[datetime, datetime]]:
    """Step 1: anchor T0 and Tnow, return the [T0 - 15min, Tnow] window."""
    t0 = _parse_ts(t0_iso)
    tnow = _parse_ts(tnow_iso)
    if t0 > tnow:
        raise ValueError("T0 cannot be after Tnow")
    window = (t0 - WINDOW_LEAD_IN, tnow)
    return t0, tnow, window


def bisect_change_surface(deploys_fixture: dict, window: tuple[datetime, datetime]) -> list[dict]:
    """Step 2: pull every change event overlapping the window."""
    changes = []
    for deploy in deploys_fixture.get("deploys", []):
        if _in_window(deploy["deployed_at"], window):
            changes.append({**deploy, "kind": "deploy"})
    for change in deploys_fixture.get("infra_changes", []):
        if _in_window(change.get("changed_at", ""), window):
            changes.append({**change, "kind": "infra"})
    for change in deploys_fixture.get("rbac_changes", []):
        if _in_window(change.get("changed_at", ""), window):
            changes.append({**change, "kind": "rbac"})
    for flip in deploys_fixture.get("feature_flags", []):
        if _in_window(flip.get("flipped_at", ""), window):
            changes.append({**flip, "kind": "feature_flag"})
    for job in deploys_fixture.get("scheduled_jobs", []):
        if _in_window(job.get("ran_at", ""), window):
            changes.append({**job, "kind": "scheduled_job"})
    return changes


def _has_oom_signature(pod_events: list[dict], metrics: dict, window: tuple[datetime, datetime]) -> tuple[bool, list[str]]:
    evidence = []
    oom_events = [e for e in pod_events if e.get("reason") == "OOMKilled" and _in_window(e["t"], window)]
    if oom_events:
        evidence.append(f"{len(oom_events)} OOMKilled events on pods")
    limit = metrics.get("memory_limit_bytes", 0)
    if limit:
        peak = max((s.get("rss_bytes_p95", 0) for s in metrics.get("samples", []) if _in_window(s["t"], window)), default=0)
        if peak >= 0.9 * limit:
            evidence.append(f"RSS p95 reached {peak} bytes against limit {limit} bytes (>=90%)")
    # GC pause growth (5x baseline or higher) is a slow-burn OOM signature even without
    # an OOMKill yet. Covers the leak-trajectory case where memory hasn't quite hit the
    # limit but is heading there.
    samples_with_gc = [s for s in metrics.get("samples", []) if _in_window(s["t"], window) and "gc_pause_p99_ms" in s]
    if len(samples_with_gc) >= 2:
        baseline_gc = samples_with_gc[0]["gc_pause_p99_ms"]
        peak_gc = max(s["gc_pause_p99_ms"] for s in samples_with_gc)
        if baseline_gc > 0 and peak_gc >= 5 * baseline_gc:
            evidence.append(f"GC pause p99 grew from {baseline_gc}ms to {peak_gc}ms ({peak_gc / baseline_gc:.1f}x baseline)")
    return (len(evidence) >= 2, evidence)


def _has_dns_signature(logs: list[dict], window: tuple[datetime, datetime]) -> tuple[bool, list[str]]:
    evidence = []
    dns_error_markers = ("NXDOMAIN", "SERVFAIL", "getaddrinfo", "no such host", "dns resolution")
    matching = [log for log in logs if _in_window(log["t"], window) and any(m.lower() in log.get("msg", "").lower() for m in dns_error_markers)]
    if matching:
        evidence.append(f"{len(matching)} DNS error log lines in window")
    return (len(matching) >= 3, evidence)


def _has_cascade_signature(metrics: dict, window: tuple[datetime, datetime]) -> tuple[bool, list[str]]:
    evidence = []
    samples = [s for s in metrics.get("samples", []) if _in_window(s["t"], window)]
    if len(samples) < 2:
        return (False, evidence)
    baseline = samples[0].get("gateway_retry_rate_rps", 0)
    peak = max((s.get("gateway_retry_rate_rps", 0) for s in samples), default=0)
    if baseline > 0 and peak >= 3 * baseline:
        evidence.append(f"gateway retry rate spiked from {baseline} to {peak} rps ({peak / baseline:.1f}x baseline)")
        return (True, evidence)
    # Cascade can also surface as upstream latency growth without a retry-rate spike
    # (thread-pool saturation pattern). Detect that as a secondary check.
    upstream_p99_field = "upstream_latency_p99_ms"
    if all(upstream_p99_field in s for s in samples):
        baseline_lat = samples[0][upstream_p99_field]
        peak_lat = max(s[upstream_p99_field] for s in samples)
        if baseline_lat > 0 and peak_lat >= 3 * baseline_lat:
            evidence.append(f"upstream P99 latency grew from {baseline_lat}ms to {peak_lat}ms ({peak_lat / baseline_lat:.1f}x baseline)")
            return (True, evidence)
    return (False, evidence)


def _detect_regional_asymmetry(metrics: dict, window: tuple[datetime, datetime]) -> dict[str, Any]:
    """Detect per-region asymmetry in error rate. Empty dict if no region field in fixtures."""
    samples = [s for s in metrics.get("samples", []) if _in_window(s["t"], window) and "region" in s]
    if not samples:
        return {}
    regions: dict[str, list[float]] = {}
    for s in samples:
        regions.setdefault(s["region"], []).append(s.get("error_rate_pct", 0))
    if len(regions) < 2:
        return {}
    region_max = {r: max(vals) for r, vals in regions.items()}
    worst = max(region_max.values())
    best = min(region_max.values())
    if worst >= 5 * max(best, 0.5):
        return {
            "detected": True,
            "per_region_peak_error_rate_pct": region_max,
            "asymmetry_ratio": round(worst / max(best, 0.5), 1),
        }
    return {"detected": False, "per_region_peak_error_rate_pct": region_max}


def _deploy_correlator_evidence(changes: list[dict], failing_surface_hints: list[str]) -> list[str]:
    evidence = []
    for change in changes:
        if change["kind"] != "deploy":
            continue
        diff = change.get("diff_summary", "").lower()
        for hint in failing_surface_hints:
            if hint.lower() in diff:
                evidence.append(f"deploy {change.get('version', '?')} ({change.get('commit', '?')}) touches failing surface: {hint}")
                break
    return evidence


def classify(
    pod_events: list[dict],
    metrics: dict,
    logs: list[dict],
    changes: list[dict],
    window: tuple[datetime, datetime],
    failing_surface_hints: list[str] | None = None,
) -> tuple[str, list[str]]:
    """Step 3: classify against the four reference paths.

    Returns (path, evidence_lines). Path is one of:
    "OOM", "DNS", "cascading-failure", "deploy-correlator", "outside-reference-paths".

    When OOM is triggered by a deploy, returns "OOM" (the OOM is the root path; the
    deploy explains the trigger but the failure shape is OOM). Reflected in the
    SKILL.md guidance: classify the primary path, note the cascade or trigger as
    second-order, do not collapse into a single label.
    """
    failing_surface_hints = failing_surface_hints or []

    oom_hit, oom_evidence = _has_oom_signature(pod_events, metrics, window)
    dns_hit, dns_evidence = _has_dns_signature(logs, window)
    cascade_hit, cascade_evidence = _has_cascade_signature(metrics, window)
    deploy_evidence = _deploy_correlator_evidence(changes, failing_surface_hints)

    if oom_hit:
        evidence = oom_evidence + deploy_evidence + cascade_evidence
        return ("OOM", evidence)
    if dns_hit:
        evidence = dns_evidence + deploy_evidence
        return ("DNS", evidence)
    if cascade_hit and not oom_hit and not dns_hit:
        return ("cascading-failure", cascade_evidence)
    if deploy_evidence and not cascade_hit:
        return ("deploy-correlator", deploy_evidence)
    return ("outside-reference-paths", ["no reference-path signature met threshold"])


def confirm_signals(
    pod_events: list[dict],
    metrics: dict,
    logs: list[dict],
    changes: list[dict],
    traces: list[dict],
    window: tuple[datetime, datetime],
) -> list[dict[str, str]]:
    """Step 4: list signals supporting the hypothesis, drawn from independent sources.

    Each source contributes at most one entry; this enforces the "two signals from the
    same source count as one" rule from SKILL.md step 4.
    """
    signals: list[dict[str, str]] = []

    # orchestrator_events source
    oom_events = [e for e in pod_events if e.get("reason") == "OOMKilled" and _in_window(e["t"], window)]
    if oom_events:
        signals.append({"source": "orchestrator_events", "evidence": f"{len(oom_events)} OOMKilled pod events in window"})

    # metrics source: any of memory-at-limit, upstream-latency growth, retry-rate spike, error-rate jump.
    samples = [s for s in metrics.get("samples", []) if _in_window(s["t"], window)]
    if samples:
        metric_evidence_parts = []
        limit = metrics.get("memory_limit_bytes", 0)
        peak_rss = max((s.get("rss_bytes_p95", 0) for s in samples), default=0)
        if limit and peak_rss >= 0.9 * limit:
            metric_evidence_parts.append(f"RSS p95 peak {peak_rss}B vs limit {limit}B")
        if all("upstream_latency_p99_ms" in s for s in samples):
            up_baseline = samples[0]["upstream_latency_p99_ms"]
            up_peak = max(s["upstream_latency_p99_ms"] for s in samples)
            if up_baseline > 0 and up_peak >= 3 * up_baseline:
                metric_evidence_parts.append(f"upstream P99 latency {up_baseline}ms -> {up_peak}ms ({up_peak / up_baseline:.1f}x)")
        retry_baseline = samples[0].get("gateway_retry_rate_rps", 0)
        retry_peak = max((s.get("gateway_retry_rate_rps", 0) for s in samples), default=0)
        if retry_baseline > 0 and retry_peak >= 3 * retry_baseline:
            metric_evidence_parts.append(f"gateway retry rate {retry_baseline} -> {retry_peak} rps ({retry_peak / retry_baseline:.1f}x)")
        err_baseline = samples[0].get("error_rate_pct", 0)
        err_peak = max(s.get("error_rate_pct", 0) for s in samples)
        if err_peak >= max(1.0, 3 * err_baseline):
            metric_evidence_parts.append(f"error rate {err_baseline}% -> {err_peak}%")
        if metric_evidence_parts:
            signals.append({"source": "metrics", "evidence": "; ".join(metric_evidence_parts)})

    # change_audit / deploy_diff sources
    deploy_changes = [c for c in changes if c["kind"] == "deploy"]
    if deploy_changes:
        signals.append({"source": "deploy_diff", "evidence": f"deploy {deploy_changes[0].get('version', '?')} in window: {deploy_changes[0].get('diff_summary', '')[:120]}"})
    non_deploy_changes = [c for c in changes if c["kind"] != "deploy"]
    if non_deploy_changes:
        first = non_deploy_changes[0]
        descriptor = first.get("resource") or first.get("flag") or first.get("job") or first["kind"]
        signals.append({"source": "change_audit", "evidence": f"{first['kind']} change in window: {descriptor} - {first.get('diff_summary', '')[:120]}"})

    # traces source
    fail_traces = [t for t in traces if _in_window(t["t"], window) and t.get("outcome") == "fail"]
    if fail_traces:
        signals.append({"source": "traces", "evidence": f"{len(fail_traces)} failing distributed traces in window"})

    # logs source: DNS errors, timeout errors, thread-pool saturation, OOM logs.
    log_markers = (
        "NXDOMAIN", "SERVFAIL", "getaddrinfo", "no such host",
        "timeout", "thread pool saturated", "out of memory",
        "retry budget exhausted",
    )
    log_hits = [log for log in logs if _in_window(log["t"], window) and any(m.lower() in log.get("msg", "").lower() for m in log_markers)]
    if log_hits:
        signals.append({"source": "logs", "evidence": f"{len(log_hits)} error log lines in window (timeout / saturation / DNS / OOM patterns)"})

    return signals


def blast_radius(metrics: dict, window: tuple[datetime, datetime]) -> dict[str, Any]:
    """Step 5: quantify users / surfaces / business impact from telemetry."""
    samples = [s for s in metrics.get("samples", []) if _in_window(s["t"], window)]
    if not samples:
        return {"users_affected_pct": None, "request_rate_peak": None, "error_rate_peak_pct": None, "request_rate_growing": False}
    rates = [s.get("request_rate_rps", 0) for s in samples]
    request_rate_growing = False
    if len(rates) >= 3 and rates[0] > 0:
        request_rate_growing = rates[-1] >= 1.5 * rates[0]
    return {
        "users_affected_pct": max(s.get("error_rate_pct", 0) for s in samples),
        "request_rate_peak": max(rates),
        "error_rate_peak_pct": max(s.get("error_rate_pct", 0) for s in samples),
        "request_rate_growing": request_rate_growing,
    }


def propose_mitigation(
    classified_path: str,
    changes: list[dict],
    blast: dict[str, Any],
) -> list[dict[str, str]]:
    """Step 6: ordered mitigation list. Mitigation before root cause."""
    actions: list[dict[str, str]] = []
    # Revert the implicated change. Prefer reverting the change in window (code deploy,
    # cluster/config change, RBAC change, or feature flag flip), whichever is the implicated
    # single change. Methodology rule: mitigation before root cause.
    revertable = [c for c in changes if c["kind"] in ("deploy", "infra", "rbac", "feature_flag")]
    if revertable and classified_path in ("OOM", "deploy-correlator", "cascading-failure", "DNS"):
        change = revertable[0]
        if change["kind"] == "deploy":
            target = f"{change.get('service', '?')} to previous version"
            note = f"deploy {change.get('version', '?')} ({change.get('commit', '?')}) is in window and touches the failing surface"
        elif change["kind"] == "infra":
            target = f"{change.get('resource', '?')} to pre-{change.get('changed_at', '?')} state"
            note = f"cluster/infra change to {change.get('resource', '?')} is in window and touches the failing surface"
        elif change["kind"] == "rbac":
            target = f"{change.get('resource', '?')} RBAC change"
            note = f"RBAC change to {change.get('resource', '?')} is in window and touches the failing surface"
        else:  # feature_flag
            target = f"{change.get('flag', '?')} feature flag"
            note = f"feature flag {change.get('flag', '?')} was flipped in window"
        actions.append({"action": "revert", "target": target, "note": note})
    # Pure cascade with no in-window change: circuit-breaker / traffic-shift come first.
    if classified_path == "cascading-failure" and not revertable:
        actions.append({
            "action": "circuit_breaker",
            "target": "open circuit on the degraded upstream dependency",
            "note": "stops the retry storm from amplifying the underlying degradation. Does not fix the upstream itself.",
        })
        actions.append({
            "action": "traffic_shift",
            "target": "shed traffic from the saturating service or shift to a healthy replica",
            "note": "buys time while the upstream recovers or is scaled.",
        })
    if classified_path == "OOM":
        actions.append({
            "action": "scale_resource",
            "target": "memory limit",
            "note": "stopgap if revert is delayed. Does not address cause. Will increase per-pod cost.",
        })
    if classified_path == "DNS":
        actions.append({
            "action": "traffic_shift",
            "target": "regions / resolvers not affected by DNS issue",
            "note": "if asymmetric. Verify the unaffected path actually bypasses the broken resolver before shifting traffic.",
        })
    # Outside reference paths with sustained traffic growth: scaling is a valid first action.
    if classified_path == "outside-reference-paths" and blast.get("request_rate_growing"):
        actions.append({
            "action": "scale_resource",
            "target": "horizontal scale of the saturated service",
            "note": "request rate has been climbing across the window and headroom is exhausted. Scaling addresses the immediate cause; capacity planning addresses the root cause.",
        })
    actions.append({
        "action": "manual_intervention",
        "target": "restart affected pods / processes",
        "note": "last resort. Does not address cause. Failure will recur.",
    })
    return actions


def maybe_escalate(
    classified_path: str,
    changes: list[dict],
    signals: list[dict[str, str]],
    blast: dict[str, Any],
    t0_ambiguous: bool = False,
    regional_asymmetry: dict[str, Any] | None = None,
) -> tuple[bool, list[str]]:
    """Aggregate the FAILURE_MODES.md escalation rules.

    Implemented checks: M1 (outside reference paths), M2 (ambiguous T0),
    M3 (revert blast-radius exceeds incident), the M-bias three-independent-signals
    guard, and the regional-asymmetry surfacing as a soft signal. M4 (deploy-correlator
    confirmation bias) is handled structurally by classify(), which requires
    diff-touches-failing-surface evidence before classifying as deploy-correlator;
    a timing-only correlation falls through to outside-reference-paths and trips M1.
    The operational rules (O1 missing sources, O2 telemetry blackout, O3 multi-incident)
    are surfaced as fixture-schema follow-ups.
    """
    reasons = []
    if classified_path == "outside-reference-paths":
        reasons.append("M1: failure classified outside the four reference paths")
    if len({s["source"] for s in signals}) < 3:
        reasons.append("M-bias: fewer than three independent signal sources")
    if t0_ambiguous:
        reasons.append("M2: T0 is ambiguous; re-run with a widened window before acting on the recommended mitigation")
    # M3: revert blast radius vs incident blast radius.
    revertable = [c for c in changes if c["kind"] in ("deploy", "infra", "rbac", "feature_flag")]
    if revertable and classified_path in ("OOM", "deploy-correlator", "cascading-failure", "DNS"):
        change = revertable[0]
        bundle_size = change.get("bundle_size", 1)
        if bundle_size > 1:
            reasons.append(
                f"M3: recommended revert affects {bundle_size} bundled changes, broader blast radius than the incident; requires a human approver before executing"
            )
    if regional_asymmetry and regional_asymmetry.get("detected"):
        reasons.append(
            f"regional-asymmetry: per-region error-rate peaks {regional_asymmetry.get('per_region_peak_error_rate_pct')}; treat as a config-drift hypothesis and escalate"
        )
    return (len(reasons) > 0, reasons)


def handoff_payload(investigation: Investigation) -> dict[str, Any]:
    """Step 7: structured handoff for postmortem-author."""
    return {
        "t0": investigation.t0.isoformat(),
        "t0_ambiguous": investigation.t0_ambiguous,
        "tnow": investigation.tnow.isoformat(),
        "classified_path": investigation.classified_path,
        "evidence": investigation.classification_evidence,
        "confirming_signals": investigation.confirming_signals,
        "blast_radius": investigation.blast_radius,
        "regional_asymmetry": investigation.regional_asymmetry,
        "mitigation_recommended": investigation.mitigation_ranked[0] if investigation.mitigation_ranked else None,
        "mitigation_alternatives": investigation.mitigation_ranked[1:],
        "escalate_to_human": investigation.escalate_to_human,
        "escalation_reasons": investigation.escalation_reasons,
    }


def run_investigation(
    fixture_dir: Path,
    t0_iso: str,
    tnow_iso: str,
    failing_surface_hints: list[str] | None = None,
    t0_ambiguous: bool = False,
) -> Investigation:
    """End-to-end: load fixtures, run steps 1-7, return the structured investigation."""
    deploys = load_fixture(fixture_dir, "deploys.json")
    pod_events = load_fixture(fixture_dir, "pod_events.jsonl") if (fixture_dir / "pod_events.jsonl").exists() else []
    metrics = load_fixture(fixture_dir, "metrics.json")
    logs = load_fixture(fixture_dir, "logs.jsonl") if (fixture_dir / "logs.jsonl").exists() else []
    traces = load_fixture(fixture_dir, "traces.jsonl") if (fixture_dir / "traces.jsonl").exists() else []

    t0, tnow, window = anchor_window(t0_iso, tnow_iso)
    changes = bisect_change_surface(deploys, window)
    path, evidence = classify(pod_events, metrics, logs, changes, window, failing_surface_hints)
    signals = confirm_signals(pod_events, metrics, logs, changes, traces, window)
    blast = blast_radius(metrics, window)
    regional = _detect_regional_asymmetry(metrics, window)
    mitigation = propose_mitigation(path, changes, blast)
    escalate, escalation_reasons = maybe_escalate(path, changes, signals, blast, t0_ambiguous=t0_ambiguous, regional_asymmetry=regional)

    investigation = Investigation(
        t0=t0,
        tnow=tnow,
        window=window,
        change_surface=changes,
        classified_path=path,
        classification_evidence=evidence,
        confirming_signals=signals,
        blast_radius=blast,
        mitigation_ranked=mitigation,
        escalate_to_human=escalate,
        escalation_reasons=escalation_reasons,
        t0_ambiguous=t0_ambiguous,
        regional_asymmetry=regional,
    )
    investigation.handoff = handoff_payload(investigation)
    return investigation