# 01 — Structured Logging

Logs exist to answer questions during incidents. Every rule here optimizes
for one reader: an on-call engineer with a trace ID and 5 minutes.

## 1. Emit JSON, one event per line

Machine-parseable structure is non-negotiable. String interpolation destroys
queryability; you cannot `WHERE user_plan = 'enterprise'` on prose.

**Bad:**

```python
logger.info(f"User {user.id} checked out cart {cart.id} for ${total} in {ms}ms")
```

**Good:**

```python
logger.info("checkout_completed",
    user_id=user.id, cart_id=cart.id,
    amount_usd=total, duration_ms=ms)
```

Rules:
- Static, snake_case event name as the message; everything variable goes in
  fields. The message is a grep key, not a sentence.
- Consistent field names across the codebase: `duration_ms` everywhere, never
  a mix of `elapsed`, `time_taken`, `latency`. Maintain a field dictionary;
  prefer OTel semantic convention names (`http.response.status_code`,
  `db.system`) where they exist.
- Typed values: `duration_ms: 142` (number), not `"142ms"` (string). Units in
  the field name, not the value.
- Timestamps in UTC ISO-8601 or epoch nanos, emitted by the logger, never
  hand-formatted.
- Multi-line payloads (stack traces) belong in a single JSON field
  (`exception.stacktrace`), never as raw multi-line output that shreds into
  N orphan lines in the aggregator.

## 2. Levels: ERROR means a human must act

Level discipline is the upstream of alert discipline. If ERROR is noisy,
error-rate alerts are noise, and on-call learns to ignore both.

| Level | Contract | Examples |
|-------|----------|----------|
| FATAL | Process cannot continue; exits after logging | Config invalid at boot, can't bind port |
| ERROR | Unexpected failure; a human should investigate; counts toward error-rate SLIs | Unhandled exception, dependency hard-down after retries, data corruption detected |
| WARN | Degraded but self-handled; investigate if it trends | Retry succeeded, fallback used, deprecated API called, near a limit |
| INFO | Business-significant state change; the wide event lives here | Request completed, job finished, config reloaded |
| DEBUG | Developer detail; off or heavily sampled in prod | Cache decision, intermediate values |

**Bad** (expected events at ERROR — trains everyone to ignore ERROR):

```go
if errors.Is(err, sql.ErrNoRows) {
    log.Error("user not found", "user_id", id) // expected outcome, not an error
}
log.Error("retrying request, attempt 2/5")     // handled; WARN at most
log.Error("invalid input from client")          // client's bug → 4xx, INFO/WARN
```

**Good:** client errors (4xx) are INFO/WARN with the status in the wide
event; retries are WARN only on final failure being near; ERROR is reserved
for "this should never happen and someone must look."

Never log-and-rethrow at every layer — one exception must produce one ERROR
line (at the boundary that handles it), not five duplicates that quintuple
your error rate.

## 3. Correlation: trace_id in every line

A log line that cannot be joined to a request is gossip. Inject IDs from
context automatically — never pass them by hand.

```python
# Python: contextvars-based injection (structlog)
structlog.configure(processors=[
    structlog.contextvars.merge_contextvars,  # trace_id, request_id auto-attached
    ...,
    structlog.processors.JSONRenderer(),
])

# Middleware, once:
ctx = trace.get_current_span().get_span_context()
structlog.contextvars.bind_contextvars(
    trace_id=format(ctx.trace_id, "032x"),
    span_id=format(ctx.span_id, "016x"),
)
```

Rules:
- Use the active OpenTelemetry trace_id as the correlation ID. Do not invent
  a parallel `request_id` scheme if tracing exists; if you must keep a legacy
  request_id, log both.
- Propagate into async work: thread pools, queue consumers, cron-spawned
  tasks must restore context before logging (see rules/03 §4).
- Also bind stable dimensions once per request: `user_id` (if policy allows),
  `tenant_id`, `service.version`, `deployment.environment` — via logger
  context, not repeated at every call site.
- Audit test: pick any prod log line; you must be able to retrieve the full
  request trace and all sibling logs from it. If not, correlation is broken.

## 4. Redaction at the logger — secrets and PII never reach the sink

Call-site vigilance fails; the 200th engineer will log the request object.
Enforce centrally, fail closed.

**Bad:**

```js
logger.info('login attempt', { headers: req.headers });   // Authorization, cookies
logger.debug('user object', user);                        // email, address, hash
catch (e) { logger.error('payment failed', { request: e.config }); } // card data in axios config
```

**Good** (pino):

```js
const logger = pino({
  redact: {
    paths: ['*.password', '*.token', '*.authorization', '*.cookie',
            '*.ssn', '*.card_number', 'req.headers["x-api-key"]'],
    censor: '[REDACTED]',
  },
});
```

Rules:
- Layered defense: (1) typed serializers per domain object that emit an
  explicit allowlist of fields (`user → {id, plan}` only); (2) logger-level
  denylist for known key patterns (`password|token|secret|authorization|
  cookie|ssn|card`); (3) pipeline-level scanner (OTel Collector
  `transform`/`redaction` processor, or vendor DLP) as the last net.
- Never log: credentials, session tokens, API keys, full request/response
  bodies by default, `Authorization`/`Cookie` headers, PII beyond opaque IDs
  (email, name, address, IP where regulated), card/bank data (PCI scope
  contamination), encryption keys, signed URLs.
- Exceptions are caught objects too: exception messages and locals can embed
  connection strings and tokens. Scrub exception serializers as well.
- A secret found in logs is an incident: rotate the secret AND purge the log
  history; retention means the leak persists for the retention window.

## 5. Wide events: one canonical log line per unit of work

The single highest-leverage logging practice. Instead of 15 scattered
breadcrumb lines per request, emit ONE rich event at completion carrying
everything needed to characterize that request.

**"At completion" is doing real work in that sentence.** A line emitted mid-function
attests only that *that line ran* — not that its result survived the filter, early
return, exception path or reassignment that follows it. A count computed from a
collection the function later discards is still a computed count, and still false: one
real case logged `1 adjudicated` for weeks after the `return` beneath it stopped
including that collection. So **site a claim where the value is consumed**, derived from
what was actually returned or written: a producer may log its *intent*, only the
consumer can report the *effect*. Verify it by changing what the function returns and
reading the emitted line — for anything running unattended (cron, pipeline stage, agent
loop) the log is the **only witness**, so a log unchanged by that mutation is the
finding. Full class: `sota-code-security` rules/14 §1. Scattered lines force
join-by-timestamp archaeology; the wide event makes "show me slow checkouts
for enterprise tenants on v2.14" a single query.

```json
{
  "event": "http_request",
  "timestamp": "2026-06-12T03:14:07.121Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "http.request.method": "POST",
  "http.route": "/api/v2/checkout",
  "http.response.status_code": 502,
  "duration_ms": 4312,
  "outcome": "error",
  "error.type": "UpstreamTimeout",
  "user_id": "u_8a2f",
  "tenant_id": "t_acme",
  "tenant_plan": "enterprise",
  "cart_items": 7,
  "amount_usd": 1249.00,
  "payment_provider": "stripe",
  "retries.payment": 2,
  "cache.hit": false,
  "db.queries": 11,
  "db.total_ms": 220,
  "upstream.payment_ms": 4002,
  "feature_flags": ["checkout_v3"],
  "service.version": "2.14.1",
  "region": "eu-west-1"
}
```

Rules:
- Build the event incrementally: middleware creates a per-request accumulator
  at start; handlers and clients attach fields (`evt.set("cache.hit", true)`);
  middleware emits once in a `finally` — including on exceptions, where it
  must still fire with `outcome=error` and error fields.
- Include: identity (trace_id, route, method), outcome (status, error.type),
  timing breakdown (total + per-dependency ms), business context (tenant,
  plan, amounts, flags), infrastructure (version, region, instance).
- High dimensionality is the point — many fields per event is good. (High
  *cardinality* is fine in logs/events; it is only forbidden in metric
  labels — see rules/02 §3.)
- Same pattern for non-HTTP work: one event per consumed message, per job
  run, per batch — with queue lag, attempt number, batch size.
- Wide events at INFO are never sampled away independently of their request;
  breadcrumb DEBUG logs are the sampling target.

## 6. Sampling and cost discipline

Log spend is real money and real signal-to-noise. Defaults that are safe at
10 rps bankrupt you at 10k rps.

Rules:
- Never sample: ERROR/FATAL, wide events for failed or slow requests, audit/
  security logs (separate stream, longer retention, stricter access).
- Sample aggressively: DEBUG breadcrumbs on hot paths, wide events for
  boring successes (e.g. keep 1–10% of fast 2xx health-adjacent traffic),
  repeated identical WARNs (log-once-per-N or token bucket per event key).
- Sample per-trace, not per-line: keep or drop ALL logs of a request
  together (key the decision on trace_id, or follow the trace sampling
  decision), otherwise you get unjoinable fragments.
- Loops: never log per-item at INFO. Log batch start/end with counts, and
  per-item only at sampled DEBUG or on failure.

```python
# Bad: 1M lines per batch run
for row in rows:
    logger.info("processing row", row_id=row.id)

# Good: 2 lines + failures
logger.info("batch_started", batch_id=b, total=len(rows))
... # per-row only on failure, at WARN, with row_id
logger.info("batch_completed", batch_id=b, ok=ok, failed=failed, duration_ms=ms)
```

- Tier storage: hot/searchable 7–30 days; archive to object storage for
  compliance; route DEBUG to a cheap or ephemeral sink. Set retention per
  stream deliberately, not platform-default.
- Review the top-10 log producers (by volume and by cost) monthly; the top
  emitter is usually a forgotten DEBUG line or a health check being logged.
  Don't log load-balancer health-check requests at INFO at all.

## 7. What NOT to log

- Secrets/PII (§4) — ever.
- Per-iteration loop spam, poll ticks, "entering function X" tracing — that's
  what spans and profilers are for.
- Full request/response bodies by default. If a payload is needed for
  debugging, log it size-capped, sampled, redacted, behind a flag.
- Health-check and readiness probe traffic at INFO.
- Duplicate error reports up the call stack (§2).
- Anything you wouldn't show a contractor with log access: logs are your
  widest-read datastore with your weakest access control.

## Audit checklist

- [ ] All services emit JSON (or otherwise structured) logs; no printf prose
      on production paths.
- [ ] Field names consistent across services; a field dictionary or OTel
      semantic conventions are followed.
- [ ] Sample 20 ERROR lines from production: every one represents something
      a human should act on. No expected 4xx/no-rows/retry noise at ERROR.
- [ ] Every production log line carries trace_id (or correlation ID); IDs
      flow into async/queue/cron work.
- [ ] Redaction enforced at logger/pipeline level, not call sites; grep logs
      for `Authorization`, `password=`, `eyJ` (JWT), card-number patterns —
      zero hits.
- [ ] One wide event per request/job exists with outcome, duration breakdown,
      and business context; it fires on exceptions too.
- [ ] No per-item INFO logging in loops/batch jobs; hot-path DEBUG is sampled
      or disabled in prod.
- [ ] Sampling never drops errors or splits a request's logs; audit/security
      logs are unsampled on a separate stream.
- [ ] Log volume/cost reviewed; top producers known; retention set per
      stream; health-check traffic not logged.
- [ ] Exception serialization scrubbed (no connection strings/tokens in
      messages or stack locals).

# 02 — Metrics

Metrics are cheap, aggregated, alertable numbers. They answer "is it broken
and how badly" — traces and wide events answer "why". Design them for
queries and alerts, not for completeness.

> **Backend-neutral.** Examples below use Prometheus/PromQL because it is the
> de-facto exposition format and query language. Everything here applies
> unchanged to any Prometheus-compatible backend — **VictoriaMetrics**
> (MetricsQL is a PromQL superset), Mimir, Thanos, Cortex — and OTLP-native
> pipelines. The rules are about metric *design* (RED/USE, cardinality,
> histograms, exemplars), not the vendor.

## 1. RED and USE: the two starting templates

**RED** — for every request-driven service/endpoint:
- **Rate**: requests/sec (`http_requests_total` counter)
- **Errors**: failed requests/sec (same counter, `status` label, or
  separate; define "error" = 5xx + timeouts, not 4xx)
- **Duration**: latency distribution (histogram, never a gauge or average)

**USE** — for every resource (CPU, memory, disk, connection pool, queue,
thread pool, semaphore):
- **Utilization**: fraction of capacity in use (pool connections busy / max)
- **Saturation**: queued/waiting work (waiters on the pool, queue depth/lag)
- **Errors**: resource-level failures (connection timeouts, OOM kills)

Rules:
- Every service exposes RED per route before any bespoke metric. Most
  platforms get this free from OTel/middleware instrumentation — verify it's
  on, don't reimplement.
- Every bounded resource you own (DB pool, worker pool, internal queue)
  exposes USE. The classic 3am mystery — "service slow, CPU idle" — is a
  saturated connection pool with no saturation metric.
- Queue consumers: RED becomes consume rate / failure rate / processing
  duration, plus **lag/age of oldest message** (the consumer's real SLI).
- Business metrics on top: `orders_completed_total`, `payments_failed_total`
  — these feed the SLOs users actually care about.

## 2. Instrument semantics: counter vs gauge vs histogram

| Instrument | Use for | Query pattern | Never |
|------------|---------|---------------|-------|
| Counter | Monotonic event counts (requests, errors, bytes, retries) | `rate()`, `increase()` | never decrement; never store a value that can go down |
| Gauge | Current level of something measurable now (queue depth, pool in-use, temperature, config value) | last value, `avg/max_over_time` | never for event counts (loses events between scrapes); never for latency |
| Histogram | Distributions (latency, payload size, batch size) | `histogram_quantile()` over summed buckets | — |
| Summary (client-side quantiles) | Almost never | — | cannot be aggregated across instances; prefer histograms |

**Bad:**

```python
LATENCY = Gauge("request_latency_seconds")     # last write wins; lies under load
LATENCY.set(elapsed)

ERRORS = Gauge("errors")                        # scrape misses bursts
ERRORS.set(error_count_this_minute)
```

**Good:**

```python
LATENCY = Histogram("http_request_duration_seconds",
    buckets=(.005,.01,.025,.05,.1,.25,.5,1,2.5,5,10))
LATENCY.observe(elapsed)

ERRORS = Counter("http_requests_errors_total")
ERRORS.inc()
```

Rules:
- Counters end in `_total`; include the unit in the name (`_seconds`,
  `_bytes`); base units (seconds not ms) per Prometheus convention.
- Gauges for derived "current state" should be callbacks/observable gauges
  (sample on scrape), not values you remember to set.
- Counter resets (restarts) are handled by `rate()` — never compute deltas
  by hand from raw counter values.
- Don't emit a metric and a log line as the same signal twice on hot paths
  by hand — derive metrics from spans/wide events where the pipeline
  supports it (spanmetrics), or instrument once in middleware.

## 3. Label cardinality discipline

Each unique label combination is a separate time series held in memory by
the TSDB. Cardinality explosions are the #1 way teams take down their own
monitoring — during the incident they need it.

**Forbidden as label values:** user IDs, emails, session/request/trace IDs,
raw URLs/paths (use the route template), free-text error messages, IPs,
container IDs you don't aggregate by, anything user-controlled.

**Bad:**

```python
REQS.labels(user_id=user.id, path=request.path, error=str(exc)).inc()
# /api/users/8231, /api/users/8232 ... × users × error strings = millions of series
```

**Good:**

```python
REQS.labels(
    route="/api/users/{id}",        # template, bounded by route table
    method="GET",
    status_class="5xx",             # or exact code: bounded set
    error_type=type(exc).__name__,  # bounded by exception classes
).inc()
```

Rules:
- Budget: know each label's value-set size; total series per metric =
  product of label cardinalities × instances. Keep per-metric series in the
  hundreds/low thousands, not millions.
- High-cardinality questions ("which user?", "which exact URL?") belong in
  traces and wide events — that's the division of labor. Metrics say *that*
  p99 spiked on route X; the exemplar-linked trace says *who and why*.
- Normalize at the edge: route templates from the router, error types from
  exception classes, status classes. Add a relabeling/drop rule in the
  pipeline as a backstop against accidental unbounded labels.
- Watch `prometheus_tsdb_head_series` (or vendor cardinality reports) and
  alert on sudden series growth — that alert is cheaper than the outage.
- Unbounded label sets are also a DoS vector when user input can mint
  series. Treat label values as untrusted input.

## 4. Percentiles done right

Averages hide everything that matters; percentiles computed wrong are worse
because they look authoritative.

Rules:
- **Never average percentiles.** `avg(p99 per instance)` is mathematically
  meaningless. Aggregate histogram buckets across instances FIRST, then
  compute the quantile:

```promql
# Bad: average of per-pod p99s
avg(histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])))

# Good: sum buckets across pods, then quantile
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
```

- Choose buckets around your SLO thresholds: quantile accuracy is bounded by
  bucket boundaries. If the SLO is 300ms, you need boundaries at/near 300ms
  (e.g. .25 and .3 and .4), or the reported p99 can be off by the whole
  bucket width. Default buckets are rarely right — set them per metric.
- Prefer **native/exponential histograms** (Prometheus native histograms,
  OTel exponential histograms) where the stack supports them: automatic
  bucketing, better accuracy, cheaper series. Note: Prometheus native
  histograms are **stable since v3.8.0 (Nov 2025)** — enable ingestion via the
  `scrape_native_histograms: true` config (the old
  `--enable-feature=native-histograms` flag is now a no-op). Still verify your
  whole pipeline (remote write 2.0 — itself still experimental — and dashboards)
  handles them before switching SLI queries.
- SLO arithmetic trick: a bucket boundary exactly at the SLO threshold lets
  you compute "fraction of requests under 300ms" exactly:
  `sum(rate(..._bucket{le="0.3"}[5m])) / sum(rate(..._count[5m]))` — this is
  your latency SLI, more robust than thresholding a quantile estimate.
- Report p50/p95/p99, not the average; keep max/p99.9 visible for tail
  debugging. Track tail latency per dependency, not just at the edge.

### 4a. `now` vs `offset X` is two samples, not a trend

And it convinces *more* than a single sample, which is exactly what makes it
dangerous: a before/after pair looks like a measurement. Neither point carries an
error bar, and a bursty series will hand you whatever ratio the two landing spots
imply.

Measured (2026-08-18). After a fix that bounded a set of unbounded aggregate
queries, a write-latency p99 read **0.40 s now** against **8.83 s at `offset
24h`** — a tidy 22x improvement, ready to report. Sampling the same metric at
hourly offsets across the same day:

```
-0h 0.40 | -2h 11.59 | -4h 10.49 | -6h 9.54 | -9h 0.11 | -12h 13.40 | -18h 4.38 | -24h 8.97
```

The series swings **0.11–13.40 s**. "Now" had landed in a trough and "24 h ago" on
a peak. Across that spread the same pair of reads could have shown anything from a
**122x improvement** to a **122x regression** (13.40 / 0.11), depending only on
where the two points landed. The load-invariant signal over the same windows — the **absolute
rate** of queries slower than the threshold, which a change in total query volume
cannot move — was flat, i.e. nothing had changed. The fix was real and useful; the
22x was an artefact of two points.

Rules:
- Before quoting any before/after from production telemetry, **sample the
  intervening window**. Two points cannot distinguish a step change from a
  diurnal swing, and the diurnal swing is the common case.
- When the question is "is this still happening", prefer a **load-invariant
  absolute count** to a percentile. A percentile is a **ratio**: a traffic-mix
  shift, a retry storm, or a batch job ending moves it without anything
  improving, because the denominator moved.
- This is not the "many samples, report variance" of a benchmark
  (`sota-performance` rules/01 §3). A production series **cannot be re-run**, so
  sampling more *offsets* of the same series is the only equivalent available —
  and a deploy marker on the chart is worth more than either number.

## 5. Exemplars: metrics → traces in one click

An exemplar attaches a sampled trace_id to a histogram bucket observation.
The on-call workflow becomes: see p99 spike on the dashboard → click the
exemplar dot → land in the exact slow trace. Without exemplars, the path
from "metric anomaly" to "specific request" is manual time-window
spelunking.

```yaml
# OTel SDK: exemplars on by default when a span is active (trace-based filter).
# Prometheus server: enable storage
#   --enable-feature=exemplar-storage
# Scrape with OpenMetrics so exemplars survive:
#   honor exemplars via application/openmetrics-text
```

```text
http_request_duration_seconds_bucket{le="0.5",route="/checkout"} 1027 \
  # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 0.43 1718160847.12
```

Rules:
- Record observations inside an active span so the SDK attaches the exemplar
  automatically; verify the whole chain (SDK → exposition format → scrape →
  Grafana datasource "exemplars: on") because any broken link silently drops
  them.
- Exemplars matter most on error counters and latency histograms — the two
  things you alert on.
- If the stack can't do exemplars, get the same effect by deriving metrics
  from traces (Collector `spanmetrics` connector) so trace search by
  route+duration substitutes for the click-through.

## 6. OTel metrics specifics

When instrumenting via the OpenTelemetry metrics API (preferred for new
code — one API, exporters decide Prometheus vs OTLP):

```go
meter := otel.Meter("checkout")
reqDur, _ := meter.Float64Histogram("http.server.request.duration",
    metric.WithUnit("s"),
    metric.WithExplicitBucketBoundaries(.005,.01,.025,.05,.1,.25,.3,.5,1,2.5))
poolInUse, _ := meter.Int64ObservableGauge("db.client.connections.usage",
    metric.WithInt64Callback(func(_ context.Context, o metric.Int64Observer) error {
        o.Observe(int64(pool.InUse()), metric.WithAttributes(attribute.String("state","used")))
        return nil
    }))
reqDur.Record(ctx, elapsed.Seconds(),
    metric.WithAttributes(semconv.HTTPRoute("/checkout"),
                          semconv.HTTPResponseStatusCode(code)))
```

Rules:
- Use semantic-convention instrument names (`http.server.request.duration`
  in seconds) — backends and dashboards key on them; don't reinvent
  `my_request_time_ms`. Prometheus 3.x ingests OTLP natively and accepts
  UTF-8 metric/label names, so dotted semconv names no longer have to be
  mangled to underscores — pick one naming scheme end-to-end and stop
  maintaining translation rules.
- **Aggregation temporality**: Prometheus needs cumulative; some vendors
  want delta. Set it in the exporter, never assume — delta counters scraped
  as cumulative silently report garbage rates.
- Prefer observable (callback) instruments for state you'd otherwise poll;
  prefer synchronous instruments inside request flow (they're what exemplars
  attach to).
- Views (SDK-level) are the escape hatch to fix cardinality/buckets of
  third-party instrumentation without forking it: drop attributes, re-bucket,
  rename — at the app edge or in the Collector.
- UpDownCounter vs Gauge: use UpDownCounter for additive quantities summed
  across instances (active requests fleet-wide); Gauge for non-additive
  readings (queue depth measured by each consumer — summing it double-counts).

## 7. Operational hygiene

- Pre-register metrics at startup (zero-valued where applicable) so
  `absent()`-style alerts and rate() have a baseline; a metric that appears
  only on first error breaks "no data" vs "no errors" disambiguation.
- Every metric a dashboard or alert references must exist in code review:
  delete metrics nothing queries (they cost memory and attention), and grep
  dashboards/alerts before renaming a metric — renames are breaking changes.
- Standard resource attributes on every series: `service.name`,
  `service.version`, `deployment.environment` — version is what turns "p99
  rose at 14:02" into "the 14:00 deploy did it".
- Instrument the telemetry itself: scrape failures, exporter queue drops,
  remote-write errors. Silent telemetry loss looks identical to "all good".
- **Accept a new scrape target on the series arriving, never on discovery.** A target
  list showing `1 target` means a selector matched; read the per-target `health` and
  `lastError` beside it, then confirm the series exists
  (`sota-code-security` rules/14 §4b). **A port declaration does not state the protocol
  spoken on it**: a container advertising `metrics:9090` says nothing about TLS, and
  scraping HTTP where HTTPS is served returns `400 Client sent an HTTP request to an
  HTTPS server` — an error easily read as a broken exporter. `curl` the endpoint both
  ways before writing the scrape config. Where the certificate is self-signed by the
  component's own issuer, no CA bundle can verify it and mounting one is theatre; skip
  verification explicitly and **write the reason next to the flag**, so a later reader
  can tell a considered exception from a copied one.

## Audit checklist

- [ ] New scrape targets accepted on **series arriving**, not on discovery: per-target
      `health`/`lastError` read, endpoint probed for scheme (HTTP vs TLS) before the config
      was written, and any skipped certificate verification carries its reason inline (§7).
- [ ] RED metrics exist per service and per route (rate, errors with a
      defined error definition, duration as histogram).
- [ ] USE metrics exist for every owned bounded resource: DB/HTTP connection
      pools, worker pools, internal queues (utilization AND saturation).
- [ ] Queue consumers expose lag/oldest-message-age.
- [ ] No gauges used for latency or event counts; counters are `_total`,
      units in names, base units.
- [ ] Grep label usage: no user IDs, raw paths/URLs, error strings, or other
      unbounded values as labels; route templates used; series counts per
      metric known and bounded; cardinality-growth alert exists.
- [ ] No dashboard or alert averages percentiles; quantiles computed from
      bucket sums across instances; summaries not aggregated.
- [ ] Histogram buckets chosen around SLO thresholds (or native/exponential
      histograms in use); latency SLI computed from a bucket boundary.
- [ ] Exemplars flow end-to-end (SDK → exposition → TSDB → dashboard), or
      spanmetrics provides the metric↔trace bridge.
- [ ] `service.version` and environment present on all series; deploys are
      correlatable with metric shifts.
- [ ] Does any before/after claim from production telemetry rest on exactly two
      points (`now` vs `offset X`)? The intervening window must be sampled, and
      "is it still happening" answered with a load-invariant absolute count
      rather than a percentile, whose denominator moves on its own (§4a).
- [ ] Metrics pipeline self-monitored (scrape/export failures alerted);
      unused metrics pruned.

# 03 — Distributed Tracing (OpenTelemetry)

Tracing answers "where did this request spend its time, and which hop
failed" across service boundaries. OpenTelemetry is the standard; vendor
tracing SDKs in application code are technical debt as of 2026.

## 1. OTel architecture: API vs SDK separation

- **Libraries and shared code depend ONLY on the OTel API**
  (`opentelemetry-api`, `@opentelemetry/api`, `go.opentelemetry.io/otel`).
  The API is no-op without an SDK, so libraries instrument unconditionally
  with zero cost to non-adopters.
- **Applications configure the SDK once at the entry point**: tracer
  provider, resource attributes, sampler, exporter (OTLP). Nothing else in
  the codebase imports SDK packages.
- **Export OTLP to a Collector**, not directly to a vendor: the Collector
  owns batching, retries, tail sampling, redaction, fan-out, and vendor
  routing. Swapping backends becomes a Collector config change, not a code
  change.

```python
# Library code — API only:
from opentelemetry import trace
tracer = trace.get_tracer("payments-lib", "1.4.0")

# main.py — the ONLY place SDK appears:
provider = TracerProvider(
    resource=Resource.create({
        "service.name": "checkout", "service.version": VERSION,
        "deployment.environment": ENV}),
    sampler=ParentBased(TraceIdRatioBased(0.1)),
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
```

Rules:
- Start with auto-instrumentation (HTTP frameworks, DB drivers, HTTP/gRPC
  clients, queue clients) — it covers 80% with correct semantic conventions.
  Add manual spans only for business logic the auto layer can't see.
- **Follow OTel semantic conventions** for span names and attributes
  (`http.request.method`, `db.system`, `messaging.operation`,
  `error.type`). Invented names break backend UIs, spanmetrics, and every
  query anyone else writes. HTTP and database conventions are now stable
  (messaging is still in development); when upgrading instrumentation that
  predates stabilization, migrate via `OTEL_SEMCONV_STABILITY_OPT_IN`
  rather than breaking dashboards in one jump.
- `service.name`, `service.version`, `deployment.environment` set as
  resource attributes always — they are the join keys to metrics and logs.

## 2. Span design: what deserves a span

A span = a meaningful unit of work whose duration and failure you'd want to
see in a waterfall. Span-per-function is noise; span-per-service is blind.

**Gets a span:** inbound request handling (server span); every outbound
network call — HTTP, DB query, cache op, queue publish (client/producer
span); message consume + processing (consumer span); expensive internal
phases (render, batch chunk, ML inference); each retry attempt (so retries
are visible as siblings).

**Does NOT get a span:** trivial pure functions, getters, per-item work in
large loops (span the batch, count the items), logging itself.

**Attributes vs events vs status:**
- **Attributes** = dimensions you'd filter/group by: route, db.system,
  tenant_id, cache.hit, retry.count. Set on the span, bounded-ish values,
  no payloads, no PII (same redaction policy as logs — rules/01 §4).
- **Events** = point-in-time happenings inside the span: `retry_scheduled`,
  `lock_acquired`, and especially **exception events**
  (`span.record_exception(e)`). Note OTel deprecated the Span Event API
  (`AddEvent`/`RecordException`) in 2026: the end-state is events and
  exceptions emitted as span-correlated **logs** via the Logs API.
  `record_exception` stays the acceptable default during the transition;
  migrate instrumentation via `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN=logs`
  (or `logs/dup` to emit both), and SDK-level routing can surface log-based
  events as span events for backends that need them. Exceptions-as-logs
  matches rules/01 §1 (`exception.stacktrace` as a structured field).
- **Status**: set `ERROR` only for genuine failures of that unit of work.
  A 404 on a lookup endpoint is not span-error; an unhandled exception is.
  Marking expected outcomes as errors poisons tail sampling and spanmetrics.

```python
# Good: one span per outbound call, rich attributes, honest status
with tracer.start_as_current_span(
    "charge_card", kind=SpanKind.CLIENT,
    attributes={"payment.provider": "stripe", "amount_usd": total,
                "retry.max": 3}) as span:
    try:
        resp = stripe.charge(...)
        span.set_attribute("payment.id", resp.id)
    except StripeTimeout as e:
        span.record_exception(e)
        span.set_status(Status(StatusCode.ERROR, "provider timeout"))
        raise
```

```python
# Bad
with tracer.start_as_current_span("process"):          # name says nothing
    for item in items:                                   # 50k spans per request
        with tracer.start_as_current_span(f"item-{item.id}"):  # high-card name
            ...
```

Rules:
- Span names are low-cardinality templates: `GET /users/{id}`,
  `SELECT orders` — never interpolated IDs/URLs (breaks grouping and
  spanmetrics cardinality).
- Always end spans — `with`/`defer`/`try-finally`. A leaked span orphans its
  whole subtree.
- Record the queue/pool wait time as either a separate span or an attribute;
  "slow handler" that is really "waited 2s for a connection" is the classic
  misdiagnosis tracing exists to prevent.

## 3. Context propagation: W3C traceparent everywhere

One dropped hop splits the trace and you're back to timestamp archaeology.

- **HTTP/gRPC**: W3C Trace Context (`traceparent`, `tracestate`) is the
  standard. Auto-instrumented clients/servers handle it; verify any
  hand-rolled HTTP client injects it, and configure the B3↔W3C composite
  propagator only where legacy systems require it.
- **Queues/streams (the usual gap)**: inject context into message
  headers/attributes at publish; extract at consume and start the consumer
  span with the extracted context as **link or parent**:

```python
# Producer
carrier = {}
propagate.inject(carrier)
channel.basic_publish(..., properties=BasicProperties(headers=carrier))

# Consumer
ctx = propagate.extract(message.headers)
with tracer.start_as_current_span("orders process", context=ctx,
                                  kind=SpanKind.CONSUMER):
    ...
```

  For batch consumers, use **span links** to all source messages rather than
  picking one parent. For long-delay queues, prefer a new trace linked to
  the producer trace over a single week-long trace.
- **Background jobs / cron / fan-out**: thread pools and task queues
  (Celery, Sidekiq, asyncio tasks) must capture context at submit time and
  restore at execution (`contextvars` copy, `context.with_current()`,
  framework instrumentation). A scheduled job starts a fresh root span —
  give it a real name and the standard resource attributes.
- **Trust boundary**: at the public edge, decide policy — typically accept
  incoming `traceparent` for continuity but never trust sampling flags from
  the internet blindly; strip/regenerate at the edge proxy if traces could
  be forced-sampled by clients.
- Audit test: one user action that crosses HTTP → queue → worker → DB must
  appear as ONE trace (or explicitly linked traces). If you see N
  single-service traces for one action, propagation is broken at a hop.

## 4. Sampling: head vs tail

100% tracing at scale is unaffordable; naive sampling discards exactly the
traces you need (errors, tail latency). Decide deliberately.

| Strategy | How | Pros | Cons |
|----------|-----|------|------|
| Head (TraceIdRatioBased + ParentBased) | Decide at root, propagate decision | Cheap, simple, consistent per-trace | Blind: drops 99% of errors/slow traces at 1% rate |
| Tail (Collector `tailsamplingprocessor`) | Buffer whole trace, decide on completion | Keep 100% of errors + slow + rare routes, sample boring successes | Collector memory/state; needs all spans of a trace at one collector instance (load-balancing exporter by trace_id) |

Recommended composite policy (tail, in the Collector):

```yaml
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors        # keep all failed traces
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow          # keep all traces over SLO threshold
        type: latency
        latency: {threshold_ms: 300}
      - name: baseline      # 5% of everything else
        type: probabilistic
        probabilistic: {sampling_percentage: 5}
```

Rules:
- Always `ParentBased` in SDKs so a trace is kept or dropped whole; mixed
  decisions produce broken partial traces.
- Pair head sampling at a moderate rate with tail sampling downstream
  (head controls SDK/network cost, tail controls storage and preserves
  signal). Pure 1% head sampling on a low-error service means you will have
  zero traces of the incident.
- Log the effective sampling config; during incidents, support a runtime
  knob to raise sampling on a specific route/tenant.
- Remember sampled-out requests still need their wide event (rules/01 §5) —
  logs are the unsampled record; traces are the deep-dive.

## 5. Limits, overhead, and pipeline hardening

Tracing must never take down the service it observes.

- **Span/attribute limits**: configure SDK limits (max attributes, events,
  links per span; max attribute length). A bug that attaches a 2MB response
  body or 10k events to a span should be truncated by config, not crash the
  exporter. Default limits exist — verify they're sane, don't raise them
  casually.
- **BatchSpanProcessor always** in production (never SimpleSpanProcessor —
  it exports synchronously on the request path). Size queue and export
  batches for peak; monitor the SDK's dropped-span counter: silent drops
  during the incident are when you needed traces most.
- Exporter failures must be non-blocking and bounded (timeout + queue, drop
  on overflow). Telemetry backpressure must never propagate into request
  latency.
- Collector deployment: at least an agent (daemonset/sidecar) + gateway
  pair for tail sampling; the gateway tier needs trace-ID-routed load
  balancing (`loadbalancingexporter`) so tail decisions see whole traces.
  The Collector itself exports its own metrics — alert on
  `otelcol_processor_dropped_spans` and exporter queue saturation.
- Redaction in the pipeline: a Collector `attributes`/`redaction` processor
  deny-listing token/PII-shaped attributes is the backstop for instrument-
  ation mistakes — same philosophy as logger-level redaction (rules/01 §4).
- Shutdown: flush the provider on SIGTERM (`provider.shutdown()` /
  `ForceFlush`) or you lose the last batch of every deploy — which is
  exactly the window where regressions live.

## 6. Baggage: handle with caution

Baggage propagates key-values alongside the trace context to all downstream
services — and into every outbound header.

- Use only for small, low-sensitivity routing/context values needed across
  services: `tenant.tier`, synthetic-test flag, experiment bucket.
- **Never** put PII, tokens, or anything large in baggage: it is forwarded
  to EVERY downstream hop — including third-party APIs your HTTP client
  calls with propagation enabled. That's a data-leak primitive.
- Baggage is not span attributes: receivers must explicitly read baggage and
  stamp it onto spans (Collector/SDK baggage-to-attribute processors) for it
  to appear in trace data.
- Cap count/size; treat inbound baggage at trust boundaries as untrusted
  input and strip it at the edge.

## Audit checklist

- [ ] Application code depends on OTel API only; SDK configured in exactly
      one place per service; exporters point at a Collector, not hardcoded
      vendors.
- [ ] Auto-instrumentation enabled for frameworks, DB, HTTP clients, queue
      clients; semantic conventions used for names/attributes.
- [ ] Every outbound network call (HTTP, DB, cache, queue) produces a span;
      retries visible; span names are low-cardinality templates.
- [ ] Span status ERROR only on real failures; exceptions recorded as span
      events or span-correlated logs (Span Event API deprecated 2026); no
      payloads/PII in attributes.
- [ ] Trace continuity verified end-to-end across HTTP → queue → worker →
      cron paths (one trace or linked traces per user action).
- [ ] Sampling is deliberate: documented policy; errors and slow traces are
      retained (tail sampling or equivalent); ParentBased everywhere;
      sampling rate adjustable during incidents.
- [ ] trace_id present in logs (cross-link works both ways: log→trace,
      trace→logs, metric exemplar→trace).
- [ ] Baggage usage reviewed: no PII/secrets, stripped or validated at trust
      boundaries, not leaking to third-party APIs.
- [ ] Collector pipeline monitored (dropped spans, queue saturation, export
      failures); tail-sampling memory sized for peak.

# 04 — SLOs & Alerting

Alerts are interrupts on humans; SLOs are the contract that decides which
interrupts are justified. The failure mode to design against is alert
fatigue: a page that is ignorable trains on-call to ignore pages.

## 1. SLI selection: measure user journeys, not infrastructure

An SLI is a ratio: good events / valid events, defined from the user's
perspective.

Rules:
- Derive SLIs from **critical user journeys** (login, search, checkout,
  API call succeeds fast), not from components (CPU, pod count, replication
  lag). Users don't experience CPU; they experience "checkout failed".
- Standard SLI shapes:
  - **Availability**: non-5xx (and non-timeout) responses / valid requests.
  - **Latency**: requests faster than threshold / valid requests — computed
    from a histogram bucket boundary at the threshold (rules/02 §4), not
    from a p99 estimate.
  - **Freshness/lag** (pipelines): events processed within X / all events;
    or age of oldest unprocessed message under threshold.
  - **Correctness/durability** where it matters (e.g., job produced valid
    output).
- Measure as close to the user as possible (load balancer / edge), because
  server-side-only SLIs miss connection failures; complement with synthetic
  probes for low-traffic journeys.
- Define "valid events" explicitly: exclude health checks, exclude 4xx
  caused by clients (but COUNT 429s you caused by under-provisioning),
  document the exclusions in the SLO spec.
- Per-journey SLOs, few of them: 2–5 SLOs per service. Twenty SLOs means
  none of them governs decisions.

**Bad SLIs:** "CPU < 80%", "p99 of an internal function", "pod restart
count", "cache hit rate" (these are diagnostics, fine on dashboards, wrong
as SLOs).

## 2. SLOs and error budgets

- SLO = SLI target over a window: "99.9% of checkout requests succeed,
  rolling 30 days." Rolling windows beat calendar windows for alerting
  (no end-of-month amnesty).
- **Error budget** = 1 − target. At 99.9%/30d: 0.1% ≈ 43.2 minutes of full
  outage, or proportionally longer partial degradation.
- The budget is a spending account that makes reliability negotiable with
  engineering velocity:
  - Budget healthy → ship faster, run chaos experiments, take risks.
  - Budget exhausted → feature freeze / reliability-only work per a
    **documented error budget policy** agreed with product BEFORE the first
    breach. An SLO without consequences is decoration.
- Choose targets from user need and current reality, not vanity: don't
  promise 99.99% when your single-region dependency offers 99.9%; don't set
  an SLO you've never historically met (start at achievable, ratchet).
- Review SLOs quarterly: targets, exclusions, whether the SLI still matches
  the journey, and whether anyone used the budget to make a decision.

## 3. Burn-rate alerts: the multi-window pattern

Threshold alerts ("error rate > 1% for 5m") are either too twitchy or too
slow. Burn-rate alerting fixes both by alerting on the **rate of budget
consumption**.

Burn rate = (observed error ratio) / (1 − SLO). Burn rate 1 = budget exactly
exhausted at window end; 14.4 = monthly budget gone in ~2 days.

Multi-window multi-burn config for a 30d SLO. The Google SRE Workbook's canonical
recommendation is the three bold tiers (14.4/1h, 6/6h, 1/72h); the 3x/24h ticket tier
is a widely-used community extension (Sloth/Pyrra) that fills the gap between fast page
and slow burn — keep or drop it per taste:

| Severity | Burn rate | Long window | Short window | Budget consumed |
|----------|-----------|-------------|--------------|-----------------|
| **PAGE** | 14.4 | 1h | 5m | 2% of 30d budget in 1h |
| **PAGE** | 6 | 6h | 30m | 5% in 6h |
| TICKET | 3 | 24h | 2h | 10% in 24h (community extension) |
| **TICKET** | 1 | 72h | 6h | slow steady burn |

```yaml
# Prometheus: fast-burn page (99.9% SLO ⇒ 1-SLO = 0.001)
- alert: CheckoutSLOFastBurn
  expr: |
    ( sum(rate(http_requests_total{route="/checkout",code=~"5.."}[1h]))
      / sum(rate(http_requests_total{route="/checkout"}[1h])) ) > (14.4 * 0.001)
    and
    ( sum(rate(http_requests_total{route="/checkout",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{route="/checkout"}[5m])) ) > (14.4 * 0.001)
  labels: {severity: page, slo: checkout-availability}
  annotations:
    summary: "Checkout burning 30d error budget at 14.4x"
    runbook: "https://runbooks.internal/checkout-availability"
    dashboard: "https://grafana.internal/d/checkout-slo"
```

Rules:
- The **short window** is the key fatigue fix: the alert auto-resolves when
  the burn stops, instead of paging for an hour about a blip that ended.
- Both PAGE tiers and both TICKET tiers; nothing else pages on this SLI.
- Use recording rules for the SLI ratios; alert expressions stay readable.
- Sparse-traffic services: burn rates go wild on 3 requests/min. Add a
  minimum-traffic guard or use synthetic probes as the SLI source.

## 4. Alert quality rules

Every alert must pass ALL of these or be deleted/demoted:

1. **Actionable**: there is a specific action a human takes on receipt. "FYI"
   alerts go to dashboards or logs, not notification channels.
2. **Symptom-based, not cause-based**: page on user pain (SLO burn, journey
   failure), not on causes (CPU high, pod restarted, disk 80%, node down).
   Causes belong on the diagnostic dashboard the symptom alert links to.
   Exception: page on causes only for *imminent, irreversible* user pain
   with lead time to act — e.g. "disk full in 4h at current rate", cert
   expiring, backup job failed (durability has no symptom until too late).
3. **Runbook-linked**: every alert carries a runbook URL with: meaning,
   verification step, dashboard link, mitigation steps, escalation. An
   alert nobody can act on at 3am without tribal knowledge is unfinished.
4. **Severity-routed**:
   - **PAGE** (wake a human): user-visible harm now or imminently, and
     human action can help. Fast/medium burn rates, journey down, security
     incident.
   - **TICKET** (next business day): slow burn, capacity trends, flaky
     dependency, single-instance failures the platform self-healed.
   - **NONE** (dashboard/log only): everything else. Most "warning"
     channels should not exist.
5. **Owned**: an alert routes to the team that can fix it. Unowned alerts
   are deleted, not muted.

**Bad:**

```yaml
- alert: HighCPU
  expr: cpu_usage > 0.85          # cause-based; autoscaler's job; no user impact
- alert: PodRestarted
  expr: increase(kube_pod_container_status_restarts_total[5m]) > 0  # self-healed
- alert: ErrorsInLogs
  expr: rate(log_errors_total[5m]) > 0   # any single error pages someone
```

## 5. Fighting alert fatigue (operational practice)

- **Track interrupts as a metric**: pages per on-call shift (target: < 2 per
  shift, each genuinely actionable), ack times, % of pages that led to
  action. Review in a monthly alert review; every page from the last period
  gets a verdict: keep / tune / demote / delete.
- Every incident retro asks two alert questions: did we get paged for the
  symptom (if not, add SLO coverage)? did we get paged for noise during it
  (if so, delete it)?
- Auto-resolve correctness: alerts must clear when the condition clears
  (multi-window helps); stale firing alerts get silenced and then fixed.
- Use inhibition/grouping: when the edge SLO pages, suppress downstream
  cause alerts; group per service per incident, don't send 40 notifications
  for one outage.
- Silences are temporary and expiring with an owner and a reason. A
  permanent silence is a deletion in denial.
- Never alert on every ERROR log line; alert on the SLI. Error logs are for
  diagnosis after the symptom alert fires. (Crash/error-tracker spike
  notifications are tickets, not pages — rules/05 §5.)
- Protect the alerting path itself: dead-man's-switch (an always-firing
  alert whose absence pages via an independent channel) so a broken
  Prometheus/Alertmanager doesn't equal silence-as-success.

## 6. SLO spec template and recording rules

Every SLO is a versioned document next to the alert code:

```yaml
# slo/checkout-availability.yaml
slo: checkout-availability
owner: payments-team
journey: "User completes checkout"
sli:
  kind: availability
  good:  sum(rate(http_requests_total{route="/checkout",code!~"5.."}[$window]))
  valid: sum(rate(http_requests_total{route="/checkout"}[$window]))
  exclusions: "health checks (excluded at scrape), synthetic logged-out probes"
target: 99.9
window: 30d rolling
budget_policy: https://wiki.internal/payments/error-budget-policy
dashboard: https://grafana.internal/d/checkout-slo
runbook: https://runbooks.internal/checkout-availability
```

Precompute the ratios as recording rules so alerts and dashboards share one
definition (no drift between "the alert's error rate" and "the dashboard's"):

```yaml
groups:
- name: slo-checkout
  rules:
  - record: slo:checkout_error_ratio:rate5m
    expr: sum(rate(http_requests_total{route="/checkout",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{route="/checkout"}[5m]))
  - record: slo:checkout_error_ratio:rate1h
    expr: ...   # same, 1h window; repeat for 30m, 6h, 24h, 72h
```

Latency SLOs use the same machinery with the SLI flipped to a bucket ratio:
`1 - (rate(..._bucket{le="0.3"}) / rate(..._count))` is the "too slow" ratio
and burns budget identically to errors. One pair of burn-rate alerts per SLI
— availability and latency page independently because they fail
independently.

## 7. Page vs ticket decision table

| Situation | Route |
|-----------|-------|
| SLO fast burn (14.4x/1h or 6x/6h) | PAGE |
| SLO slow burn (3x/24h, 1x/72h) | TICKET |
| Critical journey down per synthetic probe | PAGE |
| Disk/quota/cert exhaustion with hours of lead time | PAGE if action needed before next business day, else TICKET |
| Single pod OOM, replaced automatically | NONE (dashboard); TICKET if recurring trend |
| Dependency degraded, fallback holding, SLI fine | TICKET |
| Backup/DR job failed | TICKET same-day; PAGE if RPO about to be violated |
| New error type spike in tracker, SLI fine | TICKET |
| Telemetry pipeline down (flying blind) | PAGE — blindness during a real incident is unbounded risk |

## 8. Burn-rate math reference

For SLO target T over window W, with budget B = 1 − T:

- Burn rate b means the W-budget is exhausted in W/b.
- Alert threshold for "consume fraction f of budget in time t":
  b = f × (W/t); error-ratio threshold = b × B.
- Sanity examples for 99.9%/30d (B = 0.001):
  - 2% in 1h → b = 0.02 × 720 = 14.4 → ratio > 1.44%
  - 5% in 6h → b = 0.05 × 120 = 6 → ratio > 0.6%
  - 10% in 24h → b = 0.10 × 30 = 3 → ratio > 0.3%
- Detection time at full outage (ratio = 1): t_detect ≈ threshold × window;
  the 14.4×/1h pager detects total outage in ~86 seconds — verify yours.

## Audit checklist

- [ ] SLOs exist, are user-journey-based, and are written down (target,
      window, SLI query, exclusions, owner); 2–5 per service, not 0, not 20.
- [ ] Latency SLI computed from histogram bucket boundary at the threshold,
      not from averaged quantiles.
- [ ] Error budget policy documented and signed off by product; evidence it
      has actually gated a decision at least once.
- [ ] Paging alerts are multi-window burn-rate (or equivalently
      symptom-based with auto-resolve); no raw "error rate > X for 5m"
      pagers; no cause-based pagers (CPU/restarts/disk%) except
      lead-time-to-irreversible cases.
- [ ] Sample 10 recent pages: every one was actionable, runbook-linked, and
      led to action; pages per shift within budget; monthly alert review
      happens.
- [ ] Every alert has: severity label, owner/route, runbook URL (resolving,
      current), dashboard link.
- [ ] Inhibition/grouping configured; silences expire and carry reasons.
- [ ] Dead-man's-switch on the alerting pipeline; monitoring-down pages via
      an independent channel.
- [ ] Low-traffic journeys covered by synthetic probes feeding SLIs.
- [ ] Alert definitions in version control, code-reviewed, deployed like
      code (no hand-edited live alerts).

# 05 — Operational Readiness

The surfaces operators and orchestrators use: health endpoints, degradation
visibility, debug/profiling access, crash reporting, dashboards. These ship
WITH the feature, not after the first incident.

## 1. Health endpoints: liveness ≠ readiness ≠ startup

Three probes, three different questions, three different consequences:

| Probe | Question | On failure | Checks |
|-------|----------|-----------|--------|
| Liveness | Is the process irrecoverably wedged? | RESTART | Process-internal only: event loop responsive, no deadlock. Usually just "return 200". |
| Readiness | Can this instance serve traffic NOW? | Remove from LB (no restart) | Required dependencies, warm caches, not draining, not overloaded |
| Startup | Has init finished? | Keep waiting (gates liveness) | Migrations applied, config loaded, connections established |

**The cardinal sin — dependency checks in liveness:**

```yaml
# Bad: database hiccup → every pod fails liveness → cluster-wide restart
# storm → thundering-herd reconnects → outage amplified
livenessProbe:
  httpGet: {path: /health, port: 8080}   # /health pings Postgres + Redis
```

```yaml
# Good
livenessProbe:
  httpGet: {path: /livez, port: 8080}    # process self-check only
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: {path: /readyz, port: 8080}   # checks required deps, cheap/cached
  periodSeconds: 5
startupProbe:
  httpGet: {path: /startupz, port: 8080}
  failureThreshold: 30                    # allows slow boot without lying livez
```

Rules:
- Restarting a process does not fix its database. Liveness restarts must
  only fire for conditions a restart actually fixes.
- Readiness checks only **required** dependencies (without which serving is
  impossible). Optional dependencies (cache, recommendations) degrade
  gracefully (§2) and must NOT fail readiness — or a Redis blip removes
  your whole fleet from the load balancer.
- Health checks are cheap and bounded: cached dependency status (TTL a few
  seconds), strict timeouts, no real queries against production tables, no
  writes. The probe must not be the load.
- Whole-fleet readiness failure on a shared dependency takes everything out
  of rotation simultaneously — for shared deps, prefer serving degraded
  (§2) over failing ready. Decide per dependency, on purpose.
- Expose a verbose authenticated variant (`/readyz?verbose`) listing each
  check's status for humans; the orchestrator gets the cheap boolean.
- Health endpoints: no auth for the orchestrator path, but not internet-
  exposed; excluded from access logs and request metrics (or labeled out).

## 2. Graceful degradation must be visible

Silent fallbacks rot: the cache that's been bypassed for a week, the
secondary provider quietly serving 100%. Degradation without telemetry is a
latent outage.

Rules:
- Every fallback/circuit-breaker/feature-kill-switch emits, when active:
  a WARN log (rate-limited), a metric
  (`degradation_active{feature="recs",reason="redis_down"}` gauge and a
  fallback counter), and a span attribute (`fallback: true`).
- Circuit breaker state is a metric (`circuit_breaker_state{dep="stripe"}`
  0=closed/1=half/2=open) with state-transition events logged.
- Wide events carry `degraded: true` + which features, so you can quantify
  user impact of running degraded ("12% of requests served without
  personalization").
- Long-running degradation alerts as TICKET (not page, if SLI holds):
  fallbacks are for surviving the night, not for permanent operation.
- Test degradation paths in CI or chaos drills; an unexercised fallback is
  assumed broken.
- **One shared helper, deduped per cause.** Route every degradation through a
  single `degraded(component, reason)` call rather than ad-hoc warnings, and
  dedupe per (component, reason) — not per request. Per-request warnings get
  filtered by operators and stop being read, which returns the system to silent
  failure. This matters most for **security controls**: a scanner or policy
  engine running inert must be a distinct health state, not a quiet default
  (`sota-code-security` rules/10).

## 3. Debug endpoints: powerful and dangerous

`/debug/pprof`, `/actuator`, `/metrics`, heap dumps, env dumps, GraphQL
introspection, `phpinfo` — chronic real-world breach and DoS vectors.

Rules:
- **Never on the public listener.** Bind debug/admin surfaces to a separate
  port/interface reachable only via internal network + authn (mTLS, SSO
  proxy). `kubectl port-forward` beats an exposed route.
- Spring Boot Actuator: explicit include-list only (`health,info,
  prometheus`); `env`, `heapdump`, `threaddump`, `mappings`, `shutdown`
  stay disabled or hard-authed — heapdump and env leak secrets outright.
- Go `net/http/pprof`: importing it registers on `DefaultServeMux` — if
  your app serves DefaultServeMux publicly, you just published profiling
  (info leak + trivial CPU DoS). Register pprof on a dedicated internal-only
  mux/port.
- `/metrics` is internal: scraped by the collector, not world-readable
  (metric names and label values leak topology, tenants, versions).
- Dynamic debug togglers (log level change endpoints, feature inspection)
  are admin APIs: authenticated, audited, rate-limited.
- Audit move: enumerate every listening port and route table; diff against
  "intended public surface". Anything debug-shaped reachable without auth is
  a CRITICAL finding.

## 4. Profiling in production

Metrics say the service is slow; profiles say which function. As of 2026,
**continuous profiling** is a standard fourth signal, not an exotic one.

Rules:
- Run an always-on low-overhead profiler (eBPF agents — Parca, Pyroscope,
  Elastic Universal Profiling; or language-native continuous profilers:
  Go pprof-based, JFR for JVM, py-spy-based). Overhead at ~1–2% CPU buys
  you "what was on-CPU at 3:14am" forever.
- OTel's profiles signal entered public **alpha** in early 2026 (OTLP
  profiles format, Collector pprof receiver, official eBPF-profiler
  distribution) — watch it as the future standard transport, but don't bet
  production profiling on it yet; the established agents above remain the
  stable path until profiles reach GA.
- Profile types: CPU + allocation at minimum; add lock-contention and
  off-CPU/wall where supported (most "slow but idle CPU" mysteries are
  off-CPU: locks, I/O waits, pool waits).
- Tag profiles with `service.version` so a regression diff is "compare
  profile of v2.14 vs v2.13" — flamegraph diffing is the fastest perf-
  regression root-cause tool that exists.
- Keep on-demand deep capture available (pprof endpoint on the internal
  port, JFR trigger) for incidents needing higher resolution.
- Memory-leak workflow: alleged leak → allocation profile + heap diff over
  time, not guess-and-redeploy.

## 5. Crash reporting & error tracking (Sentry-style)

Error trackers answer "what exceptions exist, are they new, who do they
hit" — a different job from logs (search) and alerts (interrupts).

Rules:
- Every unhandled exception in every runtime is captured: backend services,
  workers, frontend JS (with sourcemaps uploaded per release — minified
  stacks are useless), mobile (with dSYM/mapping files). Crash without a
  report = invisible user pain.
- **Release tagging is mandatory**: every event carries `release` and
  `environment`. The killer queries — "new in this release", "regressed
  after being resolved" — depend on it. Wire deploy notifications so the
  tracker knows release boundaries.
- Attach context: trace_id (link back to the trace!), user-impact key
  (opaque user/tenant id per privacy policy), feature flags. Scrub PII via
  the SDK's server-side + client-side scrubbing — same redaction bar as
  logs.
- **Grouping hygiene is the difference between signal and landfill:**
  - Fix groups that lump distinct bugs (over-grouping) or shatter one bug
    into hundreds of issues (under-grouping — usually dynamic strings in
    exception messages; move variables to structured context, keep messages
    static).
  - Every issue gets triaged: assign, resolve-in-release, or ignore-with-
    reason. "5,000 open unassigned issues" means the tracker is dead;
    institute a weekly triage rota and resolve-by-default policies for
    stale noise.
  - Resolved-then-reoccurred ("regression") notifications ON — that's the
    highest-signal notification type the tool has.
- Notification policy: new issue / regression / spike → TICKET (or chat),
  not page. Pages come from SLOs (rules/04); the tracker tells you WHICH
  exception is burning the budget.

## 6. Shutdown, crashes, and the last 10 seconds

The least-observed moments of a process are its first and last seconds —
and that's where deploy regressions and OOM mysteries live.

Rules:
- **Graceful shutdown is observable**: on SIGTERM log `shutdown_started`
  (with reason if known), flip readiness to failing, drain in-flight work
  with a deadline, flush telemetry exporters (spans, metrics, logs, error
  tracker), then log `shutdown_completed{drained=n, aborted=m}`. A deploy
  that loses its final telemetry batch hides exactly the requests it broke.
- **Crash forensics**: panics/fatal errors write a structured last-gasp
  line (and error-tracker event where the SDK supports fatal handling)
  before exit; container stdout is the channel of record — never only a
  file inside the dying container.
- **OOM kills are invisible to the app** — detect them from the outside:
  kube_state_metrics `OOMKilled` reason, exit code 137 tracking, and a
  memory-usage-vs-limit panel per workload. Recurring OOM = TICKET with the
  allocation profile attached (§4), not a silent restart loop.
- **CrashLoopBackOff has a budget**: restarts are a metric; > N restarts/h
  on one workload tickets the owner even if replicas mask user impact.
- Startup is logged once, structured: version, config hash (not values),
  migrations applied, listening ports. "What exactly is running right now"
  must be answerable from logs alone.

## 7. Dashboards that answer questions

A dashboard is a pre-computed answer to a question you expect to ask under
stress. A wall of 40 unlabeled graphs is a vanity wall, not a tool.

Rules:
- Name the question. Each dashboard (and ideally each row) answers
  something specific: "Is checkout healthy?" "Why is checkout slow right
  now?" "Are we keeping up with the queue?"
- Standard per-service layout, top to bottom = symptom to cause:
  1. SLO status + burn rate (is it broken? how badly?)
  2. RED per route (where is it broken?)
  3. Dependency latency/errors (is it them?)
  4. USE/saturation: pools, queues, CPU/mem (is it us, resource-wise?)
  5. Deploy/config-change annotations overlaid on everything (was it a
     change? — it usually was).
- Every paging alert's runbook links a dashboard whose top row confirms the
  symptom and whose rows below bisect causes.
- Link down the stack: dashboard panel → exemplar trace → logs by trace_id.
  A panel that can't lead anywhere deeper is a dead end at 3am.
- Dashboards as code (Grafana provisioning/Jsonnet/Terraform), reviewed,
  versioned. Hand-edited live dashboards drift and die.
- Delete dashboards nobody opened in 90 days (usage stats exist). Curation
  is a feature: the on-call landing page lists THE five dashboards that
  matter.
- No averaged percentiles, no per-instance p99 walls (rules/02 §4); prefer
  route/tenant breakdowns over instance breakdowns for symptom dashboards.

## 7a. The question with no instrument, and the substitute that answers a different one

§7 assumes the data exists. The harder failure is a question with **no** instrument
behind it — and it never presents as a gap, because somebody always finds a proxy
and the proxy returns a number.

The canonical case is a removal decision. *"Is this feature still used?"* is a
question about **requests**, and it is answerable only from request-level telemetry
at the edge: gateway/ingress/load-balancer access logs, or per-route and per-field
usage metrics (`sota-api-design` rules/02 §5 step 4, rules/03 §11 — *without
per-field usage data you can never delete anything*). With those absent, the
reachable substitute is the **stored data**: query the corpus, count what carries
the feature's shape, conclude.

That answers *"does data shaped like this exist"* — a different question, with a
different answer and a **known direction of error**. Stored data outlives its last
reader, so the corpus systematically over-reports use and the substitute is biased
toward keep-it. The shape recurs: commit count standing in for maintenance, a
dependency's presence in a manifest standing in for it being reached (`sota-devsecops`
rules/10), a dashboard existing standing in for someone opening it.

Rules:
- **Edge access logs are a required telemetry stream**, not an optional one, wherever
  a gateway/ingress/LB fronts a versioned or deprecable surface. Sample if volume
  demands, but **retain across one deprecation runway** — rules/02 §5 publishes a
  ≥ 6-month runway, so 30-day retention cannot support the decision it exists for.
  Log the route *template* and the principal *id*, not the raw path and identity
  (cardinality and PII: rules/01).
- **Instrument the question you will be asked, not only the ones you are asked
  today.** "Which surfaces can we retire?" is asked of every system that lives long
  enough; it needs a per-route/per-field usage counter from the day the surface
  ships (rules/02 §5's "usage metric the day its successor ships").
- **When you substitute, say so in the same sentence as the number.** Name the
  question you could answer, the question you were asked, and the direction the
  substitution errs. *"Zero rows carry this field"* is evidence; *"nobody uses this
  feature"* is a claim that measurement does not support.
- **Prefer instrumenting forward over inferring backward.** If the signal is absent
  and the decision is reversible, adding the counter and waiting one runway is
  usually cheaper and always sounder than building a more elaborate proxy. Record
  the gap as a finding in its own right — *a decision was made on a substitute
  measure* is a durable observability defect, and it recurs on the next removal.

## 8. Synthetic monitoring

Real-user telemetry goes silent exactly when traffic does — overnight
low-traffic windows, broken signup flows (no users get far enough to emit
errors), pre-launch features.

Rules:
- Probe every critical journey end-to-end (not just `/health`): scripted
  login → action → assert on response content, from outside your network,
  from the regions users are in.
- Tag synthetic traffic (`synthetic: true` header → wide-event field) so it
  is excludable from SLIs/business metrics while feeding its own
  availability SLI for low-traffic journeys (rules/04 §1).
- Probe failures page only on consecutive failures from multiple locations
  (single-location flaps are network noise).
- Certificates, DNS, and domain expiry are synthetic checks too — classic
  "no symptom until total outage" causes with perfect lead time.

## 8a. Test writes and production writes must not share a sink

§8 tags synthetic *probe* traffic so it is excludable from SLIs. The same requirement
holds one layer down and is met far less often: **telemetry a test run can write must be
distinguishable from telemetry production writes, at the point of collection.**

- Either a **separate sink** — a distinct directory, table, index, bucket prefix or
  dataset chosen by config — or a **stamped marker on every record** (`env: test`, the
  provider identity, the run id). Prefer the separate sink; a marker only helps a reader
  who already knows to filter on it.
- **Never a naming convention.** One filename pattern for both, told apart by "test runs
  happen to have a round row count", is not a filter — it is post-hoc archaeology,
  available only to someone who already suspects the problem. The same goes for filtering
  on a *value*: excluding rows by duration or size infers the population from the data
  instead of recording it.
- The stamp goes on at **write** time, in the emitting code. A field the reader adds can
  only classify what the reader already understands, which is the case that was never in
  doubt.
- **Check the ordering, not just the presence, of the stamp.** In this library's own eval
  harness the runner recorded its denominator *before* branching into `--selftest`, so a
  self-test row and a measurement row were byte-identical apart from the elapsed time —
  the marker existed and was written on the wrong side of the branch (2026-09-02).
- Git-ignoring or gitignoring a local sink is not isolation: it keeps the file out of the
  repo, not the test rows out of the aggregate.
- The reader-side obligation — every aggregate over a shared sink states its exclusion
  filter, and an unexplained jump in n is contamination rather than power — is
  `sota-code-security` rules/11 §2.7.

## Audit checklist

- [ ] Liveness, readiness, startup probes distinct; liveness contains NO
      dependency checks; readiness fails only on required deps; probes are
      cheap, cached, and bounded.
- [ ] Optional-dependency failure degrades service without failing
      readiness; shared-dependency behavior (fail vs degrade) is a
      documented decision.
- [ ] All degradation paths (fallbacks, breakers, kill switches) emit
      metric + log + span attribute when active; long-running degradation
      tickets someone.
- [ ] Port/route inventory done: no pprof/actuator/metrics/heapdump/env/
      introspection endpoints reachable without auth from outside the
      internal network; Go DefaultServeMux not publicly served with pprof
      imported.
- [ ] Continuous profiling running with version tags; on-demand capture
      path documented; off-CPU/lock profiling available where supported.
- [ ] No sink receives both test-suite and production telemetry without a write-time
      marker or a separate destination; any analysis over a shared sink states its
      exclusion filter (§8a).
- [ ] Error tracker captures all runtimes incl. frontend with sourcemaps;
      release+environment on every event; trace_id linked; PII scrubbed.
- [ ] Issue grouping healthy (no message-interpolation shatter); triage
      rota exists; regression notifications enabled; open-untriaged count
      is bounded.
- [ ] Per-service dashboard follows symptom→cause layout with deploy
      annotations; paging alerts link runbook + dashboard; panels
      click-through to traces/logs.
- [ ] Dashboards and alerts are code-reviewed and provisioned, not
      hand-edited; stale dashboards pruned.
- [ ] Edge access logs (gateway/ingress/LB) exist for every deprecable surface,
      with per-route/per-field usage counters and retention covering a full
      deprecation runway — and where a usage question was answered from stored
      data instead, the substitution and its direction of error are stated
      beside the number, not left implied (§7a).

# 06 — Observability Audit Playbook

How to assess a codebase/deployment's observability posture. The verdict
hinges on two questions, answered with evidence, not vibes:

1. **"Why is this request slow?"** — for an arbitrary single production
   request, can an engineer reconstruct where time went, today, with
   existing signals?
2. **"What broke at 3am?"** — would the team be paged for real user pain,
   and could a non-author on-call localize the cause in minutes?

## 1. Method: trace real paths, distrust claims

Do not grade by which tools are installed; OTel-in-requirements.txt proves
nothing. Pick 2–3 **critical user journeys** (the money paths) and follow
each through the code:

1. Entry point: is there middleware emitting a wide event / server span /
   RED metrics? Read the actual middleware config, not the framework docs.
2. Each outbound hop (DB, cache, HTTP, queue): spans? context injected?
   latency recorded per dependency?
3. Async continuation (consumer, job): context restored? its own wide
   event? lag metric?
4. Failure path: throw an exception mentally at each layer — where is it
   logged (once?), is the span marked, does the wide event still fire, does
   the error tracker see it, which alert (if any) fires, and at what user
   impact threshold?
5. The 3am simulation: pick last quarter's worst incident (or invent
   "dependency X p99 ×10"); walk the on-call path: which alert fires →
   which runbook → which dashboard → which trace/log query. Every missing
   link is a finding.

If you have live access, prefer empirical checks: pull one trace and count
services in it; grep an hour of logs for trace_id coverage; list firing and
recently-fired alerts; open the on-call dashboard cold and try to answer
"is checkout healthy right now".

## 2. Evidence-gathering greps and probes

Adapt to language/stack; these locate the load-bearing code fast.

```bash
# Logging reality: structured vs printf
grep -rEn 'print\(|console\.log|System\.out|fmt\.Print' --include='*.{py,js,ts,go,java}' src/ | head
grep -rEn 'logger\.|log\.|slog\.|zap\.|structlog|pino|winston' src/ | head

# Level abuse: expected events at error level
grep -rEn 'log(ger)?\.(error|Error)' src/ | grep -iE 'not found|retry|invalid input|missing param'

# Secrets/PII risk at call sites
grep -rEn 'log.*\b(password|token|secret|authorization|api_key|ssn|card)' -i src/
grep -rEn 'log.*(req\.(headers|body)|request\.(headers|json)|to_dict\(\)|JSON\.stringify\(req)' src/

# Correlation: who binds trace/request IDs, and is it middleware or hand-passed
grep -rEn 'trace_id|traceId|correlation|request_id|traceparent' src/ | head -30

# OTel posture: API vs SDK, where configured, propagators
grep -rn 'opentelemetry' --include='*' -l | head
grep -rEn 'TracerProvider|set_tracer_provider|NodeSDK|sdktrace' src/   # should hit ~1 file/service
grep -rEn 'inject|extract' src/ | grep -i propagat                      # queue propagation exists?

# Metrics cardinality: label values from variables = suspect
grep -rEn '\.labels\(|With\(prometheus\.Labels|attributes=' src/ | grep -vE '"(GET|POST|2xx|4xx|5xx)"'

# Health endpoints: what do they actually check
grep -rEn '(livez|healthz|readyz|/health|/ready|liveness|readiness)' src/ k8s/ deploy/ charts/

# Debug surface exposure
grep -rEn 'pprof|actuator|debug|heapdump|/metrics' src/ k8s/ ingress* charts/ | grep -ivE 'test|_test'

# Alerting/dashboards as code present at all?
find . -path ./node_modules -prune -o -name '*.y*ml' -print | xargs grep -lE 'alert:|groups:|burn|slo' 2>/dev/null
find . -name '*.json' | xargs grep -l '"panels"' 2>/dev/null | head

# Error tracking
grep -rEn 'sentry|rollbar|bugsnag|crashlytics|captureException' src/ | head
```

Then read the hits in full context — a grep match is a lead, not a finding.

## 3. Common gaps catalog

Pattern-match against these; each maps to the rules file that specifies the
fix. Severity per SKILL.md conventions.

**Logging (rules/01):**
- G1. printf/console logging on prod paths; unparseable. [MEDIUM]
- G2. No correlation/trace ID in logs; async/queue work logs orphaned. [HIGH]
- G3. Secrets/PII reachable in logs (headers, bodies, user objects dumped);
  no logger-level redaction. [CRITICAL]
- G4. ERROR noise: 4xx/no-rows/successful-retry at ERROR; log-and-rethrow
  duplicates. [MEDIUM, HIGH if error-rate alerts feed off it]
- G5. No wide event; debugging = joining 15 breadcrumbs by timestamp. [HIGH]
- G6. Per-item loop logging, health checks logged, no sampling/retention
  policy; cost unowned. [MEDIUM]

**Metrics (rules/02):**
- G7. No RED per route; or "error" undefined (4xx counted, timeouts not). [HIGH]
- G8. No saturation metrics on pools/queues — the "slow but CPU idle"
  blindspot. [HIGH]
- G9. Unbounded labels (IDs, raw paths, error strings). [HIGH; CRITICAL if
  user-mintable]
- G10. Latency as gauge/average; p99s averaged across instances; buckets
  never tuned to SLO thresholds. [MEDIUM–HIGH]
- G11. No exemplars / no metric→trace path. [MEDIUM]

**Tracing (rules/03):**
- G12. No tracing at all on a multi-service system. [HIGH]
- G13. Broken propagation: traces shatter at queue/job boundaries; one user
  action = N disconnected traces. [HIGH]
- G14. Vendor SDK calls scattered through business code; SDK config
  duplicated per module. [MEDIUM]
- G15. Naive 1% head sampling: zero traces of any error. [MEDIUM–HIGH]
- G16. PII in span attributes or baggage; baggage leaking to third parties. [CRITICAL]

**SLOs & alerting (rules/04):**
- G17. No SLOs; alerting is ad-hoc thresholds someone set in 2022. [HIGH]
- G18. Cause-based paging (CPU, restarts, disk%) with no symptom coverage;
  or every ERROR log pages. [HIGH]
- G19. Alerts without runbooks/owners; permanent silences; > a few pages
  per shift; on-call confirms they ignore some alerts. [HIGH]
- G20. SLO exists but no error budget policy; never influenced a decision. [MEDIUM]
- G21. No dead-man's-switch: monitoring outage = silence = "all good". [HIGH]

**Cross-cutting:**
- GX1. Signals exist but don't link: no trace_id in logs, no exemplars, no
  release tags — three databases that can't be joined. [HIGH]
- GX2. Observability only on HTTP: cron jobs, consumers, and batch
  pipelines emit nothing (check: does the nightly job's failure surface
  anywhere within 24h?). [HIGH]
- GX3. Staging-only telemetry config drift: sampling/exporters differ from
  prod so nothing is rehearsed where it matters. [MEDIUM]
- GX4. Tribal-knowledge debugging: the one engineer who knows the magic
  log query is the real observability system. Runbooks/dashboards absent
  or stale. [MEDIUM]

**Operational readiness (rules/05):**
- G22. Liveness probe checks dependencies (restart-storm primitive); or
  readiness == liveness == "return 200". [CRITICAL/HIGH]
- G23. Debug/profiling/metrics endpoints reachable without auth from
  outside the internal network. [CRITICAL]
- G24. Silent fallbacks: degradation with no metric/log/alert. [HIGH]
- G25. No error tracker, or tracker is a landfill (thousands untriaged, no
  release tags, no sourcemaps). [HIGH/MEDIUM]
- G26. No profiling story; perf incidents debugged by redeploy-and-pray. [MEDIUM]
- G27. Dashboard sprawl: vanity walls, no deploy annotations, no
  symptom→cause structure, hand-edited. [MEDIUM]
- G28. Telemetry pipeline unmonitored (export drops invisible); single
  collector SPOF. [HIGH]

## 4. Time-boxed audit plans

Match depth to the time available; always deliver the two verdicts.

**90 minutes (smoke audit):** run §2 greps; read the request middleware,
one queue consumer, the probe manifests, and the alert rules file; check
for trace_id in a sample log line; deliver verdicts + top-5 findings.

**1 day (standard):** full §1 journey trace for two paths incl. failure
path; label-cardinality review of every custom metric; alert inventory
against the rules/04 quality bar; debug-surface enumeration; scored rubric
+ full findings report.

**1 week (deep):** everything above, plus: live verification (break a
sandbox dependency and watch the signals), page-history analysis for a
quarter (actionability rate, pages/shift), telemetry cost review (top log/
metric/trace producers vs value), pipeline failure-mode testing (kill the
collector — does anyone notice?), and on-call interviews (the single
highest-signal source: ask "which alerts do you ignore?" and "what do you
wish you could see?").

Sequencing rule: hunt CRITICALs first (secrets in telemetry, liveness
dep-checks, exposed debug endpoints) — they are cheap to find with greps
and unacceptable to miss regardless of time box.

## 5. Scoring rubric

Score each pillar 0–3; report the profile, not just a total.

| Pillar | 0 — Blind | 1 — Basic | 2 — Solid | 3 — SOTA |
|--------|-----------|-----------|-----------|----------|
| Logging | printf, no IDs | structured, partial IDs | JSON + trace_id + redaction | + wide events, sampling, cost-managed |
| Metrics | none/host-only | some app metrics | RED+USE, bounded labels, real histograms | + exemplars, SLO-tuned buckets, cardinality guards |
| Tracing | none | edge-only spans | E2E propagation incl. queues, semconv | + tail sampling, profile/log/metric linkage |
| SLOs/alerts | ad-hoc thresholds | SLIs defined | SLOs + burn-rate paging + runbooks | + budget policy in use, alert reviews, DMS |
| Op readiness | none | health endpoint exists | correct probes, error tracker, secured debug | + continuous profiling, degradation visibility, dashboards-as-code |

Verdicts for the two questions:
- **"Why is this request slow?"** YES = trace_id from any log/event → full
  trace with per-dependency timing → profile if CPU-bound. PARTIAL = some
  hops visible. NO = timestamp archaeology.
- **"What broke at 3am?"** YES = symptom page fires at real impact, runbook
  + dashboard localize within minutes. PARTIAL = paged but must improvise.
  NO = customers are the alerting system.

## 6. Report structure

```
# Observability Audit — <system> — <date>

## Verdict
"Why is this request slow?"  — YES/PARTIAL/NO + one-line evidence
"What broke at 3am?"         — YES/PARTIAL/NO + one-line evidence
Pillar scores: Logging 2/3, Metrics 1/3, Tracing 0/3, SLOs 1/3, OpReady 2/3

## Critical & High findings
[findings in SKILL.md format, sorted by severity, with file:line evidence]

## Medium & Low findings
[...]

## Shortest path to YES
1–5 ordered moves with highest MTTR-reduction per effort, e.g.:
1. Add trace_id injection middleware + wide event (S) — unlocks correlation
2. Replace CPU pagers with 2-tier burn-rate alerts on checkout SLI (M)
3. Move liveness dep-checks to readiness (S) — removes restart-storm risk
```

Rules for findings:
- Every finding carries verbatim evidence (file:line snippet, alert YAML,
  probe config). No evidence → no finding.
- Distinguish "absent" from "present but broken" — broken telemetry that
  the team trusts is worse than a known gap.
- Note what is GOOD too: the team must know what not to break, and audits
  that only criticize get ignored.
- Prioritize by incident impact, not by purity: a missing wide event on the
  checkout path outranks printf logging in an internal cron.

## Audit checklist (meta — did the audit itself cover everything)

- [ ] 2–3 critical user journeys traced through actual code, entry to
      async tail, including the failure path.
- [ ] The two verdict questions answered with named evidence.
- [ ] All five pillars scored; gaps mapped to catalog IDs and rules files.
- [ ] Secrets/PII telemetry scan performed (logs, span attributes, baggage,
      error tracker payloads).
- [ ] Probe configs (liveness/readiness) read from deploy manifests, not
      assumed; debug surface enumerated from route/port truth.
- [ ] Alert inventory reviewed against actionability/runbook/owner bar;
      on-call interviewed or page history sampled if accessible.
- [ ] Telemetry pipeline itself assessed (sampling config, export loss,
      collector SPOF, dead-man's-switch).
- [ ] Findings carry file:line evidence, severity, concrete fix, effort;
      "shortest path to YES" list delivered.