# 01 — REST/HTTP API Design

Scope: resource modeling, method/status semantics, pagination, filtering, partial
responses, idempotency, conditional requests, hypermedia, error format, spec-first
workflow, contract testing.

## 1. Resource modeling

- Model **nouns, not verbs**. URLs identify resources; methods supply the verb.
  - Bad: `POST /createUser`, `POST /users/123/activate-account-now`
  - Good: `POST /users`, `POST /users/123/activations` (action reified as a sub-resource)
- When an operation genuinely isn't CRUD (search with a huge body, batch mutation,
  state transition), reify it as a resource: `POST /searches`, `POST /transfers`,
  `POST /orders/{id}/cancellations`. This gives the operation an ID, a status, and
  auditability for free.
- For pure reads that merely outgrow the URL, the **QUERY method** (RFC 10008, 2026)
  is the standards-track alternative — safe, idempotent, cacheable — where your
  proxies/gateways/frameworks support it; `POST /searches` stays the compatibility fallback.
- Use plural nouns consistently (`/users`, `/users/{id}`). Never mix `/user/{id}`
  and `/users`.
- Limit nesting to **two levels** (`/orgs/{org}/projects/{id}`). Deeper nesting bakes
  hierarchy into clients; prefer top-level collections with filters:
  `/comments?post_id=42` over `/posts/42/threads/7/comments/9`.
- IDs: opaque, non-enumerable (ULID/UUIDv7 or prefixed IDs like `cus_01H...`).
  Sequential integer IDs leak volume and invite IDOR scanning. Prefixed IDs
  (Stripe-style) make logs and support tickets self-describing.
- Resource representation = stable contract, not a DB row dump. Never serialize ORM
  entities directly; map through an explicit response type so schema migrations don't
  silently change the API.

## 2. HTTP method semantics

| Method | Safe | Idempotent | Use |
|---|---|---|---|
| GET | yes | yes | Read. Never mutate on GET — caches/prefetchers will trigger it. |
| HEAD | yes | yes | Metadata/existence checks. |
| QUERY | yes | yes | Safe read with a request body (RFC 10008, 2026); responses cacheable. Searches/filters too large for a URL. Ecosystem support is early — fall back to `POST /searches` (§1) where proxies/frameworks lack it. |
| PUT | no | yes | Full replace at a client-chosen or known URL. |
| PATCH | no | **no** (unless designed so) | Partial update. Prefer JSON Merge Patch (RFC 7396) for simple cases; JSON Patch (RFC 6902) when array surgery is needed. |
| POST | no | no | Create under a collection; non-idempotent actions. Pair with idempotency keys (§6). |
| DELETE | no | yes | Delete. Repeat DELETE returns 404 or 204 — both acceptable; pick one and document it. |

- `PUT` with client-generated IDs (`PUT /documents/{uuid}`) gives you idempotent
  create-or-replace for free — often better than `POST` + idempotency key.
- Don't tunnel everything through POST "because firewalls". If a gateway blocks
  PATCH, use `POST` + `X-HTTP-Method-Override` only as a documented escape hatch.
- **Per-route method allowlist** (OWASP REST): each route permits an explicit set
  of methods and returns `405` for the rest — don't let the framework expose
  `PUT`/`DELETE`/`TRACE` on a read-only route by default. Re-check authorization
  **per method**, not just per path (a user allowed to `GET` an order is not
  thereby allowed to `DELETE` it).

## 3. Status codes — the ones that matter

- `200` body returned; `201` + `Location` header on create; `202` accepted for async
  work (return a status resource URL); `204` success, no body.
- `400` malformed request (syntax, types); `422` well-formed but semantically invalid
  (validation). Pick one convention and hold it — many shops use `400` for both;
  fine, but never both interchangeably.
- `401` not authenticated (missing/bad credentials, send `WWW-Authenticate`);
  `403` authenticated but not allowed. Returning `404` instead of `403` is a valid
  **deliberate** choice to hide resource existence across tenants — document it.
- `404` not found; `409` state conflict (duplicate, version clash); `410` gone
  permanently (deprecated endpoint after sunset).
- `412` precondition failed (ETag mismatch); `428` precondition required (you demand
  `If-Match` and it's missing).
- `429` rate limited + `Retry-After`; `413` body too large; `415` wrong content type
  (request); `406` when you can't satisfy the client's `Accept` (response negotiation).
- **Content negotiation hygiene** (OWASP REST): validate `Content-Type` on writes,
  honour `Accept` for the response and `406` when unsupported — but **never reflect
  the client's `Accept` value into the response `Content-Type`** (a path to XSS /
  content sniffing). Serve JSON as `application/json`, never `application/javascript`.
- **Declare each operation's media types; refuse the rest with `415`.** Every
  operation lists the request and response types it supports (the OpenAPI
  `requestBody.content` / `responses.*.content` keys, a framework `consumes`/
  `produces`), and anything else is rejected rather than parsed. Otherwise the
  framework picks a parser from the client's `Content-Type`, and a JSON endpoint
  quietly becomes an XML one the day an XML library lands on the classpath
  (Spring's default converter set, for one, adds an XML converter when Jackson
  XML or JAXB is present) — the route to XXE and parser-differential bugs.
  OWASP: REST Security cheat sheet.
- `500` your bug; `502/503/504` upstream/overload/timeout. **Never** return `200`
  with `{"error": ...}` in the body — it breaks retries, monitoring, caching, and
  every generic client.

```http
POST /users HTTP/1.1            HTTP/1.1 201 Created
Content-Type: application/json  Location: /users/usr_01HZX4
                                Content-Type: application/json
{"email":"a@b.co"}              {"id":"usr_01HZX4","email":"a@b.co"}
```

## 4. Pagination — cursor over offset

- **Offset pagination (`?page=3&per_page=50`) is broken at scale**: O(n) skip cost in
  the DB, and rows shift under the client when items are inserted/deleted mid-scan
  (skipped or duplicated records). Acceptable only for small, admin-facing,
  rarely-changing datasets.
- **Cursor (keyset) pagination is the default.** Cursor encodes the position
  (e.g., `(created_at, id)` of last item), opaque and signed/encoded:

```http
GET /orders?limit=50&cursor=eyJpZCI6Im9yZF8wMUha...
HTTP/1.1 200 OK
{
  "data": [...],
  "next_cursor": "eyJpZCI6...",   // null when exhausted
  "has_more": true
}
```

- Cursor rules: opaque (base64 of internal position — clients must not parse it),
  short-lived validity is fine, must tolerate the anchor row being deleted (seek
  semantics: `WHERE (created_at, id) < (?, ?)`), and **must embed the sort/filter**
  or reject cursors used with different query params.
- Always enforce a server-side `limit` max (e.g., 100). Reject or clamp larger values
  — document which.
- Return total counts only when cheap and genuinely needed; `COUNT(*)` on large
  tables is a self-DoS. Offer `?include_total=true` as opt-in, or an estimate.
- Stable sort always requires a unique tiebreaker (`ORDER BY created_at, id`).

## 5. Filtering, sorting, partial responses

- Filtering: flat query params for the common case (`?status=active&customer_id=...`).
  For richer needs adopt one convention and document it: bracketed operators
  (`?created_at[gte]=2026-01-01`) or a defined mini-language. Never accept raw query
  expressions you interpolate into SQL.
- Filter/search payloads too large for a URL: QUERY (RFC 10008) where supported,
  else reified `POST /searches` (§1).
- Validate every filter/sort field against an **allowlist**. `?sort=password_hash`
  or sorting on an unindexed column is an availability bug.
- Sorting: `?sort=-created_at,name` (leading `-` = desc). Multi-field allowed, all
  fields allowlisted and indexed.
- Partial responses: `?fields=id,email,profile.name` (sparse fieldsets) for large
  resources; saves bandwidth and discourages clients from coupling to fields they
  don't need. Define behavior for unknown fields (ignore or 400 — pick one).
- Expansion: `?expand=customer,items.product` to inline related resources instead of
  forcing N+1 client round-trips. Cap expansion depth.

## 6. Idempotency keys for unsafe operations

POST that creates a payment/order/email must be safely retryable — networks fail
after the server commits.

```http
POST /payments HTTP/1.1
Idempotency-Key: 6c1f6c1e-9c5a-4f7e-b2f3-1d2a3b4c5d6e
{"amount": 5000, "currency": "EUR"}
```

Server contract (Stripe semantics; the IETF `draft-ietf-httpapi-idempotency-key-header`
captured the same convention but expired without becoming an RFC):
- Key + endpoint + auth principal scope the dedupe record.
- First request: store key + request hash **before** executing; execute; store the
  response; return it.
- Retry with same key + same body: replay the **stored response** (same status, body).
- Same key + **different body**: `422`/`409` — never execute.
- Concurrent duplicate while first is in flight: `409` (or block) — never run twice.
- Keys expire (24h typical). Persist in the same transaction as the side effect or
  use an outbox; a Redis-only record that can vanish independently of the DB write
  is a false guarantee.

## 7. Conditional requests (ETags) and concurrency

- Emit `ETag` on single-resource GETs. Strong ETags from a version column or content
  hash; `Last-Modified` as a coarse fallback.
- Reads: `If-None-Match` → `304 Not Modified` (bandwidth + cache validation).
- Writes (optimistic concurrency — prevents lost updates):

```http
PUT /articles/42                 HTTP/1.1 412 Precondition Failed
If-Match: "v17"                  (someone wrote v18 meanwhile)
```

- For resources where lost updates are costly, **require** `If-Match` and return
  `428 Precondition Required` when absent.
- Cache headers are part of the design: `Cache-Control: private, max-age=0,
  must-revalidate` for per-user data; explicit `no-store` for sensitive payloads;
  `public, max-age=...` only for genuinely shared data. Unstated caching = whatever
  intermediaries feel like.

## 8. HATEOAS — pragmatic dose

Full HATEOAS (clients discover all transitions from hypermedia) rarely pays off for
machine clients with generated SDKs. The pragmatic subset that does:
- `Location` on 201/202.
- Pagination links (`next_cursor` or RFC 8288 `Link` headers).
- Status/affordance hints on workflow resources:
  `{"status":"pending_approval","actions":["approve","reject"]}` — clients render
  capabilities without hardcoding the state machine.
- Absolute or root-relative URLs in link fields; never make clients string-build URLs
  from IDs when you can hand them the URL.
Skip generic `_links` envelopes (HAL/Siren) unless your clients actually walk them.

## 9. Error format — RFC 9457 (problem+json)

One error shape across the whole API. RFC 9457 (obsoletes 7807) is the standard:

```http
HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json
{
  "type": "https://api.example.com/errors/validation",
  "title": "Validation failed",
  "status": 422,
  "detail": "2 fields failed validation.",
  "instance": "/payments/req_01HZX4",
  "errors": [
    {"pointer": "/amount", "code": "min", "message": "must be >= 100"},
    {"pointer": "/currency", "code": "unsupported", "message": "JPY not supported"}
  ],
  "trace_id": "a1b2c3d4e5f6"
}
```

- `type` is a stable machine-readable identifier (URI; need not resolve). Clients
  branch on `type`/`code`, never on `detail` text.
- Extend with custom members (`errors`, `trace_id`, `retry_after`) — the RFC allows it.
- Include a correlation/trace ID in every error for support.
- **Never leak internals**: no stack traces, SQL, file paths, or dependency versions
  in any environment reachable by clients. 500s get a generic body + trace ID.
- Validation errors: report **all** failures in one response, not first-failure.

## 10. OpenAPI-first vs code-first; contract testing

- **Spec-first (OpenAPI 3.1+)** is the default for public/partner APIs: the spec is
  the contract, reviewed like code, before implementation. Enables parallel
  client/server work, generated SDKs, mock servers, and breaking-change detection in
  CI (e.g., `oasdiff`). OpenAPI 3.2 (released Sep 2025) is a non-breaking superset of
  3.1 adding first-class streaming (SSE/JSON-Lines via `itemSchema`), hierarchical
  tags, and custom HTTP methods; adopt it when tooling supports it. (4.0 "Moonwalk"
  is still in design — not a target.)
- **Code-first with generated spec** is acceptable for internal APIs *iff* the
  generated spec is committed, diffed in CI, and breaking diffs fail the build. A
  spec nobody diffs is documentation, not a contract.
- Either way, non-negotiable: the served API and the spec **must not drift**.
  Validate requests/responses against the schema in test (or runtime middleware in
  staging).
- Contract testing: schema validation of real responses in CI at minimum;
  consumer-driven contracts (Pact) when you control both sides and have many
  consumers; record real consumer expectations, verify provider against them in the
  provider's pipeline.
- OpenAPI hygiene: every operation has `operationId`, every response (incl. errors)
  schematized, `additionalProperties` intent explicit, enums marked extensible where
  evolution is expected (see rules/02), auth schemes declared in `securitySchemes`.
- **Every operation states its own `security` requirement** (which scheme, which
  scopes), not only the top-level default. In OpenAPI 3.1 an operation's
  `security` overrides the global one, an empty array (`security: []`) removes
  it, and an empty object (`{}`) in the list makes auth optional, so each of
  these is a reviewed decision, never an accident of inheritance. That
  per-operation table is the source of truth for authorisation regression tests:
  generate a case per operation (no credential, wrong scope, right scope) and
  have the gateway or middleware enforce the same declarations. OWASP:
  Authorization Regression Testing and Microservices based Security Arch Doc
  cheat sheets.

## 11. Async operations (202 pattern)

Anything that can exceed a few seconds (report generation, bulk import, video
transcode) must not hold a synchronous connection.

```http
POST /reports HTTP/1.1                  HTTP/1.1 202 Accepted
{"type":"annual","year":2025}           Location: /operations/op_01HZX9
                                        {"id":"op_01HZX9","status":"pending"}

GET /operations/op_01HZX9               HTTP/1.1 200 OK
                                        {"id":"op_01HZX9","status":"succeeded",
                                         "result_url":"/reports/rep_42",
                                         "created_at":"...","finished_at":"..."}
```

- Operation resource carries: `status` (`pending|running|succeeded|failed`),
  result link on success, problem+json-shaped `error` on failure, timestamps,
  and percentage/progress where meaningful.
- Clients poll with backoff; offer `Retry-After` on the operation GET, and/or a
  webhook (rules/06) for completion. Operations are listable per principal and
  retained long enough for slow pollers (≥24h).
- The initial POST still takes an Idempotency-Key (§6) — duplicate submissions
  must return the same operation, not start a second job.

## 12. Bulk and batch endpoints

- Prefer many small idempotent requests over bespoke batch endpoints; HTTP/2
  multiplexing removes most of the round-trip motivation.
- When batch is justified (mobile constraints, atomic multi-item writes): cap
  batch size, define atomicity explicitly (all-or-nothing vs per-item), and for
  per-item semantics return `207`-style per-item results — each entry carrying
  its own status + problem details, in input order:

```json
{"results":[
  {"status":201,"id":"usr_1"},
  {"status":422,"error":{"type":".../validation","detail":"email taken"}}
]}
```

- Never report overall `200` while hiding per-item failures the client can't
  detect without parsing prose.

## 13. Misc non-negotiables

- JSON: UTF-8, ISO 8601 / RFC 3339 timestamps **with offset** (`2026-06-12T09:30:00Z`),
  never epoch-seconds-as-float. Money as integer minor units or string decimal —
  never IEEE 754 floats.
- Field naming: pick `snake_case` or `camelCase` once, enforce with lint.
- Booleans over flag-strings; enums as strings, not magic ints.
- Nulls vs absent fields mean different things in PATCH — define it (Merge Patch:
  `null` deletes).
- Request size limit, response compression (`gzip`/`br`), and a per-request timeout
  budget exist on every endpoint (details: rules/07).
- **The secure use has to be the easy one — your API's ergonomics are a control you own.**
  If a caller must read the docs, remember a rule, or pass an extra argument to be safe, the
  interface has moved your security into their discipline. Two shapes recur and both are
  audited elsewhere in the library, listed here because this is where they are *designed*:
  a parameter selecting a trust boundary whose default is the **unsafe** value — make the
  closed side the default and the open side explicit and keyword-only, which also makes
  every privileged call site a `grep -c`
  (`sota-code-security` rules/14 §6a) — and letting a **caller or the payload choose the
  security-critical algorithm**, the `alg` pattern, where the fix is a server-side allowlist
  rather than a documented warning (`sota-code-security` rules/02). "It is documented" is
  not a mitigation; neither is "no one would do that".

## Audit checklist

- [ ] URLs are nouns; no verbs except reified action sub-resources; consistent plural casing.
- [ ] No GET/HEAD endpoint mutates state.
- [ ] Status codes semantically correct; no `200 {"error":...}` anti-pattern anywhere.
- [ ] **Media types allowlisted per operation (§3) — MEDIUM, HIGH if an XML parser is reachable**: undeclared `Content-Type` → `415`; the spec lists every operation's request/response types. Spring write mappings with no `consumes` (each hit accepts whatever converters are registered; confirm by sending `Content-Type: application/xml`):
      `grep -rnE '@(Post|Put|Patch)Mapping' --include='*.java' --include='*.kt' . | grep -v 'consumes'`
- [ ] 201 responses include `Location`; async operations return 202 + status resource.
- [ ] Collection endpoints use cursor pagination with opaque cursors, enforced max limit, stable sort with unique tiebreaker.
- [ ] No unbounded `COUNT(*)`/total on large collections by default.
- [ ] Filter/sort fields allowlisted; sorts hit indexes; no raw query-language injection path.
- [ ] Every non-idempotent unsafe operation (payments, orders, sends) supports idempotency keys with stored-response replay and same-key-different-body rejection.
- [ ] Concurrent duplicate idempotency-key requests cannot double-execute (locking/unique constraint verified).
- [ ] ETags emitted; mutating endpoints on contended resources honor `If-Match`/412.
- [ ] Cache-Control explicitly set; sensitive responses `no-store`.
- [ ] Errors are RFC 9457 problem+json, single shape API-wide, machine-readable `type`/`code`, all validation errors batched, trace ID present.
- [ ] No stack traces / SQL / internal paths in any client-visible error.
- [ ] IDs opaque and non-enumerable; no sequential integers exposed.
- [ ] Resources mapped through explicit DTOs, not raw ORM serialization.
- [ ] OpenAPI spec exists, is in CI, breaking-change diff fails the build, and matches the served API (spot-check 3 endpoints).
- [ ] **Per-operation security (§10) — HIGH when a write is unauthenticated**: every operation declares scheme and scopes and feeds the authz regression suite. Operations that drop or relax auth (each hit must be an intended public endpoint):
      `grep -rnE '"?security"?[[:space:]]*:[[:space:]]*\[[[:space:]]*\]|"?security"?[[:space:]]*:[[:space:]]*\[[[:space:]]*\{[[:space:]]*\}|^[[:space:]]*-[[:space:]]*\{[[:space:]]*\}[[:space:]]*$' --include='*.yaml' --include='*.yml' --include='*.json' .`
- [ ] Timestamps RFC 3339 with timezone; money not floats.

# 02 — Versioning, Evolution & Deprecation

Scope: additive-change discipline, what counts as breaking, version placement
tradeoffs, tolerant readers, deprecation lifecycle (Deprecation/Sunset headers),
enum and schema evolution.

## 1. The prime directive: evolve, don't version

A new major version is a **failure mode**, not a tool. Every major version doubles
maintenance, splits traffic and docs, and strands clients. Design v1 so that years
of change stay additive. Teams that internalize this ship `v1` forever (Stripe runs
one URL version + dated changes); teams that don't accumulate `v1`–`v4` zombies.

## 2. What is breaking vs additive

**Additive (safe, ship anytime):**
- New optional request field, new request header, new query param (with default).
- New response field. New endpoint. New optional enum *input* value.
- Relaxing validation (accepting more than before).
- New error `code` under the existing error shape.

**Breaking (requires deprecation cycle or new version):**
- Removing/renaming any field, endpoint, or query param.
- Type change (`string`→`int`), format change (epoch→ISO), semantic change of an
  existing field ("amount" switches units).
- Tightening validation (rejecting what used to pass).
- New **required** request field. Changing defaults. Changing auth requirements.
- Changing status codes for existing situations (404→410 is breaking for clients
  that branch on it).
- New value in a response enum **is breaking for clients that switch exhaustively**
  — see §6.
- URL structure changes, pagination scheme changes, error shape changes.

Gray zone — treat as breaking unless contract says otherwise: field ordering
(never promise it), response timing/latency characteristics clients depend on,
rate-limit reductions.

Enforce mechanically: OpenAPI/proto diff in CI (`oasdiff`, `buf breaking`) that
fails the pipeline on breaking diffs. Human review alone misses these.

## 3. Tolerant reader / never break parsers

Both sides of the contract:

**Servers must promise:** unknown response fields may appear at any time. Document
this loudly: *"Clients MUST ignore unknown fields."* This single sentence is what
makes additive evolution legal.

**Clients you write must:**
- Ignore unknown JSON fields (configure deserializers: don't `FAIL_ON_UNKNOWN_PROPERTIES`,
  don't `additionalProperties: false` on response validation).
- Handle unknown enum values (map to `UNKNOWN`, don't crash — §6).
- Never depend on field order or absence-vs-null distinctions not in the contract.

**Servers receiving requests:** decide and document unknown-request-field policy.
Ignoring is lenient and traditional; rejecting (`400`) catches client typos
(`"ammount"` silently ignored = money bug). Best practice 2026: **reject unknown
request fields on write operations**, ignore on reads/filters. Either way: explicit.

## 4. Version placement: URL vs header

| | URL (`/v1/users`) | Header (`Api-Version: 2026-06-01`) |
|---|---|---|
| Visibility | obvious in logs, curl, docs | hidden; needs tooling |
| Caching/routing | trivial (path-based) | needs `Vary: Api-Version` |
| "REST purity" | one resource, many URLs | one URL, content negotiation |
| Granularity | big-bang major versions | per-change dates possible |
| Client mistake mode | can't forget it | forgot header → which default? |

**Recommendation:**
- Public API: coarse URL major (`/v1/`) that you intend never to bump, **plus**
  date-based header versioning for behavioral changes (Stripe model:
  `Stripe-Version: 2026-06-12`; account pinned to first-seen date; upgrades opt-in
  per account). This combines discoverability with fine-grained evolution.
- Internal/service-to-service: no URL version; additive-only + tolerant readers +
  CI breaking-change gates. gRPC: package version in proto (`package billing.v1`).
- Never: version in query param (`?v=2`) — cache-key and default ambiguity; per-endpoint
  version mixing (`/v1/users` calls `/v2/orders` semantics) without explicit design.
- If header-versioned: an **explicit, documented default** for missing header
  (oldest-supported or account-pinned — never "latest", which breaks clients on
  every release).

## 5. Deprecation lifecycle

Killing anything (field, endpoint, version, auth scheme) follows a pipeline:

1. **Announce** — changelog, email to affected key owners, migration guide with
   before/after examples.
2. **Mark in spec** — OpenAPI `deprecated: true`; proto `[deprecated = true]`;
   GraphQL `@deprecated(reason: "Use X. Removed 2026-12-01.")`.
3. **Signal in responses** — standard headers:

```http
HTTP/1.1 200 OK
Deprecation: @1767225600                     # RFC 9745: deprecated as of (unix ts)
Sunset: Mon, 01 Jun 2027 00:00:00 GMT        # RFC 8594: will stop working at
Link: <https://api.example.com/docs/migrate-orders-v2>; rel="deprecation"
```

4. **Measure** — per-key metrics on deprecated-surface usage. You cannot sunset
   what you can't attribute. Dashboards by API key/account. If that telemetry does
   not exist, the substitute — counting stored data that carries the feature's shape
   — answers a *different* question and errs toward keep-it: `sota-observability`
   rules/05 §7a.
5. **Nag** — targeted emails to remaining callers; for stragglers, scheduled
   **brownouts** (return `410`/`503` for 5 minutes, then 1 hour, announced in
   advance) — converts ignored emails into pager alerts on the client side.
6. **Remove** — after the sunset date, return `410 Gone` with a problem+json body
   pointing at the migration guide. Not `404` (looks like a client bug).

Minimum runway: 6 months public APIs (12 for auth/breaking-payment changes),
1–3 months internal with known consumers. Put the policy in your docs *before*
you need it.

## 6. Enum and schema evolution

Enums are the most common evolution trap.

- **Response enums**: adding a value breaks exhaustive-matching clients. Options:
  (a) document from day one that enums are **open** — clients must treat unknown
  values as a defined fallback (`"other"`/ignore); (b) gate new values behind the
  version/date header so only opted-in clients see them. Do (a) always, (b) for
  high-impact enums like `status`.
- **Request enums**: adding accepted values is safe.
- Booleans become enums: model tri-state-prone fields as string enums from the start
  (`"state": "active"` not `"is_active": true`) — you cannot evolve a boolean.
- Nullable changes: optional→required on responses is safe; anything→nullable on
  responses is breaking. On requests, the reverse.
- Width/format: never repurpose a field. New semantics ⇒ new field name
  (`amount_cents` alongside deprecated `amount`), dual-write during transition,
  sunset the old one via §5.
- Proto specifics (field numbers, `reserved`) in rules/04. GraphQL deprecation in
  rules/03.

## 7. Running multiple versions

If you do end up with versions:
- **Translate at the edge, one core**: maintain a single internal model; each
  version is a request/response transformation layer (Stripe's
  version-change-modules pattern). Never fork business logic per version.
- Version compatibility tests: golden request/response fixtures per supported
  version, run in CI forever.
- Cap concurrent supported versions (2–3). Each old version must have an owner, a
  metric, and a sunset date the day its successor ships.
- New features land **only** in the newest version — carrot for migration.

## 8. Date-based header versioning — concrete mechanics

The Stripe model, since it's the one worth copying:

```http
GET /v1/subscriptions/sub_9 HTTP/1.1
Authorization: Bearer sk_live_...
Api-Version: 2026-06-01
```

- Each dated version = a small, named change module ("2026-06-01: `status`
  gains value `paused`; `discount` becomes a list"). The core serves only the
  newest shape; modules transform responses backwards (and requests forwards)
  in a chain. New version = new module; old modules are never edited.
- Account pinning: the first version an account ever uses becomes its default;
  requests without the header get the pinned version. Upgrading the pin is an
  explicit, reversible dashboard/API action — ideally with a "preview newest
  against my recent traffic" diff.
- Each module ships with: changelog entry, migration note, and a pair of
  round-trip tests (new-shape → module → old-shape fixtures).
- This machinery costs real engineering. If you can't fund it, the honest
  alternative is: strict additive-only forever + the §5 deprecation pipeline —
  which is what most internal APIs should do anyway.

## 9. Compatibility testing in practice

Breaking-change CI diffs (§2) catch *schema* breaks; behavioral breaks need
golden fixtures:

```text
tests/contract/
  2025-09-01/
    list-orders.request.json      # frozen real request
    list-orders.response.json     # frozen expected response (shape-asserted)
  2026-06-01/
    list-orders.request.json
    list-orders.response.json
```

- Replay every supported version's fixtures against every release; assert shape
  and semantics (field present, type, enum within documented set) — not
  byte-equality (timestamps/IDs vary; over-strict goldens rot).
- Add a fixture the day a version ships; delete it the day the version sunsets.
  The fixture set *is* your supported-version inventory.
- For consumer-driven contracts (Pact et al.): consumer teams publish
  expectations from their actual usage; provider CI verifies before deploy.
  This beats golden files when you control the consumers — it tests what's
  *used*, enabling deletion of what isn't.

## 10. Changelogs and client communication

- Machine-readable changelog (one entry per change: date, surface affected,
  additive/deprecation/breaking, link). Generated from spec diffs where
  possible; hand-written prose drifts.
- Every API key/app has a registered owner contact; deprecation mail goes to
  owners of keys that *actually called* the deprecated surface in the last 90
  days (from §5 metrics) — not a newsletter blast everyone filters.
- SDKs are part of the contract: regenerate and release SDKs with every spec
  change; an SDK that lags the API teaches clients to bypass it. Pin SDK major
  versions to API behavior expectations and document the mapping.
- Sandbox mirrors production versioning — clients must be able to test against
  the version they'll be pinned to, and against the newest, in sandbox.

## Good/bad example

```jsonc
// BAD v1 design (forces v2 later)
{
  "name": "Ada Lovelace",          // can't split into first/last additively
  "is_active": true,               // boolean — can't add "suspended"
  "created": 1718180000,           // epoch int; tz-ambiguous
  "type": 2                        // magic int enum
}

// GOOD v1 design (evolves forever)
{
  "display_name": "Ada Lovelace",
  "given_name": "Ada",
  "family_name": "Lovelace",
  "status": "active",              // open string enum, documented fallback
  "created_at": "2026-06-12T09:33:20Z",
  "type": "organization"
}
```

```jsonc
// BAD deprecation (silent breakage)
//   v1.1 release notes: "renamed `name` to `display_name`"  -> 500s for everyone

// GOOD deprecation (dual-field transition)
{
  "name": "Ada Lovelace",          // still served; marked deprecated in spec,
                                   // Deprecation/Sunset headers on responses
  "display_name": "Ada Lovelace"   // new field, added 2026-06; docs updated
}
// ...usage metric for `name` readers -> 0 -> remove after Sunset date
```

## Audit checklist

- [ ] Documented compatibility policy exists: what's additive vs breaking, unknown-field rules for both directions.
- [ ] Clients-must-ignore-unknown-fields stated in docs; server is free to add response fields.
- [ ] Server's unknown-request-field policy explicit (reject on writes recommended); typo'd field names cannot be silently dropped on money/critical writes.
- [ ] CI runs spec diff (oasdiff / buf breaking / GraphQL schema check) and fails on breaking changes.
- [ ] Version scheme is deliberate: URL major never bumped casually; header/date versioning has a safe documented default (never implicit "latest").
- [ ] No business logic forked per version — version adapters at the edge only.
- [ ] Deprecated surface marked in spec AND signaled at runtime (`Deprecation`, `Sunset`, `Link` headers).
- [ ] Per-key usage metrics exist for deprecated endpoints/fields; sunset decisions are data-driven.
- [ ] Published deprecation runway (≥6mo public) with migration guides; removed endpoints return `410` + pointer, not `404`.
- [ ] Response enums documented as open; client SDKs you ship tolerate unknown enum values and unknown fields.
- [ ] No repurposed fields in history (same name, changed semantics) — grep changelog/spec history.
- [ ] State-like fields are string enums, not booleans; timestamps RFC 3339; no magic-int enums.
- [ ] Old supported versions have golden contract tests, owners, and sunset dates.

# 03 — GraphQL

Scope: when (not) to use it, schema design, N+1/dataloaders, query cost controls,
persisted queries, error handling, security, evolution.

## 1. When GraphQL is the wrong choice

Choose GraphQL when: many heterogeneous clients (web + mobile + partners) with
divergent data shapes; aggregation over multiple backends (BFF/federation); rapid
frontend iteration where the server team is a bottleneck.

It is the **wrong** choice when:
- **Server-to-server APIs** — fixed call shapes; REST/gRPC are simpler, cacheable,
  and have better tooling for this.
- **One client, one team** — you pay the complexity tax (resolvers, dataloaders,
  cost limits, caching workarounds) without the flexibility benefit.
- **File upload/download heavy** — multipart-over-GraphQL is a hack; use signed
  URLs + REST.
- **HTTP-cache-dependent** — POST-based queries bypass CDN/browser caches; if your
  win is `Cache-Control: public`, GraphQL erases it (GET + persisted queries
  partially restores it, §5).
- **Hard latency/throughput budgets** — resolver fan-out and JSON weight lose to
  gRPC.
- **Public unauthenticated APIs without resourcing** — you are signing up for
  cost-analysis, depth limits, and abuse engineering; if you can't staff that,
  expose REST.

A GraphQL layer that fronts exactly one REST API 1:1 is pure overhead — delete it
or give it a real aggregation job.

## 2. Schema design

- **Schema-first, design-reviewed**: the SDL is the contract; review it like an
  OpenAPI spec. Run schema checks (Apollo/Hive/`graphql-inspector`) in CI against
  production traffic to catch breaking changes.
- Nullability is a promise: **non-null (`!`) only when the server can always
  deliver**, because a null in a non-null field destroys the whole parent selection
  (error bubbling). Nullable-by-default on object fields backed by other services;
  non-null for IDs and intrinsic scalars.
- Connections, not naked lists: Relay connection spec (`edges/node/pageInfo`,
  `first/after`) for anything that can grow. Naked `[Order!]!` is an unbounded
  query waiting to happen. Enforce a max `first`.
- Mutations: one **input object** per mutation (`input PlaceOrderInput`), one
  **payload type** per mutation containing the changed entity + `userErrors:
  [UserError!]!` (§6). Name mutations verb-first: `placeOrder`, `cancelSubscription`.
- Mutually exclusive input variants: use a **`@oneOf` input object** (standardized
  in the September 2025 spec edition) — exactly one field set, enforced by the type
  system — instead of multiple nullable fields plus runtime validation.
- Global object identification: `id: ID!` globally unique (encode type+id),
  `node(id:)` lookup — enables client-side normalized caching.
- Model relationships as fields, not foreign keys: `order.customer: Customer`, not
  `order.customerId: ID` (expose the ID too if clients need it cheaply).
- Custom scalars for semantics: `DateTime`, `EmailAddress`, `Money`/`BigInt` — not
  stringly-typed `String`.
- Avoid god-queries (`viewer { everything }`) and generic JSON scalars — `JSON`
  fields are contract escape hatches that defeat the type system.

## 3. N+1 and dataloaders

The default resolver execution model is N+1 by construction: a list of 100 orders
each resolving `customer` issues 100 queries.

- **Every field resolver that hits a data source goes through a batch loader**
  (DataLoader pattern: per-request cache + batch window). No exceptions for
  "small" fields — query shapes you didn't predict will combine them.
- Loaders are **per-request** (auth context, no cross-user cache leaks). Never
  process-global with user data.
- Batch by the access pattern: `userById`, `ordersByCustomerId` (one loader per
  key shape). For SQL backends, loaders translate to `WHERE id = ANY($1)`.
- Lookahead optimization where it pays: inspect the selection set to JOIN/preload
  instead of loading lazily — but only after loaders, not instead of them.
- **Test it**: assert query counts in integration tests (e.g., run a 2-level query
  over a list of 50, assert ≤ small constant DB calls). N+1 regressions are silent
  until production.

```text
BAD : resolve Customer per order  -> SELECT * FROM customers WHERE id = $1   x100
GOOD: DataLoader batches one tick -> SELECT * FROM customers WHERE id = ANY($1) x1
```

## 4. Query cost controls — non-negotiable on any exposed endpoint

A single unauthenticated query can be a DoS (`{ users { friends { friends {
friends ... }}}}`). Layered defenses, all of them:

1. **Depth limit** (typically 8–12) — blocks recursive nesting.
2. **Complexity/cost analysis** — assign cost per field (list fields multiply by
   `first`), reject above budget *before execution*; charge cost against
   rate-limit quotas (points/minute, à la GitHub/Shopify).
3. **Breadth/alias limits** — cap aliases and root fields per document
   (alias-based amplification: `a1: heavyField a2: heavyField ...` defeats naive
   depth limits).
4. **Paginate everything** + max page size (§2).
5. **Timeout per request** and per resolver; cancel downstream work on abort.
6. **Disable introspection and GraphiQL in production** for non-public schemas;
   disable field suggestions ("Did you mean `creditCard`?") which leak schema.
7. **Reject batched arrays of operations** or cap batch size — batching multiplies
   everything above and bypasses per-request rate limits. Beyond cost-DoS this is a
   **rate-limit-bypass brute-force vector**: aliases/batches let one HTTP request
   carry hundreds of `login`/OTP/token/password-reset attempts past per-request
   throttles. Disable aliasing/batching on auth operations and rate-limit those
   fields **per object/account**, not just per request.

## 5. Persisted queries

Two distinct tools — know which you're deploying:

- **APQ (automatic persisted queries)**: client sends SHA-256 hash, falls back to
  full text on miss. A bandwidth/cache optimization only — **not** security; anyone
  can register any query.
- **Persisted-query allowlist (trusted documents)**: queries extracted from client
  code at build time, registered server-side; production endpoint **rejects
  arbitrary query text** and accepts only known document IDs. This is the SOTA
  posture for first-party-client APIs: kills query-shape abuse, most cost-attack
  surface, and enables GET + CDN caching of hot queries.
- Default for apps you control: allowlist in production, free-form only in dev.
  Public third-party APIs can't allowlist — that's when §4 must carry the load.

## 6. Error handling

GraphQL transports over HTTP 200; do not let that destroy observability.

- **Two error channels, used deliberately**:
  - Top-level `errors` array: *exceptional/system* failures (auth, downstream
    outage, cost-limit). Include machine-readable `extensions.code`
    (`UNAUTHENTICATED`, `FORBIDDEN`, `RATE_LIMITED`, `BAD_USER_INPUT`, `INTERNAL`).
  - **Domain errors as schema**: expected business failures are data, not errors —
    `userErrors: [UserError!]!` on mutation payloads, or union result types
    (`PlaceOrderResult = Order | InsufficientFunds | OutOfStock`). Clients get
    typed, exhaustive handling; your error contract evolves with schema checks.
- Mask internals: never propagate raw exception messages/stack traces into
  `errors[].message`; log server-side with a `trace_id` echoed in `extensions`.
- Partial data is a feature: nullable field fails → field is null + entry in
  `errors` with `path`; clients render the rest. This is why §2 nullability matters.
- Use `application/graphql-response+json` (GraphQL-over-HTTP spec, a Stage 2 draft
  as of 2026-09-26): request errors
  (parse/validation) may use 4xx; field errors stay 200 with `errors`. Ensure your
  monitoring counts GraphQL errors, not just HTTP 5xx — otherwise outages look
  like 100% success.

## 7. Authorization & multi-tenancy

- Authenticate at the transport (HTTP middleware), **authorize per field/object in
  resolvers** (or directive/policy layer). The graph is reachable from many roots
  — `node(id:)`, nested relations — so object-level checks must live with the
  object, not the entry point.
- Never trust IDs from arguments: `order(id:)` must verify tenant/ownership
  (classic GraphQL IDOR).
- `node(id:)` global lookup is an IDOR superhighway if type-level authz is missing.
- Don't expose internal mutations/fields on the public schema — split schemas
  (public vs admin) rather than relying on authz alone; what's not in the schema
  can't be probed.
- **CSRF on a cookie-authenticated endpoint**: reject mutations sent over `GET`
  (the GraphQL-over-HTTP draft says MUST NOT execute; `405` recommended) and accept
  only `application/json` bodies — `text/plain`, `multipart/form-data` and
  `application/x-www-form-urlencoded` are CORS "simple requests" with no preflight.
  A server that parses those as JSON is exploitable (CVE-2025-64166 /
  GHSA-v66j-6wwf-jc57: Mercurius, fixed in 16.4.0). Then apply the CORS and CSRF
  rules in rules/07.

## 8. Schema snippets — good/bad

```graphql
# BAD
type Query {
  orders: [Order]                    # unbounded, nullable-soup list
}
type Order {
  customerId: String!                # FK instead of relationship
  status: Boolean!                   # can't evolve; meaningless name
  meta: JSON                         # contract escape hatch
}
type Mutation {
  updateOrder(id: String!, status: Boolean, note: String): Order  # arg sprawl
}

# GOOD
type Query {
  orders(first: Int! = 50, after: String, filter: OrderFilter): OrderConnection!
  node(id: ID!): Node
}
type Order implements Node {
  id: ID!
  customer: Customer                 # relationship; nullable: separate service
  status: OrderStatus!               # enum, documented open
  createdAt: DateTime!
}
type Mutation {
  cancelOrder(input: CancelOrderInput!): CancelOrderPayload!
}
type CancelOrderPayload {
  order: Order
  userErrors: [UserError!]!          # domain failures as data
}
```

## 9. Subscriptions

- Transport: GraphQL-over-WebSocket (`graphql-ws` protocol — not the legacy
  `subscriptions-transport-ws`) or **GraphQL over SSE** (`graphql-sse`) — prefer
  SSE for server-push-only subscriptions per rules/05 §1. Everything in rules/05
  applies: auth (`connection_init` payload = first-message auth, with timeout),
  heartbeats, reconnect/backoff, backpressure.
- Subscriptions are for **small, fast deltas** ("order 42 changed"), not data
  transfer: push the ID/patch, let the client query for the full shape, or
  resolve a narrow selection. Fat subscription payloads multiply resolver fanout
  by subscriber count.
- Each subscription field maps to a pub/sub topic with object-level authz at
  subscribe time **and** per-event filtering (the topic may carry other tenants'
  events; filter before resolve).
- Server restarts drop all subscriptions silently — clients must treat
  subscription data as a cache over a re-fetchable source (snapshot on
  reconnect), never the system of record. No replay buffer exists by default;
  if gaps matter, add sequence numbers + re-query (rules/05 §2.3).
- Cap concurrent subscriptions per connection and per principal; subscription
  resolvers go through the same cost analysis as queries.

## 10. Caching and federation notes

- Lose HTTP caching, replace deliberately: persisted queries over GET for
  CDN-cacheable public reads; `@cacheControl`-style hints feeding a response
  cache keyed by (document, variables, auth scope) — never share cached
  responses across principals; entity-level caching inside dataloaders.
- Client caches normalize by `id` — another reason for global object IDs (§2).
- Federation (multiple subgraphs behind one router) only when multiple teams own
  distinct subdomains of one graph. It buys ownership boundaries and costs: a
  router on the hot path, cross-subgraph N+1 (entity resolution must be
  batched), and composition checks that **must** run in CI (a subgraph deploy
  can break the supergraph). Don't federate a single-team schema; modularize the
  codebase instead.

## 11. Evolution

- Additive is easy (new fields/types). Removal: `@deprecated(reason:)` →
  field-level usage metrics (which clients, which operations) → contact → remove
  when traffic is zero. Schema registries give you this; without per-field usage
  data you can never delete anything.
- Never change a field's type or nullability in place — add `fieldV2`/new name,
  deprecate old.
- Enums: same open-enum discipline as REST (rules/02 §6) — clients must tolerate
  unknown values.

## Audit checklist

- [ ] GraphQL is earning its keep (multiple clients/aggregation) — flag 1:1 REST-wrapper deployments.
- [ ] Schema checks run in CI against real traffic; breaking changes blocked.
- [ ] All list fields paginated (connections) with enforced max page size; no unbounded lists.
- [ ] Non-null used deliberately; cross-service-backed fields nullable (error-bubbling blast radius understood).
- [ ] Every data-fetching resolver goes through per-request DataLoaders; integration test asserts query counts (no N+1).
- [ ] Depth limit, cost analysis (with list multipliers), and alias/root-field caps active in production — verify by sending a hostile query in staging over **both** the HTTP path and the WebSocket/SSE subscription path (cf. CVE-2026-30241: Mercurius enforced queryDepth on HTTP but not WS subscriptions; fixed in 16.8.0).
- [ ] Operation batching capped or disabled; cost charged against rate limits.
- [ ] Introspection + field suggestions disabled in production for private schemas; no GraphiQL exposed.
- [ ] First-party clients use a persisted-query allowlist (trusted documents); APQ not mistaken for security.
- [ ] Mutations use input objects + payload types with `userErrors`/result unions; domain errors are schema, not exceptions.
- [ ] `errors[].extensions.code` machine-readable; raw exception messages masked; trace ID present.
- [ ] Monitoring counts GraphQL-level errors (200-with-errors), not just HTTP status.
- [ ] Object-level authorization in resolvers (incl. `node(id:)` and nested paths); tenant checks on every ID argument.
- [ ] Public vs admin schema split; no internal fields on public schema.
- [ ] Cookie-authenticated endpoint: a mutation over `GET` is refused (`405`) and a non-`application/json` body (`text/plain`, form, multipart) is rejected — send both in staging.
- [ ] Deprecations carry reasons + removal dates; per-field usage metrics exist before removals.
- [ ] Subscriptions use `graphql-ws`/`graphql-sse` (not legacy protocol), authz at subscribe + per-event filtering, small delta payloads, client snapshot-on-reconnect; concurrent subscriptions capped.
- [ ] Response caches keyed by auth scope (no cross-principal cache hits); federation (if present) has CI composition checks and batched entity resolution.

# 04 — gRPC, Protobuf Evolution & Protocol Choice

Scope: proto evolution rules, deadlines/cancellation, streaming patterns, errors,
gRPC-Web/Connect, load balancing, and the REST vs GraphQL vs gRPC decision.

## 1. Choosing the protocol

| Criterion | REST/HTTP+JSON | GraphQL | gRPC |
|---|---|---|---|
| Public/partner API | **default** | many diverse clients | rarely (tooling burden on partners) |
| Service-to-service | fine | wrong tool | **default** (perf, codegen, deadlines) |
| Browser clients | native | native | needs gRPC-Web/Connect proxy layer |
| Streaming | SSE/WS bolt-on | subscriptions bolt-on | first-class (4 modes) |
| Payload efficiency | verbose | verbose | binary, ~5-10x smaller, cheap parse |
| HTTP caching/CDN | excellent | poor | none |
| Human debuggability | curl-able | tooling | needs grpcurl/buf curl |
| Contract | OpenAPI (optional) | SDL (intrinsic) | proto (intrinsic, enforced) |

Practical rule: **gRPC inside the datacenter, REST at the edge, GraphQL only when
client-shape diversity demands it.** Mixed estates are normal; transcode at the
gateway (grpc-gateway / Connect / Envoy gRPC-JSON transcoder) rather than
hand-maintaining parallel APIs. Consider Connect-RPC where you want gRPC semantics
plus curl-able JSON over plain HTTP without a proxy.

## 2. Proto evolution — the field-number contract

Wire compatibility lives in **field numbers and types**, not names.

- **Never change a field number. Never reuse one.** A reused number deserializes
  old data into the wrong field — silent corruption, not an error.
- **Never change a field's type** (except documented wire-compatible sets like
  int32/uint32/int64/bool — and even those change language-level semantics; avoid).
- Renaming a field is wire-safe but breaks JSON transcoding and generated code —
  treat as breaking if protojson or codegen consumers exist.
- **Deleting a field: `reserved` the number AND the name**, forever:

```protobuf
message User {
  reserved 4, 9 to 11;
  reserved "ssn", "internal_score";
  string id = 1;
  string display_name = 2;
}
```

- New fields: optional semantics, new unique number. Required fields don't exist
  in proto3 — and proto2's `required` was removed for a reason: required is forever.
- **Enums: entry 0 is `FOO_UNSPECIFIED`** always; never renumber; `reserved`
  removed values. Open enums: receivers keep unknown values (proto3) — code must
  `default:` handle them.
- Scalars have no presence in proto3 (0/""/false ≡ unset). When "unset vs zero"
  matters (PATCH-like updates, money), use `optional` (proto3 field presence) or
  wrapper types, plus `google.protobuf.FieldMask` for partial updates.
- Don't repurpose semantics under the same field; add a new field, deprecate old
  (`[deprecated = true]`) and follow rules/02 §5.
- **Enforce with `buf breaking` in CI** against the previous commit/main. Protos
  live in one schema repo/module with code review; consumers pin generated
  artifacts. Hand-edited generated code is an audit finding.
- Package versioning: `package billing.v1;` — a true breaking change means
  `billing.v2` side-by-side, both served during migration.

Evolution example — the corruption trap:

```protobuf
// v1                                  // v2 — WRONG
message Payment {                      message Payment {
  string id = 1;                         string id = 1;
  int64 amount_cents = 2;                string customer_note = 2;  // reused #2!
}                                      }
// Old senders' amount_cents bytes now parse as customer_note garbage —
// or worse, varint-compatible types silently produce wrong values. No error.

// v2 — RIGHT
message Payment {
  reserved 2;
  reserved "amount_cents";
  string id = 1;
  Money amount = 3;          // new field, new number, richer type
}
```

## 3. Deadlines everywhere

gRPC's killer operational feature — and the most commonly omitted.

- **Every client call sets a deadline.** No deadline = infinite default = threads
  and connections wedged behind a slow dependency until restart.
- Propagate, don't reset: deadlines flow with the context across hops. A service
  with 200ms left gives its downstream ≤200ms (gRPC propagates `grpc-timeout`
  automatically when you pass the inbound context — so *pass the inbound context*).
- Budget top-down: edge sets the total (e.g. 1s); interior hops inherit the
  remainder. Per-hop floors guard against doing pointless work: if remaining
  budget < your P99, fail fast with `DEADLINE_EXCEEDED`.
- **Servers must check cancellation** (`ctx.Done()` / `context.isCancelled()`)
  before/within expensive work and abort DB queries — otherwise clients give up
  but servers keep burning, the classic cascading-overload pattern.
- Retries: only on idempotent methods, only with backoff + jitter, only within the
  original deadline; honor `RESOURCE_EXHAUSTED`/pushback. Use gRPC service config
  retry policy or a mesh — not hand-rolled loops. Hedging only for read-only calls.

```go
// BAD: no deadline, background context
resp, err := client.GetUser(context.Background(), req)

// GOOD: propagated inbound ctx + explicit ceiling
ctx, cancel := context.WithTimeout(inboundCtx, 300*time.Millisecond)
defer cancel()
resp, err := client.GetUser(ctx, req)
```

## 4. Streaming patterns

Four modes; pick the simplest that works — unary is right ~90% of the time.

- **Server streaming**: large result sets (chunked instead of giant unary
  responses — keep messages < ~1-4 MB; set `maxReceiveMessageSize` deliberately),
  watch/subscribe feeds, progress updates.
- **Client streaming**: uploads, batched telemetry ingestion.
- **Bidirectional**: chat-like sessions, interactive sync protocols. Costly to get
  right (ordering, flow control, app-level acks) — require justification.
- Rules for any stream:
  - Deadlines still apply; long-lived streams need an explicit max age + graceful
    re-establish (also re-balances load after topology changes).
  - Cap the **number of messages** a client may send on one stream, alongside
    per-message size and max age: a stream of many small messages passes a
    size limit. Over the cap, end the stream with `RESOURCE_EXHAUSTED`, the
    canonical code for an exhausted per-user resource. The cap is enforced in
    the handler or a stream interceptor; server options bound message size and
    connection age, not message count (grpc-go's `ServerOption` set has no
    per-stream message limit). OWASP: gRPC Security cheat sheet.
  - **Flow control is real backpressure** — never buffer unboundedly around a
    slow receiver; respect the write-availability signal (`onReady`/blocking send).
  - Design resumability at the app layer (resume tokens / cursors in the request)
    — streams will drop; reconnect must not re-send or skip data.
  - Heartbeat long quiet streams (HTTP/2 PING via keepalive config — respect
    server `keepalive` enforcement policy or get `GOAWAY ENHANCE_YOUR_CALM`).
- Don't use streaming as a database replication protocol or for >minutes-long
  transfers where object storage + signed URL is simpler and restartable.

## 5. Service design patterns (steal from AIPs)

Google's API Improvement Proposals (aip.dev) are the de-facto style guide for
resource-oriented gRPC; follow them unless you have a reason:

```protobuf
service OrderService {
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc ListOrders(ListOrdersRequest) returns (ListOrdersResponse);
  rpc CreateOrder(CreateOrderRequest) returns (Order);
  rpc UpdateOrder(UpdateOrderRequest) returns (Order);   // uses FieldMask
  rpc CancelOrder(CancelOrderRequest) returns (Order);   // custom verb, reified
}

message ListOrdersRequest {
  int32 page_size = 1;        // server clamps; 0 => default
  string page_token = 2;      // opaque cursor — same rules as REST (01 §4)
  string filter = 3;          // constrained filter expression, allowlisted fields
  string order_by = 4;
}
message ListOrdersResponse {
  repeated Order orders = 1;
  string next_page_token = 2; // empty => done
}
```

- Standard verbs (`Get/List/Create/Update/Delete`) + reified custom verbs;
  request/response message per RPC (never share request messages across RPCs —
  they evolve independently; never return bare scalars — you can't add fields
  to an `int64`).
- `Update` takes `google.protobuf.FieldMask update_mask` — explicit partial
  update beats "absent means unchanged" guessing (§2 presence).
- Long-running work returns `google.longrunning.Operation` (the gRPC analogue
  of REST's 202 pattern, rules/01 §11) — pollable, cancellable, with typed
  metadata/result.
- `buf lint` in CI for naming/package/style; one lint config org-wide.
- Pagination, idempotency (request IDs on Create), and tenant scoping rules
  from 01/07 apply unchanged — the wire format doesn't exempt you.

## 6. Errors and metadata

- Use canonical codes correctly; clients branch on code:
  `INVALID_ARGUMENT` (bad request), `NOT_FOUND`, `ALREADY_EXISTS`,
  `FAILED_PRECONDITION` (state conflict), `PERMISSION_DENIED` vs
  `UNAUTHENTICATED`, `RESOURCE_EXHAUSTED` (rate/quota), `UNAVAILABLE`
  (retryable infra), `DEADLINE_EXCEEDED`, `INTERNAL` (your bug).
  **Never** stuff app errors into `UNKNOWN` or encode them in message strings.
- Rich detail: `google.rpc.Status` + `error_details.proto` types —
  `BadRequest.FieldViolation` for validation, `RetryInfo` for pushback,
  `ErrorInfo{reason, domain, metadata}` for machine-readable causes. This is the
  gRPC analogue of RFC 9457.
- Don't leak internals in messages (mirrors rules/01 §9). Include trace IDs via
  metadata/interceptors.
- Interceptors (client+server) are the standard home for auth, deadline floors,
  logging, metrics, panic-to-`INTERNAL` conversion — not per-method copy-paste.

## 7. gRPC-Web, Connect, and the browser

- Browsers can't speak native gRPC (no control over HTTP/2 frames/trailers).
  Options: **gRPC-Web** (Envoy/in-process proxy; no client streaming, server
  streaming is text/binary framed), **Connect protocol** (POST/JSON or binary,
  curl-able, no proxy needed, interops with gRPC servers), or **JSON transcoding**
  at the gateway (`google.api.http` annotations → REST surface).
- Public-facing rule: don't hand partners raw gRPC unless they asked for it —
  transcode to REST at the gateway from the same protos so there's one source of
  truth.

## 8. Load balancing & operations

- gRPC's persistent HTTP/2 connections defeat L4 load balancers: all RPCs from a
  client ride one connection to one backend. Use **client-side LB** (resolver +
  `round_robin`/weighted), a **service mesh** (Envoy/linkerd L7 per-RPC balancing),
  or an L7 proxy. Audit flag: gRPC behind a plain TCP/L4 LB with hot-spotting.
- Set `MAX_CONNECTION_AGE` (+ grace) on servers so connections cycle and rebalance.
- Health: standard `grpc.health.v1.Health` service wired into LB/mesh checks;
  reflection enabled in non-prod for grpcurl, deliberate decision for prod.
- Keepalive tuned on both sides and consistent with proxy idle timeouts (the #1
  cause of mysterious `UNAVAILABLE` storms).
- TLS everywhere; mTLS for service-to-service is the 2026 default (mesh-issued
  certs). Per-call auth (JWT/OAuth) in metadata via interceptors, validated
  server-side — transport identity ≠ caller authorization.
- Observability: standard interceptor stack exporting per-method RPC metrics
  (rate, error-by-code, latency histograms) and trace propagation
  (W3C `traceparent` / OpenTelemetry gRPC instrumentation). `DEADLINE_EXCEEDED`
  and `UNAVAILABLE` rates are your two most important alerts — they precede
  user-visible outages. For security, break metrics down **per client identity**
  too (request rate, and `UNAUTHENTICATED`/`PERMISSION_DENIED` rates) and alert on
  a spike in authn/authz failures, calls to unknown methods (`UNIMPLEMENTED`)
  and `RESOURCE_EXHAUSTED` bursts. Per-method totals hide one client probing.
  Keep the client label bounded (service or tenant ID, not raw IP) so metric
  cardinality stays manageable. OWASP: gRPC Security cheat sheet.

Reference keepalive alignment (mismatches cause `UNAVAILABLE` storms):

```text
client keepalive_time            60s   # >= server's min allowed (else GOAWAY)
server KEEPALIVE_ENFORCEMENT min 30s
server MAX_CONNECTION_IDLE       5m
server MAX_CONNECTION_AGE        30m (+5m grace)
LB / proxy idle timeout          > client keepalive_time AND > longest idle gap on a stream
                                 # AWS ALB: default 60s, and HTTP/2 PING does NOT reset it (ALB
                                 # does not support PING) — keepalive cannot hold an ALB idle
                                 # connection; raise idle_timeout or send app-level traffic.
                                 # 350s is the NLB TCP default, not ALB's. Verify at the LB docs.
```

## Audit checklist

- [ ] Protocol choice justified: gRPC internal / REST edge / GraphQL only for client diversity; no raw gRPC forced on unwilling partners.
- [ ] `buf breaking` (or equivalent) gates proto changes in CI; protos versioned in a shared module; no hand-edited generated code.
- [ ] No field number ever reused; removed fields have `reserved` numbers AND names (check git history of .proto files).
- [ ] Enums have `*_UNSPECIFIED = 0`; consumer code handles unknown enum values with a default branch.
- [ ] Field presence handled deliberately (proto3 `optional`/FieldMask) where unset≠zero matters — especially money and PATCH-style updates.
- [ ] Every outbound RPC sets a deadline derived from the inbound context; grep for `context.Background()`/missing `withDeadline` at call sites.
- [ ] Servers check cancellation in expensive paths and propagate aborts to DB/downstream calls.
- [ ] Retry policy: idempotent-only, backoff+jitter, deadline-bounded, via service config/mesh — no naive retry loops.
- [ ] Status codes used canonically; rich errors via `error_details.proto`; no app errors in `UNKNOWN`/message-string parsing; no internal leak in messages.
- [ ] Max message sizes configured; large transfers use streaming chunks or signed URLs, not giant unary messages.
- [ ] Streams: bounded buffering honoring flow control, app-level resume tokens, max stream age, keepalive config consistent with proxies.
- [ ] Client-streaming and bidi handlers cap messages per stream and end over-cap streams with `RESOURCE_EXHAUSTED` — MEDIUM. Read each handler's receive loop (`Recv()`, `onNext`, `async for`) for a counter.
- [ ] Metrics per client identity with alerts on authn/authz-failure spikes, `UNIMPLEMENTED` probing and `RESOURCE_EXHAUSTED` bursts — MEDIUM.
- [ ] Per-RPC L7 load balancing (client-side/mesh/proxy) — not a bare L4 LB; `MAX_CONNECTION_AGE` set.
- [ ] Standard health service wired to infra; reflection disabled or justified in prod.
- [ ] TLS/mTLS on all links; per-call authn validated in server interceptors; authz not inferred from transport alone.
- [ ] Browser/partner access goes through gRPC-Web/Connect/transcoding generated from the same protos.
- [ ] AIP-style hygiene: per-RPC request/response messages (none shared, no bare scalar returns), `page_size`/`page_token` listing, `FieldMask` updates, LRO `Operation` for long work, `buf lint` in CI.
- [ ] Create/mutating RPCs carry request IDs (idempotency) where retries are configured.

# 05 — WebSockets, SSE & Realtime

Scope: transport selection, WS lifecycle (auth, heartbeats, reconnection, ordering,
backpressure, close semantics), SSE patterns, scaling fanout and presence.

## 1. Choosing the transport

| | SSE | WebSocket | Long-polling | WebTransport |
|---|---|---|---|---|
| Direction | server→client | bidirectional | server→client | bidi, streams+datagrams |
| Protocol | plain HTTP | upgrade, own framing | plain HTTP | HTTP/3 (QUIC) |
| Auto-reconnect+resume | **built-in** (`Last-Event-ID`) | DIY | DIY | DIY |
| Infra friendliness | excellent (it's HTTP) | proxies/LBs need care | excellent | fair (Baseline since Safari 26.4, Mar 2026; UDP/443 blocked in some networks) |
| Binary | no (text/UTF-8) | yes | no | yes |

Decision rules:
- **Server-push only** (notifications, live feeds, dashboards, job progress, LLM
  token streaming) → **SSE**. It's the most under-used right answer; HTTP/2 removes
  the old 6-connection limit.
- **True bidirectional, low-latency, frequent client→server** (chat with typing,
  collaborative editing, multiplayer, trading) → **WebSocket**.
- Client→server messages are *occasional*? SSE down + plain POSTs up beats a WS —
  you keep HTTP auth, retries, and observability.
- **Long-polling**: fallback only, behind an abstraction, for ancient
  proxies/networks. Don't design for it first in 2026.
- **WebTransport**: adopt when you need unreliable/unordered delivery (game state,
  media) or stream multiplexing without head-of-line blocking. Browser support is
  no longer the blocker (Baseline since Safari 26.4 shipped it, Mar 2026) — keep a
  WS fallback for UDP-blocking middleboxes/networks.
- If you'd rather not own any of this: managed layers (Ably/Pusher/Momento) or
  infra (e.g. Centrifugo) are legitimate — reconnection/resume/fanout are where
  homegrown realtime dies. Check a self-hosted server is maintained before adopting
  it: Soketi's last release and commit were 2024-03-25 (GitHub, checked 2026-09-26).

## 2. WebSocket lifecycle done right

### 2.1 Auth before (or immediately after) upgrade

- **`wss://` only in production** — never `ws://`. A plaintext socket leaks
  tokens/session data and is trivially MITM'd; treat `ws://` carrying credentials
  as a High finding (mirrors the mTLS-internal baseline).
- **Disable `permessage-deflate` compression** when the stream can carry secrets
  alongside attacker-influenced data — compression-ratio side channels are the
  CRIME/BREACH class applied to WebSockets. Leave it off unless you've reasoned
  about what shares the frame.
- The upgrade request is a GET: **no custom `Authorization` header from browser
  `WebSocket()`**. Your options, in order of preference:
  1. **Cookie auth** (same-origin apps): session cookie rides the upgrade; **must**
     verify `Origin` server-side — WS is exempt from CORS/SOP, so a malicious page
     can open `wss://yourapp` with the victim's cookies (Cross-Site WebSocket
     Hijacking). Origin allowlist or per-connection CSRF ticket is mandatory.
  2. **Short-lived one-time ticket**: client POSTs to `/realtime/ticket` (normal
     auth), gets a 30s single-use token, connects `wss://...?ticket=...`. Tokens in
     query strings land in logs — that's why it must be one-time + short-lived.
  3. **First-message auth**: accept the socket, require an `auth` frame within
     ~5s, process nothing else before it, hard-close on timeout. Keep
     pre-auth state tiny (DoS surface).
  - Do **not** smuggle tokens via `Sec-WebSocket-Protocol` (logged, semantically
    wrong, breaks subprotocol negotiation).
- Authorize per action after connect (subscribing to channel X = its own check),
  not just once at handshake.
- **Token expiry mid-connection**: long-lived sockets outlive JWTs. Either close
  with a specific code (e.g. 4401) when the token expires and let the client
  re-auth on reconnect, or support an in-band token-refresh frame. Ignoring expiry
  means a revoked user stays connected for hours — common audit finding.
- **Revocation reaches open sockets, not just the next request.** Expiry is the
  slow path; logout, admin session revocation, password reset, account disable and
  a permission or role change are the fast ones. Keep a registry from session ID
  (and user ID) to live WS/SSE connections, shared across nodes through the same
  pub/sub backplane as fanout (§5), and have every revocation path publish a
  "kill" event that closes those connections at once (WS close `1008` policy
  violation or an app-range code the client treats as "do not reconnect with this
  credential"; SSE: end the response). A permission change may instead re-check
  and drop only the subscriptions no longer allowed. Test it: log out in one tab
  and assert the socket in another closes within seconds.
  OWASP: WebSocket Security cheat sheet.
- **Do not tunnel raw TCP services over a browser-reachable socket** (VNC, SSH,
  RDP, FTP, database consoles; bridges such as websockify or a terminal sharer
  such as ttyd). Any script running on an allowed origin, an XSS included,
  can drive the tunnel with the victim's session. Where a tunnel is needed, the
  tunnelled service keeps its own authentication and access control, and an
  authenticated socket is not treated as access to the service behind it.
  OWASP: WebSocket Security cheat sheet.
- Security events on the socket (open/close, auth decisions, violations) go to
  the audit stream: rules/07 §6.

### 2.2 Heartbeat / ping-pong

- TCP keepalive is not enough; intermediaries silently kill idle connections
  (typical LB idle timeout 60s). **Server pings every 20–30s; missing pong within
  the timeout ⇒ close.** Browsers auto-answer protocol pings, so server-side ping
  detects dead clients; clients also need an app-level liveness check (expect
  *some* frame every N sec) to detect half-open sockets where TCP looks alive
  but nothing flows.
- Align: heartbeat interval < LB/proxy idle timeout < your own idle close.

### 2.3 Reconnection with backoff + resume

Client side:
- Reconnect on any abnormal close with **exponential backoff + full jitter**
  (e.g. 1s base, ×2, cap 30s, `delay = rand(0, min(cap, base*2^n))`). No jitter ⇒
  thundering herd after every server deploy.
- Reset backoff only after a connection has been healthy for some seconds (not on
  connect — a connect-then-die loop must keep backing off).
- Honor close codes: don't reconnect on 1008/4401 (auth) without re-authing; don't
  reconnect at all on "kicked/replaced" codes.

Server side — **resume tokens**:
- Every server→client message carries a monotonically increasing per-channel
  sequence ID. Client sends `last_seq` on reconnect; server replays from a bounded
  per-channel buffer (size/time-limited ring, e.g. 1000 msgs / 5 min).
- If the gap exceeds the buffer: tell the client explicitly (`resume_failed`) so
  it re-fetches a snapshot via REST and re-subscribes. **Silent gap = corrupted
  client state**; the snapshot-then-stream pattern (fetch state, then apply
  buffered events with seq > snapshot version) is the correct cold-start too.

### 2.4 Ordering and delivery

- A single WS connection preserves order; your **backend fanout usually doesn't**
  (multiple publishers, pub/sub partitions, worker pools). Don't promise global
  order — promise **per-channel/per-entity order** and enforce it: one logical
  publisher sequence per channel, sequence numbers checked by clients.
- Delivery is at-most-once on a raw socket. If you need at-least-once, add acks +
  redelivery + **idempotent client handlers** (dedupe on message ID). State your
  guarantee explicitly in the protocol doc; "we never thought about it" is the
  usual answer and the usual bug.

### 2.5 Backpressure on slow consumers

The #1 realtime-server OOM: a slow client (bad network, background tab) can't
drain, and you buffer unboundedly on its behalf.

- **Bound every per-connection send queue.** On overflow, choose per product:
  - *Drop-and-coalesce* (tickers, presence, cursors: keep only latest per key) —
    usually right for state-shaped data;
  - *Disconnect* with close code "too slow" — right for event streams where loss
    is unacceptable; client reconnects and resumes/re-snapshots (§2.3).
- Check transport buffer signals (`bufferedAmount` in browsers, write deadlines /
  `ws.send` callbacks server-side); never `send()` blind in a loop.
- Per-connection **inbound** limits too: max message size enforced (reject with
  1009), rate-limit client frames, cap subscriptions per connection.
- **Connection caps**: a maximum number of concurrent sockets per authenticated
  user (per IP only where there is no user yet, e.g. before first-message auth)
  and a total per node, enforced at the upgrade. Over the cap, refuse the
  handshake with an HTTP error (`429` or `503`) or close at once with a
  documented app code, so the client backs off instead of looping. Without them,
  one account opening thousands of tabs or scripted sockets exhausts file
  descriptors for everyone on the node. OWASP: WebSocket Security cheat sheet.

### 2.6 Close semantics

- Close deliberately with codes + reason: 1000 normal, 1001 going away (deploys),
  1008 policy/auth, 1009 too big, 1011 server error, 1012 service restart;
  4000–4999 app-defined (document them: e.g. 4401 token-expired, 4290
  rate-limited, 4100 replaced-by-newer-session).
- Graceful shutdown on deploy: stop accepting, send close (1001/1012), drain with
  a deadline, rely on client jittered reconnect to spread load to new instances.
  Mass hard-kill = reconnect stampede.

### 2.7 Message protocol design

WS gives you frames, not a protocol — you must design one and write it down.

```jsonc
// BAD: shapeless blobs, no type, no seq, no correlation
{"order": 9, "s": "shipped"}

// GOOD: enveloped, versioned, sequenced, correlatable
{
  "type": "order.updated",       // namespaced event type, same registry as webhooks
  "seq": 4174,                   // per-channel sequence (resume, gap detection)
  "channel": "orders:acct_7",
  "data": {"id": "ord_9", "status": "shipped"},
  "ts": "2026-06-12T09:30:00Z"
}
// client->server commands carry a client-generated id; server replies ack/nack:
{"type":"subscribe","id":"c-17","channel":"orders:acct_7","last_seq":4170}
{"type":"ack","id":"c-17"}     |    {"type":"nack","id":"c-17","error":{"code":"forbidden"}}
```

- Every frame has a `type`; unknown types are ignored (tolerant reader,
  rules/02 §3) — that's what lets you evolve the protocol.
- Request/response over WS needs explicit correlation IDs + per-command timeout
  client-side; without them you've built RPC with no error handling.
- Errors over WS follow the same machine-readable code discipline as RFC 9457
  (rules/01 §9): `{"type":"error","code":"rate_limited","retry_after":5}`.
- Version the protocol (subprotocol negotiation `Sec-WebSocket-Protocol:
  app.v1` or a `hello` exchange) so you can change framing later.
- One multiplexed connection with channels beats N connections per page — but
  then channel-level authz (§2.1) is mandatory per subscribe.
- **State-changing client commands carry a nonce or timestamp**, and the server
  rejects a repeated nonce or a stale timestamp (e.g. outside a short window,
  with seen nonces kept for that window). This stops replay of a captured or
  re-injected command. It is a separate job from §2.4 delivery dedupe, which
  protects the *client* from duplicate server events. OWASP: WebSocket Security
  cheat sheet.
- **Isolate each message's failure.** Parse, validate and dispatch every inbound
  frame inside its own error boundary: a malformed or hostile message produces a
  `nack`/error frame, or at worst closes *that* connection (1007 invalid data,
  1008 policy), and never throws out of the read loop into the worker or event
  loop that serves other connections. OWASP: WebSocket Security cheat sheet.

Client reconnect skeleton (the part everyone gets wrong):

```js
let attempt = 0;
function connect() {
  const ws = new WebSocket(url);
  let stableTimer;
  ws.onopen = () => { stableTimer = setTimeout(() => attempt = 0, 10_000); };
  ws.onclose = (e) => {
    clearTimeout(stableTimer);
    if (e.code === 4401) return reauthThenConnect();   // don't loop on auth
    if (e.code === 4100) return;                       // replaced: stay dead
    const delay = Math.random() * Math.min(30_000, 1000 * 2 ** attempt++);
    setTimeout(connect, delay);                        // full jitter
  };
}
```

## 3. SSE patterns

```http
GET /v1/events?stream=orders HTTP/1.1        HTTP/1.1 200 OK
Accept: text/event-stream                    Content-Type: text/event-stream
Last-Event-ID: 4173                          Cache-Control: no-store
                                             X-Accel-Buffering: no

                                             id: 4174
                                             event: order.updated
                                             data: {"id":"ord_9","status":"shipped"}
                                             retry: 5000
```

- Set `id:` on every event — the browser's `EventSource` reconnects automatically
  and sends `Last-Event-ID`, giving you resume **for free**; implement server-side
  replay from it (same bounded-buffer logic as §2.3).
- Send a comment heartbeat (`: ping\n\n`) every 15–30s to defeat idle timeouts.
- Disable proxy buffering (`X-Accel-Buffering: no`, no gzip on the stream, flush
  per event) — buffered SSE silently becomes batch delivery.
- `EventSource` can't set headers: use cookies or a ticket param (same rules as
  §2.1), or the fetch-based SSE client pattern for header auth (you then own
  reconnect+`Last-Event-ID` yourself).
- **Client side, the stream URL comes from trusted configuration**, never from
  a query parameter, fragment, `postMessage` or other user input. `EventSource`
  makes a CORS request, and with `withCredentials: true` it sends cookies, so an
  attacker-chosen URL can feed the page events from the attacker's server. Treat
  `event.data` as data (parse, never inject as HTML or evaluate). Each event's
  `origin` is the origin of the stream's final URL after redirects, so where a
  stream can be redirected or come from another origin, check `event.origin`
  against an allowlist before acting. OWASP: HTML5 Security cheat sheet.
- LLM/token streaming: SSE is the de-facto standard; include a terminal event
  (`event: done`) — clients must distinguish "stream complete" from "connection
  dropped", or they'll render truncated answers as final.

## 4. Long-polling (fallback only)

If you must support it (legacy proxies stripping upgrade headers, restrictive
corporate networks): `GET /poll?last_seq=N` holds the request up to ~25s
(< all intermediary timeouts), returns immediately when events exist or `204`
on timeout; client re-polls instantly on data, with small delay + jitter on
204/error. Same sequence/resume semantics as §2.3 — long-polling is just the
transport. Hide all three transports behind one client abstraction with
automatic downgrade (WS → SSE+POST → long-poll); never let product code know
which transport is active.

## 5. Scaling realtime

- **One process is a lie at scale**: users on node A must receive events published
  on node B. Standard architecture: stateless-ish WS/SSE edge nodes + **pub/sub
  backplane** (Redis Pub/Sub or Streams, NATS, Kafka for replayable channels).
  Edge node subscribes to channels its sockets need, fans out locally.
- **Sticky sessions** (LB affinity) only pin the TCP connection to a node — fine
  and often needed; they do **not** solve cross-node fanout and must not be used
  to fake it (node dies ⇒ its "state" dies).
- Connection state (which user, which channels, last_seq) belongs in the node's
  memory + recoverable from the client on reconnect — not in a shared DB on the
  hot path.
- Capacity realities: each node has FD/memory ceilings (tune ulimits; budget
  ~tens of KB per connection); deploys cause full reconnect waves — see §2.6 and
  jitter; autoscaling on connection count and on send-queue depth, not CPU alone.
- **Presence** (who's online): heartbeat-driven entries with TTL in Redis
  (`SETEX presence:{channel}:{user}`), refreshed by the edge node, expiry = offline;
  debounce flapping (grace period before broadcasting "left") and coalesce
  presence broadcasts (full roster sync on join + deltas after).
- Hot channels (1 publisher → 100k subscribers): coalesce/throttle per channel at
  the publisher side (max N msgs/sec, latest-wins), shard the channel across
  backplane partitions, and never fan out faster than your slowest tier absorbs.

## 6. WebRTC (when you self-host the infrastructure)

WebRTC is the path for peer-to-peer audio/video/data channels. **Scope first:** if
you only consume a CPaaS SDK/API (Twilio, LiveKit Cloud, Daily…), the provider owns
most of this — these rules apply when you **run your own TURN, media, or signaling
servers**. Media itself is encrypted by mandate (DTLS-SRTP); the risks are in the
*servers* around it. (ASVS v5.0 V17; RFC 8826/8827 security architecture.)

- **TURN server — relay abuse is SSRF over UDP/TCP.** A TURN relay forwards packets
  on behalf of a client; an open one lets an attacker reach your internal network
  and reserved ranges. **Allowlist relay peers to non-special addresses only** —
  deny loopback/RFC1918/link-local/broadcast/CGNAT, IPv4 **and** IPv6 (same blocklist
  as SSRF, sota-code-security rules/01 §5). Bound per-user port/allocation counts so
  one client can't exhaust the relay (resource exhaustion).
- **Media servers (SFU/MCU/recording) — only if you host them.** Pure browser-to-
  browser P2P is out of scope here. Use **approved DTLS-SRTP cipher suites + protection
  profiles**, manage the DTLS cert key under your key-management policy (rules/04),
  and **verify SRTP authentication** so an attacker can't inject RTP (media insertion
  or DoS). The server must **survive malformed and flooded SRTP** (input validation,
  bounded buffers, drop excess packets) and must not be vulnerable to the **DTLS
  `ClientHello` race-condition DoS** (Enable Security, 2020). Bind the DTLS cert to the
  **SDP fingerprint** and terminate the stream on mismatch (authenticity).
- **Signaling server.** Signaling carries no media but sets up the session — **rate-limit
  it** and **handle malformed messages gracefully** (integer-overflow/buffer-safe parsing)
  so a flood or a bad frame can't deny session setup. Authenticate and authorize
  signaling like any other API channel (the WS/§2 rules apply if signaling rides WS).

## Audit checklist

- [ ] Transport choice justified; server-push-only features use SSE, not a WS with one direction unused.
- [ ] WebRTC (self-hosted only): TURN relay allowlists non-special IPs (v4+v6) and caps per-user allocations; media server uses approved DTLS-SRTP + SRTP auth, survives malformed/flooded SRTP, not ClientHello-race vulnerable, checks DTLS cert vs SDP fingerprint; signaling rate-limited + malformed-input-safe. (CPaaS-only consumers: provider-owned — N/A.)
- [ ] WS auth: Origin checked server-side (CSWSH), or one-time short-lived tickets, or first-message auth with timeout; no long-lived tokens in query strings; nothing smuggled via `Sec-WebSocket-Protocol`.
- [ ] Per-channel/per-action authorization after connect, not handshake-only; token expiry mid-connection handled (close code or refresh frame).
- [ ] **Revocation closes live sockets (§2.1) — HIGH**: logout/revoke/disable paths close that session's WS/SSE connections on every node. Locator for revocation code that never touches a connection registry (a hit is a file to read, not yet a finding):
      `grep -rliE 'logout|revoke|invalidate_?session' --include='*.py' --include='*.js' --include='*.ts' --include='*.go' --include='*.java' --include='*.kt' --include='*.rb' --include='*.cs' --include='*.php' . | xargs -r grep -LiE 'socket|connection|sse|disconnect'`
- [ ] Server pings with pong timeout; intervals < LB idle timeout; half-open detection on client.
- [ ] Client reconnect: exponential backoff with jitter, cap, backoff reset only after stable connection, close codes honored (no reconnect loop on auth failure).
- [ ] Resume protocol exists: sequence IDs, bounded replay buffer, explicit `resume_failed` → snapshot path; no silent gaps.
- [ ] Ordering/delivery guarantees documented; per-channel sequencing enforced; at-least-once paths have acks + idempotent dedupe.
- [ ] Per-connection send queues bounded with a defined overflow policy (coalesce or disconnect); no blind unbounded sends.
- [ ] Inbound limits: max message size (1009), frame rate limit, max subscriptions per connection, pre-auth resource caps.
- [ ] Concurrent-connection caps per user (per IP pre-auth) and per node, enforced at the upgrade with a clean refusal (§2.5) — MEDIUM.
- [ ] State-changing client commands carry a nonce/timestamp and replays or stale ones are rejected; each frame's parse/validate/dispatch is isolated so one bad message cannot crash the worker (§2.7) — MEDIUM.
- [ ] **Raw TCP tunnelled over WebSocket (§2.1) — HIGH if browser-reachable**: the tunnelled service keeps its own authn/authz. Locator for bridges:
      `grep -rniE 'websockify|novnc|ttyd' .`
- [ ] **SSE client URL (§3) — MEDIUM**: `EventSource` URLs come from config, `event.origin` checked where the stream can be cross-origin or redirected. Constructors fed from the location, query or params:
      `grep -rnE 'new[[:space:]]+EventSource\([^)]*(location|[pP]arams|query|\.search|\.hash|input)' --include='*.js' --include='*.jsx' --include='*.ts' --include='*.tsx' --include='*.vue' --include='*.svelte' .`
- [ ] Close codes documented incl. app-range (4xxx); deploys drain gracefully (1001/1012) instead of mass-killing.
- [ ] SSE: `id:` on every event with `Last-Event-ID` replay, comment heartbeats, proxy buffering disabled, terminal `done` event on finite streams.
- [ ] Cross-node fanout via pub/sub backplane; sticky sessions not abused as state storage; node death recoverable from client-held resume state.
- [ ] Presence uses TTL + heartbeat with flap debouncing; roster sync on join + deltas.
- [ ] Hot-channel throttling/coalescing exists; reconnect-stampede tested (kill a node in staging, watch the herd).
- [ ] Frames are enveloped (type/seq/channel), unknown frame types ignored, commands correlated by ID with ack/nack and client-side timeouts; protocol versioned.
- [ ] Realtime errors machine-readable (code + retry hints), mirroring the HTTP error discipline.

# 06 — Webhooks

Scope: both roles — **provider** (you deliver webhooks to user-supplied URLs) and
**consumer** (you receive them). Signing, replay protection, retries, ordering,
idempotent consumption, SSRF defense.

## 1. Payload & contract design

- Webhook = event notification, same schema discipline as the API: versioned event
  types (`invoice.paid`), documented JSON schema, additive evolution (rules/02).
- Envelope every event:

```json
{
  "id": "evt_01HZX4K7",            // globally unique — consumer dedupe key
  "type": "invoice.paid",
  "created_at": "2026-06-12T09:30:00Z",
  "api_version": "2026-06-01",
  "data": { "object": { ... } }
}
```

- **Thin vs fat payloads**: fat (full object) is convenient but delivers stale
  state out of order and leaks data through the consumer's logs/infra. **Thin
  payloads (ID + type, consumer fetches current state via API)** are the safer
  default for sensitive domains and neutralize most ordering pain (§5). Offer fat
  payloads only where the fetch round-trip genuinely hurts.
- Never put secrets/PII you wouldn't put in an email into a fat payload — you do
  not control the receiving infrastructure.
- Let users subscribe per event type; don't firehose everything to every endpoint.

## 2. Signing — HMAC with timestamp

Unsigned webhooks are unauthenticated POSTs from the internet; any auditor flags
them instantly.

**Provider:**
- Sign `id + "." + timestamp + "." + raw_body` with HMAC-SHA256 using a per-endpoint
  secret (the Standard Webhooks signed-content format). Send id + timestamp +
  signature in headers. The **Standard Webhooks** spec is the 2026 convention —
  adopt it instead of inventing:

```http
POST /hooks/billing HTTP/1.1
webhook-id: evt_01HZX4K7
webhook-timestamp: 1781256600
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=
```

- Sign the **raw bytes** you send. Include the timestamp **inside** the signed
  string (binds it — attacker can't re-send old body with fresh timestamp).
- Version-prefix signatures (`v1,`) and support **multiple signatures per
  delivery** — that's what makes zero-downtime secret rotation possible (sign with
  old+new during overlap; consumers verify against any of their stored secrets).
- Per-endpoint secrets (one compromised consumer ≠ all), shown once at creation,
  rotatable via API/dashboard.
- mTLS or OAuth-to-consumer are heavier alternatives for enterprise consumers;
  HMAC remains the baseline everyone must have.

**Consumer verification — the order matters:**
1. Read the **raw body bytes** (before any JSON parsing/middleware re-serialization
   — re-serialized JSON ≠ signed bytes; the most common "signature randomly fails"
   bug).
2. Check timestamp within tolerance (±5 min) → otherwise reject (replay window).
3. Compute HMAC, compare with **constant-time comparison** (`hmac.compare_digest`,
   `crypto.timingSafeEqual`) — `==` on signatures is a timing-oracle finding.
4. Only then parse and process.
- Also verify the event is *for you* (right endpoint/tenant) if the provider
  multiplexes.

Reference consumer verification (Python; pattern is language-agnostic):

```python
# BAD — three findings in four lines
body = request.json()                              # parsed before verification
expected = hmac_sha256(secret, json.dumps(body))   # re-serialized != signed bytes
if request.headers["X-Sig"] == expected: ...       # timing-unsafe ==; no timestamp check

# GOOD
raw = request.raw_body()                                   # exact bytes
ts = int(request.headers["webhook-timestamp"])
if abs(time.time() - ts) > 300: raise Reject(400)          # replay window
msg = f"{request.headers['webhook-id']}.{ts}.".encode() + raw
for secret in active_secrets:                              # rotation: try all
    digest = hmac.new(secret, msg, hashlib.sha256).digest()
    for sig in parse_v1_signatures(request.headers["webhook-signature"]):
        if hmac.compare_digest(digest, sig):               # constant-time
            return process(json.loads(raw))
raise Reject(401)
```

## 3. Replay protection

Signature alone doesn't stop re-delivery of a *validly signed* old request
(captured by a proxy, leaked from logs).

- Timestamp tolerance (§2) bounds the window.
- Within the window: **dedupe on `webhook-id`/event `id`** — store processed IDs
  with TTL ≥ tolerance window (and ideally ≥ provider's max retry horizon);
  reject/no-op duplicates. This doubles as idempotency (§4) — one mechanism, two
  jobs.
- Providers: never reuse event IDs, even across retries of the same event
  (same id on retry — that's the point), and rotate-don't-share secrets.
- **Prove the rejections in CI, not only the happy path.** The receiver's tests
  send (a) a forged callback (wrong secret, or one flipped body byte), (b) a
  correctly signed but stale one outside the tolerance, (c) a duplicate event ID
  inside the window, and (d) a callback validly signed for endpoint or tenant A
  replayed to endpoint B. Each must be rejected, and the test asserts that no
  payload field was read or acted on (no handler call, no row written).
  Verification that silently stopped enforcing (a refactor that skips it on a
  parse error, a test-mode flag left on) otherwise shows up only in production.
  OWASP: AI-Powered Advertising Systems Security cheat sheet.

## 4. Delivery, retries & idempotent consumers

**Provider:**
- Deliveries come from a **queue with persistent state**, never inline in the
  request path that generated the event (use outbox pattern: event row committed
  in the same transaction as the domain change, dispatcher drains it — otherwise
  you emit webhooks for rolled-back transactions or drop events on crash).
- Success = any 2xx within a short timeout (5–10s total, no following redirects —
  §8). Everything else (timeout, 4xx except 410, 5xx) ⇒ retry with **exponential
  backoff + jitter** over hours-to-days (e.g. 1m, 5m, 30m, 2h, 5h, 10h, 24h…
  ~3 days like Stripe).
- `410 Gone` from consumer ⇒ auto-disable the endpoint. Persistent failures
  (e.g. >95% over 3 days) ⇒ disable + notify the owner. Track per-endpoint health.
- Expose to consumers: delivery logs, manual redelivery button/API, and a
  **reconciliation API** (`GET /events?after=...`) so consumers can backfill
  gaps — webhooks are at-least-once-ish, never guaranteed; the events API is the
  source of truth.
- Don't let one dead endpoint block others: per-endpoint queues/concurrency
  isolation.

**Consumer:**
- **Ack fast, process async**: verify signature → enqueue → return `2xx` in
  <1s. Doing real work inline ⇒ provider timeout ⇒ retry storm ⇒ duplicate
  processing of slow work. Return 5xx only when you failed to durably enqueue.
- **Idempotency is mandatory** — retries guarantee duplicates. Dedupe on event
  `id` (unique constraint in DB beats best-effort cache), and make handlers safe
  to re-run anyway (upserts, state-machine guards).
- Don't trust webhook *content* for money-moving decisions when thin-fetch is
  available: signature proves origin, but fetching current state by ID closes
  staleness/ordering gaps.

Provider dispatch architecture (reference shape):

```text
domain tx ──commit──> outbox row
outbox poller ──> event store (immutable events, powers GET /events)
             └─> per-endpoint delivery queues ──> sender workers
                                                  | resolve->validate IP->pin->POST
                                                  | record attempt (status, latency)
                                                  └─ on failure: schedule retry (backoff+jitter)
```

- The event store is canonical; delivery attempts reference it. Redelivery and
  reconciliation read from it — never regenerate payloads from live data (the
  object may have changed; signatures/audits must match what was sent).
- Sender workers are stateless and horizontally scalable; per-endpoint
  concurrency = 1..N with ordering *not* guaranteed (and documented as such, §5).
- Emit provider-side metrics per endpoint: success rate, P95 delivery latency,
  retry depth, disabled-endpoint count — this is your consumer-health dashboard.

## 5. Ordering caveats

**Webhooks are unordered. Period.** Retries, parallel dispatchers, and network
races mean `invoice.paid` can arrive before `invoice.created`. Providers should
not promise ordering (per-key serial delivery throttles throughput to the slowest
consumer and still breaks on retry).

Consumer strategies:
- Treat events as triggers, **fetch current state** by ID (thin-payload mindset)
  — ordering becomes irrelevant.
- Or compare `created_at`/sequence in the event against last-applied state and
  drop stale updates (last-writer-wins per object).
- Or buffer out-of-order events briefly (park `*.paid` until `*.created` is seen,
  with a timeout that falls back to fetching).
- Never build a state machine that errors on out-of-order arrivals — that's a
  design bug, not a provider bug.

## 6. Consumer-side endpoint hygiene

- Dedicated route, **no session/cookie auth required** but signature verification
  enforced; reject unsigned/badly signed with `401`, don't reveal verification
  details in error bodies.
- Enforce body size limit before reading fully; `415` non-JSON; rate-limit per
  source as backstop.
- Log every delivery (id, type, signature result, latency, outcome) — disputes
  with providers are settled with logs.
- Secrets in a secret manager, per provider, rotated when staff leave; support
  two active secrets to absorb provider rotation (§2).

## 7. Webhooks vs alternatives — when not to webhook

- Consumer needs *every* event, in order, with replay → give them an **events
  API to poll** (`GET /events?after=evt_x`, cursor-paginated) or a streaming
  channel; webhooks alone are a notification optimization on top of that, not a
  reliable transport.
- Very high volume to one consumer (>~100/s sustained) → batch events per
  delivery, or switch to a queue/stream integration (consumer-owned SQS/Kafka
  topic, Event Grid). Per-event HTTP POSTs don't scale linearly forever.
- Consumer is inside your own estate → use the message bus directly; webhooks
  through the public internet between your own services is an architecture smell.
- Zapier-style fan-in platforms: support thin payloads + reconciliation API and
  they'll integrate fine; don't build platform-specific hacks.

## 8. SSRF — provider delivering to user-supplied URLs

A webhook sender is **an HTTP client that attackers point anywhere** ("give me
your cloud metadata, please"). Egress controls are non-negotiable:

- **Validate at registration AND at every send** (DNS changes between the two —
  rebinding): resolve the hostname, reject private/reserved ranges —
  `10/8, 172.16/12, 192.168/16, 127/8, 169.254/16` (cloud metadata!), `::1`,
  `fc00::/7`, `fe80::/10`, and IPv4-mapped IPv6 forms.
- **Pin the resolved IP for the actual connection** (resolve→check→connect to
  that IP with Host/SNI set) — checking then re-resolving is a TOCTOU rebinding
  hole.
- **Don't follow redirects** (or re-validate every hop — simpler: don't). A public
  URL 302→`http://169.254.169.254/` is the classic bypass.
- HTTPS-only endpoints (allow plain HTTP at most for explicit dev mode); modest
  timeouts; cap response bytes read (you only need the status); never include
  response bodies from consumer endpoints in your logs/UI unsanitized.
- Best: dispatch from an **isolated egress** (dedicated VPC/proxy with deny-all
  to internal ranges) so a validation bug still hits a wall.
- Prove ownership before sending real data to a new URL (challenge token echo /
  verification event) — stops using your webhook system to spam/probe third
  parties.
- Per-tenant send rate limits — your webhook sender must not be rentable DDoS
  infrastructure.

## Audit checklist

**Provider role:**
- [ ] HMAC-SHA256 over `id.timestamp.raw_body` (message id included), versioned scheme, per-endpoint secrets, multi-signature rotation support (Standard Webhooks-compatible preferred).
- [ ] Outbox/queue-backed dispatch (no inline sends, no events from rolled-back transactions); per-endpoint isolation so one dead consumer can't starve others.
- [ ] Exponential backoff + jitter retries over days; 410 auto-disables; failing endpoints disabled with owner notification.
- [ ] Delivery logs, manual redelivery, and an events reconciliation API exposed to consumers.
- [ ] Event IDs unique and stable across retries; no ordering promised in docs.
- [ ] SSRF defenses: private/metadata IP ranges blocked with resolve-pin-connect (no TOCTOU), redirects not followed, HTTPS-only, response size cap, isolated egress, URL ownership verification, per-tenant send rate limits.
- [ ] No secrets/excess PII in payloads; thin payloads for sensitive domains; per-event-type subscriptions.

**Consumer role:**
- [ ] Signature verified on raw bytes before parsing; constant-time comparison; timestamp tolerance enforced (±5 min).
- [ ] Dedupe on event ID with durable store (TTL ≥ retry horizon); handlers idempotent (unique constraints/upserts), out-of-order-safe.
- [ ] **Negative signature tests (§3) — MEDIUM**: CI proves forged, stale, duplicate and cross-endpoint callbacks are rejected before any payload field is used. Webhook test files with none of those cases (a lead to read):
      `grep -rliE 'webhook' --include='*test*' --include='*spec*' . | while IFS= read -r f; do grep -qiE 'invalid[_ -]?sig|bad[_ -]?sig|forged|tamper|stale|expired|replay' "$f" || echo "$f"; done`
- [ ] Ack-fast/process-async: 2xx only after durable enqueue, well under provider timeout.
- [ ] State derived by fetching current object where staleness matters, not from fat payload alone.
- [ ] Endpoint has body size limits, no cookie-auth dependency, full delivery logging; secrets in a manager with dual-secret rotation support.

# 07 — API Security & Operations

Scope: authn scheme selection, request signing, rate limiting & quotas, bot
management, request limits, timeout budgets, CORS for APIs, audit logging,
multi-tenant isolation at the API layer, API response headers, API inventory.

## 1. Authentication schemes — choosing

| Scheme | Use for | Notes |
|---|---|---|
| API keys | server-side partner/B2B access, simple integrations | bearer secrets: hash at rest, prefix for identification (`sk_live_…`), scoped, rotatable, never in URLs |
| OAuth2 client credentials | M2M where you want standard issuance, expiry, scopes, central revocation | short-lived JWT access tokens; the upgrade path from raw API keys |
| OAuth2 auth code + PKCE | acting on behalf of end users (third-party apps) | never password-grant; never implicit |
| mTLS | high-assurance B2B (finance/health), service mesh internal | strongest binding; cert lifecycle is the cost; pairs with OAuth (RFC 8705 cert-bound tokens) |
| Session cookies | first-party browser frontends | then CSRF defenses apply; don't mix with bearer on the same endpoints without thought |

Rules regardless of scheme:
- Credentials in the `Authorization` header (or mTLS), **never in query strings**
  (logs, referers, history).
- API keys: store only a hash (treat like passwords), display once, support
  ≥2 concurrent keys per principal for zero-downtime rotation, track `last_used_at`
  (enables dead-key cleanup and incident scoping), scope to least privilege
  (read-only vs write keys), expire or force-rotate stale keys.
- An API key identifies and meters a caller; it is not a whole access-control
  story. It is never the only guard on a sensitive or high-value resource (add
  OAuth scopes, mTLS or per-object authz), and it is revoked when the holder
  breaks the usage terms or abuses the API, not only when it leaks. HTTP Basic
  auth resends a reusable secret, only base64-encoded, on every call: avoid it,
  and where a legacy client forces it, serve it over TLS only (§8 on plaintext).
  OWASP: REST Security and Web Service Security cheat sheets.
- JWTs: validate `iss`, `aud`, `exp`, algorithm allowlist (no `alg:none`, no
  HS/RS confusion); access tokens ≤15–60 min; revocation story decided (short
  expiry + denylist for the rest).
- **Authn ≠ authz**: every handler authorizes object-level access
  (BOLA/IDOR — still the #1 API vulnerability class) and function-level access
  (admin routes). Centralize in middleware/policy, deny by default; a missing
  authz check should be a compile/lint/review failure, not a runtime surprise.
- Internal ≠ trusted: service-to-service calls also authenticate (mesh mTLS +
  workload identity). "It's behind the VPN" is an audit finding.
- **Sign the message, not only the channel, when one request moves money or
  authority or crosses several hops** (B2B writes, payment and settlement calls,
  agent/MCP JSON-RPC traffic): TLS ends at each proxy, a signature does not. This
  is the rules/06 §2 webhook model applied to requests. With HTTP Message
  Signatures (RFC 9421), the receiver requires a minimum covered set: `@method`,
  `@target-uri` (or `@authority` + `@path`), the tenant and audience fields,
  `created`/`expires`, and `content-digest`. RFC 9421 does not cover the body by
  itself (Section 7.2.8), so the sender adds an RFC 9530 `Content-Digest` and the
  receiver recomputes it over the bytes it actually received. A digest header
  that verifies but is never recomputed still allows the body to be swapped.
  Prefer asymmetric keys whose `keyid` maps to one registered sender, reject
  replays by nonce or `created` window (rules/06 §3), and fail closed when a
  signature is missing. Any field outside the covered set that can change an
  amount or an authorisation decision is a defect. OWASP: ASVS 5.0 V4.1.5; MCP
  Security, AI-Powered Advertising Systems Security, Bot Management and
  Anti-Automation, Multi Tenant Security cheat sheets.

## 2. Rate limiting

- **Key by authenticated principal** (API key/account), not IP, for authed
  traffic — IPs are shared (NAT/CGNAT) and rotated by attackers. IP-based limits
  are the backstop for unauthenticated surfaces (login, signup, token endpoint).
- Algorithm: **sliding window counter** (or GCRA/token bucket) — fixed windows
  allow 2x bursts at boundaries; pure sliding logs are memory-heavy. Token bucket
  when you explicitly want burst allowances atop a sustained rate.
- Enforce in a shared store (Redis + atomic Lua / built into the gateway) —
  per-instance in-memory limits multiply by replica count and reset on deploy.
- Respond `429` with headers, both legacy and the IETF standard:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 13
RateLimit-Policy: "default";q=100;w=60
RateLimit: "default";r=0;t=13          # IETF draft-ietf-httpapi-ratelimit-headers
Content-Type: application/problem+json

{"type":"https://api.example.com/errors/rate-limited","title":"Rate limited",
 "status":429,"detail":"Limit 100/min exceeded.","retry_after":13}
```

- Include limit headers on **successful** responses too — clients should
  self-throttle before hitting 429.
- **Carve-out: credential, signup and other anti-automation surfaces** (login,
  token, password reset, OTP, gift-card or voucher checks). Answer an exceeded
  limit with a plain `429` and a generic body: no bucket name, no attempts
  remaining, and no `Retry-After` or `RateLimit` reset precise enough to schedule
  the next burst against. Precise counters on these routes let an attacker tune
  a credential-stuffing run to stay just under the threshold. OWASP: Bot
  Management and Anti-Automation cheat sheet.
- Tiered limits: per-endpoint-class (cheap reads vs expensive writes vs auth
  endpoints), per-plan, and a global per-principal ceiling. Expensive operations
  (search, export, GraphQL) cost more than 1 unit (cost-based limiting).
- **Quotas** are a separate layer: monthly/daily entitlements (billing), enforced
  eventually-consistent, distinct error message ("quota exhausted, resets
  2026-07-01 / upgrade") vs rate ("slow down, retry in 13s"). Don't conflate them.
- Server-side concurrency caps (max in-flight per principal) catch slow-request
  abuse that req/sec limits miss.
- **Key set and identity cost.** Beyond principal, IP and endpoint, key buckets
  on session and on ASN or geography where datacenter traffic is unexpected.
  Login uses two independent buckets, per account and per source, never one
  bucket on the `(ip, user)` pair, which lets one IP try every username. Cut a
  client's allocation automatically when its behaviour turns anomalous (a sudden
  spike, unusual target patterns), and restore it by policy, not by hand. A limit
  per identity is only as strong as the cost of a new identity: tie API and agent
  identities to a verified operator or account, so minting many keys does not
  multiply the allowance.
- **A per-route override that weakens the global limit is a finding.** Framework
  defaults are often empty: Django REST Framework's `DEFAULT_THROTTLE_CLASSES`
  is empty unless configured, and a view setting `throttle_classes = []` or
  `@throttle_classes([])` runs with no throttle. Every override needs a reason
  in review. OWASP: AML Sanctions AI Agent Payments, Bot Management and
  Anti-Automation, Django REST Framework cheat sheets.
- **Bot management is layered; rate limits are one layer.** Edge: IP and ASN
  reputation, TLS (JA3/JA4) and HTTP/2 fingerprints. Application: session-aware
  limits, honeypot fields and bait paths, challenges escalated by risk.
  Business: velocity rules, fraud scoring, review queues. A request can pass one
  layer and fail the next (ten checkouts in thirty seconds, each with a good IP
  and a solved challenge). Business flows enforce a minimum realistic interval
  between steps (a checkout or signup completed faster than a person could is
  rejected or stepped up). User-generated content goes through submitter
  reputation and delayed publishing. The aim is to raise attacker cost, not to
  block every bot: search crawlers, uptime monitors and accessibility tools
  must keep working, and responses are graduated (log, step up, tarpit, block)
  rather than all-or-nothing. Depth: `sota-code-security` rules/02 (signup) and
  rules/07 §2.1 (detection points). OWASP: ASVS 5.0 V2.4.2; Bot Management and
  Anti-Automation cheat sheet.

## 3. Request size limits & input hygiene

- **Explicit max body size on every route** (gateway default e.g. 1 MB, raised
  per-route for uploads) → `413`. Also: max header size, max URL length, max
  query params, max multipart parts.
- JSON parsing limits: max depth, max keys/array length — deeply nested payloads
  are a CPU/stack DoS. Decompression limits (zip-bomb body with
  `Content-Encoding: gzip`): cap the *decompressed* size.
- Uploads: don't proxy large files through the API — issue pre-signed URLs to
  object storage; validate type by magic bytes not extension/Content-Type alone.
- Schema-validate all input at the boundary (the OpenAPI/proto schema from
  rules/01–04 is the enforcement artifact); reject unknown fields on writes
  (rules/02 §3).

### 3a. HTTP message framing & request smuggling

Smuggling lives in the gap between two hops that disagree on where one request
ends and the next begins; the front hop's checks see one request, the back hop
runs two. Treat ambiguous framing as an attack, not as something to be lenient about.

- **HTTP/1.1**: a request carrying both `Transfer-Encoding` and `Content-Length`
  gets rejected (`400`) and the connection closed. RFC 9112 Section 6.1 permits
  processing it by `Transfer-Encoding` alone but requires the connection close
  either way, and its Section 6.3 says such a message "ought to be handled as an error". The
  same goes for a request whose `Transfer-Encoding` does not end in `chunked`, and
  for an invalid or self-contradicting `Content-Length` list (Section 6.3 makes
  both a `400`-and-close).
- **HTTP/2 and HTTP/3**: a message carrying a connection-specific field
  (`Transfer-Encoding`, `Connection`, `Keep-Alive`, `Upgrade`, `Proxy-Connection`;
  `TE` only as `trailers`) is malformed, and so is a `content-length` that differs
  from the sum of the DATA frame payloads (RFC 9113 Sections 8.1.1 and 8.2.2, RFC
  9114 Sections 4.1.2 and 4.2). An intermediary must not forward either. This matters most where
  an HTTP/2 edge **downgrades** to HTTP/1.1 towards the origin: a field that the
  binary framing carried harmlessly becomes framing again on the old wire (the
  CR/LF/NUL variant: `sota-code-security` rules/01 §11).
- **Generating**: never emit a `Content-Length` that disagrees with what the
  framing actually sends, and never both headers on one message (RFC 9112 Section 6.2).
  Hand-set length headers on a streamed or compressed body are the usual source.
- **One parser posture on every hop**: LB, CDN, WAF, gateway and app server must
  all be strict. A relaxed-parsing switch on any one of them reopens the gap,
  e.g. Node's `insecureHTTPParser: true` / `--insecure-http-parser` (whose docs
  list accepting both headers among its leniencies) or HAProxy's
  `option accept-unsafe-violations-in-http-request` (formerly
  `accept-invalid-http-request`, now deprecated). Prefer HTTP/2 end to end to an
  HTTP/1.1 backend link, and do not reuse a backend connection after a framing error.
- Audit by pairing hops, not by reading one config: list every hop's server and
  version, find where the protocol changes (h2 in front, h1 behind), and run a
  desync scanner against staging through the real edge. (Needs verification
  per stack: which exact inputs each proxy normalises differs by product and version.)

OWASP: ASVS 5.0 V4.2.1, V4.2.2, V4.2.3.

## 4. Timeout budgets

Every request has an end-to-end budget; every hop fits inside it.

- Order matters: client timeout > LB timeout > app server timeout > downstream
  call timeouts (sum or max along the path) > DB statement timeout. An inner
  timeout exceeding an outer one means work continues after the caller is gone.
- Per-route, not global: `GET /users/{id}` 2s; `POST /reports` should be async
  (202 + status resource, rules/01 §3) rather than a 10-minute synchronous wait.
- Server-side: read/write/idle timeouts on the listener (slowloris), statement
  timeouts in the DB, **cancellation propagation** — client disconnect aborts
  downstream work (rules/04 §3).
- Retries (client or mesh) live *inside* the budget, idempotent routes only,
  backoff + jitter, with circuit breaking — otherwise retries amplify outages 3x.
- Return `504` (upstream) / `503 + Retry-After` (load shedding) honestly; do not
  hold connections open hoping.

## 5. CORS for APIs

- CORS is a **browser** mechanism: it doesn't protect the API (curl ignores it);
  it protects *users* from malicious origins riding their credentials. Server-side
  authz must never depend on it.
- Token-auth APIs for known frontends: explicit origin **allowlist** (exact
  origins from config), `Access-Control-Allow-Headers: Authorization,
  Content-Type`, only needed methods, `Access-Control-Max-Age: 600`+ to cut
  preflights.
- **Never** `Access-Control-Allow-Origin: *` with `Allow-Credentials: true`
  (spec forbids it — so libraries "helpfully" reflect the Origin header instead,
  which is *worse*: any site can ride cookies). Reflecting arbitrary origins with
  credentials is a critical finding.
- `*` without credentials is acceptable for genuinely public, unauthenticated,
  read-only APIs.
- Cookie-auth'd APIs additionally need CSRF defenses (`SameSite=Lax/Strict` +
  token or custom-header check) — CORS preflights don't cover
  form/simple-request CSRF.
- Don't blanket-expose headers; `Access-Control-Expose-Headers` only what clients
  read (e.g. `RateLimit`, `Sunset`, `Location`).

```text
# BAD (found in the wild constantly)
Access-Control-Allow-Origin: <echo of request Origin>   # reflection
Access-Control-Allow-Credentials: true                  # + credentials = any site rides cookies
Access-Control-Allow-Headers: *
Access-Control-Allow-Methods: *

# GOOD (token-auth SPA frontend)
Access-Control-Allow-Origin: https://app.example.com    # from explicit allowlist
Vary: Origin
Access-Control-Allow-Methods: GET, POST, PATCH, DELETE
Access-Control-Allow-Headers: Authorization, Content-Type, Idempotency-Key
Access-Control-Expose-Headers: RateLimit, Retry-After, Location
Access-Control-Max-Age: 7200
```

## 6. Audit logging

- Two streams, different lifecycles: **ops logs** (debugging, short retention)
  and the **audit trail** (security/compliance: append-only, long retention,
  integrity-protected, restricted read access).
- Audit-log these always: authn events (success/failure, key used), authz
  denials, all writes to sensitive resources (who/what/when/before-after or
  diff-ref), admin/break-glass actions, key/secret lifecycle, data exports,
  rate-limit and quota trips, webhook endpoint changes. Also, often missed:
  service or API token creation with the **scopes/entitlements granted**; explicit
  logout (with a hash of the session ID, never the ID itself); file and upload
  deletion; data imports; creation and deletion of system-level objects
  (tenants, projects, API clients); receipt and processing of user-generated
  content and uploads; out-of-sequence steps in a multi-step flow and fraud
  signals. Event names: `sota-code-security` rules/07 §2.1.
- **High-risk operations write two entries**: an intent record before the action
  (who, what, which object, request ID) and an outcome record after it. An
  action that crashes, times out or is killed midway then still leaves a trace,
  and an intent with no outcome is itself an alertable signal. OWASP: Logging
  Vocabulary, Logging and REST Security cheat sheets.
- **WebSocket and other long-lived channels** (rules/05): the HTTP access log
  sees only the upgrade. Log connection open and close (user, IP, `Origin`),
  auth and authz decisions at the handshake and per message, rate-limit and
  message-validation violations, abnormal disconnects and protocol errors, never
  message bodies or tokens. OWASP: WebSocket Security cheat sheet.
- Every entry: timestamp, actor (principal + acting-on-behalf-of), tenant,
  action, object type+ID, outcome, source IP, user agent, **trace/request ID**
  correlating to ops logs. Plus the "where": application ID and version,
  hostname, protocol and port, request method and URI, region, and client port
  (behind NAT or CGNAT the source IP alone does not identify a client; RFC 6302
  recommends logging the source port with a traceable timestamp). Clocks on
  every node are time-synced and drift is alerted on, or events from two hosts
  cannot be ordered (`sota-observability` rules/01 §1). On sensitive and
  anti-automation endpoints, add ASN, country, TLS/HTTP2 fingerprint, a hashed
  session ID, and the bot or risk decision **with the signals behind it**
  (`bot_score`, rule name). An unlogged anti-bot decision cannot be tuned.
  Hash or truncate fingerprints before storage. OWASP: Logging Vocabulary and
  Bot Management and Anti-Automation cheat sheets.
- **Never log**: credentials, bearer tokens, full API keys (log the key *prefix*),
  passwords, cookie values, full card/SSN data, raw request bodies of sensitive
  endpoints. Centralized redaction middleware, not per-handler discipline; test
  it (send a fake secret, grep the logs in CI/staging).
- Logs are an injection target: encode/escape user-controlled strings (CRLF/log
  forging); treat log viewers as XSS sinks.
- Request IDs: accept inbound `traceparent` (W3C Trace Context), generate if
  absent, return an ID header on every response (incl. errors — rules/01 §9), and
  propagate downstream.

## 7. Multi-tenant isolation at the API layer

Cross-tenant data leakage is the worst API bug class. Defense in depth:

- **Tenant from the credential, never the request**: derive tenant ID from the
  authenticated principal (token claim/key record). A `tenant_id` in the body or
  query is at most a *consistency check* against the credential — never the
  source of truth. (`X-Tenant-Id` headers trusted from clients = critical
  finding.)
- **Scope every query structurally**: tenant filter applied by a repository
  layer/ORM global scope or Postgres RLS (`SET LOCAL app.tenant_id` or
  `set_config('app.tenant_id', …, true)` inside the request's transaction — a
  plain `SET` leaks to the next borrower under transaction pooling, see
  `sota-databases` rules/04; policies on every table) — not by remembering
  `WHERE tenant_id = ?` in each handler. RLS as a second enforcement layer
  catches the handler someone forgot.
- Resource IDs: lookups are always `(tenant_id, id)`; return `404` (not `403`)
  for other tenants' resources to avoid existence oracles — and make that
  consistent (a timing or message difference is still an oracle).
- Isolation applies to *everything*, not just primary GETs: list filters, search,
  exports, aggregations/counts, **webhooks** (events only to the owning tenant's
  endpoints), realtime channels (rules/05 — channel authz), idempotency-key
  scopes, ETag values, and cache keys (a shared cache without tenant in the key
  is a leak machine).
- Noisy-neighbor: rate limits and quotas per tenant (§2), per-tenant concurrency
  caps, fair-queuing on expensive shared resources.
- Cross-tenant admin/support access: separate audited surface with explicit
  on-behalf-of recording (§6) — not super-tenant credentials in the normal API.
- **A shared audit store is tenant data too.** Reads filter by the caller's
  tenant, taken from the credential. Reading across tenants needs a separate
  platform-auditor permission that no tenant admin role holds, and that access is
  itself audited. Writes take the tenant from the verified context: a tenant, or
  a service acting for one, cannot append entries to another tenant's stream,
  which would let it plant or bury evidence. OWASP: Multi Tenant Security cheat
  sheet.
- **Test it continuously**: automated suite that, for every endpoint, attempts
  access to tenant B's resources with tenant A's credentials and asserts 404 —
  the highest-ROI security test an API team can own.

## 8. Gateway placement & defense in depth

- Centralize cross-cutting controls at the gateway/edge (TLS termination, authn
  verification, rate limits, size limits, CORS, request-ID injection,
  WAF/bot rules); keep **authorization and tenant scoping in the service** —
  the gateway doesn't know your object model. The gateway may still do
  **coarse** authorisation as the first layer: route or scope checks such as
  "`POST /admin/*` needs `admin:write`" or "this client may call only these
  operations". That drops obviously unauthorised traffic early, but it adds to
  the service-level and object-owner checks and never replaces them. OWASP:
  Microservices Security cheat sheet.
- The gateway is one layer, not the boundary: services must reject unauthenticated
  traffic even from "inside" (a path that bypasses the gateway — internal port,
  mesh misconfig, SSRF pivot — must hit a second wall). Verify: call a service
  pod directly in staging without gateway headers; it must 401.
- Never trust gateway-injected identity headers (`X-User-Id`) unless the link
  is mTLS-pinned and the header is stripped from external requests at the edge
  — header-smuggling of identity is a recurring critical.
- **mTLS that ends at an LB or CDN stops being mTLS at that hop.** Behind it the
  service holds no certificate, only a forwarded client-certificate header, and
  RFC 8705 Section 6.5 explicitly leaves how that metadata travels safely out of scope.
  So the header gets the `X-User-Id` treatment above: honoured only on the link
  from the terminating proxy (itself authenticated), with any client-sent copy
  removed at the edge (the proxy-side settings: `sota-code-security` rules/04 §5).
  Where the caller's identity *is* the authorisation (payments, agent-initiated
  actions, B2B writes), do not let the header carry it alone: bind identity into
  the message, e.g. a sender-constrained token whose `cnf` thumbprint
  (RFC 8705 `x5t#S256`, or DPoP, RFC 9449) the service checks itself, or a signed
  request (HTTP Message Signatures, RFC 9421); or pass TLS through to the service.
  OWASP: AML Sanctions AI Agent Payments cheat sheet.
- TLS posture: TLS 1.2+ only, HSTS on API hosts, no plaintext listeners except
  health checks on loopback.
- **No transparent HTTP-to-HTTPS redirect on API endpoints.** Only hosts that
  people open in a browser redirect. An API host answers plaintext with an error
  (or does not listen on port 80). A client misconfigured to `http://` has
  already sent its token in cleartext, and a silent redirect makes it work, so
  nobody notices the leak. OWASP: ASVS 5.0 V4.1.2.
- **Defensive headers on JSON responses** a browser may fetch, as defence in
  depth: `Content-Security-Policy: default-src 'none'; frame-ancestors 'none'`
  (nothing in an API response should load, run or be framed),
  `Permissions-Policy` with empty allowlists (`camera=(), geolocation=()`…),
  `Referrer-Policy: no-referrer`, `X-Content-Type-Options: nosniff`, and
  `Cache-Control: no-store` on sensitive data. This differs from HTML pages
  on purpose. A page links to other sites and needs `strict-origin-when-cross-origin`
  (the browser default per the W3C Referrer Policy spec) plus a real CSP. An API
  response should trigger no further requests, so it can refuse everything.
  Non-browser clients ignore these headers; they cost nothing. Page baseline:
  `sota-code-security` rules/05. OWASP: REST Security cheat sheet.

## 9. OWASP API Security Top 10 mapping (2023 list, still canonical)

| OWASP | This skill |
|---|---|
| API1 Broken Object Level Auth | §1, §7 — credential-derived tenant, per-object checks, cross-tenant test suite |
| API2 Broken Authentication | §1 — scheme selection, JWT validation, key handling |
| API3 Object Property Level Auth | rules/01 §1 (explicit DTOs — no mass assignment/ORM dumps), §1 authz |
| API4 Unrestricted Resource Consumption | §2–4 — rate limits, quotas, size limits, timeout budgets; rules/03 §4 |
| API5 Broken Function Level Auth | §1 — deny-by-default policy, admin surface separation |
| API6 Unrestricted Access to Sensitive Business Flows | §2 cost-weighted limits + flow-specific throttles; **enforce multi-step flow order server-side** — model the flow as a state machine, validate the current state on every step, reject out-of-order/replayed steps (don't trust client sequencing). Business-logic depth: sota-code-security |
| API7 SSRF | rules/06 §8 — webhook/user-URL egress controls |
| API8 Security Misconfiguration | §5 CORS, §8 gateway/TLS; rules/03 §4 introspection |
| API9 Improper Inventory Management | rules/02 §5 — versioned, measured, sunset surfaces; spec-as-truth (rules/01 §10) |
| API10 Unsafe Consumption of APIs | rules/06 consumer role; rules/04 §3 deadlines on upstream calls |

Use this table to structure a security-focused audit report when the requester
wants OWASP-mapped findings.

**API9 inventory, concretely.** Each API host has an inventory row with host,
version, environment (production, staging, test, development) and intended
audience (public, partner, internal), plus its auth, rate-limit and CORS posture.
To audit it, compare three lists: the endpoints and parameters the server code
routes (extract from routing code, OWASP Noir is one extractor), the published
spec, and the URLs that shipped client JS/HTML bundles reveal (LinkFinder and
jsluice are examples — both without a commit since early 2024, checked
2026-09-26 on GitHub, so treat them as unmaintained). An entry on one list and missing from another is an
undocumented or orphaned surface. Also check the server URLs a published
description advertises (OpenAPI `servers`, WSDL `soap:address`): each must be
intended and live, with no staging, localhost or private-range host.
OWASP: WSTG-APIT-01; Django REST Framework cheat sheet.

## Audit checklist

- [ ] Auth scheme appropriate per consumer type; no credentials in query strings anywhere (grep logs/gateway config).
- [ ] API keys hashed at rest, prefixed, scoped, dual-key rotation supported, `last_used_at` tracked.
- [ ] JWT validation complete (iss/aud/exp/alg allowlist); access tokens short-lived; revocation story exists.
- [ ] Object-level (BOLA) and function-level authz on every handler, deny-by-default middleware/policy — sample 5 endpoints incl. one obscure one.
- [ ] Internal services mutually authenticated (mTLS/workload identity); no network-position trust.
- [ ] Rate limiting keyed per principal, sliding-window/GCRA in shared store; 429 + `Retry-After` + RateLimit headers; limits visible on successes; unauth endpoints (login/token) IP-limited.
- [ ] Quotas separate from rate limits with distinct errors; expensive ops cost-weighted; per-principal concurrency caps.
- [ ] Body/header/URL/multipart size limits explicit per route (413); JSON depth/key caps; decompressed-size caps; uploads via pre-signed URLs.
- [ ] Timeout hierarchy verified outer>inner end-to-end (client→LB→app→downstream→DB statement); long work is async 202, not long synchronous holds.
- [ ] Retries idempotent-only, budget-bounded, jittered, circuit-broken.
- [ ] CORS: explicit origin allowlist; no origin reflection with credentials; no `*`+credentials; cookie APIs have CSRF defenses; preflight cache set.
- [ ] Append-only audit trail covering authn, authz denials, sensitive writes, admin actions, exports — with actor/tenant/object/outcome/trace ID.
- [ ] No secrets/tokens/PII in logs (verified by test, not policy); log output encoded against CRLF/log injection.
- [ ] Trace/request ID on every response and propagated downstream (W3C Trace Context).
- [ ] Tenant derived from credential only; structural scoping (repo layer or RLS) — not per-handler WHERE clauses; cross-tenant probes return consistent 404.
- [ ] Tenant isolation covers search, exports, counts, webhooks, realtime channels, idempotency keys, and cache keys.
- [ ] Automated cross-tenant access test suite exists and runs in CI.
- [ ] Services reject direct (gateway-bypassing) traffic; identity headers from the gateway are mTLS-bound and stripped from external requests at the edge.
- [ ] TLS 1.2+ everywhere, HSTS on API hosts; no plaintext listeners beyond loopback health checks.
- [ ] **Framing (§3a) — HIGH**: every hop strict; a relaxed HTTP parser anywhere on the path is a finding:
      `grep -rnE 'insecureHTTPParser[[:space:]]*:[[:space:]]*true|--insecure-http-parser|accept-(invalid-http|unsafe-violations-in-http)-request' .`
      — then walk the hop chain for an h2-front/h1-back downgrade and for hand-set `Content-Length` on streamed bodies.
- [ ] **API keys and Basic auth (§1) — MEDIUM**: no high-value resource guarded by an API key alone; abuse leads to revocation; Basic auth absent or TLS-only. Locator:
      `grep -rnE 'Authorization:[[:space:]]*Basic|WWW-Authenticate:[[:space:]]*Basic|HTTPBasicAuth|BasicAuthentication' .`
- [ ] **Message signing (§1) — HIGH on money or authority paths**: high-value, B2B and agent requests signed with a receiver-enforced minimum covered set including `content-digest`, digest recomputed over received bytes. Verifiers that never mention the digest:
      `grep -rliE 'signature-input' . | while IFS= read -r f; do grep -qi 'content-digest' "$f" || echo "$f"; done`
- [ ] **Login-surface 429s (§2) — MEDIUM**: generic body, no bucket name, attempt count or precise reset on credential/signup/OTP routes:
      `grep -rniE 'remaining[_ -]?attempts|attempts[_ -]?(left|remaining)' .`
- [ ] **Throttle overrides (§2) — HIGH on auth or expensive routes**: a global default is configured and no view disables it:
      `grep -rnE 'throttle_classes[[:space:]]*=[[:space:]]*(\[[[:space:]]*\]|\([[:space:]]*\)|None)|@throttle_classes\([[:space:]]*(\[[[:space:]]*\]|\([[:space:]]*\))[[:space:]]*\)' --include='*.py' .`
- [ ] Rate-limit keys include session and ASN/geo where relevant; login uses separate per-account and per-source buckets; anomalous clients auto-downgraded; new identities cost a verified operator. Bot defence layered (edge, app, business), with minimum-realistic-interval checks on business flows and reputation plus delayed publishing for UGC — MEDIUM.
- [ ] Audit events (§6) include token issuance with scopes, logout (hashed session ref), file deletion, imports, system-object create/delete, UGC processing, sequence and fraud signals, and WebSocket open/close, decisions and violations; high-risk operations log intent before and outcome after — MEDIUM.
- [ ] Log records carry app ID, hostname, protocol/port, method/URI, region and client port; nodes time-synced with drift alerting; anti-bot decisions logged with their signals and hashed fingerprints — MEDIUM.
- [ ] **Shared audit store (§7) — HIGH**: reads tenant-filtered, cross-tenant reads need a platform-auditor permission no tenant admin has, and writes cannot target another tenant's stream. Audit-table reads with no tenant term on the line (a lead to read):
      `grep -rniE 'from[[:space:]]+audit_?(log|events?|trail)' . | grep -viE 'tenant'`
- [ ] Gateway does coarse route/scope authz as a first layer only; service-level and object checks still present (§8) — MEDIUM.
- [ ] **Plaintext on API hosts (§8) — MEDIUM**: `curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' http://<api-host>/<path>` returns an error or refuses the connection; a 301/302/307/308 to `https://` is the finding.
- [ ] JSON responses a browser can reach carry `default-src 'none'; frame-ancestors 'none'`, empty-allowlist `Permissions-Policy`, `Referrer-Policy: no-referrer`, `nosniff` (check with `curl -sI`) — LOW.
- [ ] **Inventory (§9) — MEDIUM**: every host has version, environment and audience; routed vs spec vs client-bundle endpoint lists reconciled; no stray published server URL:
      `grep -rnE 'url"?[[:space:]]*:[[:space:]]*"?https?://[^"[:space:]]*(staging|localhost|127\.0\.0\.1|internal|\.local[:/"]|10\.[0-9]+\.|192\.168\.)' --include='*.yaml' --include='*.yml' --include='*.json' .`
- [ ] **Forwarded client certificate (§8) — HIGH**: every read of a forwarded cert header is honoured only from the terminating proxy, stripped at the edge, and not the sole basis of an identity-authorised write. Locator:
      `grep -rniE 'forwarded-client-cert|ssl[-_]client[-_](cert|escaped)|client[-_]cert(ificate)?[-_]?header|x-client-cert' .`