# 01 — Methodology: Measure, Budget, Decide, Protect

Performance work without measurement is guessing; measurement without a budget
is trivia. This file defines how to measure, what numbers mean, and how to
decide what is worth fixing.

## 1. Measure first — with the right instrument

Pick the instrument for the question. Using a CPU profiler to debug an I/O-bound
service tells you nothing.

| Question | Instrument |
|---|---|
| Where does CPU time go? | Sampling CPU profiler → flamegraph |
| Why is wall time >> CPU time? | Off-CPU / wall-clock profiler, async-aware tracing |
| Where does latency go across services? | Distributed tracing (OpenTelemetry spans) |
| Is it the kernel/syscalls? | `strace -c`, eBPF (bpftrace), `perf trace` |
| Memory growth? | Heap profiler, allocation profiler, heap snapshots diff |
| Is this function faster now? | Micro-benchmark harness with statistics |
| Is the system faster? | Load test + latency distribution comparison |

Per-runtime profilers (sampling, production-safe unless noted):

- **Linux native / mixed**: `perf record -g -F 99`, eBPF tools, flamegraphs via
  `perf script | flamegraph.pl` or `samply`.
- **Go**: built-in `pprof` (CPU, heap, mutex, block, goroutine), continuous
  profiling via Pyroscope/Parca. Always enable `net/http/pprof` in services.
- **JVM**: async-profiler (CPU + alloc + locks, no safepoint bias), JFR
  (always-on flight recorder, < 2% overhead).
- **Node.js**: `node --prof`, `--cpu-prof`, Chrome DevTools, `0x` for
  flamegraphs; `perf_hooks.monitorEventLoopDelay` for event-loop lag
  (rules/04 §9). Clinic.js is unmaintained (per its own README) — flag it
  where found; don't adopt.
- **Python**: `py-spy` (attach to live process, no code change), `cProfile`
  (deterministic, high overhead — dev only), `memray` for allocations.
- **Rust/C++**: `perf` + flamegraph, `heaptrack`/`valgrind --tool=massif` (dev).
- **Browser**: Chrome DevTools Performance panel, Lighthouse (lab), RUM (field).

**Rule: profile in production or production-like conditions.** Dev machines
have empty caches, tiny datasets, no contention, and different CPUs. Sampling
profilers at 49–99 Hz are safe in production; prefer continuous profiling so
the data already exists when an incident starts.

## 2. Reading flamegraphs

- **Width = time** (samples). The x-axis is alphabetical, NOT chronological.
- Look for **wide plateaus**: a single wide frame is your hottest code.
- Look for **wide-but-thin towers repeated** under many parents: a hot utility
  (serialization, logging, regex) called from everywhere — fix once, win
  everywhere.
- **CPU flamegraph flat/idle but requests slow?** The time is off-CPU: I/O,
  locks, GC, scheduler. Switch to off-CPU analysis or tracing.
- Inverted (icicle) view answers "which leaf functions burn the most total CPU".

## 3. Benchmarks that don't lie

Micro-benchmarks are adversarial: the compiler, CPU, and OS all conspire to
give you fiction.

```text
BAD                                  GOOD
start = now()                        Use a harness: JMH (JVM), criterion
f()                                  (Rust), go test -bench + benchstat,
print(now() - start)   # one run,    pytest-benchmark, mitata/tinybench (JS).
                       # cold cache, They handle warmup, multiple samples,
                       # no variance # outlier rejection, and statistics.
```

Non-negotiables for any benchmark:

1. **Warm up** until JIT/branch predictors/caches stabilize (harnesses do this).
2. **Many samples, report variance.** A result is `median ± MAD` or a
   confidence interval, never a single number. Two results differ only if the
   intervals don't overlap (Go: `benchstat`, p < 0.05).
3. **Prevent dead-code elimination**: consume results (blackhole/`black_box`).
4. **Realistic data**: production-shaped sizes and distributions. Sorting
   already-sorted arrays or hashing tiny strings benchmarks nothing.
5. **Pin the environment**: fixed CPU governor (`performance`), no turbo
   variance for comparisons, no laptop on battery, no noisy CI neighbors —
   or use dedicated runners / ratio-based comparisons.
6. **Benchmark the distribution, not the mean** of latency-sensitive code.
7. **Avoid coordinated omission** in load tests: closed-loop testers that wait
   for each response before sending the next silently pause during slow
   periods, deleting the worst samples from your data. Use open-loop /
   constant-arrival-rate load (wrk2, vegeta, k6 `constant-arrival-rate`) when
   measuring latency under a target throughput.

**A production metric you cannot re-run has no "run it 30 times".** §3's remedy
assumes a repeatable benchmark. For a live series the equivalent is sampling more
*offsets*, never a `now` vs `offset 24h` pair — `sota-observability` rules/02 §4a
has the measured case where such a pair showed 22x on a series that had not moved.

## 4. Percentiles, not averages

Averages are arithmetic fiction for latency. Latency distributions are
long-tailed; the mean sits between p50 and p99 and describes no real request.

- **p50**: typical experience. **p95/p99**: the experience of your heaviest
  users (who are often your biggest customers — bigger carts, more data).
- **Tail amplification**: if one page fans out to 50 backend calls, the page
  hits a backend's p99 on `1 - 0.99⁵⁰ ≈ 39%` of loads. Per-service p99 IS the
  user's median when fan-out is high. Budget backends at p999 when fan-out > 10.
- p99/p50 ratio > ~10× signals queueing, GC pauses, lock contention, or cache
  misses — not uniformly slow code.
- Never average percentiles across hosts; aggregate histograms (HDRHistogram,
  Prometheus native histograms / t-digest), then compute percentiles.

## 5. Orders of magnitude — internalize this table

Approximate 2020s hardware; exact values vary, ratios don't.

| Operation | Latency |
|---|---|
| L1 cache hit | ~1 ns |
| L2 cache hit | ~4 ns |
| L3 cache hit | ~10–40 ns |
| Main memory (DRAM) | ~60–100 ns |
| Mutex lock/unlock, uncontended | ~20 ns |
| Syscall (getpid, round trip) | ~100–300 ns |
| NVMe SSD random read | ~20–100 µs |
| Same-DC network round trip | ~100–500 µs |
| Memory read of 1 MB sequential | ~10–50 µs |
| Disk read of 1 MB sequential (NVMe) | ~50–200 µs |
| Cross-AZ round trip | ~1–2 ms |
| Same-region DB query (indexed, warm) | ~0.5–2 ms |
| HDD seek | ~5–10 ms |
| Cross-continent round trip (US↔EU) | ~70–90 ms |
| TLS 1.3 full handshake (cross-continent) | ~1 RTT + crypto ≈ 80–100 ms |

Consequences:

- One avoidable network round trip (~0.5 ms in-DC) costs the same as ~5,000
  DRAM accesses or ~500k L1 hits. **Round trips dominate; count them first.**
- RAM is the new disk: a cache-missing pointer chase (100 ns) is 100× an L1
  hit. Data layout (rules/03) matters for hot loops.
- Anything touching cross-region links is 100,000× slower than memory — cache
  it, move it, or batch it.

## 6. USE and RED

**USE** (Brendan Gregg) — for every hardware/software *resource*:
- **U**tilization: % busy (CPU %, disk busy %, pool in-use/size).
- **S**aturation: queued work (run-queue length, pool wait time, queue depth).
- **E**rrors: error counts (TCP retransmits, pool timeouts, OOM kills).

Saturation, not utilization, predicts latency: 80% CPU with an empty run queue
is fine; 60% CPU with a growing run queue is an incident. Check USE on: CPU,
memory, network, disk, connection pools, thread pools, worker queues, locks.

**RED** — for every *service/endpoint*:
- **R**ate (req/s), **E**rrors (failed/s), **D**uration (latency histogram).

Audit rule: a service without RED metrics per endpoint and USE on its pools is
unauditable at runtime — flag that as a finding itself (Medium).

## 7. Latency budgets

Work backwards from the user:

1. Pick the user-facing SLO: e.g. "search responds in ≤ 300 ms p99".
2. Subtract fixed costs you don't control: client RTT (~50 ms), TLS (resumed,
   ~0), CDN/proxy hops (~5 ms). Remainder = server budget (~245 ms).
3. Decompose across the critical path: auth 10 ms + query 100 ms + ranking
   80 ms + serialization 10 ms = 200 ms, leaving 45 ms slack (keep ≥ 20% slack
   for variance).
4. Assign each component's budget to its owning team/module; enforce in CI
   and alerting per component, not just end-to-end.

A new feature that adds a sequential dependency must fit the remaining slack
or buy budget by optimizing something else. "We'll just add one more call" is
how 300 ms endpoints become 900 ms over two years.

## 8. Amdahl's law — what's worth optimizing

Speedup from optimizing a fraction *p* of total time by factor *s*:
`Speedup = 1 / ((1 − p) + p/s)`.

- Optimizing 10% of runtime **infinitely** yields at most 1.11×. Don't touch
  anything under ~20% of the profile unless it's a one-line fix.
- Corollary for parallelism: 5% serial fraction caps speedup at 20× regardless
  of core count. Find and shrink the serial section (locks, single-threaded
  stages) before adding cores.
- Inverse use: a component that is 60% of latency is where 2× effort yields
  1.43× end-to-end — start there. The flamegraph tells you *p*.

## 9. Premature optimization vs known pathology

The Knuth quote has a second half: "...yet we should not pass up our
opportunities in that critical 3%". Operationalize it:

**Fix on sight, no profiler needed (known pathologies):**
- O(n²)+ on input that can grow (user data, DB rows, list endpoints).
- N+1 queries / RPC-in-a-loop.
- Unbounded memory: caches without eviction, accumulating listeners, reading
  unbounded result sets into memory.
- Blocking calls on async event loops.
- Per-request creation of poolable resources (connections, clients, regexes).
- Missing timeouts/limits.
- Sequential awaits on independent I/O.

These are correctness-adjacent: they work in dev and fail at scale.

**Profile first (speculative optimization):**
- Rewriting idiomatic code into "fast" contorted code.
- Caching something not yet shown to be hot or expensive.
- Micro-tuning (manual loop unrolling, bit tricks, custom allocators).
- Adding concurrency/complexity for an unmeasured win.

Decision test: *"Does this code's cost grow with production scale in a way dev
testing won't reveal?"* Yes → pathology, fix now. No → demand a profile.

One more invariant: optimizations must not erode security — identity-keyed
caching, constant-time comparisons, and validation/size limits are not
overhead to shave (rules/05 §9, sota-code-security).

## 9a. Duration as a correctness signal, not just a cost one

Profiling asks "why is this slow?". Turn it around once per pipeline: **is any
stage suspiciously *fast*?** A step that reports "nothing to do" far quicker than
its claimed work allows did not do the work — a scan that returns 0 findings in
2 s over 40k files, a migration that returns instantly, a backup that finishes in
seconds. It is the cheapest diagnostic available and needs no code reading.

Two forms worth timing deliberately:

- **Duration vs claimed work** — record the wall time *and* the input size, then
  compare against the order of magnitude the work implies.
- **Duration constant across scales** — if a 100-item and a 100k-item input take
  the same time, size is not reaching the work. Same test as §"cross-scale delta".

A stage that got dramatically faster while reporting the same result is a
regression signal, not a win, until you can say what work was removed. Full class
and the evidence bar: `sota-code-security` rules/11 §2.1.

**And the inverse: a duration that is suspiciously *slow* indicts the measurement before
it indicts the subject.** "Suspiciously fast" impugns the work; "suspiciously slow" usually
impugns your harness — and it is the more dangerous direction, because *"the suite got 6x
slower"* is a far more exciting finding than *"I measured it wrong"*, which is exactly why
it gets written down first.

Before recording a slowdown against a recorded baseline, establish that the two runs are
**comparable**: scheduling priority, machine load, and whether the run was backgrounded at
all. **A job launched into the background may be niced by whatever launched it** — measured
on one agent harness, backgrounded jobs ran at `nice 5` while the foreground shell in the
same invocation was `nice 0`, under a load average of 5.9–8.6. An unprivileged user cannot
renice back down, so the measurement cannot be rescued in place; it has to be re-run in the
foreground.

```sh
ps -o pid=,nice=,stat= -p "$PID"     # SN / RN in the state column is the tell
```

Field-reported, and **corrected by the reporter when the run finished — the correction is the
more useful half**. A test lane sitting at 2% after twelve minutes was extrapolated to a **6x**
regression against a ~38-minute baseline, and that false claim was written to a durable
project-memory file before anyone checked the process. The lane actually finished in 3190s =
53m10s: **1.41x, not 6x.** Two errors were stacked, and the bigger one was not the nice
penalty:

1. **Arithmetic on a progress percentage is not a measurement.** A test runner's early
   progress is dominated by collection and front-loaded heavy cases, so it is **not linear**
   and multiplying it out means nothing. This produced the 6x.
2. The run *was* also genuinely niced — a real effect worth the remaining ~41%.

**So check the shape of your instrument before you extrapolate from it**: a percentage that
moves non-uniformly is a progress *indicator*, not a clock, and the first 2% of a suite is the
least representative slice of it. The baseline had been measured in the foreground.
**A wall-clock number from a backgrounded run is not comparable to one from a foreground
run** — and note the asymmetry with §3 item 5 ("pin the environment"): that rule is framed
for a deliberate benchmark with a harness, and this case is someone running a test suite
and glancing at the clock, which is where it does not think to apply.

## 10. Performance regression testing in CI

Performance regressions ship silently; functional tests pass at any speed.

1. **Micro-benchmarks in CI** for hot library code. Compare against the base
   branch with statistical tooling (`benchstat`, JMH + jmh-compare,
   criterion's built-in comparisons). Gate on regressions beyond noise
   (e.g. > 10% with p < 0.05), don't gate on raw thresholds that rot.
2. **Macro load tests** (k6, Locust, Gatling, vegeta) on a fixed-size staging
   environment, nightly or per-release: assert p95/p99 and throughput against
   the budget, with the same dataset every run.
3. **Counting tests beat timing tests in noisy CI.** Assert *invariants* that
   don't depend on machine speed: number of DB queries per request (tools:
   `assertNumQueries` in Django, n+1 detectors like Prosopite/Bullet in Rails,
   query counters in tests), allocation counts per op (Go `testing.AllocsPerOp`,
   JMH GC profiler), bytes over the wire, bundle size (size-limit). These are
   deterministic and catch the most common regressions (a new N+1).
4. **Frontend budgets in CI**: Lighthouse CI with budgets.json (LCP, TBT,
   bundle bytes); fail PRs that exceed them.
5. **Continuous profiling in prod** + alerting on RED p99 per endpoint catches
   what CI misses; keep before/after flamegraphs for every major release.

CI timing noise mitigation: dedicated runners, multiple iterations with
median-of-runs, compare ratios vs base commit on the same machine in the same
job, never compare absolute times across runner generations.

## Audit checklist

- [ ] **Any recorded slowdown compared like-for-like before it was believed?** (§9a) A
      duration that is suspiciously *slow* indicts the measurement first — scheduling
      priority, machine load, foreground vs backgrounded (`ps -o nice=,stat=`; a
      backgrounded job may carry a nice penalty you cannot undo as a normal user). A
      regression written down from a non-comparable run is a false finding about the
      subject, and it reads as a much better finding than the truth.
- [ ] Is there any profiling/tracing data, or is all perf discussion folklore?
      No data on a "slow" system → first finding: add RED metrics + profiler.
- [ ] Are SLOs/budgets defined in percentiles? Any dashboards showing averages
      only → flag (Medium): averages hide the tail.
- [ ] Compute p99/p50 per hot endpoint; ratio > 10× → investigate queueing,
      GC, locks, cache misses.
- [ ] High fan-out call graphs: is the per-dependency percentile budget set
      accordingly (p999 for fan-out > 10)?
- [ ] Do benchmarks exist? Do they use a statistical harness, warmup,
      realistic data, and report variance? Single-run timing → flag.
- [ ] Does CI gate on any perf signal (query counts, alloc counts, bundle
      size, benchmark deltas)? None → flag (Medium): regressions ship blind.
- [ ] For each proposed/past optimization: what fraction of total time was it?
      (< 20% of profile and non-trivial → likely wasted effort — Amdahl.)
- [ ] Are USE metrics available for pools/queues (saturation especially)?
      Pool wait time unmeasured → flag.
- [ ] Were any "optimizations" added without before/after measurements?
      Treat them as suspect complexity; consider recommending removal.
- [ ] Count network round trips on the critical user journey; verify each is
      necessary, parallelized where independent, and inside the budget.

# 02 — Algorithmic & Data-Structure Performance

Complexity bugs are the only perf bugs that get *worse* on their own: traffic
doubles, data doubles, and O(n²) quadruples. They hide behind innocent-looking
one-liners. This file catalogs the patterns to write and the signatures to hunt.

## 1. N+1 anything

The N+1 pattern is one operation to fetch a list, then one operation *per
element*. It applies to DB queries, HTTP calls, cache gets, file reads, and
RPC — any boundary with per-call overhead.

```python
# BAD — 1 + N queries; 200 orders ≈ 201 round trips ≈ 200+ ms
orders = db.query("SELECT * FROM orders WHERE user_id = %s", uid)
for o in orders:
    o.items = db.query("SELECT * FROM order_items WHERE order_id = %s", o.id)

# GOOD — 2 queries total, regroup in memory
orders = db.query("SELECT id, ... FROM orders WHERE user_id = %s", uid)
items = db.query("SELECT ... FROM order_items WHERE order_id = ANY(%s)",
                 [o.id for o in orders])
by_order = group_by(items, key=lambda i: i.order_id)   # O(N) hash regroup
for o in orders: o.items = by_order.get(o.id, [])
```

Variants to recognize:
- **ORM lazy loading** in a loop (`order.items` triggering a query per access).
  Fix: eager loading (`select_related`/`prefetch_related`, `includes`,
  JPA fetch joins) or a dataloader.
- **HTTP N+1**: calling a microservice per item. Fix: batch endpoint
  (`GET /users?ids=1,2,3`), or GraphQL dataloader pattern (collect IDs within
  one tick, issue one batched fetch).
- **Cache N+1**: `cache.get(key)` per item over the network. Fix: `MGET` /
  pipelined multi-get.
- **N+1 writes**: INSERT per row. Fix: multi-row INSERT / `COPY` / bulk APIs —
  typically 10–100× faster.

Detection by reading: any loop body containing `await`, `query`, `fetch`,
`get`/`post`, `client.`, or an ORM relationship access. Detection by measuring:
queries-per-request metric; test assertion on query count.

## 2. Accidental quadratic

O(n²) created by composing two O(n) things. The code reads as linear.

**String building in a loop** (immutable strings copy on every concat):

```java
// BAD — O(n²): each += copies the whole accumulated string
String csv = "";
for (Row r : rows) csv += r.toLine() + "\n";

// GOOD — O(n)
StringBuilder sb = new StringBuilder(rows.size() * 64); // pre-size
for (Row r : rows) sb.append(r.toLine()).append('\n');
```

Same trap: Python `s += chunk` in a loop (use `''.join(parts)` or
`io.StringIO`), JS heavy `+=` in hot loops (use `parts.push(...); parts.join('')`),
Go `s += x` (use `strings.Builder`), repeated list concatenation
(`list = list + other` vs `extend`), `array.unshift`/`list.insert(0, x)` in a
loop (O(n) per op → O(n²); use append + reverse, or a deque).

**Linear scan inside a loop** — `O(n·m)` that should be `O(n+m)`:

```javascript
// BAD — for 10k users × 10k allowed ids = 10⁸ comparisons (~seconds)
const active = users.filter(u => allowedIds.includes(u.id));

// GOOD — build the hash set once: ~20k operations (~ms)
const allowed = new Set(allowedIds);
const active = users.filter(u => allowed.has(u.id));
```

Signatures: `.includes(`, `.indexOf(`, `in some_list`, `.find(` / `.filter(`,
`array_search`, `list.count(x)`, `.contains(` on a List — *inside another
loop or array method*. The fix is almost always: hoist a `Set`/`Map`/dict
built once, O(1) lookups after.

**Other quadratic generators:**
- Sorting or deduplicating inside a loop (`sort()` per iteration).
- `dict(list)` rebuilt per call instead of cached.
- Nested ORM/collection traversal: `for a in A: for b in a.related_b_filtered_in_python`.
- Recomputing an aggregate per element (`sum(items)` inside `for item in items`)
  — use a running total or prefix sums.
- Regex with catastrophic backtracking (`(a+)+$`) — exponential, not even
  quadratic; user-controlled input + nested quantifiers = ReDoS finding (High).
- Deep-copying or JSON-serializing a growing accumulator every iteration.

**Threshold guidance:** n ≤ 100 with no growth path: leave linear scans alone —
a linear scan over 32 elements often beats a hash map (cache locality, no
hashing cost). n is user-data-sized or grows with the business: fix on sight.

## 3. Choose structures by access pattern

| Need | Structure | Cost |
|---|---|---|
| Membership / lookup by key | Hash set/map | O(1) avg; no ordering |
| Lookup + ordered iteration / range queries | B-tree / skip list / sorted structure (`TreeMap`, `BTreeMap`, `sorted containers`) | O(log n) |
| Min/max repeatedly | Heap | O(log n) push/pop; beats re-sorting |
| FIFO / both-ends ops | Deque / ring buffer | O(1); never `shift()` an array |
| Top-K of huge stream | Bounded heap of size K | O(n log K), constant memory |
| Append-heavy, index access | Dynamic array | amortized O(1) append |
| Many membership checks, some false positives OK | Bloom/cuckoo filter | O(1), ~10 bits/elem |
| Prefix search | Trie / sorted array + binary search | — |
| Counting distinct at scale | HyperLogLog | KBs for billions |

Hash vs tree decision: need range scans, ordered traversal, floor/ceiling, or
predictable worst-case (no rehash spikes, adversarial keys)? → tree (O(log n)).
Pure point lookups → hash. At n < 10³ the difference rarely matters; at n > 10⁶
or in a hot loop, it defines the design. Also note constant factors: hash maps
cost a hash + probe (~20–50 ns); arrays with linear scan win below ~50 elements.

Hidden costs to know:
- Rehash/resize spikes: pre-size hash maps and arrays when the size is known
  (`make(map, n)`, `new ArrayList<>(n)`, `dict` over-allocation is automatic).
- Hashing long strings is O(len): hashing 1 KB keys in a hot loop is the work.
  Intern or pre-hash hot keys.
- `LinkedList` is almost never the answer: O(n) cache-hostile traversal loses
  to array shifting in practice for all but huge mid-list insert workloads.

## 4. Hidden complexity in innocent calls

Library calls have complexity classes too; they just don't print them.

| Call | Hidden cost |
|---|---|
| `list.remove(x)` / `array.splice(i,1)` | O(n) scan + O(n) shift |
| `len(set(a) & set(b))` per pair in a loop | rebuilds sets every iteration |
| `sorted(x)[0]` / `.sort()` then take first | O(n log n) for an O(n) `min` |
| `Object.keys(obj).length` in a loop (JS) | allocates the key array each time |
| `str.split()` / regex compile inside loop | recompiled/re-allocated per iteration — hoist `re.compile`/`Pattern.compile` |
| `LinkedList.get(i)` in an indexed loop (JVM) | O(n) per get → O(n²) loop |
| `in` on a Python list / `.contains` on List | O(n) — use set/dict |
| Spread-accumulate `acc = [...acc, x]` / `{...acc}` in reduce (JS) | copies accumulator per element → O(n²); mutate or push |
| `COUNT(*)` per item, `EXISTS` in app loop | a query per element — batch with `GROUP BY`/`ANY` |

**Top-K / partial results**: need the 10 largest of 10M? `heapq.nlargest`,
`partial_sort`, `select_nth` — O(n log K) instead of full O(n log n) sort.
Need "is there at least one match?" — short-circuit (`any`, `LIMIT 1`),
don't count or materialize everything.

**Precompute and reuse across iterations**: anything loop-invariant (compiled
regex, parsed config, dictionary built from a constant list, formatted
prefix) hoists out. Memoize pure expensive functions with *bounded* caches
(see rules/05 for eviction discipline).

## 5. Batching

Per-operation overhead (syscall, RTT, transaction, lock acquisition) amortizes
over batch size. Throughput scales until the batch's marginal cost dominates.

- **Batch the boundary, not the CPU**: batching matters where each op carries
  fixed overhead — network, disk, locks, GPU kernel launches.
- Typical wins: multi-row INSERT 10–100× vs row-at-a-time; Redis pipeline of
  100 commands ≈ 1 RTT instead of 100; `writev` over 100 `write` calls.
- **Bound batches** by count AND bytes AND time (e.g. "≤ 500 items, ≤ 1 MB, or
  every 50 ms, whichever first"). Unbounded batching trades latency and memory
  for throughput and breaks tail latency.
- Amortize, don't serialize: batch building must not add a full batch-interval
  to p99 of latency-sensitive paths — use small time windows (1–10 ms) or
  opportunistic batching (take whatever is queued now, send immediately).

## 6. Streaming vs materializing

Materializing = build the entire result in memory, then process/send.
Streaming = process elements as they arrive, O(1)–O(batch) memory.

```python
# BAD — loads every row into memory; 10M rows × 1 KB = 10 GB, OOM
rows = cursor.fetchall()
return json.dumps([transform(r) for r in rows])

# GOOD — constant memory, first byte leaves immediately
def generate():
    yield '['
    for i, r in enumerate(cursor):          # server-side cursor / chunked fetch
        yield (',' if i else '') + json.dumps(transform(r))
    yield ']'
return StreamingResponse(generate())
```

Stream when: result size is unbounded or user-controlled; the consumer can
start before the producer finishes (time-to-first-byte matters); data passes
through (file upload → object storage; DB → CSV export). Use: generators,
async iterators, Node streams with `pipeline` (backpressure handled), Go
`io.Reader` chains, SAX/streaming JSON parsers for huge documents.

Materialize when: you need multiple passes, sorting, or random access; data is
known-small; retry semantics require buffering anyway.

**Backpressure is part of the design**: a fast producer + slow consumer +
unbounded queue = memory leak. Use bounded channels/queues and blocking or
shedding when full.

## 7. Move work out of the hot path

When an operation is both necessary and expensive, relocate it in time:

- **Precompute on write** (read-heavy data): maintain the aggregate/denormalized
  view as writes happen (counter columns, materialized views, search indexes)
  instead of computing per read. One O(1) update per write beats O(n) per read
  when reads ≫ writes.
- **Defer off the request path**: anything the user doesn't need in the
  response (emails, analytics, thumbnail generation, fan-out) goes to a queue.
  Request latency = critical path only.
- **Incremental over recompute**: update running totals/deltas rather than
  rescanning; cache invalidation by dependency rather than recompute-all.
- The inverse also holds: don't precompute combinatorial spaces "just in case"
  (precompute cost × cardinality must beat lazy compute × actual hit count).

## 8. Database quick pointers

Detailed rules live in the **sota-databases** skill; in a perf audit, flag
these on sight:

- **N+1 queries** — §1 above. The #1 real-world perf bug.
- **Missing indexes**: any `WHERE`/`JOIN`/`ORDER BY` column on a large table
  without a supporting index → full scan. Verify with `EXPLAIN (ANALYZE)`;
  look for `Seq Scan` on big tables, `rows` estimates in the millions.
- **`SELECT *`**: drags unneeded (possibly TOASTed/large) columns over the
  wire, defeats covering indexes, breaks when schema grows. Select named
  columns on hot paths.
- **Chatty transactions**: many small queries + app think-time inside one
  transaction → locks held for the round-trip sum, pool exhaustion under load.
  Keep transactions short, no external calls inside them, batch the reads.
- **Unbounded queries**: no `LIMIT` on list endpoints; `OFFSET` pagination on
  deep pages (O(offset) per page) → use keyset/cursor pagination.
- **Pool sizing**: connections per instance × instances > DB max_connections
  is an outage; pool too small is artificial saturation — measure pool wait.

## Audit checklist

- [ ] Grep loop bodies for I/O: `await`, `fetch`, `query`, `exec`, `.get(`,
      `client.`, ORM relation access → N+1 candidates (High on hot paths).
- [ ] Grep for `+=` on strings, `concat`, `unshift`, `insert(0,` inside
      loops → accidental quadratic.
- [ ] Grep for `.includes(`, `.indexOf(`, ` in [`/`in list`, `.find(`,
      `.contains(` inside loops/`.filter` → O(n·m); recommend Set/Map hoist.
- [ ] Any sort, dedupe, aggregate, deep copy, or serialization recomputed
      per-iteration?
- [ ] Regexes on user input with nested quantifiers/backreferences →
      catastrophic backtracking risk.
- [ ] Collections pre-sized where final size is known? Rehash/regrow in hot
      loops?
- [ ] `fetchall()` / `findAll()` / `.ToList()` / reading whole files where the
      result is unbounded → demand streaming or LIMIT.
- [ ] Writes performed row-at-a-time where bulk APIs exist?
- [ ] Batch jobs bounded by count, bytes, and time? Queues bounded with
      backpressure?
- [ ] DB: EXPLAIN available for hot queries? `SELECT *`, missing LIMIT,
      OFFSET pagination, long transactions with app logic inside?
- [ ] For every flagged loop: what is realistic production n? Document the
      math (n × per-op cost) in the finding.

# 03 — Memory: Allocation, Layout, Leaks, GC

Memory problems present as CPU problems (GC burn, cache misses), latency
problems (pauses, page faults), and reliability problems (OOM). This file
covers allocation discipline, data layout, leak patterns per runtime, and GC
tuning principles.

## 1. Allocation pressure

Every allocation costs: the allocation itself (~10–50 ns fast path), future GC
work proportional to allocation *rate*, and cache pollution. In managed
runtimes, **allocation rate is the GC tax base** — halving allocations/sec
roughly halves GC CPU.

Hot-path rules (a "hot path" runs ≥ ~10⁴ times/sec or inside a per-request loop):

- **Don't allocate per iteration what can be allocated per batch/request.**

```go
// BAD — allocates a new buffer per call; 50k req/s × 64 KB = 3 GB/s churn
func handle(w io.Writer, r *Req) {
    buf := make([]byte, 64*1024)
    process(buf, r, w)
}

// GOOD — reuse via sync.Pool (Go), ThreadLocal/ringbuffer (JVM), or
// preallocated per-worker buffers
var bufPool = sync.Pool{New: func() any { return make([]byte, 64*1024) }}
func handle(w io.Writer, r *Req) {
    buf := bufPool.Get().([]byte)
    defer bufPool.Put(buf)
    process(buf, r, w)
}
```

- **Pre-size growable containers** when the size is known or estimable:
  repeated regrowth of a vector copies O(n log n) bytes total and fragments.
- **Avoid hidden allocators**: boxing (Java `Integer` in hot loops, Go
  `interface{}` conversions), closure capture creating heap escapes, string
  formatting/concat, iterator/lambda allocation per call in some runtimes,
  substring/slice APIs that copy, `JSON.parse`/reflect-based serialization of
  large objects per message.
- **Escape analysis is your friend** (Go `-gcflags=-m`, JVM does it silently):
  keep values stack-allocatable — don't return pointers to locals needlessly,
  don't store short-lived values into longer-lived structures.
- Measure, don't guess: Go `testing.AllocsPerOp` / pprof alloc profile; JVM
  async-profiler `-e alloc`; .NET `dotnet-counters` alloc rate; Python
  `memray`/`tracemalloc`. Allocation profiles are usually more actionable
  than CPU profiles in managed services.

## 2. Object pooling — when it helps and when it hurts

Pooling pays when the object is **expensive to create** (connections, TLS
sessions, big buffers, compiled regexes, ML sessions) or when allocation rate
is a measured GC bottleneck.

Pooling **hurts** when:
- Objects are cheap: modern allocators/GCs make small short-lived objects
  nearly free (bump-pointer alloc + die-young = generational hypothesis).
  Pooling them adds synchronization, retention, and bugs for nothing.
- Pooled objects hold stale state → correctness bugs (the classic "user A sees
  user B's data" is often a dirty pooled buffer). Reset on return, always.
- The pool is unbounded → it's a leak; or sized wrong → contention point.
- In GC'd runtimes, long-lived pools promote objects to old-gen, making major
  GCs scan more — pooling can *increase* GC cost if the objects are small.

Rules: bound the pool, reset state on return/acquire, measure before and
after, prefer per-worker (sharded) pools over one global locked pool.

## 3. Arena allocation (concept)

Arena/region allocation: allocate many objects from one contiguous block with
a bump pointer; free them **all at once** by resetting the arena. Fits
phase-structured work: per-request, per-frame, per-compilation-unit.

- Wins: allocation ≈ pointer increment (~1–2 ns); zero per-object free cost;
  perfect locality (objects allocated together sit together); no fragmentation.
- Native: explicit arenas (Rust `bumpalo`, C `talloc`/APR pools, jemalloc
  arenas). Managed analogs: reusing one large buffer + indices, .NET
  `ArrayPool`+spans, flatbuffers-style serialization into one slab, Go arena
  experiment (frozen — use pooled slabs instead).
- Constraint: nothing allocated in the arena may outlive it. Escaping pointers
  are use-after-free (native) or force copies (managed). Design the lifetime
  boundary first (request scope is the natural one).

## 4. Cache locality — layout is performance

DRAM access is ~100 ns; L1 is ~1 ns. The CPU fetches 64-byte cache lines and
prefetches sequential patterns. Hot-loop throughput is usually bounded by
memory layout, not instruction count.

- **Sequential beats random**: iterating a contiguous array can be 10–100×
  faster than chasing pointers (linked lists, object graphs) over the same
  elements. Prefer arrays/vectors of values over collections of heap pointers
  in hot loops.
- **Smaller is faster**: shrinking a hot struct from 80 to 48 bytes puts more
  elements per cache line; use compact field types, reorder fields to kill
  padding (largest first), use indices (u32) instead of 8-byte pointers.
- **Row-major vs column-major**: iterate 2D data in memory order; wrong order
  multiplies cache misses (classic `a[j][i]` vs `a[i][j]` — up to 10× on big
  matrices).
- **False sharing**: two threads writing different variables on the same
  64-byte line ping-pong the line between cores (~100× slowdown on counters).
  Pad/align per-thread hot data (`#[repr(align(64))]`, `@Contended`,
  cache-line padding in counter arrays).

### Struct-of-Arrays vs Array-of-Structs

```text
AoS: [ {x,y,z,vx,vy,vz,hp,name…}, … ]   — natural OO layout
SoA: { x:[…], y:[…], z:[…], hp:[…], … } — column layout
```

- Loop touches *few fields of many records* (analytics, physics, filtering by
  one column) → **SoA**: only needed bytes enter cache, SIMD vectorizes
  naturally. This is why columnar formats (Arrow, Parquet, DuckDB) win for
  scans.
- Loop touches *all fields of one record* (per-entity logic, OLTP row access)
  → **AoS**: one cache line per record.
- Hybrid (AoSoA) for SIMD kernels. In managed languages, SoA = parallel
  primitive arrays instead of object lists — also removes per-object headers
  (12–16 bytes/object on JVM) and pointer chasing.

## 5. Memory leak patterns per runtime

A leak in GC'd runtimes = unintended *reachability*. Hunt the references.

**JavaScript / Node / browser**
- Closures capturing large scopes: a small callback retaining a parsed 50 MB
  document because it references one field. Extract the field before closing.
- `addEventListener` / `on(...)` without removal — especially on long-lived
  emitters (sockets, window, global stores) from short-lived components.
  Symptom: `MaxListenersExceededWarning`; fix: `removeEventListener`,
  `AbortSignal`-based listeners, framework cleanup hooks.
- Timers: `setInterval` never cleared retains its closure forever.
- Module-level caches (`const cache = new Map()`) without eviction.
  Use bounded LRU (`lru-cache`) or `WeakMap` keyed by the owning object.
- Detached DOM nodes retained by JS references (browser).
- Tooling: heap snapshot diff in DevTools, `--inspect`, look at "Retainers".

**Python**
- Module-level dict/list accumulators; `functools.lru_cache` on methods
  (retains `self` for every instance — use `cached_property` or bounded
  per-instance caches); default mutable args accumulating.
- Reference cycles with `__del__` (delays collection), C-extension leaks.
- Large object retained by an exception traceback held in a variable.
- Tooling: `tracemalloc` snapshots diff, `memray`, `objgraph` for retainers.

**JVM**
- `static` collections/caches without eviction (the canonical Java leak).
- `ThreadLocal` not removed on thread-pool threads (threads live forever →
  values live forever).
- Listener/observer registration without deregistration; inner-class instances
  retaining outer `this`.
- ClassLoader leaks on redeploy (web containers).
- Unclosed resources (use try-with-resources): direct ByteBuffers and native
  handles leak off-heap.
- Tooling: heap dump + Eclipse MAT "dominator tree"; JFR allocation/leak views.

**Go**
- Goroutine leaks: a goroutine blocked forever on a channel nobody closes or a
  context never cancelled — each retains its whole stack and referenced heap.
  Audit every `go func` for a guaranteed exit path; `pprof/goroutine` count
  must plateau.
- Subslices retaining huge backing arrays (`small := big[:3]` keeps all of
  `big`) — copy when keeping a sliver. Same for substrings pre-1.21 patterns.
- `time.Ticker` not stopped; maps that only grow (maps never shrink — replace
  the map to reclaim).

**General (all runtimes)**: any cache/map/registry with `put` but no
`remove/TTL/LRU` is a leak by construction — flag without running anything.
Symptom signature in metrics: sawtooth baseline that ratchets upward after
each GC; old-gen/heap floor climbing across days.

## 6. Serialization and copies — the silent memory tax

Serialization sits on nearly every hot path and is routinely the top
allocator in service profiles.

```javascript
// BAD — three full materializations of a 20 MB payload per request:
// object graph → JSON string → Buffer
const data = await loadBigReport(id);          // 20 MB of objects
const json = JSON.stringify(data);             // +20 MB string
res.end(Buffer.from(json));                    // +20 MB buffer

// GOOD — stream rows; peak memory = one chunk
res.setHeader("content-type", "application/json");
await pipeline(reportRowStream(id), jsonArrayStringify(), res);
```

- Prefer encoders that write to the output stream (`json.NewEncoder(w)` in
  Go, Jackson streaming, `serde` to writer) over encode-to-string-then-write.
- Don't round-trip for deep copy (`JSON.parse(JSON.stringify(x))` →
  `structuredClone` or targeted copies).
- Parse selectively for huge documents: streaming/SAX parsers, or formats
  with lazy/zero-copy access (flatbuffers, Cap'n Proto, Arrow for columnar).
- Intermediate collection chains (`filter().map().map()` materializing each
  stage on big inputs) → fuse into one pass or use lazy iterators
  (generators, Rust iterators, Java streams are already lazy until collect).

## 7. GC tuning principles

Tune in this order — most GC problems are application problems:

1. **Reduce allocation rate first** (§1). No GC flag beats allocating less.
2. **Right-size the heap.** Too small → constant collection; absurdly large →
   long full collections and wasted RAM. Aim for live-set × 2–4 as a starting
   point. Set container limits and GC heap % consistently
   (`-XX:MaxRAMPercentage`, `GOMEMLIMIT`, `--max-old-space-size`) — a JVM/Go
   process that doesn't know its cgroup limit OOMs instead of collecting.
3. **Pick the collector for the goal**: throughput batch jobs → throughput
   collector (Parallel); latency-sensitive services → low-pause concurrent
   collectors (G1 default, ZGC/Shenandoah for < 1 ms pauses on big heaps;
   modern ZGC is generational). Go: one GC, tune `GOGC` (collection frequency
   vs heap growth) and `GOMEMLIMIT` (hard cap).
4. **Watch promotion, not just pauses**: short-lived objects surviving into
   old gen (because of pools, caches, or batch lifetimes) make major GCs
   expensive. Generational hypothesis: die young or live forever — avoid the
   middle.
5. **Measure GC like latency**: pause time distribution (p99), GC CPU %, and
   allocation rate. GC CPU > ~10% or pauses inside your latency budget →
   act. Enable GC logs/JFR in prod; they're nearly free.
6. Don't cargo-cult flags. Every GC flag pasted from a blog without a
   before/after measurement is a liability — audit finding if you see a wall
   of unexplained `-XX:` flags.

## Audit checklist

- [ ] Grep for caches/maps/registries with insertion but no eviction, TTL, or
      removal path → leak by construction (High if keyed by user/request data).
- [ ] Event listeners, subscriptions, timers: every registration paired with a
      cleanup on the owner's lifecycle?
- [ ] Go: every spawned goroutine has a guaranteed exit (context cancel,
      channel close)? Tickers stopped?
- [ ] JVM: `static` collections, `ThreadLocal` on pooled threads without
      `remove()`, unclosed resources?
- [ ] JS: `setInterval` cleared? Listeners on global/long-lived objects
      removed? Module-level Maps bounded?
- [ ] Python: `lru_cache` on methods? Module-level accumulators?
- [ ] Hot paths: per-iteration allocation of buffers/objects that could be
      pooled or hoisted? Containers pre-sized?
- [ ] Any pooling of cheap objects (adds complexity, no win) or pools without
      bounds/reset (bug factory)?
- [ ] Hot data structures: pointer-chasing collections where contiguous arrays
      would do? Hot loops touching few fields of fat structs (SoA candidate)?
- [ ] Shared mutable counters/flags written by multiple threads without
      cache-line padding (false sharing)?
- [ ] Heap/RSS trend over days: does the floor after GC ratchet upward?
- [ ] Runtime knows its memory limit (`GOMEMLIMIT`, `MaxRAMPercentage`,
      `--max-old-space-size` aligned with container limit)?
- [ ] GC metrics exported (pause p99, GC CPU %, alloc rate)? Unexplained GC
      flag soup in deploy configs?

# 04 — I/O & Network Performance

Everything crossing the kernel boundary or the wire costs 10³–10⁶× more than
memory work. The strategy is always the same: **fewer, bigger, reused,
concurrent** — fewer round trips, bigger batches, reused connections,
concurrent independent operations.

## 1. Syscalls: batch and buffer

A syscall costs ~100–300 ns plus icache/TLB pollution. Unbuffered I/O turns
one logical write into thousands of syscalls.

```python
# BAD — one write() syscall per line; 1M lines ≈ 1M syscalls ≈ seconds
for line in lines:
    os.write(fd, line.encode())

# GOOD — buffered: ~64 KB per syscall; 1M lines ≈ hundreds of syscalls
with open(path, "w", buffering=1 << 16) as f:
    f.writelines(lines)
```

- Always wrap raw fds/sockets in buffered writers (`bufio.Writer`,
  `BufferedOutputStream`, default Python buffering) — and **flush at
  boundaries** (message end, before fsync, before close). A missing flush is a
  correctness *and* latency bug (data sits in the buffer until it fills).
- Vectored I/O (`writev`/`readv`) sends multiple buffers in one syscall —
  header + body without concatenating.
- `fsync` is the expensive one (~ms on SSD): batch durability points (group
  commit), don't fsync per record unless the contract demands it.
- Reads: read in ≥ 64 KB chunks; `mmap` for large read-mostly files with
  random access; `posix_fadvise`/`readahead` for known sequential scans.
- io_uring (Linux) batches submission *and* completion — relevant for
  syscall-bound services at 10⁵+ IOPS; most apps get 90% of the win from
  plain buffering.

## 2. Zero-copy

Each unnecessary copy burns memory bandwidth and CPU. Classic file-to-socket
path copies 4× (disk→page cache→user→socket buffer→NIC); zero-copy paths skip
the user-space bounce:

- `sendfile()` / `splice()`: serve static files kernel-to-socket. Exposed as
  Go `io.Copy` (uses sendfile/splice when src/dst are *os.File/TCPConn), Java
  `FileChannel.transferTo`, Node `fs.createReadStream().pipe(res)` (still
  user-space but chunked), nginx `sendfile on`.
- Don't read a file into memory just to write it elsewhere; pipe/stream it.
- In-process: pass slices/views (`ByteBuffer.slice`, Go subslices, Rust
  `Bytes`) instead of copying; beware retention (rules/03 §5 Go subslice trap).
- Serialization is the hidden copy machine: JSON encode→string→buffer→socket
  can copy a payload 3–4×. Encoders that write directly to the output stream
  (`json.NewEncoder(w)`, streaming serializers) remove the intermediate
  strings.

## 3. Connection pooling and reuse

A new connection costs: TCP handshake (1 RTT) + TLS handshake (1 RTT on
TLS 1.3, 2 on 1.2) + slow start (small initial congestion window) + server-side
session setup (a Postgres connection fork costs ~ms and ~5–10 MB). Per-request
connections can triple latency and crush the backend.

```javascript
// BAD — new client (and pool) per invocation; handshakes every call,
// leaks sockets under load
async function getUser(id) {
  const client = new pg.Client(cfg); await client.connect();
  const r = await client.query("SELECT ...", [id]); await client.end();
  return r.rows[0];
}

// GOOD — module-level pool, bounded, reused across requests
const pool = new pg.Pool({ ...cfg, max: 10, idleTimeoutMillis: 30_000 });
const getUser = (id) => pool.query("SELECT ...", [id]).then(r => r.rows[0]);
```

- One pooled client per process for each upstream (HTTP client with
  keep-alive, DB pool, Redis client). Grep for `new .*Client(`, `connect(`,
  `createConnection` inside request handlers — finding on sight.
- HTTP: ensure keep-alive is actually on (Node needs an `Agent` with
  `keepAlive: true` pre-v19 defaults; Python `requests.Session` vs bare
  `requests.get`).
- **Bound pools and measure wait time.** Pool too small = invisible queueing
  (saturation); too large = overload the upstream. DB pools: start ~2–4× CPU
  cores of the DB-bound work, verify with pool-wait metrics. Total across all
  instances must fit DB max_connections (use a server-side pooler like
  PgBouncer beyond that).
- Set idle timeouts below any NAT/LB idle cutoff (~350 s on some clouds) and
  enable TCP keepalive, or you'll pay for silently dead connections (first
  request hangs until timeout).

## 4. Round trips, concurrency, and coalescing

In-DC RTT ~0.5 ms; cross-region ~80 ms. Sequential round trips add linearly;
**latency of concurrent calls is the max, not the sum**.

```typescript
// BAD — 3 sequential awaits on independent data: ~3 × RTT
const user = await getUser(id);
const orders = await getOrders(id);
const prefs = await getPrefs(id);

// GOOD — concurrent: ~1 × RTT (max of the three)
const [user, orders, prefs] =
  await Promise.all([getUser(id), getOrders(id), getPrefs(id)]);
```

Same in Python (`asyncio.gather`), Go (errgroup), Java (CompletableFuture /
structured concurrency). **Bound the concurrency** when fanning out over
collections (semaphore / `errgroup.SetLimit` / `p-limit`) — unbounded fan-out
is a self-DDoS.

**Request coalescing / singleflight**: when many concurrent callers need the
same expensive fetch, let one do the work and share the result (Go
`singleflight`, promise memoization in JS, distributed locks). Essential in
front of caches (rules/05 §4) and for config/metadata fetches.

**Pagination over the wire**: offset pagination re-scans O(offset) rows and
skews under concurrent writes; cursor/keyset pagination (`WHERE (created, id)
> ($1, $2) ORDER BY created, id LIMIT $3`) is O(page) at any depth and stable.
APIs: return opaque cursors, enforce max page size, never offer unbounded
`?limit=`. Deep-paging a 10M-row table by offset is a Critical finding on hot
paths.

**Timeouts/retries shape tail latency**: every remote call needs a deadline
propagated end-to-end; retries need backoff + jitter + budget (retry storms
amplify outages); hedged requests (send a second attempt at ~p95) cut tail
latency for idempotent reads at small extra load.

## 5. HTTP/2 and HTTP/3

- **HTTP/1.1**: one request in flight per connection (pipelining is dead) →
  browsers open 6 connections/origin; head-of-line (HoL) blocking at the
  application layer. Domain sharding and asset spriting are obsolete hacks.
- **HTTP/2**: multiplexes streams over one TCP connection, header compression
  (HPACK), stream prioritization. Removes app-layer HoL but keeps **TCP-layer
  HoL**: one lost packet stalls all streams. Use one connection per origin;
  enable on all public endpoints and internal LBs (gRPC requires it).
- **HTTP/3 (QUIC)**: streams over UDP — packet loss stalls only the affected
  stream; 1-RTT handshake combining transport+TLS; 0-RTT resumption;
  connection migration (Wi-Fi↔cellular without reconnect). Biggest wins on
  lossy/mobile/high-RTT networks (typically 5–15% p95 improvement, more at
  p99 on bad networks). Serve via CDN/edge (broad support); advertise with
  `Alt-Svc`.
- Server-internal hops: HTTP/2 (gRPC) for multiplexing; HTTP/3 rarely matters
  in-DC where loss ≈ 0.
- Don't let a proxy downgrade you: check the whole chain (CDN→LB→app) actually
  negotiates h2/h3, not h2 outside and 1.1 inside with per-request connections.

## 6. Compression tradeoffs

Compression trades CPU for bytes. Win condition: `time_saved_on_wire >
compress_time + decompress_time`. Always true cross-internet for text; often
false in-DC on 10 Gbps+ links for already-small payloads.

| Codec | Ratio (text) | Compress speed | Use |
|---|---|---|---|
| gzip -6 | baseline | ~50–100 MB/s | Legacy compatibility |
| brotli -4..5 | ~+10–15% vs gzip | comparable to gzip | Dynamic web responses |
| brotli -11 | ~+20–25% vs gzip | very slow (~1 MB/s) | **Static assets, precompressed at build** |
| zstd -3 (default) | ≈ gzip -6 or better | ~300–500 MB/s | APIs, internal traffic, storage, logs |
| zstd -19 + dict | best-in-class | slow compress, fast decompress | Precompressed artifacts; small payloads w/ dictionary |
| lz4 | lowest | GB/s | In-memory / latency-critical, RPC in-DC |

Rules:
- **Precompress static assets at build time** (brotli -11 + gzip fallback);
  serve with `Content-Encoding` negotiation. Never compress per-request what
  never changes.
- Dynamic responses: brotli 4–5 or zstd 3; gzip 6 as floor. `Accept-Encoding:
  zstd` is now sent by major browsers — support br + zstd + gzip.
- Don't compress: already-compressed media (JPEG/AVIF/WebP/MP4/ZIP — you burn
  CPU for ~0%), payloads < ~1 KB (header overhead, MTU fits anyway).
- zstd **dictionaries** give 2–5× better ratios on small similar payloads
  (per-message JSON/events) — train on a sample, version the dictionary.
- **Compression Dictionary Transport** (RFC 9842: `dcb`/`dcz` encodings,
  `Use-As-Dictionary`/`Available-Dictionary` headers) delta-compresses new
  asset versions against ones the client already cached — big wins on
  frequently redeployed JS bundles and templated HTML. Chromium-only (not
  Baseline); serve as progressive enhancement over the br/zstd/gzip ladder.
- Compression level is a live tuning knob under CPU pressure: dropping a level
  is a cheap capacity lever.
- Security: BREACH-style attacks — don't compress responses mixing secrets
  with attacker-reflected input (or mask tokens).

## 7. TLS efficiency

- TLS 1.3 everywhere: 1-RTT full handshake (vs 2 in 1.2), modern ciphers.
- **Session resumption** (session tickets): returning clients skip the full
  handshake; verify ticket keys rotate and resumption rate is monitored
  (target > 50% on browser traffic).
- **0-RTT early data**: resumed clients send the request in the first flight —
  saves a full RTT. Replay-unsafe: enable only for idempotent GETs and ensure
  the app/CDN rejects 0-RTT for mutations (`Early-Data` header / 425 status).
- OCSP stapling on; certificate chain minimal (every extra cert is bytes in
  the handshake, can overflow initcwnd).
- Internal mTLS meshes: handshake cost × per-request connections is a classic
  hidden tax — pooling (§3) matters double under mTLS.

## 8. CDN strategy

The fastest request is one that terminates ~10 ms from the user instead of
~150 ms.

- **Static assets**: immutable URLs (content hash in filename) +
  `Cache-Control: public, max-age=31536000, immutable`. Cache hit ratio on
  static should be > 95%.
- **Dynamic acceleration**: even uncacheable APIs benefit — TLS terminates at
  edge, long-lived warm connections edge→origin, better congestion control on
  the long haul.
- **Cacheable APIs/HTML**: `s-maxage` +`stale-while-revalidate`; purge by
  surrogate key/tags on writes (rules/05 §3). Short TTL (30–60 s) +
  request collapsing at the CDN shields origins from thundering herds.
- Normalize cache keys (strip marketing query params, normalize
  `Accept-Encoding`) or hit ratio dies of key fragmentation.
- Origin shield / tiered caching: one designated mid-tier reduces origin
  fan-in from hundreds of edge POPs to one.
- Edge compute for personalization-at-edge (rules/06 §7).

## 9. Async runtimes: never block the loop

In Node, Python asyncio, and single-threaded reactors, one blocked event loop
blocks **every** in-flight request.

- Grep for sync APIs on hot paths: Node `fs.readFileSync`, `zlib.gzipSync`,
  `crypto.pbkdf2Sync`, `child_process.execSync`, `JSON.parse` of multi-MB
  bodies; Python `time.sleep`, `requests.*`, blocking DB drivers inside
  `async def`.
- CPU-heavy work (hashing, compression, image resize, big serialization) →
  worker threads / process pool / dedicated service.
- Measure event-loop lag (Node `monitorEventLoopDelay`, asyncio debug slow
  callbacks); p99 loop delay > ~20 ms = something is blocking.
- Sync I/O in threaded runtimes is fine **if** the thread pool is sized for
  the blocking (and bounded); mixing blocking calls into a small shared pool
  (e.g. seda-style executors) causes whole-service stalls.

## Audit checklist

- [ ] Connections/clients created per request anywhere? (`new Client`,
      `connect(` in handlers.) Keep-alive verified on HTTP clients?
- [ ] Pools bounded? Pool wait time and saturation measured? Total
      connections fit upstream limits? Idle timeout < NAT/LB cutoff?
- [ ] Sequential awaits on independent operations? Fan-out concurrency
      bounded?
- [ ] Every remote call has a timeout; retries have backoff + jitter +
      budget; deadlines propagate?
- [ ] Offset pagination on large tables / unbounded page sizes on APIs?
- [ ] Unbuffered writes to files/sockets in loops? Missing flushes? fsync per
      record where group commit would do?
- [ ] Files read fully into memory only to be streamed out (sendfile/pipe
      candidates)? Serializers writing to intermediate strings vs streams?
- [ ] h2/h3 negotiated along the entire chain? gRPC/internal hops multiplexed
      or per-request 1.1 connections?
- [ ] Compression: static precompressed (brotli)? Dynamic on gzip-only when
      zstd/brotli available? Compressing compressed media or < 1 KB bodies?
- [ ] TLS 1.3? Resumption rate monitored? 0-RTT restricted to idempotent
      requests?
- [ ] CDN: immutable hashed assets with long max-age? Cache key normalized?
      stale-while-revalidate / surrogate-key purge in place? Hit ratio known?
- [ ] Any sync/blocking calls reachable from the event loop? Event-loop lag
      measured?
- [ ] Singleflight/coalescing in front of expensive shared fetches?

# 05 — Caching: Hierarchy, Invalidation, Stampedes

A cache is a bet that the past predicts the future, paid for with staleness
and operational complexity. Every cache must declare: what's cached, the key
schema, where it lives, how long it lives, how it's invalidated, what happens
when 10,000 requests miss at once, and how its hit ratio is watched. A cache
missing any of these answers is an incident on a timer.

## 1. The cache hierarchy — place data deliberately

| Layer | Latency | Scope | Coherence | Use for |
|---|---|---|---|---|
| CPU/data layout | ~1–40 ns | core | hardware | rules/03 §4 |
| In-process (map/LRU) | ~100 ns–1 µs | one instance | none across instances | hot config, compiled artifacts, per-entity hot reads |
| Distributed (Redis/Memcached) | ~0.3–1 ms in-DC | fleet-wide | single source | sessions, computed views, API responses |
| CDN/edge | ~5–30 ms to user | global | purge/TTL | static assets, cacheable HTML/API |
| Browser/client | 0 ms (hit) | one user | HTTP semantics | assets, API GETs, app state |

Rules:
- **Cache as close to the consumer as coherence allows.** Each layer down
  adds ~10–1000× latency.
- In-process caches are 100–1000× faster than Redis but multiply staleness by
  instance count and eat heap — use small bounded LRUs with short TTLs
  (1–60 s) for ultra-hot keys, backed by the distributed layer (L1/L2
  pattern). Per-instance hit ratio drops as the fleet scales out; don't expect
  L1 to carry a 200-instance fleet.
- Distributed cache is also a *shared failure domain*: a Redis hiccup becomes
  everyone's latency spike. Set aggressive client timeouts (~50–100 ms) and a
  fallback path; a cache that can take down the service is an availability
  finding, not just perf.
- The browser/CDN layers are governed by HTTP caching headers — get
  `Cache-Control`, `ETag`, and `Vary` right before adding server caches
  (rules/04 §8, rules/06).

## 2. Cache key design

Bad keys cause the two real cache bugs: **wrong data served** (key misses a
dimension) and **hit ratio collapse** (key includes a needless dimension).

- Key = every input that changes the value, and nothing else.
  `user:{id}:profile:v2:{locale}` — entity, qualifier, **schema version**,
  variant dimensions.
- **Version the schema in the key** (`:v2`): deploying a new shape then
  invalidates by abandonment — old keys age out, no purge needed, instant
  rollback (old code still reads `:v1`).
- Normalize inputs before keying: sort query params, lowercase where
  case-insensitive, strip irrelevant params (tracking junk), canonicalize
  `Vary` dimensions. Unnormalized keys fragment one logical value into
  hundreds of entries.
- Never build keys from raw user input without bounding/hashing (cardinality
  explosion + key injection via delimiter characters — hash long/dirty parts).
- Watch cardinality: a key including `user_id × endpoint × locale` may be
  fine; adding `?page&filter&sort` permutations can make every entry
  single-use (hit ratio → 0, memory → full). Audit: top key *prefixes* by
  count and by hit ratio.

## 3. Invalidation strategies

Pick per data class — there is no universal answer, only explicit tradeoffs:

1. **TTL-only**: simplest; staleness bounded by TTL. Right for data where
   bounded staleness is acceptable (prices refresh in 60 s). Wrong alone for
   anything users expect to see change immediately (their own edits).
2. **Write-through / write-invalidate**: on write, update or delete the cache
   entry. **Prefer delete over update** (update races: two concurrent writes
   can land in cache out of order; delete + lazy refill is idempotent).
   Still keep a TTL as backstop — invalidation paths have bugs.
3. **Event-driven**: writes publish invalidation events (or CDC from the DB)
   consumed by cache layers / CDN purge API. Scales to many caches; adds
   pipeline lag (purge latency = staleness window) and a component to monitor.
4. **Versioned/generation keys** (invalidation by abandonment): bump a
   generation counter (`tenant:{id}:gen`) on write; readers include the
   generation in keys. O(1) "invalidate everything for tenant" without
   scanning keys. Costs an extra lookup (cacheable in-process) and leaves
   garbage for LRU/TTL to sweep. CDN equivalent: surrogate keys / cache tags.
5. **Never use wildcard scans (`KEYS pattern*`) for invalidation** in Redis —
   O(N) blocking scan. If you need group invalidation, design for it
   (sets of keys, generations, tags).

**Read-your-own-writes**: after a user mutates data, route their next reads
around the cache (session flag, short-lived bypass) or write-through —
"I edited it and it didn't change" is the most-reported staleness bug.

**TTL discipline**: every entry gets a TTL even with active invalidation
(backstop). Choose TTL from a staleness budget ("how stale is acceptable?"),
not vibes. Document it next to the cache code.

## 4. Stampede protection (dogpile)

When a hot key expires (or cache restarts), every concurrent request misses
and hits the origin simultaneously. A 99% hit-ratio cache in front of a DB
sized for 1% traffic produces a 100× origin spike — the cache *caused* the
outage.

Defenses — layer them:

1. **Singleflight / request coalescing** (per instance): first miss computes;
   concurrent misses for the same key wait for that result.

```go
var g singleflight.Group
func Get(ctx context.Context, key string) (Val, error) {
    if v, ok := cache.Get(key); ok { return v, nil }
    v, err, _ := g.Do(key, func() (any, error) {
        val, err := loadFromOrigin(ctx, key)      // exactly one caller runs this
        if err == nil { cache.Set(key, val, ttlWithJitter()) }
        return val, err
    })
    if err != nil { return zero, err }
    return v.(Val), nil
}
```

   JS equivalent: memoize the *promise*, not the value (concurrent callers
   share one in-flight promise; clear on settle/error). Distributed
   singleflight: short-TTL lock key (`SET lock:k token NX PX 3000`), losers
   serve stale or wait briefly.

2. **Jittered TTLs**: identical TTLs synchronize expiry of keys populated
   together (deploy, cache flush, midnight cron). `ttl = base × (0.9 + 0.2 ×
   rand())` (±10%) decorrelates expirations. Apply everywhere by default.

3. **Soft TTL / refresh-ahead (stale-while-revalidate)**: store
   `(value, soft_expiry)` with a longer hard TTL. After soft expiry, serve the
   stale value immediately and refresh asynchronously (one refresher via
   singleflight). Users never wait on origin latency; origin sees ~1 request
   per key per refresh period. This is the SOTA default for hot keys; HTTP's
   `stale-while-revalidate` and CDN request collapsing are the same idea.
   Probabilistic early expiration (XFetch: refresh early with probability
   rising as expiry nears) avoids even the soft-expiry synchronization.

4. **Cold-start protection**: cache warmers for known-hot keys before taking
   traffic; rate-limit/queue origin concurrency (bounded semaphore around
   origin loads) so a cache wipe degrades latency instead of toppling the DB.

Audit shortcut: find the hottest cached key, ask "what happens the instant it
expires under full load?" If the answer is "everyone queries the origin",
that's a High finding regardless of current hit ratio.

## 5. Write strategies and consistency

How writes interact with the cache determines both performance and the bugs
you'll debug at 3 a.m.:

- **Cache-aside (lazy)** — default. App reads cache, on miss loads origin and
  fills; writes go to origin + delete the key. Simple, origin is source of
  truth. Race to know: read-miss loads stale row, a write lands, the stale
  fill overwrites the delete. Mitigate with short TTLs, compare-and-set
  (`SET ... NX` for fills), or version checks on fill.
- **Write-through** — write cache + origin synchronously. Reads never miss
  hot data; writes pay double latency. Use for read-heavy keys with strict
  read-your-writes needs.
- **Write-behind (write-back)** — write cache, flush to origin async. Fastest
  writes, but the cache becomes the durability story: a crash loses acked
  writes. Only with replicated cache tiers and idempotent replay; treat as a
  queue, monitor flush lag. Rarely the right call for business data.
- **Refresh-ahead** — §4.3; reads never pay origin latency for hot keys.

Distributed caches are not transactional with your DB. Any "update DB and
cache together" code has a window where one succeeded and the other didn't —
prefer delete-on-write + TTL, or CDC-driven invalidation, over dual writes.

## 6. Negative caching

Cache "not found" / empty results too — otherwise every lookup for a missing
key (typos, deleted users, enumeration attacks, retry loops on 404s) goes to
origin forever. A miss-storm on nonexistent keys is indistinguishable from a
DoS at the database.

- Store an explicit sentinel (`NOT_FOUND`) — distinguish "cached absence" from
  "cache miss". Don't conflate with `null`/nil returns.
- **Short TTL** (5–60 s): absence changes when the entity is created, and you
  usually have no invalidation event wired for creations. Long negative TTLs
  cause "I just signed up and the site says I don't exist".
- Bound negative-cache memory (attackers can enumerate infinite missing keys):
  separate small LRU, or a Bloom filter of *existing* keys in front of the
  cache for cheap "definitely absent" answers.
- Cache errors with care: brief caching of upstream 5xx (1–5 s) acts as a
  circuit-breaker valve; never cache them as long as successes.

## 7. Metrics — a cache without metrics is unfalsifiable

Track per logical cache (not just per Redis instance):

- **Hit ratio** — but interpret it: 95% hits with a 1 ms origin saves little;
  60% hits with a 2 s origin is gold. The number that matters is
  `misses × origin_cost` (origin offload).
- Origin load with/without cache (what happens if it flushes?).
- Latency of the *cache path itself* (p99 — a slow Redis adds latency to every
  request, hit or miss).
- Eviction rate & memory: high eviction = undersized or key-cardinality
  explosion; near-zero eviction with full memory = TTLs too long.
- Stale-serve rate (if soft TTL) and invalidation pipeline lag (if event-driven).

## 8. When caching is the wrong fix

Caching hides cost; it doesn't remove it. Reach for it after cheaper, more
honest fixes:

- **A missing index is not a caching problem.** A 2 s query that should be
  20 ms with an index gets an index, not a Redis layer. Cache-on-top leaves
  the pathology to detonate on every miss.
- **N+1 is not a caching problem** — batching removes the cost; caching N+1
  multiplies key cardinality and stampede surface.
- **Don't cache cheap things**: if origin cost ≈ cache cost (~sub-ms indexed
  PK lookup vs ~0.5 ms Redis hop in another AZ), you added staleness and an
  invalidation bug surface for zero latency win.
- **Low hit ratios** (< ~50–80% depending on origin cost): per-user
  rarely-repeated data, long-tail key distributions, highly personalized
  responses — the cache is mostly overhead. Compute it cheaper or precompute
  (materialized views, denormalization) instead.
- **Correctness-critical reads** (balances, inventory, auth/permissions):
  staleness is a security/correctness bug. If you must cache authz, keep TTL
  seconds-short and provide kill-switch invalidation.
- **Caching to mask a leak/regression**: if latency degraded recently, find
  the regression; a cache on top converts a visible problem into a latent one.
- Smell: caches added inside the same process directly in front of another
  cache (cache-on-cache) without distinct purpose — usually one layer is
  unmanaged.

Decision order: fix the query/algorithm → batch the calls → precompute on
write → then cache, with the full contract from this file's intro.

## 9. Optimization must not erode security

Caching and other speedups are where security regressions hide — the code
still "works", just for the wrong user:

- **Identity in the key, or no shared cache.** Any authz-sensitive or
  per-user/per-tenant response cached in a shared layer (Redis, CDN, shared
  in-process map) must carry the user/tenant — and role/scope where the value
  varies by it — in the cache key. A key missing the identity dimension serves
  user A's response to user B: a cache-key authorization bypass. Same family
  as web cache deception (sota-code-security rules/05): cacheability must
  never be decided by URL shape alone. Default for personalized responses:
  `Cache-Control: private` at the HTTP layer, identity-scoped keys elsewhere.
- **Constant-time comparisons stay constant-time.** Secret/token/MAC checks
  (`hmac.compare_digest`, `crypto.timingSafeEqual`,
  `subtle.ConstantTimeCompare`) are deliberately "inefficient" — "optimizing"
  them into early-exit `==`/memcmp reintroduces the timing oracle. Never flag
  them as a perf finding; never accept a patch that replaces them.
- **Fast paths keep the guards.** Streaming/zero-copy parsing, parallel
  handlers, and request coalescing must preserve the input validation, size
  and recursion limits, and auth checks the slow path performed. An
  optimization PR that touches a validation path needs a security review, not
  just a benchmark.

## Audit checklist

- [ ] Inventory every cache (in-process maps, memoizers, Redis, CDN, HTTP
      headers). For each: key schema, TTL, bound/eviction, invalidation path,
      stampede defense, metrics. Any blank cell is a finding.
- [ ] Unbounded in-process caches (plain `Map`/dict with no LRU/TTL) → memory
      leak (rules/03), High.
- [ ] TTLs constant (no jitter)? Mass-populated keys expiring in sync
      (deploy/cron)?
- [ ] Hot keys: singleflight or soft-TTL in place? What happens on expiry
      under peak load — and on full cache flush/restart?
- [ ] Promise/value memoization in JS: are *errors* cached forever
      (rejected promise never cleared)?
- [ ] Keys: schema-versioned? Inputs normalized? Cardinality bounded?
      User-controlled fragments hashed/escaped?
- [ ] Invalidation: delete-vs-update on writes? TTL backstop present even
      with event-driven purge? Any `KEYS pattern*` scans? Read-your-own-writes
      handled after mutations?
- [ ] Negative caching present for high-miss lookups? Sentinel distinct from
      miss? Short TTL? Bounded against enumeration?
- [ ] Cache client timeouts set (~50–100 ms) and a degradation path if the
      cache tier is down? Or does cache-down = site-down?
- [ ] Hit ratio, eviction rate, and origin offload measured per logical
      cache? Any cache with hit ratio < 50% — should it exist?
- [ ] Any cache papering over an uninvestigated slow query / N+1 / missing
      index? Recommend the honest fix first.
- [ ] Authz/financial data cached? TTL and invalidation justified in writing?
- [ ] Shared-cache keys for per-user/per-tenant responses include the
      identity/tenant (and role where it varies)? Any personalized response
      cacheable by URL alone (cache deception / key bypass —
      sota-code-security rules/05)?
- [ ] Any optimization that replaced a constant-time compare or relaxed
      validation/size limits for throughput? Treat as Critical, not perf win.

# 06 — Frontend & Web Performance

Frontend performance is measured at the user's device — a $150 Android phone
on 4G, not your M-series laptop on fiber. Budget against **field data at p75**
(CrUX / your RUM), use lab tools (Lighthouse, WebPageTest) for diagnosis and
CI gating.

## 1. Core Web Vitals — current thresholds (p75, field)

| Metric | Good | Needs improvement | Poor | Measures |
|---|---|---|---|---|
| **LCP** (Largest Contentful Paint) | ≤ 2.5 s | 2.5–4.0 s | > 4.0 s | Loading: when the main content renders |
| **INP** (Interaction to Next Paint) | ≤ 200 ms | 200–500 ms | > 500 ms | Responsiveness: worst-case interaction latency (replaced FID in 2024) |
| **CLS** (Cumulative Layout Shift) | ≤ 0.1 | 0.1–0.25 | > 0.25 | Visual stability |

Supporting diagnostics: TTFB ≤ 800 ms (LCP can't be good on a 2 s TTFB),
FCP ≤ 1.8 s, Total Blocking Time (lab proxy for INP).

**LCP decomposition** — fix the dominant phase, not all of them:
TTFB → resource load delay → resource load time → render delay.
- LCP image must be discoverable in initial HTML (no CSS `background-image`
  for hero, no JS-inserted `<img>`, never `loading="lazy"` on the LCP image).
  Use `<img fetchpriority="high">` + `<link rel="preload">` if late-discovered.
- Lazy-load everything below the fold (`loading="lazy"`), never above it.

**INP** is caused by long main-thread tasks (> 50 ms):
- Break up long JS: `scheduler.yield()` / `await` chunking; show feedback
  within 100 ms even if work continues.

```javascript
// BAD — 5,000 items processed in one task: ~800 ms frozen main thread,
// every click during it counts against INP
items.forEach(render);

// GOOD — yield between chunks; first paint of feedback within one frame
for (const chunk of chunks(items, 200)) {
  chunk.forEach(render);
  await scheduler.yield();          // or: await new Promise(r => setTimeout(r))
}
```

- Avoid synchronous layout thrash (read layout → write style → read again in
  a loop forces reflow per iteration — batch reads, then writes).

```javascript
// BAD — read/write interleaved: forced synchronous reflow per element
for (const el of els) el.style.height = el.offsetHeight * 2 + "px";

// GOOD — phase 1 read all, phase 2 write all: one reflow total
const heights = els.map(el => el.offsetHeight);
els.forEach((el, i) => { el.style.height = heights[i] * 2 + "px"; });
```
- Heavy work off the main thread: Web Workers for parsing/crypto/diffing.
- Hydration and rerender storms are the top INP killers in SPAs (§6).
- Debounce input handlers; use CSS (`content-visibility`, transforms,
  animations on compositor) over JS where possible.

**CLS**: reserve space — explicit `width`/`height` (or `aspect-ratio`) on
images/embeds/ads; `font-display` strategy + size-matched fallback fonts (§5);
never insert banners above existing content; animate with `transform`, not
top/left/height.

## 2. Bundle size budgets

JS is the most expensive byte: 200 KB of JS costs download + parse + compile +
execute (~3–5× the cost of 200 KB of image on a mid-range phone's CPU).

Budgets (compressed, over-the-wire) — adjust to audience, enforce in CI:
- Initial critical-path JS: **≤ 150–200 KB** (≈ 450–600 KB uncompressed).
  An app shipping 1 MB+ initial JS will not hit INP/LCP targets on median
  mobile hardware.
- Initial CSS: ≤ 50 KB; inline critical CSS if render-blocking matters.
- Per-route async chunks: ≤ 100 KB each.
- Track *first-load JS per route* (Next.js build output does this natively).

Enforcement: `size-limit`, `bundlesize`, Lighthouse CI `budgets.json`, webpack
`performance.maxAssetSize` — fail the PR, don't dashboard it. Diagnose with
`webpack-bundle-analyzer` / `source-map-explorer` / `vite-bundle-visualizer`.

Top offenders to grep for: moment (→ date-fns/dayjs/Temporal), lodash
full-import (→ `lodash-es` named imports), big charting/editor libs loaded
eagerly, polyfills for evergreen browsers, duplicate dependency versions
(`npm dedupe`, lockfile audit), source-map/dev artifacts shipped to prod,
importing a server SDK into client code.

## 3. Code splitting

- **Route-based splitting is the floor**: every router-level view is its own
  chunk (framework defaults: Next/Nuxt/SvelteKit do this; verify it isn't
  defeated by a barrel file importing everything into a shared layout).
- **Interaction-based splitting**: heavy components behind user intent —
  modals, editors, charts, maps — `import()` on open/hover/viewport
  (`React.lazy`, `defineAsyncComponent`).

```tsx
// BAD — 280 KB editor in the initial bundle of every page that *might* edit
import { RichTextEditor } from "@acme/editor";

// GOOD — loads only when the user opens the editor; prefetch on hover
const RichTextEditor = lazy(() => import("@acme/editor"));
<Suspense fallback={<EditorSkeleton />}>{editing && <RichTextEditor />}</Suspense>
```
- Preload likely-next chunks on idle/hover (`rel="prefetch"`, router
  prefetching) so splitting doesn't add interaction latency.
- Don't over-split: hundreds of tiny chunks add request overhead and waterfall
  depth even on h2/h3; group by route/feature (~30–100 KB chunks).
- Barrel files (`index.ts` re-exporting a directory) defeat tree-shaking in
  many setups — import from concrete modules or configure
  `optimizePackageImports`/`sideEffects: false`.

## 4. Images

Usually the largest bytes on the page and the most common LCP element.

- **Formats**: AVIF first (≈ 30–50% smaller than JPEG at equivalent quality),
  WebP fallback (≈ 25–35% smaller than JPEG), JPEG/PNG last resort. SVG for
  icons/illustrations. Use `<picture>` + `type` negotiation or an image CDN
  that negotiates via `Accept`.
- **Responsive sizing**: `srcset` + `sizes` so a phone doesn't download the
  2400 px desktop hero. Serving a 2000 px image into a 400 px slot is a
  ~10–20× byte waste — the most common image finding.
- Compress: quality 60–75 covers most photographic content; use an image
  CDN/pipeline (resize, format, quality per request) instead of committed
  pre-baked assets.
- LCP image: `fetchpriority="high"`, preload, no lazy (§1). Everything below
  fold: `loading="lazy" decoding="async"`.
- Always `width`/`height`/`aspect-ratio` (CLS). Poster images for videos;
  `preload="none"` on below-fold video.

## 5. Fonts

Web fonts block or shift text rendering.

- **WOFF2 only** (≈ 30% smaller than WOFF; universal support).
- **Subset** to used scripts/characters (`pyftsubset`, `glyphhanger`):
  a full font with CJK + symbols can be 1 MB+; a Latin subset ~15–30 KB.
- **Self-host** with `Cache-Control: immutable`; third-party font CSS adds a
  connection + CSS round trip on the critical path (and cache partitioning
  killed the shared-cache benefit years ago).
- `<link rel="preload" as="font" type="font/woff2" crossorigin>` for the 1–2
  critical fonts only.
- `font-display: swap` (text visible immediately) or `optional` (no swap
  flash; best CLS). Tame swap-induced CLS with metric-compatible fallbacks:
  `size-adjust`/`ascent-override` on a fallback `@font-face` (tooling:
  fontaine, Next.js `next/font` does all of this automatically).
- Limit families/weights: every weight is a file; use variable fonts when you
  need > 2–3 weights.

## 6. Hydration cost

SSR HTML that then hydrates pays twice: server render + client re-execution
of the entire component tree (download → parse → execute → attach). On
mid-range mobile, hydrating a large tree costs 1–3 s of main-thread time —
the page *looks* ready but ignores taps (INP/TBT killer, "uncanny valley").

Mitigations, in order of leverage:
1. **Ship less component code**: server-only rendering for static parts —
   React Server Components / Astro-style zero-JS-by-default mean
   non-interactive components ship **no** client JS at all.
2. **Islands / partial hydration**: hydrate only interactive widgets (Astro,
   Fresh, eleventy-is-land); the static 90% of a content page stays HTML.
3. **Lazy/deferred hydration**: hydrate on visibility/interaction
   (`client:visible`, `astro:idle` equivalents; React `lazy` + Suspense
   boundaries) — below-fold widgets shouldn't hydrate during load.
4. **Streaming SSR + selective hydration** (React 18+): flush HTML early
   (TTFB/LCP win), hydrate islands as their code arrives, prioritize the one
   the user touches.
5. Resumability (Qwik) skips replay-style hydration entirely — niche but the
   conceptual benchmark.

Audit signals: framework runtime + app code re-executing everything on load;
`hydration mismatch` warnings (double render); interactive-but-dead period in
traces (long tasks right after LCP); TBT ≫ 300 ms in lab.

## 7. Edge rendering & delivery

- **Static-first**: anything renderable at build time (marketing, docs, blogs)
  ships as CDN-cached static HTML — TTFB ~20–50 ms globally, origin can be
  down. ISR/SWR regeneration keeps content fresh without rebuild-the-world.
- **Edge SSR** for personalized-but-light pages: render at the POP (~10–30 ms
  from user) vs origin (~100–300 ms cross-region). Constraints: limited
  runtime APIs, and **data locality** — an edge function calling a
  single-region DB pays the cross-region RTT anyway (worse than origin
  rendering). Edge SSR only wins when data is also at the edge (edge KV,
  regional replicas) or the page needs ≤ 1 origin fetch.
- Hybrid default: static shell from CDN + cached API + client/edge
  personalization; or streaming SSR from origin with early-flushed `<head>`.
- TTFB budget ≤ 800 ms is mostly an architecture decision: cache HTML where
  possible, stream when not, terminate TLS at edge always (rules/04 §8).

## 8. Delivery hygiene (fast wins)

- `<script defer/type=module>` — no sync scripts in `<head>`; third-party tags
  async or via a tag manager loaded post-LCP, or `web worker`-ized (Partytown)
  when feasible. Third-party JS is the most common externally-caused INP/TBT
  regression — audit the tag list quarterly.
- Resource hints: `preconnect` to critical third-party origins (≤ 2–3);
  `preload` only what's provably late-discovered (preload spam steals
  bandwidth from the LCP resource).
- `103 Early Hints` for preconnect/preload while origin thinks.
- Compression and caching per rules/04 §6/§8: brotli/zstd, immutable hashed
  assets, h2/h3 end-to-end.
- Measure RUM (web-vitals JS library → your analytics) segmented by device
  class and country — averages across devices hide the phones where you fail.
- Know the SPA blind spot: classic CWV attributes everything after the initial
  load to that first page — soft (in-app) navigations aren't measured. Chrome's
  Soft Navigations API (launching unflagged from Chrome 151; the final API
  differs from earlier origin-trial shapes) extends LCP/CLS/INP to SPA route
  changes — adopt via the web-vitals library once it ships.

Calibrate against real conditions — what a byte budget means on the wire:

| Condition | Bandwidth | RTT | 200 KB JS arrives in |
|---|---|---|---|
| Fast 4G / median mobile | ~9 Mbps | ~60–170 ms | ~0.3–0.5 s + parse/exec ~0.5–1.5 s on mid-range CPU |
| Slow 4G / crowded network | ~1.6 Mbps | ~150 ms | ~1.2 s + parse/exec |
| 3G (emerging markets, roaming) | ~0.4–0.7 Mbps | ~300–400 ms | ~3–4 s before a line of your code runs |

Lab-test with throttling (Lighthouse's default is throttled mobile for a
reason); a page that's "instant" unthrottled and 6 s on slow 4G is a 6 s page
for a real cohort of users.

## Audit checklist

- [ ] Field CWV (CrUX/RUM) at p75 per key page: LCP ≤ 2.5 s, INP ≤ 200 ms,
      CLS ≤ 0.1? Which metric/phase dominates the failure?
- [ ] LCP element: discoverable in initial HTML? `fetchpriority="high"`?
      Not lazy-loaded? TTFB ≤ 800 ms on that route?
- [ ] Initial compressed JS per route ≤ ~200 KB? CI budget enforcement
      (size-limit/Lighthouse CI) present and failing builds?
- [ ] Bundle analyzer output: moment/full-lodash/duplicate versions/eagerly
      loaded heavy libs/barrel-file tree-shaking defeats?
- [ ] Route-level code splitting working (check chunk map)? Heavy widgets
      (editor/chart/map/modal) behind dynamic `import()`? Likely-next routes
      prefetched?
- [ ] Images: AVIF/WebP negotiated? `srcset`/`sizes` present? Dimensions set
      (CLS)? Below-fold lazy, above-fold not? Any image > ~200 KB on the wire?
- [ ] Fonts: WOFF2, subset, self-hosted, ≤ 2 preloaded, `font-display` +
      metric-compatible fallback (or next/font/fontaine)?
- [ ] Long tasks > 50 ms during load and on interaction (trace)? Layout
      thrash loops? Heavy work that belongs in a Worker?
- [ ] Hydration: does static content ship client JS (RSC/islands candidate)?
      Below-fold components hydrating eagerly? TBT after LCP?
- [ ] Third-party scripts: inventoried, async/deferred, measured for
      main-thread cost? Any sync `<head>` script?
- [ ] HTML caching strategy: static/ISR where possible? Edge SSR only where
      data is edge-local? Early-flushed streaming where origin-rendered?
- [ ] RUM in place, segmented by device/geo — or is the team flying on
      laptop Lighthouse runs only?