# 01 — Concurrency Models & Structured Concurrency

## Choosing a model

There are four models. Pick by workload shape, not by familiarity.

| Model | Parallelism | Memory | Best for | Worst for |
|---|---|---|---|---|
| Event loop (async/await) | None (single thread) | Shared, but single-threaded — no data races, still race conditions | Many concurrent I/O waits (10k sockets) | CPU work, blocking libraries |
| Threads | Yes (runtime permitting) | Shared — full data-race exposure | Parallel CPU work, blocking-API integration | Massive fan-out (stack cost), correctness at scale |
| Processes | Yes, full | Isolated — share via IPC/serialization | CPU-bound in GIL-locked runtimes; fault isolation | Chatty workloads (serialization tax) |
| Actors (message passing) | Yes | Isolated per actor; communicate via mailboxes | Stateful entities at scale, distribution, supervision trees | Simple request/response pipelines (overkill) |

### Decision tree

```
What dominates the workload?
├── Waiting on I/O (network, disk, DB)
│   ├── Concurrency level high (100s+)  → event loop / async
│   └── Low, and libraries are blocking → small thread pool is fine
├── Computing (CPU-bound)
│   ├── Runtime has parallel threads (Go, Rust, Java, C#,
│   │   Python 3.13+ free-threaded)    → thread pool, size ≈ cores
│   └── GIL-constrained (CPython w/ GIL, Node JS-land)
│       → process pool / worker_threads (V8 isolates count as this);
│         subinterpreter pool on CPython 3.14+ (see Python note)
├── Both (typical server: I/O front, CPU spikes)
│   → async front-end + bounded worker pool for CPU (rule 04)
└── Long-lived stateful entities, supervision, distribution
    → actor model (Erlang/Elixir, Akka/Pekko, Actix, or hand-rolled
      single-owner task + mailbox channel — see below)
```

**Rationale:** the event loop wins I/O fan-out because a parked coroutine costs
~KB versus ~MB-of-stack per thread; threads win CPU because an event loop has
exactly one core's worth of compute. Mixing them backwards produces the two
classic failures: blocked loops (rule 04) and 10,000 threads.

### Per-runtime notes

- **Python:** asyncio coroutines for I/O; `ProcessPoolExecutor` for CPU under
  the GIL. Python 3.13+ free-threaded builds make thread pools viable for CPU,
  but C extensions must declare support — verify before relying on it.
  Python 3.14+ adds subinterpreters (PEP 734: `concurrent.interpreters`,
  `concurrent.futures.InterpreterPoolExecutor`) — per-interpreter-GIL
  parallelism in one process, cheaper than a process pool, but objects cross
  the boundary by pickling, not by reference.
- **Node:** one JS thread per isolate. CPU work goes to `worker_threads`
  (or `piscina`); never compute on the main loop.
- **Go:** goroutines are M:N scheduled — blocking syscalls don't block the
  scheduler, so "async vs threads" mostly disappears. Your problems shift to
  unbounded goroutine spawns and channel misuse (rules 03, 06).
- **Rust:** `tokio` (or `smol`) for I/O; `rayon` or `spawn_blocking` for CPU.
  Don't run rayon work on the tokio runtime threads — it starves the reactor.
- **JVM:** virtual threads (Loom) give event-loop economics with thread
  programming model; pinning on `synchronized` blocks around blocking calls is
  the main hazard on JDK ≤23 (use `ReentrantLock` instead in hot paths) —
  fixed in JDK 24+ (JEP 491), but JDK 21 LTS deployments still hit it.

## Structured concurrency is the default

A task whose lifetime exceeds the scope that created it is an **orphan**: its
exceptions vanish, its resources leak, and nothing cancels it on shutdown.
Structured concurrency makes task lifetime lexical: a scope (nursery, task
group, errgroup, JoinSet) owns its children, propagates their errors, and
cancels siblings on failure.

**Rule: every spawn is inside a scope that joins it.** Spawning into the void
(`asyncio.create_task` with a discarded handle, bare `go func()`,
un-awaited promise, dropped `JoinHandle`) requires: (a) an attached error
handler, (b) registration with a shutdown mechanism, (c) a comment saying why.

```python
# BAD — orphan: exception is swallowed, task outlives the request,
# nothing cancels it on shutdown. CPython may even GC the task mid-flight
# because create_task only holds a weak reference.
async def handle(req):
    asyncio.create_task(audit_log(req))     # fire-and-forget
    return await process(req)

# GOOD — scope owns both; if process() raises, audit_log is cancelled;
# neither can outlive handle().
async def handle(req):
    async with asyncio.TaskGroup() as tg:
        tg.create_task(audit_log(req))
        t = tg.create_task(process(req))
    return t.result()
```

```go
// BAD — goroutine leaks if ctx is cancelled before send; errors lost.
func fetchAll(urls []string) []Result {
    out := make(chan Result)
    for _, u := range urls {
        go func() { out <- fetch(u) }()     // who joins this? nobody.
    }
    ...
}

// GOOD — errgroup: bounded, joined, first error cancels the rest.
func fetchAll(ctx context.Context, urls []string) ([]Result, error) {
    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(16)
    results := make([]Result, len(urls))
    for i, u := range urls {
        g.Go(func() error {
            r, err := fetch(ctx, u)
            results[i] = r                  // disjoint index: no race
            return err
        })
    }
    return results, g.Wait()
}
```

```rust
// BAD — handle dropped: task detaches, panics are silently lost.
tokio::spawn(async move { sync_to_remote(item).await });

// GOOD — JoinSet ties tasks to the owning scope; aborts on drop.
let mut set = tokio::task::JoinSet::new();
for item in items {
    set.spawn(sync_to_remote(item));
}
while let Some(res) = set.join_next().await {
    res??; // surface panics and errors
}
```

```js
// BAD — floating promise: rejection becomes unhandledRejection,
// possibly crashing the process at a random later time.
function handle(req) {
  auditLog(req);            // async fn, not awaited, no .catch
  return process(req);
}

// GOOD — join both; Promise.allSettled if audit failure is non-fatal.
async function handle(req) {
  const [audit, result] = await Promise.allSettled([auditLog(req), process(req)]);
  if (result.status === "rejected") throw result.reason;
  return result.value;
}
```

### Scope semantics to know

- **Error propagation:** TaskGroup/errgroup cancel siblings on first error and
  re-raise. If you need "collect all results, even failures", use
  `Promise.allSettled` / gather with `return_exceptions=True` / collect from
  `JoinSet` — and then *actually inspect* the failures.
- **Nesting:** scopes nest; cancellation flows down, errors flow up. Deadlines
  attach naturally to scopes (rule 05).
- **Background services** (true daemons: metrics flusher, heartbeat) are the
  one legitimate long-lived spawn. Pattern: create them in `main`'s top-level
  scope, store handles, cancel them in shutdown. They are still owned — by the
  application scope, not by nobody.

### Hand-rolled actor (single-owner state)

When multiple tasks need the same mutable state, the cheapest correct design
is often: one owner task, one bounded mailbox, no locks.

```go
// GOOD — counter actor: state confined to one goroutine; callers
// communicate via channel. No mutex, no race, natural backpressure.
type op struct{ delta int; reply chan int }
func counter(ctx context.Context, ops <-chan op) {
    n := 0
    for {
        select {
        case <-ctx.Done():
            return
        case o := <-ops:
            n += o.delta
            o.reply <- n
        }
    }
}
```

Use this instead of a mutex when: state has invariants spanning multiple
fields; operations must serialize anyway; or you need to bound the request
rate to the state (mailbox = built-in backpressure).

## Sizing rules

- Thread/process pool for CPU: `n_cores` (maybe `n_cores ± 1`); more adds
  context-switch overhead, not throughput.
- Thread pool wrapping blocking I/O: size by `concurrency_target ×
  avg_wait_fraction`, cap it, and queue behind a semaphore — not "unbounded
  cached pool".
- Async fan-out: never `gather(*[f(x) for x in million_items])`. Bound with a
  semaphore or worker pool reading from a bounded queue (rule 06).

```python
# BAD — a million concurrent connections, file descriptors, and timers.
await asyncio.gather(*(fetch(u) for u in urls))

# GOOD — bounded fan-out inside a scope.
sem = asyncio.Semaphore(50)
async def bounded_fetch(u):
    async with sem:
        return await fetch(u)
async with asyncio.TaskGroup() as tg:
    tasks = [tg.create_task(bounded_fetch(u)) for u in urls]
```

## Audit checklist

- [ ] Workload classified? CPU-bound code on an event loop or I/O fan-out on
      a thread-per-request model is an architecture-level finding.
- [ ] Grep for orphan spawns: `create_task(`/`ensure_future(` with unused
      result, bare `go func(`, `tokio::spawn` with dropped handle, async calls
      and `.then(` chains with no await/`.catch`.
- [ ] For every spawn: who joins it? Who sees its exception? Who cancels it on
      shutdown? Three answers or it's a finding (HIGH if it does I/O or holds
      resources).
- [ ] `asyncio.create_task` results stored in a strong reference (or
      TaskGroup)? Weak-ref GC of running tasks is a real CPython footgun.
- [ ] Fan-out loops bounded by semaphore/errgroup limit/pool size?
- [ ] `gather`/`Promise.all` failure mode considered — does first failure
      strand or leak the siblings? (Plain `asyncio.gather` does not cancel
      siblings on error unless they're in a TaskGroup.)
- [ ] Background daemons registered for shutdown cancellation?
- [ ] Pools sized with a stated rationale, not defaults-by-accident?
- [ ] Shared mutable state: could it be owned by one task + mailbox instead of
      a lock? (Not mandatory, but flag invariant-spanning state guarded by
      multiple separate locks.)

# 02 — Correctness: Races, Atomicity, Memory Ordering, Deadlock

## Data race vs race condition — different bugs, different fixes

- **Data race:** two threads access the same memory location concurrently,
  at least one writes, with no synchronization. In C/C++/Rust(unsafe)/Go this
  is undefined or corrupting behavior — torn reads, impossible values. Fix
  with synchronization (atomics, locks) or by removing sharing.
- **Race condition:** a correctness bug from *ordering*, even with perfectly
  synchronized individual accesses. A program can be 100% data-race-free and
  still race. Fix by making the whole multi-step operation atomic, not by
  adding more locks around the individual steps.

Single-threaded event loops eliminate data races but **not** race conditions:
every `await` is a yield point where other tasks run and mutate shared state.

```python
# RACE CONDITION, zero threads — classic async check-then-act.
# Task A and Task B both see cache miss, both fetch, B overwrites A.
# Worse with non-idempotent actions (double-charge, double-send).
async def get_user(uid):
    if uid not in cache:               # check
        cache[uid] = await fetch(uid)  # await = interleave point; act
    return cache[uid]

# GOOD — collapse to single-flight: first caller stores a future,
# concurrent callers await the same future. Check+act with no await between.
async def get_user(uid):
    fut = cache.get(uid)
    if fut is None:
        fut = asyncio.ensure_future(fetch(uid))
        cache[uid] = fut               # no await between check and act
    try:
        return await fut
    except Exception:
        cache.pop(uid, None)           # don't cache failures
        raise
```

## Atomicity: find the invariant, protect the whole transition

`count += 1` is read-modify-write: three steps, racy everywhere (yes, also in
CPython — the GIL serializes bytecodes, not statements; and `+=` on an
attribute is multiple bytecodes). The unit of protection is the **invariant**,
not the variable.

```go
// BAD — both fields individually atomic, invariant (sum constant) still
// violated: a reader between the two Stores sees money created/destroyed.
a.balance.Store(a.balance.Load() - amt)   // also a lost-update race itself
b.balance.Store(b.balance.Load() + amt)

// GOOD — one lock spans the whole invariant-preserving transition.
mu.Lock()
a.balance -= amt
b.balance += amt
mu.Unlock()
```

Heuristics:
- If two fields must change together, one lock (or one owner task) covers both.
- Atomics are for single-word counters/flags/pointers. The moment logic reads
  an atomic and then writes based on it, you need CAS-loop or a lock.
- Compound map ops (`check-then-insert`, `get-then-update`) need the map's
  lock across the compound, or a primitive that is compound-atomic
  (`dict.setdefault`, `sync.Map.LoadOrStore`, `compute_if_absent`, UPSERT).

## Check-then-act / TOCTOU

Any `if <state> then <act-on-state>` where state is shared (memory, file
system, DB, remote API) is suspect. The gap between check and act is the race
window — and on the filesystem it's also a security hole (symlink swap between
`access()` and `open()`).

| Bad pattern | Atomic replacement |
|---|---|
| `if not exists(path): create(path)` | `open(path, O_CREAT|O_EXCL)` / `mkdir` and handle EEXIST |
| `if key not in map: map[key] = v` | `setdefault` / `LoadOrStore` / `putIfAbsent` |
| `SELECT` then `INSERT` | `INSERT ... ON CONFLICT` / unique constraint + handle violation |
| `if balance >= amt: balance -= amt` | `UPDATE ... SET balance = balance - amt WHERE balance >= amt` and check rows-affected |
| `if not lock_file_exists: write lock_file` | `O_EXCL` create, or flock, or a real lease with expiry |

**Rule:** push atomicity to the system that owns the state (DB constraint,
atomic syscall, compound-atomic API). Checking first and acting second is only
valid as an optimization *before* an atomic operation, never as the guard.

## Memory visibility & ordering (the 20% you need)

On multicore hardware, writes by one thread are not visible to others in
program order unless synchronization establishes **happens-before**. Without
it: stale reads forever (compiler hoists the load), reordered writes
(initialization seen after the published pointer).

- Every lock release **happens-before** the next acquire of the same lock;
  channel send happens-before the corresponding receive; thread/task join
  happens-before the joiner's next read. Use these — they're why properly
  locked code "just works".
- A plain boolean `done` flag set by one thread and polled by another with no
  sync is broken in C/C++/Rust/Go/Java. Use an atomic with acquire/release
  (or the language default: Java `volatile`, Rust `AtomicBool` with
  `Release`/`Acquire`, Go — use `sync/atomic` or a channel; Go has no benign
  data races, the race detector is the law).
- `Relaxed`/`memory_order_relaxed` is for statistics counters only. If any
  control flow or other memory depends on the value, you need acq/rel. When
  in doubt, use the default sequentially-consistent ordering — correctness
  first, then profile.
- Double-checked locking is broken without an acquire-load/release-store on
  the pointer. Don't hand-roll it: use `Once`/`OnceLock`/`lazy_static`,
  `sync.Once`, Java holder idiom, `functools.cache` (rule 03).
- Python/JS note: the GIL / single thread gives you sequential consistency for
  free *within* the interpreter; this whole section applies to Python threads
  with free-threaded builds, to native extensions, and to shared
  ArrayBuffers (`Atomics.*` in JS).

```go
// BAD — data race AND visibility bug: `done` may never be observed as true
// by the reader (load hoisted out of the loop), and even when it is, the
// write to `result` is not guaranteed visible (no happens-before).
var result *Report
var done bool
go func() { result = compute(); done = true }()
for !done { runtime.Gosched() }   // may spin forever; result may be nil
use(result)

// GOOD — channel send happens-before receive: result is fully visible.
ch := make(chan *Report, 1)
go func() { ch <- compute() }()
use(<-ch)
```

The pattern generalizes: **publish via a synchronizing edge** (channel, lock,
atomic release-store, task join), never via a plain flag. The same bug in
Java is a non-volatile `done`; in C++ a non-atomic bool (UB); in Rust the
compiler simply refuses — which is the correct default to internalize.

## Deadlock

Requires all four Coffman conditions: mutual exclusion, hold-and-wait, no
preemption, circular wait. Break any one — in practice you break **circular
wait** (ordering) or **hold-and-wait** (timeouts/trylock).

```text
T1: lock(A); lock(B)        T2: lock(B); lock(A)     → cycle → deadlock
```

Prevention rules, in order of preference:
1. **One lock.** Two locks whose regions ever overlap are a design question
   before they are an ordering question.
2. **Global lock order.** Document it (e.g., "always account-lock before
   ledger-lock; for two accounts, lock lower ID first"). Enforce in debug
   builds if possible (lock-rank assertions, `tokio-console`, TSan).
3. **Never call unknown code while holding a lock** — callbacks, virtual
   methods, user-supplied closures, logging frameworks that might lock, and
   above all `await` (rule 07: lock-across-await).
4. **Hierarchies with timeouts at the boundary.** `try_lock` with backoff
   only as a last resort — it converts deadlock into livelock if everyone
   retries in lockstep (add jitter).

```python
# GOOD — canonical two-resource ordering by stable key.
first, second = sorted((acct_a, acct_b), key=lambda a: a.id)
with first.lock, second.lock:
    transfer(acct_a, acct_b, amt)
```

Async deadlocks need no locks at all: two tasks awaiting each other's results;
a task awaiting a queue item that only it would produce; `gather` inside a
handler waiting on a pool slot held by its own parent; a bounded channel
where the consumer awaits the producer's reply on the same full channel.
Audit the **wait-for graph**, not just the mutexes.

## Livelock & starvation

- **Livelock:** everyone busy, nobody progresses (mutual try-lock-retry,
  retry storms in lockstep). Fix: randomized backoff (jitter), or impose
  asymmetry (one side wins ties).
- **Starvation:** some waiter never gets the resource. Sources: RwLock
  readers starving writers (rule 03), unfair locks under high contention,
  priority tasks monopolizing the loop, a `select` with a always-ready branch
  shadowing others. Fix: fair/queued primitives, bounded work per turn, or
  explicit rotation.

## Idempotency under retries

Every retry (and every at-least-once queue) means your operation can execute
**twice**. Concurrency-correct code is idempotent or deduplicated:

- Give mutations an **idempotency key**; store processed keys transactionally
  with the effect (same DB transaction — a separate "did I do this" check is
  TOCTOU again).
- Make handlers natural no-ops on repeat: `SET state = 'shipped'` not
  `count = count + 1`; `INSERT ... ON CONFLICT DO NOTHING`.
- A timeout does **not** mean the operation failed — it means you don't know.
  Retrying a timed-out non-idempotent call without a dedupe key is a
  double-charge bug (HIGH in audits).

## Async-signal-safety (pointer)

Signal handlers (POSIX) may only call async-signal-safe functions — no
malloc, no printf, no locks (the interrupted thread may hold them → instant
self-deadlock). Correct pattern: handler sets a `volatile sig_atomic_t` flag
or writes one byte to a self-pipe/eventfd; the main loop reacts. In
Python/Go/Rust runtimes, use the runtime's signal facilities
(`asyncio.loop.add_signal_handler`, `signal.Notify`, `tokio::signal`) — never
do real work in a raw handler.

## Audit checklist

- [ ] Every shared mutable variable: what synchronizes it? "It's just a flag"
      is not an answer in threaded code (visibility).
- [ ] Grep for read-modify-write on shared state: `+=`, `count++`,
      `x = x + `, `append` to shared slices/lists from multiple tasks.
- [ ] Check-then-act sweep: `if ... in` / `exists(` / `has(` followed by a
      mutation of the same state — is the pair atomic? Includes
      filesystem (TOCTOU) and SELECT-then-INSERT.
- [ ] In async code: any await between a check and its act? Any shared-state
      invariant temporarily broken across an await?
- [ ] Multi-lock code: is there a documented order? Build the lock graph from
      the call sites; any cycle is CRITICAL.
- [ ] Callbacks/virtual calls/logging/awaits inside lock regions?
- [ ] Wait-for cycles without locks: tasks awaiting each other, self-feeding
      queues, pool-within-pool acquisition.
- [ ] RwLocks under write-heavy or reader-storm load: starvation analysis.
- [ ] Every retried or queued mutation: idempotency key or natural
      idempotence? Timeout treated as "unknown", not "failed"?
- [ ] Atomics: any load-then-write-based-on-it that isn't a CAS loop? Any
      `Relaxed` ordering guarding non-counter data?
- [ ] Signal handlers doing more than set-flag/write-byte?

# 03 — Primitives: Locks, Semaphores, Condvars, Channels, Select, Once

## Picking the primitive

| Need | Use | Not |
|---|---|---|
| Protect invariant across fields | Mutex (one, spanning the invariant) | Two atomics |
| Single-word counter/flag | Atomic | Mutex |
| Bound concurrent access to N | Semaphore | Spin-checking a counter |
| Hand data between tasks | Bounded channel/queue | Shared list + lock + sleep-poll |
| Wait for a state change | Condvar (with predicate loop) or channel/event | Polling loop |
| Read-mostly shared data | RwLock — or better, immutable snapshot swap (arc-swap, copy-on-write) | RwLock by reflex |
| One-time init | Once/OnceLock/lazy | Hand-rolled double-checked locking |

Prefer the highest-level primitive that fits: channel > semaphore > mutex >
atomic > condvar (condvars are last because they're the easiest to misuse).

## Mutex vs RwLock — RwLock is not "mutex but faster"

- RwLock pays for its bookkeeping; under short critical sections a plain mutex
  often outperforms it. Choose RwLock only for **read-heavy, long-ish reads,
  measurable contention**.
- **Writer starvation:** with reader-preferring RwLocks, a continuous stream
  of readers can block a writer indefinitely. Writer-preferring
  implementations (and most modern defaults: Rust `parking_lot`, Go
  `sync.RWMutex`) instead make *new readers* wait once a writer queues — which
  means **recursive read-lock acquisition deadlocks** (reader holds read lock,
  writer queues, reader re-acquires read → waits behind writer → cycle).
  Never re-acquire a read lock you might already hold.
- Read-mostly data is often better served lock-free: build a new immutable
  snapshot, atomically swap the pointer (`ArcSwap`, `AtomicReference`,
  Go `atomic.Pointer`). Readers never block; writers pay for copy.

```rust
// BAD — recursive read lock; deadlocks when a writer queues between them.
fn total(&self) -> u64 {
    let g = self.map.read();
    g.keys().map(|k| self.get(k)).sum()   // get() also takes map.read()
}
```

Upgradable locks (read→write upgrade) deadlock when two readers both try to
upgrade. If your design "needs" an upgrade, restructure: release, re-acquire
write, **re-validate** (state may have changed — TOCTOU otherwise).

## Semaphores: the bounding primitive

A semaphore's job in modern code is **admission control**: cap in-flight
work, connections, memory-heavy operations.

```python
# GOOD — cap concurrent outbound calls without restructuring the caller.
sem = asyncio.Semaphore(20)
async def call(api):
    async with sem:                  # context manager: released on raise too
        return await api.get()
```

Rules:
- Release on every path — context manager / `defer` / RAII, never bare
  `acquire` ... `release` pairs separated by raisable code.
- Acquire with the request's timeout/cancellation, not unconditionally:
  a semaphore queue with no deadline converts overload into infinite latency
  (rule 06 — shed instead).
- Don't acquire a second semaphore (or pool) while holding one unless ordering
  is global (same deadlock rules as locks). Pool-within-pool (HTTP pool inside
  DB-transaction holder) is a classic production deadlock.
- Weighted semaphores (`golang.org/x/sync/semaphore`) for memory-proportional
  bounding: acquire `n = estimated_bytes`, not 1.

## Condition variables: two famous bugs

**Bug 1 — spurious wakeups & stolen wakeups.** `wait()` can return without a
notify, and another thread may consume the state between notify and your
wakeup. Therefore: **always wait in a loop re-checking the predicate.**

```java
// BAD                                 // GOOD
synchronized (lock) {                  synchronized (lock) {
    if (queue.isEmpty())                   while (queue.isEmpty())
        lock.wait();                           lock.wait();
    item = queue.remove();                 item = queue.remove();
}                                      }
```

**Bug 2 — lost notify.** Notifying before the waiter waits (or mutating the
predicate outside the mutex) means the waiter sleeps forever. Therefore:
**mutate the predicate and notify while holding the same mutex the waiter
checks under.** The predicate-loop also defends against this — the waiter
checks state before sleeping.

`notify_one` vs `notify_all`: `notify_one` is an optimization that is only
correct when all waiters are interchangeable and one item satisfies exactly
one waiter. When waiters wait for different conditions on one condvar, or you
are not sure: `notify_all` (correctness first; thundering herd is rule 06's
problem).

In async code, prefer events/channels over condvars; if you use
`asyncio.Condition`, the same predicate-loop law applies (`await
cond.wait_for(pred)` encodes it).

## Channels & queues: bounded or it's a bug

**An unbounded channel is a memory leak with extra steps.** If the producer
is ever faster than the consumer — and under load it will be — the queue
absorbs the difference until OOM. Bounding turns the failure into visible
backpressure at the source (rule 06).

```go
// BAD — unbounded buffering via goroutine-per-send, or huge buffer "to be safe"
ch := make(chan Event, 1_000_000)

// GOOD — small bound; the send blocks (or selects to drop) when behind.
ch := make(chan Event, 128)
select {
case ch <- ev:
case <-ctx.Done():
    return ctx.Err()
}
```

```rust
// BAD: tokio::sync::mpsc::unbounded_channel() in a request path.
// GOOD: mpsc::channel(cap) — and handle `send().await` taking time,
// or try_send + shed when latency matters more than completeness.
```

Channel discipline:
- **Capacity is a policy decision**: 0/1 = handoff/rendezvous (tightest
  coupling, best backpressure), small N = burst absorption, large N = latency
  hiding + delayed failure. Document the choice.
- **Close from the producer side only**; receivers detect closure
  (`for range ch`, `recv() -> None`). Go: send on closed channel panics;
  multiple producers need a `sync.WaitGroup` + closer goroutine, or don't
  close at all and signal via context.
- **Every blocking send/receive pairs with cancellation** (select on
  ctx.Done / `asyncio.wait_for` / AbortSignal), or it's a leak when the other
  side is gone. A goroutine blocked forever on a channel nobody reads is the
  canonical Go leak.
- Python `asyncio.Queue`: `Queue(maxsize=N)` — default is unbounded. Use
  `queue.join()` + `task_done()` for drain semantics; remember `get()` must be
  cancellable during shutdown.
- JS has no native channel; bounded behavior comes from async iterators with
  pull semantics or a library — `Array.push` into a shared array consumed by
  an interval is an unbounded queue in disguise.

## Select / race patterns

`select` (Go), `tokio::select!`, `asyncio.wait(FIRST_COMPLETED)`,
`Promise.race` — wait on multiple sources, proceed with the first.

Hazards:
- **Loser leakage.** `Promise.race([op(), timeout()])` does not cancel the
  loser: `op()` keeps running, holding sockets. Pair race with cancellation —
  AbortSignal into `op`, cancel the losing asyncio task, drop the future in
  Rust (where drop *is* cancel — see below).
- **Partial-completion loss (tokio):** `select!` drops the non-winning
  futures. If a dropped future had buffered state (half-read message,
  half-sent write), it's gone. Loop-with-select must keep such futures
  outside the loop or use cancellation-safe operations only — check each
  awaited method's cancellation-safety documentation. This is the #1 subtle
  tokio bug class.
- **Starvation by ordering:** Go's `select` picks randomly among ready cases
  (good); `tokio::select!` is also randomized by default; hand-rolled
  "check A then B" loops starve B. If one branch must win ties (e.g.,
  shutdown), use `biased;` (tokio) or check it explicitly first.
- **asyncio:** `asyncio.wait(..., return_when=FIRST_COMPLETED)` returns
  pending tasks — you must cancel and await them. Forgetting is fire-and-forget.

```python
# GOOD — race with proper loser cleanup.
done, pending = await asyncio.wait(
    {asyncio.create_task(primary()), asyncio.create_task(fallback())},
    return_when=asyncio.FIRST_COMPLETED)
for t in pending:
    t.cancel()
await asyncio.gather(*pending, return_exceptions=True)  # reap, don't leak
result = done.pop().result()
```

## Once / lazy init

One-time initialization under concurrency must be a primitive, not a flag:

- Rust: `OnceLock` / `LazyLock`. Go: `sync.Once` / `sync.OnceValue`.
  Java: holder class idiom or `volatile` + DCL (prefer the holder).
  Python threads: `threading.Lock` around init or module import-time init.
  JS single-threaded: a memoized promise.
- **Memoize the promise/future, not the value** in async code — otherwise N
  concurrent first-callers each run init (the cache-stampede micro version):

```js
// BAD — three concurrent callers → three connections, last one wins.
let conn;
async function getConn() {
  if (!conn) conn = await connect();   // await between check and act
  return conn;
}

// GOOD — store the promise synchronously; everyone awaits the same one.
let connPromise;
function getConn() {
  if (!connPromise) {
    connPromise = connect().catch(e => { connPromise = undefined; throw e; });
  }
  return connPromise;                  // note: failure resets for retry
}
```

- Beware **init that blocks inside Once** while other threads wait on it:
  if init can call back into something needing the same Once (or lock),
  that's a self-deadlock; `sync.Once` re-entry deadlocks.

## Reentrancy & lock scope hygiene

- Recursive/reentrant mutexes paper over design problems and break invariant
  reasoning (the invariant may be broken when you re-enter). Avoid; refactor
  into `fooLocked()` internal methods called under the lock.
- Keep critical sections minimal: compute outside, lock, mutate, unlock. No
  I/O, no allocation-heavy work, no logging, no callbacks, no awaits inside.
- Guard data, not code: name the lock after the data it protects and keep
  them adjacent (`struct { mu sync.Mutex; cache map[...]... }`).

## Audit checklist

- [ ] Unbounded anything: `Queue()` no maxsize, `unbounded_channel`,
      `make(chan T, huge)`, arrays used as queues, channels with
      goroutine-per-send. Each is MEDIUM minimum; HIGH on network-fed paths.
- [ ] Condvar waits: `if` instead of `while` around `wait()` (HIGH);
      predicate mutated outside the mutex; `notify_one` with heterogeneous
      waiters.
- [ ] RwLock: recursive read acquisition; reader-storm writer starvation;
      upgrade patterns; would snapshot-swap be simpler?
- [ ] Semaphores: released on all paths (RAII/finally)? Acquired with
      timeout/cancellation? Nested under another semaphore/pool?
- [ ] Channel sends/receives without a cancellation branch; close() called
      from consumer side or from multiple producers (Go panic).
- [ ] `Promise.race` / `select!` / `wait(FIRST_COMPLETED)`: are losers
      cancelled and reaped? Are tokio `select!` arms cancellation-safe?
- [ ] Lazy init: value memoized instead of future (stampede)? Failed init
      cached forever (permanent outage)? DCL without acquire/release?
- [ ] Critical sections containing I/O, awaits, callbacks, or logging.
- [ ] Reentrant locks or `fooLocked` conventions violated (double-acquire).

# 04 — Event-Loop Hygiene: Never Block the Loop

## The contract

An event loop multiplexes thousands of tasks onto one thread. Any task that
holds the thread for more than ~10–50ms steals latency from **every** other
task: timers fire late, heartbeats miss, health checks fail, sockets back up.
One blocking call in one handler degrades the whole process. There is no
"just this once".

Latency math: a 200ms sync call on a loop serving 500 req/s queues ~100
requests behind it. Blocking is a tail-latency multiplier, not a local cost.

## The blocklist — what must never run on the loop thread

1. **Sync I/O:** file reads/writes (`open`/`read`, `fs.readFileSync`),
   sync HTTP clients (`requests`, `urllib`, `http.client`), sync DB drivers
   (psycopg2 without async wrapper, `sqlite3`, blocking JDBC), `subprocess.run`,
   `socket` without non-blocking mode. Note: on Linux, regular-file I/O is
   *always* potentially blocking — there is no readiness API for files; even
   "fast" reads stall on cold page cache or NFS.
2. **Sync sleeps:** `time.sleep` (use `asyncio.sleep`), `std::thread::sleep`
   in async fns (use `tokio::time::sleep`), busy-wait loops.
3. **CPU-bound work:** parsing/serializing multi-MB JSON, compression,
   image/video processing, template rendering of huge documents, regex
   catastrophic backtracking, crypto: `bcrypt.hashSync`, `pbkdf2Sync`,
   `scrypt` sync variants, RSA keygen. Password hashing is *designed* to take
   ~100ms — it is the canonical loop-killer.
4. **Blocking lock acquisition** on a contended threading mutex
   (`threading.Lock.acquire` in a coroutine, `Mutex::lock` of `std::sync` in
   async Rust when contended), and `queue.Queue.get()`/`.join()` from a
   coroutine.
5. **Sync DNS:** Node's `dns.lookup` uses a tiny libuv threadpool (default 4)
   — slow DNS exhausts it and stalls *fs and crypto too*; size
   `UV_THREADPOOL_SIZE` or use `dns.resolve`.

```python
# BAD — three loop-blockers in one handler.
async def register(req):
    data = requests.post(KYC_URL, json=req.json).json()   # sync HTTP
    pw = bcrypt.hashpw(req.password, bcrypt.gensalt())     # CPU ~100ms
    time.sleep(0.1)                                        # sync sleep
    ...

# GOOD
async def register(req):
    async with httpx.AsyncClient() as c:
        data = (await c.post(KYC_URL, json=req.json)).json()
    pw = await asyncio.to_thread(bcrypt.hashpw, req.password, bcrypt.gensalt())
    await asyncio.sleep(0.1)
```

## Offloading correctly

| Runtime | I/O-blocking call | CPU-bound work |
|---|---|---|
| Python asyncio | `asyncio.to_thread` / `loop.run_in_executor(None, f)` | `run_in_executor(ProcessPoolExecutor)` (GIL); thread pool OK on free-threaded 3.13+ |
| Node | rewrite to async API (almost always exists) | `worker_threads` / `piscina` pool |
| Rust tokio | `tokio::task::spawn_blocking` | `spawn_blocking` for occasional; `rayon` pool + channel back for heavy |
| Go | nothing — runtime parks blocked goroutines | nothing special; cap with semaphore if it floods cores |
| JVM virtual threads | fine, *unless* pinned: native calls pin the carrier; `synchronized` pins only on JDK ≤23 (JEP 491 fixed it in 24+) — on JDK 21 LTS use `ReentrantLock` | platform-thread pool sized to cores |

Offloading rules:
- **Bound the offload pool and its queue.** `spawn_blocking` has a large
  default cap (512) — that's 512 concurrent blocking calls before
  backpressure; put a semaphore in front for expensive operations.
  Python's default executor: similar story (`min(32, cpu+4)` threads, unbounded queue).
- **Data crossing to a process pool is serialized** — don't ship 1GB to save
  50ms of compute. Sometimes the sync path is the right call: measure.
- **Don't offload trivially small work**; thread-hop overhead (~µs–ms) can
  exceed the work. The threshold that matters is the p99 of the call, not the
  mean — offload anything that *can* take >10ms.
- **Values returned from workers are snapshots.** Don't mutate shared loop
  state from worker threads; return data and apply it on the loop
  (`call_soon_threadsafe` / channel back / `postMessage`).

## Long-task chunking

When CPU work must run on the loop (small, frequent, not worth a pool),
chunk it so other tasks interleave:

```js
// BAD — 5M-row aggregation freezes the loop for seconds.
for (const row of rows) total += score(row);

// GOOD — yield to the macrotask queue every N items.
let i = 0;
for (const row of rows) {
  total += score(row);
  if (++i % 10_000 === 0) await new Promise(r => setImmediate(r));
}
```

```python
# Python equivalent: await asyncio.sleep(0) every N iterations.
```

Caveats: `await asyncio.sleep(0)` / `setImmediate` yields a *turn*, it doesn't
bound the chunk's own cost — size N so each chunk stays under ~10ms. Chunking
mutable shared state reintroduces interleaving races (rule 02): your data
structure is now visible mid-aggregation.

## Microtask vs macrotask (JS — and the asyncio analogue)

- **Microtasks** (promise callbacks, `queueMicrotask`, `await` resumption)
  run to exhaustion after the current task, *before* rendering, I/O events,
  or timers. A microtask that schedules microtasks in a loop **starves the
  event loop completely** — worse than a sync loop, and `await` won't save
  you because resolved-promise `await` only hops the microtask queue:

```js
// BAD — this never yields to I/O; `await` of an already-resolved promise
// stays inside the microtask queue.
while (!done) { await Promise.resolve(); doSmallStep(); }

// GOOD — setImmediate/setTimeout(0) reaches the macrotask queue,
// letting I/O and timers run.
while (!done) { await new Promise(r => setImmediate(r)); doSmallStep(); }
```

- **Macrotasks** (`setTimeout`, `setImmediate`, I/O callbacks) interleave with
  the loop phases. Use them to yield; use microtasks only for ordering within
  a tick.
- `process.nextTick` runs even before microtasks — recursive `nextTick`
  starves everything; treat it as internals-only.
- **asyncio analogue:** `call_soon` callbacks run in FIFO per iteration —
  closer to macrotask behavior, so `await asyncio.sleep(0)` does yield to I/O.
  But a coroutine that awaits only immediately-ready awaitables can still hog;
  the loop runs ready callbacks in batches.

## Sync-over-async: the self-deadlock

Blocking the loop on a result *the loop itself must produce* doesn't just
degrade — it deadlocks instantly. The loop thread waits on a future; the
future needs the loop thread to advance; nothing ever runs again.

```python
# BAD — called from a coroutine (so the loop thread):
fut = asyncio.run_coroutine_threadsafe(work(), loop)
data = fut.result()          # loop thread blocks waiting on itself → hang

# BAD — RuntimeError at best, deadlock pattern at heart:
loop.run_until_complete(work())   # inside a running loop
```

```rust
// BAD — panics on tokio ("Cannot block_on inside a runtime") or
// deadlocks a current-thread runtime.
let data = futures::executor::block_on(fetch(url));
```

The C# classic is `.Result`/`.Wait()` on a task needing the captured context;
the JS analogue is `Atomics.wait` on the main thread. Variants to flag:

- Sync facade over async internals: `def get(self): return
  asyncio.get_event_loop().run_until_complete(self._aget())` — works in
  scripts, deadlocks when called from inside a server's loop.
- Worker thread calling back into the loop **synchronously** and the loop
  meanwhile blocking on the worker pool (pool ↔ loop cycle). Offloaded
  functions must be pure sync: no `fut.result()` back into the loop with the
  loop's own thread saturated waiting for the pool.
- Library code that "detects" a running loop and silently spins a nested one
  (`nest_asyncio`) — a workaround that hides the architecture bug; flag it.

**Rule:** the boundary between sync and async worlds belongs at the top of
the program (one `asyncio.run` / `#[tokio::main]` / single runtime), not
sprinkled through the call graph. Crossings mid-stack are findings.

## Detecting a blocked loop (build it in; check for it in audits)

- **Python:** run with `loop.set_debug(True)` + `loop.slow_callback_duration =
  0.05` in staging — logs any callback >50ms with its name. `aiodebug` /
  `aiomonitor` for production. On 3.14+, `python -m asyncio ps|pstree PID`
  inspects a live process's task tree (`pstree` also reports await-graph
  cycles, i.e. async deadlocks); `asyncio.print_call_graph()` in-process.
- **Node:** monitor event-loop lag/utilization (`perf_hooks.monitorEventLoopDelay`,
  `performance.eventLoopUtilization()`); alert at p99 lag >100ms.
  `blocked-at` pinpoints offenders in dev.
- **Rust:** `tokio-console` shows tasks with long poll times; a single poll
  >100ms is a blocking bug.
- An audit of an async service that has **no loop-lag metric** should flag
  that as a MEDIUM observability finding by itself.

## Sneaky blockers (audit-grade list)

- Logging handlers doing sync file/network writes on the loop (Python
  `logging` to a slow disk; use `QueueHandler`).
- `os.getaddrinfo` via sync resolution inside "async" libs; certificate
  loading; `random.SystemRandom`/`/dev/random` stalls.
- Lazy module imports inside handlers (Python import lock + disk I/O on
  first request).
- ORMs: "async" facades over sync drivers offload to a hidden, small,
  unbounded-queue threadpool — know your driver (asyncpg ✓; psycopg2 ✗).
- `JSON.parse`/`json.loads` of multi-MB payloads; `JSON.stringify` of huge
  responses — these are CPU items 3, not I/O.
- Accidental sync fallback: `await maybe_async()` where a code path returns a
  plain value computed synchronously for 300ms.
- Prometheus/metrics endpoints rendering huge text on the loop.

## Audit checklist

- [ ] Grep async modules for: `time.sleep`, `requests.`, `urllib`,
      `subprocess.run`, `open(`, `.read()`/`.write()` on files, `sqlite3`,
      `psycopg2`, `boto3` (sync), `Sync(` suffixed Node APIs, `hashSync`,
      `pbkdf2Sync`, `execSync`, `std::thread::sleep`, `std::sync::Mutex` in
      hot async paths, `reqwest::blocking`.
- [ ] Sync-over-async crossings mid-stack: `run_until_complete`/`block_on`/
      `.result()`/`.Result` reachable from loop threads (deadlock-capable:
      CRITICAL); `nest_asyncio` anywhere.
- [ ] Password hashing / crypto in request handlers: offloaded? (HIGH if not.)
- [ ] Large JSON/template/compression work on the loop thread?
- [ ] Offload pools: bounded? Sized deliberately? Semaphore in front of
      `spawn_blocking`/executor for expensive ops?
- [ ] Worker results applied to shared state via loop-safe mechanism
      (`call_soon_threadsafe`, channels, `postMessage`) — not direct mutation
      from the worker thread?
- [ ] Long loops over unbounded input without a yield point (and is the
      chunked state safe to observe mid-flight)?
- [ ] JS: recursive microtask/`nextTick` patterns; `await Promise.resolve()`
      used as a "yield" (it isn't).
- [ ] Node `dns.lookup` on hot paths / `UV_THREADPOOL_SIZE` left at 4 with
      heavy fs+crypto+dns use?
- [ ] Loop-lag metric exported and alerted on? Debug slow-callback logging
      available in staging?

# 05 — Cancellation, Timeouts & Shutdown Sequencing

## Every await needs a timeout policy

An await with no deadline is a promise to wait forever. Networks drop ACKs,
peers hang half-open, pools exhaust, locks queue — and your task waits
eternally, holding its own resources (this is how one hung dependency
cascades into total exhaustion).

**Policy, not reflex:** each await either (a) has an explicit timeout, or
(b) demonstrably inherits a deadline from an enclosing scope (request
deadline, task-group timeout, ctx with deadline), or (c) has a comment
defending "forever" (e.g., the main accept loop). In audits, (a|b|c) must
hold for every external call: HTTP, DB, queue ops, lock/semaphore acquires,
`channel.recv`, `process.wait`.

```python
# BAD — fetch can hang forever; so does everyone awaiting this handler.
data = await client.get(url)

# GOOD — scoped deadline covering the whole operation tree (Python 3.11+).
async with asyncio.timeout(2.0):
    data = await client.get(url)
```

```go
// GOOD — deadline rides the context through every layer.
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
data, err := client.Get(ctx, url)
```

Deadline design rules:
- **Deadlines propagate; timeouts don't compose.** Prefer one deadline at the
  boundary (request entry) that flows down, over N stacked per-call timeouts
  that can sum to more than the caller will wait. Inner calls take
  `min(own_limit, remaining_budget)`.
- **Connect timeout ≠ read timeout ≠ total timeout.** A "30s timeout" that is
  only a connect timeout still hangs forever on a stalled body. Set total
  operation deadlines.
- Timeout firing means **outcome unknown**, not failure — pair with
  idempotency before retrying (rule 02).
- A timeout that fires must also **cancel the underlying work** —
  `Promise.race` against a timer abandons the loser, it doesn't stop it
  (rule 03). Use `AbortSignal.timeout(ms)` and pass the signal into fetch;
  use `asyncio.timeout` (which cancels the task); use ctx (the callee must
  honor it — verify it does).

## Cancellation propagation — the mechanisms

| Runtime | Mechanism | Semantics |
|---|---|---|
| Python asyncio | `task.cancel()` → `CancelledError` raised at the next await point | Preemptive-at-awaits; catchable; MUST re-raise |
| Go | `context.Context` cancelled → `ctx.Done()` closes | Fully cooperative; only code that checks ctx stops |
| JS | `AbortSignal` → abort event / `signal.aborted` | Cooperative; only APIs accepting the signal stop |
| Rust async | dropping a future cancels it; nothing polls = nothing runs | Implicit at every await; sync code inside is unstoppable |
| .NET/Java | `CancellationToken` / `Thread.interrupt` | Cooperative; interrupt sets flag + wakes blocking ops |

Consequences:
- **Go/JS:** a call chain is only cancellable if **every** layer threads the
  ctx/signal through. A single `context.Background()` or signal-less fetch in
  the middle breaks the chain — grep for these in audits.
- **Python:** every `await` is a potential `CancelledError` raise site. Code
  must be exception-safe at each await (use `try/finally`, async context
  managers).
- **Rust:** every `.await` is a potential drop site. State held across awaits
  must be drop-safe; side effects "after the await" may never run. This is
  cancellation-safety (rule 03's `select!` discussion).

```python
# BAD — swallows cancellation; the task becomes unkillable and
# shutdown hangs on it.
while True:
    try:
        await poll()
    except Exception:           # CancelledError inherits BaseException in
        continue                # 3.8+, but bare `except:` or asyncio code
                                # catching BaseException still eats it
# Also BAD:
    except BaseException:
        log(...)                # caught CancelledError, didn't re-raise

# GOOD — clean up, then re-raise; cancellation is not an error.
try:
    await poll()
except asyncio.CancelledError:
    await release_lease()       # bounded cleanup only
    raise
```

## Cleanup on cancel

Cancellation arrives at the worst time by definition. Guarantees you must
preserve:

- **Resource release:** `finally` / `defer` / RAII / async context managers
  around every acquire. In Python, the `finally` block of a cancelled
  coroutine runs — but if it awaits, it can be cancelled *again*; keep
  cleanup short or shield it.
- **Shielding:** for must-complete sections (commit after the money moved),
  `asyncio.shield(commit())` — and still await it with an outer bound;
  shield-everything is just uncancellable code. Go: pass a *detached* context
  with its own short timeout (`context.WithoutCancel(ctx)` + WithTimeout) for
  cleanup RPCs — cleanup using the already-cancelled ctx instantly fails,
  a classic bug:

```go
// BAD — ctx is already cancelled; the rollback never happens.
defer store.Rollback(ctx, tx)

// GOOD — cleanup gets its own small budget, detached from the dead ctx.
defer func() {
    cctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second)
    defer cancel()
    store.Rollback(cctx, tx)
}()
```

- **Invariant restoration:** if a task mutates shared state in steps,
  cancellation between steps leaves it broken. Make mutations transactional,
  or apply them at a single commit point after all awaits.
- **Don't start what you can't stop:** spawning subprocesses / remote jobs
  requires a kill path wired to cancellation (`process.kill` on abort,
  `ctx`-aware job API).

## Shutdown sequencing

Graceful shutdown is a protocol, not a `kill`. Canonical order:

```
1. Trap signal (SIGTERM)            — flip "shutting down" flag
2. Stop accepting new work          — close listener / unsubscribe / fail
                                      readiness probe FIRST, then wait one
                                      probe interval so LBs stop routing
3. Drain in-flight work             — with a deadline (e.g., 0.8 × the
                                      orchestrator's grace period)
4. Cancel stragglers                — propagate cancellation, await briefly
5. Flush & release                  — buffers, queues, DB pools, files
6. Exit                             — nonzero if drain was forced
```

```python
# GOOD — asyncio skeleton.
async def main():
    stop = asyncio.Event()
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, stop.set)

    server = await start_server()
    await stop.wait()

    server.close()                      # 2: stop accepting
    await server.wait_closed()
    try:
        async with asyncio.timeout(25): # 3: drain (k8s grace 30s)
            await inflight.join()
    except TimeoutError:
        pass
    for t in background_tasks:          # 4: cancel daemons/stragglers
        t.cancel()
    await asyncio.gather(*background_tasks, return_exceptions=True)
    await pool.aclose()                 # 5: flush/release
```

Ordering pitfalls:
- **Closing the queue before draining it** drops accepted work; **draining
  without a deadline** hangs shutdown forever (then SIGKILL corrupts state
  anyway — your deadline must undercut the orchestrator's).
- **Producer/consumer order:** stop producers first, drain consumers, then
  stop consumers. Stopping consumers first deadlocks producers blocked on a
  full bounded queue.
- **Dependency order on release:** cancel tasks *before* closing the pools
  they use, or cancellation cleanup hits closed connections.
- Workers blocked on `queue.get()` need a wakeup: sentinel values (one per
  consumer), closing the channel (Go: `for range` exits), or cancelling the
  getter task.

## Retry interplay

Cancellation and timeout must beat retry logic: a retry loop that ignores
ctx turns one cancelled request into N background attempts.

```go
// GOOD — every retry checks the context; backoff sleep is cancellable.
for attempt := 0; attempt < max; attempt++ {
    if err := op(ctx); err == nil || !retryable(err) || ctx.Err() != nil {
        return err
    }
    select {
    case <-time.After(backoff(attempt)):   // jittered
    case <-ctx.Done():
        return ctx.Err()
    }
}
```

Full retry-storm analysis is in rules 06/07; the rule here: **the deadline is
the retry budget** — retries fit inside the caller's remaining time, never
extend it.

## Audit checklist

- [ ] Sweep external awaits (HTTP, DB, queue, lock acquire, recv, wait):
      timeout, inherited deadline, or justifying comment? Unbounded await on
      a network peer is HIGH.
- [ ] Timeouts: total-operation or just connect? Stacked timeouts that exceed
      the caller's deadline?
- [ ] Timeout firing actually cancels the work (AbortSignal wired, asyncio
      timeout context, ctx honored by callee) — or does the loser keep running?
- [ ] Grep Go for `context.Background()`/`context.TODO()` outside main/tests —
      each one severs the cancellation chain.
- [ ] Grep Python for `except BaseException`, bare `except:`, or
      `except Exception` in retry/cleanup loops around awaits — does
      CancelledError survive and re-raise?
- [ ] Cleanup paths using the already-cancelled context/signal (rollback that
      can never run)?
- [ ] `finally` blocks with unbounded awaits (cancellable cleanup hanging
      shutdown)? `shield` usage bounded?
- [ ] Shutdown: readiness flipped before listener close? Drain deadline <
      orchestrator grace? Producers stopped before consumers? Sentinels/
      closure to wake blocked getters? Background daemons cancelled and
      awaited?
- [ ] Retry loops: ctx/signal checked per attempt? Backoff sleep cancellable?
      Retries within the deadline budget?
- [ ] Subprocesses/remote jobs killed on cancellation?

# 06 — Backpressure & Flow Control

## The law of rate mismatch

Whenever a producer can outpace a consumer — and under load, some producer
always can — the difference goes somewhere: memory (unbounded queue → OOM),
latency (bounded queue → growing wait), or rejection (shedding). **You don't
choose whether to have backpressure; you choose whether it's controlled.**
Uncontrolled backpressure looks like: heap growth, GC death spirals, queue
latencies in minutes, then a crash that loses everything buffered.

Design question for every queue/channel/buffer: *what happens when it's
full?* If the answer is "it can't fill up", the design is wrong.

## Bounded queues everywhere

Every queue gets a capacity and a full-policy:

| Full-policy | Behavior | Use when |
|---|---|---|
| **Block (backpressure)** | Producer waits; pressure propagates upstream to the true source | Producer can tolerate waiting; source is throttleable (TCP does this for you) |
| **Drop-new (shed)** | Reject the incoming item, tell the caller | Request/response systems — fail fast beats slow death |
| **Drop-old (slide)** | Evict oldest, keep newest | Telemetry, frames, tickers — freshness beats completeness |
| **Coalesce/conflate** | Merge into pending item (keep-latest, sum deltas) | State updates where only the latest matters |

```python
# BAD — unbounded; a slow consumer turns this into an OOM timer.
q = asyncio.Queue()

# GOOD — bounded, and the enqueue point makes the full-policy explicit.
q = asyncio.Queue(maxsize=256)
try:
    q.put_nowait(item)              # shed:
except asyncio.QueueFull:
    metrics.dropped.inc()
    raise ServiceOverloaded()       # ...or `await q.put(item)` to block
```

Sizing: capacity ≈ `consumer_rate × tolerable_queue_delay` (Little's law),
plus burst headroom. A queue of 10,000 in front of a 100/s consumer is a
100-second latency reservoir — big buffers don't add throughput, they hide
overload and delay the inevitable. Default small (dozens–hundreds); grow only
with a measured reason.

**Blocking-policy deadlock warning:** block-on-full inside a cycle (consumer
also produces into the same queue, or A→B and B→A both bounded-blocking)
deadlocks when both fill. Cycles need shed/coalesce somewhere, or strictly
acyclic flow.

## End-to-end propagation

Backpressure only works if the pressure reaches the **source**. Audit the
chain: socket → parser → queue → workers → DB. One unbounded link (or one
fire-and-forget spawn per item — rule 07) breaks the chain, absorbing
pressure invisibly until OOM.

```go
// BAD — "absorb with goroutines": unbounded queue in disguise.
for msg := range kafkaMessages {
    go handle(msg)              // 50k msgs/s × 2s handling = 100k goroutines
}

// GOOD — worker pool: consumption rate is explicitly capped; the Kafka
// consumer naturally pauses (stops polling) when workers are busy.
sem := make(chan struct{}, 64)
for msg := range kafkaMessages {
    sem <- struct{}{}           // blocks when 64 in flight
    go func(m Msg) { defer func() { <-sem }(); handle(m) }(msg)
}
```

Natural backpressure carriers — don't defeat them:
- **TCP flow control:** not reading from a socket slows the sender. Reading
  eagerly into an unbounded buffer ("to free the socket") destroys this.
- **Pull-based consumption:** polling (Kafka, SQS) is inherently
  backpressured — poll only when capacity exists. Push-based (raw WebSocket,
  gRPC stream w/o flow control config) needs explicit bounding.
- **AMQP prefetch (`basic.qos`):** RabbitMQ *pushes*; the prefetch count (cap
  on unacked deliveries) is the demand signal. Unset/0 = unlimited prefetch =
  an unbounded buffer in the consumer. Always set it explicitly (broker-side
  semantics: sota-architecture rules/03 §7b).
- **HTTP server concurrency limits:** cap in-flight requests at the listener
  (semaphore/middleware); past the cap, 503 immediately.

## Load shedding

When demand exceeds capacity, serving fewer requests *well* beats serving all
requests *too late to matter*. A response delivered after the client's
timeout is pure waste — you paid the cost and got zero utility ("goodput").

- **Shed early, shed cheap:** reject at admission (queue full, semaphore
  unavailable, deadline already insufficient) before parsing/auth/DB work.
- **Deadline-aware:** if `remaining_budget < expected_cost`, drop now —
  the caller has already given up by the time you'd answer.
- **Signal correctly:** 503/429 + `Retry-After`; gRPC `RESOURCE_EXHAUSTED`.
  Distinguishable from real failures so clients back off rather than retry
  hot (rule 07: retry storms).
- **Prioritize:** shed batch/analytics before interactive; health checks
  never shed (or the orchestrator kills an overloaded-but-alive pod —
  conversely, *readiness* should fail under overload to divert traffic).

## Rate limiting: token bucket as the workhorse

Token bucket = sustained rate `r` + burst capacity `b`. Use it for: outbound
calls to dependencies (protect *them*), per-tenant fairness (protect *you*),
and retry/error-path throttling.

```go
// GOOD — golang.org/x/time/rate; Wait blocks (backpressure),
// Allow sheds. Pick per the full-policy table above.
lim := rate.NewLimiter(rate.Limit(100), 200) // 100/s, burst 200
if err := lim.Wait(ctx); err != nil { return err }  // cancellable!
```

Notes:
- Rate limiters **smooth**, they don't bound concurrency — a 100/s limit with
  10s handlers is 1000 in flight. Pair limiter (rate) with semaphore
  (concurrency); you usually need both.
- Per-tenant buckets need an eviction story (LRU of buckets) or they're an
  unbounded map — the leak you built to prevent leaks.
- Distributed limiting (Redis token bucket) is approximate; design for slight
  overshoot rather than coordinating exactly.
- **Adaptive concurrency** (AIMD / gradient — Netflix concurrency-limits
  style) outperforms static limits when downstream capacity varies; SOTA for
  service meshes, overkill for a single worker pool.

## Streaming with pull-based demand

The streaming SOTA is **demand-driven (pull) flow**: the consumer signals
capacity; the producer sends at most that much. Push-with-buffering is the
legacy failure mode.

- **Reactive Streams / RxJava / Reactor:** `request(n)` is the demand signal;
  an operator chain without bounded demand (`onBackpressureBuffer()`
  unbounded) is a leak.
- **Node streams:** respect `write()`'s return value — `false` means stop and
  wait for `'drain'`. Ignoring it buffers unboundedly in the writable's
  internal queue. Prefer `pipeline()`/`pipe()`, which wire this up (and
  propagate errors/teardown), over hand-rolled `on('data', ...)` +
  `write(...)` pairs. `for await (const chunk of readable)` pauses the source
  between iterations: pull semantics for free.
- **Async iterators/generators (Python & JS)** are naturally pull-based —
  the producer runs only when the consumer awaits the next item. Converting a
  callback/push source into an async iterator requires an internal queue:
  bound it and choose a full-policy (this is where "adapter" libraries leak).
- **gRPC/HTTP2:** per-stream flow-control windows give transport-level
  backpressure — but only if your handler awaits sends and reads lazily
  rather than spooling the whole stream into memory.

```js
// BAD — push: data events keep firing regardless of write capacity.
src.on("data", chunk => dst.write(transform(chunk)));

// GOOD — pull with drain handling, teardown and error propagation.
await pipeline(src, transformStream, dst);
```

```python
# GOOD — pull-based pipeline: each stage runs only on demand.
async def transformed(source):
    async for item in source:        # awaits = demand signal upstream
        yield transform(item)
```

**Slow-consumer policy for fan-out (pub/sub, WebSockets):** one slow
subscriber must not buffer the world or stall the broadcast. Per-subscriber
bounded queue + drop-old/disconnect-on-lag (this is why Go's `rxgo`-style
broadcast and NATS use lag-based eviction). Decide per endpoint: kill the
laggard, or degrade its stream.

## Circuit breakers: backpressure for failure

Retry budgets slow the bleeding; a circuit breaker stops calling a dependency
that is down, failing fast locally and giving it room to recover.

States: **closed** (normal; count failures over a rolling window) →
**open** (error rate over threshold, e.g., >50% of ≥20 calls in 10s: reject
immediately for a cooldown) → **half-open** (after cooldown, admit a few
probe calls; success closes, failure re-opens).

Implementation rules:
- Trip on **error rate over a minimum volume**, not consecutive-failure
  counts (one slow burst trips a count-based breaker spuriously).
- Treat timeouts as failures — a dependency answering slowly at 100%
  occupancy is *down* for capacity purposes.
- Half-open probes must be **bounded** (1–N concurrent), or the reopen is a
  thundering herd onto a convalescent service.
- Scope per dependency *endpoint/shard*, not globally — one bad shard
  shouldn't open the breaker for nine healthy ones.
- The breaker-open error must be distinguishable (and non-retryable at this
  layer) so callers shed or degrade rather than hammer.
- Breaker state is shared mutable state read on every call: use atomics or a
  lock-free snapshot, not a mutex on the hot path (rule 03).

```python
# GOOD — shape of the call site: breaker wraps the timeout, which wraps the op.
if not breaker.allow():
    raise DependencyUnavailable(retry_after=breaker.cooldown_remaining())
try:
    async with asyncio.timeout(0.5):
        result = await dep.call(req)
except (TimeoutError, DependencyError) as e:
    breaker.record_failure()
    raise
breaker.record_success()
```

## Batching & coalescing

Batching raises throughput (amortized syscalls/commits) at the cost of
latency. Correct batcher shape: flush on **size OR time, whichever first**
(`max_batch=100, max_delay=10ms`), bounded pending buffer, flush on shutdown
(rule 05). Size-only batchers strand the tail; time-only batchers under-fill
at high rates. Coalescing (keep-latest per key) is the strongest defense for
state-update streams: the queue can't exceed the keyspace.

## Audit checklist

- [ ] Inventory every queue/channel/buffer (incl. implicit ones: goroutine
      spawns per item, promise arrays, stream internal buffers, per-subscriber
      send queues). Each has: a bound? a full-policy? a stated size rationale?
- [ ] Unbounded queue reachable from network input = HIGH (CRITICAL if
      multiplied per-connection/per-tenant).
- [ ] Trace pressure source→sink: where does it stop propagating? Any
      fire-and-forget or eager-read-into-buffer that defeats TCP/pull
      semantics?
- [ ] Bounded-blocking queues inside cycles (deadlock) — wait-for graph check.
- [ ] Admission control on servers: in-flight cap, early shedding before
      expensive work, deadline-aware drops, correct 429/503 + Retry-After?
- [ ] Health/readiness behavior under overload: liveness must pass, readiness
      should shed.
- [ ] Rate limiters paired with concurrency limits? Per-tenant limiter maps
      evicted? Limiter waits cancellable (ctx in `Wait`)?
- [ ] Critical dependencies behind breakers? Rate-based tripping, timeouts
      counted as failures, bounded half-open probes, per-endpoint scope?
- [ ] Node: `write()` return value honored or `pipeline()` used? Custom
      `on('data')` handlers without pause/resume?
- [ ] Push→pull adapters (callback to async-iterator bridges): internal queue
      bounded?
- [ ] AMQP consumers: prefetch count set explicitly (no unlimited prefetch),
      sized to handler speed?
- [ ] Fan-out endpoints: slow-consumer policy defined (drop, lag-kick), or
      can one dead-slow WebSocket OOM the broadcaster?
- [ ] Batchers: size+time flush, bounded pending, shutdown flush?

# 07 — Audit Bug Catalog: Signatures, Severity, Fixes

How to use: grep for the signature, read each hit in context, prove the
interleaving (which two orders of execution diverge, and what breaks), then
report in the SKILL.md finding format. Severities below are baselines —
escalate for money/auth/durability state.

## 1. Fire-and-forget tasks swallowing exceptions — HIGH

**Signature greps:** `asyncio.create_task(` / `ensure_future(` with unused
return; bare `go func(` with no errgroup/WaitGroup/recover; `tokio::spawn(`
with dropped handle; `somethingAsync();` statement-position calls in JS;
`.then(` with no `.catch(`/second arg; C# `async void`, bare `Task.Run`.

**Failure mode:** the task fails, nobody observes it: silent data loss
(audit logs that never wrote, caches never invalidated), plus resource leak
if the dead task held connections. In Node, unhandled rejection kills the
process (default since v15) — a crash triggered at a *random later* tick, far
from the cause. In Go, a panic in a bare goroutine kills the process with no
request context. CPython extra: `create_task` holds only a weak ref — an
unreferenced running task can be GC'd mid-execution.

**Fix:** structured scope (TaskGroup/errgroup/JoinSet — rule 01); if truly
detached, attach an error handler + shutdown registration + comment.

```python
asyncio.create_task(send_webhook(evt))          # BAD
tg.create_task(send_webhook(evt))               # GOOD (inside TaskGroup)
```

## 2. Missing await — HIGH (often CRITICAL in tests)

**Signature:** calling an async function and using/discarding the result
without await: `result = fetch_user(id)` then `result.name` (you read a
coroutine/promise attribute → AttributeError or `[object Promise]`);
statement-position async calls; `return someAsync()` inside `try` (the
rejection escapes the catch — return-await matters in try blocks);
`if has_permission(user):` — **a coroutine/promise is always truthy**, so the
check always passes (CRITICAL: auth bypass); `forEach(async x => ...)` — JS
forEach ignores the returned promises, the loop "completes" with nothing done;
async test functions not awaited → test always passes.

**Detection:** TS `@typescript-eslint/no-floating-promises` +
`no-misused-promises`; Python `RuntimeWarning: coroutine ... was never
awaited` in logs, `asyncio` debug mode; Rust: `#[must_use]` on futures makes
this a compiler warning — heed it.

**Fix:** await it; for JS loops use `for...of` with await or
`Promise.all(arr.map(...))` (bounded — rule 01).

## 3. Shared mutable state across tasks — HIGH→CRITICAL

**Signature:** module/global-level dicts/lists/maps mutated in handlers;
`this.cache` / `self.sessions` mutated by concurrent requests; closure
variables captured by multiple spawned tasks; Go maps written from multiple
goroutines (fatal: `concurrent map writes` crash — run `-race`); loop
variables captured by reference in spawned closures (pre-Go-1.22 `for i`;
Python `lambda: f(i)` late binding; JS `var`).

**Failure mode:** lost updates, corrupted aggregates, cross-request data
bleed (one user sees another's data — CRITICAL/security), map-write crashes.
Remember: single-threaded async only protects *between* awaits; any structure
read before an await and written after is a race window (rule 02).

**Fix:** confine to one owner task + channel (rule 01); or immutable
snapshots; or a lock spanning the invariant; request-scoped state instead of
shared (contextvars / AsyncLocalStorage / explicit parameter).

```python
# BAD — interleaved handlers corrupt the running aggregate.
stats["total"] += order.amount          # read-modify-write at await scale
# GOOD — single mutation point, owned by one consumer task.
await stats_queue.put(order.amount)
```

## 4. Lock held across await — HIGH (deadlock-capable: CRITICAL)

**Signature:** `async with lock:` body containing `await` of I/O; Rust
`std::sync::Mutex`/`parking_lot` guard alive across `.await` (tokio's clippy
lint `await_holding_lock`); JS "lock" promises chained around fetches; any
`mutex.lock()` ... `await` ... `unlock()` sequence.

**Failure mode:** (a) throughput collapse — every contender parks for the
full I/O duration; (b) deadlock — the awaited operation (directly or via the
pool/queue it needs) requires the same lock or a resource held by a task
waiting on this lock; (c) in Rust, a `std` MutexGuard held across await can
block the *executor thread* when another task on the same thread contends —
freezing unrelated tasks.

**Fix:** narrow the critical section — copy what you need under the lock,
await outside, re-acquire to write (and **re-validate**: state may have
changed). If the await must be inside, use an async-aware lock and document
why; for Rust, `tokio::sync::Mutex` is the escape hatch but usually the
design wants a channel/owner-task instead.

```rust
// BAD
let g = STATE.lock().unwrap();
let val = fetch_remote(g.key).await;     // guard across await
// GOOD
let key = { STATE.lock().unwrap().key.clone() };
let val = fetch_remote(key).await;
{ let mut g = STATE.lock().unwrap(); if g.key == key { g.val = val; } }
```

## 5. Blocking call in async context — HIGH

Full treatment in rule 04. Grep list: `time.sleep`, `requests.`, `open(`,
`subprocess.run`, sync DB drivers, `readFileSync`, `execSync`, `hashSync`,
`pbkdf2Sync`, `std::thread::sleep`, `reqwest::blocking`, `block_on(` inside
async (instant deadlock on single-threaded runtimes; panics on tokio),
`.result()`/`.get()` on a future from loop thread, `loop.run_until_complete`
inside a running loop. Severity HIGH; CRITICAL when the blocked call awaits
something scheduled on the same loop (self-deadlock: e.g., sync-waiting a
future the loop must complete).

## 6. Retry storms — HIGH

**Signature:** retry loops with no backoff (`for attempt in range(5): try
... except: continue`); backoff without jitter; retries on every layer
(client retries × gateway retries × service retries = N³ amplification);
retrying non-retryable errors (400s, auth failures); no retry budget; retry
ignoring ctx/deadline (rule 05).

**Failure mode:** a downstream blip multiplies traffic exactly when capacity
is lowest, preventing recovery — the outage becomes self-sustaining
("metastable failure"). Synchronized retries (fixed backoff, cron alignment)
arrive in waves.

**Fix:** exponential backoff + **full jitter**
(`sleep(rand(0, min(cap, base·2^attempt)))`); retry budget (e.g., retries ≤
10% of requests — token bucket on the retry path); retry at **one** layer;
honor `Retry-After`; circuit breaker for persistent failure; classify errors
(retry 503/timeout, never 4xx except 429).

```python
# GOOD
for attempt in range(MAX):
    try:
        return await op()
    except RetryableError:
        if attempt == MAX - 1 or not retry_budget.try_acquire():
            raise
        await asyncio.sleep(random.uniform(0, min(CAP, BASE * 2**attempt)))
```

## 7. Thundering herd — MEDIUM→HIGH

**Variants & signatures:**
- **Cache stampede:** popular key expires → all requesters recompute
  simultaneously. Signature: `get → miss → compute → set` with TTL, no
  single-flight. Fix: request coalescing (`singleflight.Group`, memoized
  in-process future — rule 02's example), stale-while-revalidate, probabilistic
  early refresh (XFetch), lock-and-recompute-once.
- **Synchronized wakeups:** every instance polls/refreshes on the same cron
  second (`:00`), reconnects immediately after a broker restart. Fix: jitter
  every period and every reconnect delay.
- **notify_all/broadcast for one item:** N waiters wake, one wins, N−1
  re-sleep — at scale this is a CPU spike per event. Fix: `notify_one` when
  waiters are interchangeable (rule 03 caveats), or sharded queues.
- **Deploy/startup herd:** all pods warm caches/open connections at once.
  Fix: staggered rollout, jittered warmup.

## 8. Futures spawned in loops without bounding — HIGH

**Signature:** `gather(*[f(x) for x in items])` / `Promise.all(items.map(f))`
/ `for ...: tg.create_task(...)` / `for { go f() }` where `items` is
unbounded (DB rows, file lines, network input). Also: recursion that spawns
(crawlers), per-event spawn in subscription handlers.

**Failure mode:** memory (N stacks/promise chains), fd exhaustion (N
sockets), and a self-DoS of the downstream (N concurrent calls = you are the
thundering herd). Works in dev (N=10), dies in prod (N=10M).

**Fix:** semaphore-bounded fan-out, fixed worker pool over a bounded queue,
`errgroup.SetLimit`, chunked `Promise.all`, streaming instead of
collect-then-blast (rules 01, 06).

## 9. Leaked tasks & goroutines (blocked forever) — MEDIUM→HIGH

**Signature:** channel send/receive with no cancellation branch and a
receiver/sender that can exit early (rule 03); `queue.get()` workers with no
sentinel/cancel path; `Promise` executors whose resolve path can be skipped
(a promise that never settles parks every awaiter forever); event-listener
accumulation per request (`emitter.on` in handlers without `off` —
EventEmitter leak warnings); periodic timers never cleared.

**Detection in review:** for each blocking point ask "what guarantees the
other side shows up?" For each subscription/timer: "where is the matching
teardown?" Goroutine/task counts as a metric; `pprof` goroutine dumps;
`asyncio.all_tasks()` snapshots.

## 10. Double-execution & ordering assumptions — MEDIUM→HIGH

- **At-least-once handlers without idempotency** (rule 02): queue redelivery
  + side effects = duplicates. Look for consumer handlers doing
  `INSERT`/`send_email`/`charge` with no dedupe key.
- **Assumed FIFO across workers:** a queue is FIFO; N workers consuming it
  complete out of order. Look for "process events in order" logic running at
  concurrency > 1; per-key ordering needs partitioning (hash key → one
  worker/partition).
- **Ack-before-durable:** message acked, then process crashes before the
  side effect commits → silent loss. Ack must follow the commit (or use
  transactional outbox).
- **Time-of-check on task state:** `if not task.done(): task.cancel()` —
  fine; but `task.result()` after `done()` check without exception handling
  re-raises the task's exception where you didn't expect it.

## 11. Async iterator / generator cleanup — LOW→MEDIUM

`break`-ing out of an `async for` may leave the generator suspended;
finalization runs late (GC) or never — connections held open. Python: wrap
in `contextlib.aclosing(gen)`; JS: `for await` with `break` *does* call
`return()` — but a hand-rolled iterator must implement it. Rust streams:
dropping is fine (drop = cancel) but resources must be in Drop, not in
"after the loop" code.

## 12. Test-only / heisenbug smells — MEDIUM

`sleep(0.1)`-then-assert synchronization in tests or — worse — production
("wait a bit for the worker to pick it up"). Sleeps are a race with a
deadline: flaky in CI, broken under load. Replace with explicit
synchronization: events, joins, polling-with-timeout on the actual condition,
fake clocks (`tokio::time::pause`, `looptime`, Go's `testing/synctest`
bubbles — GA in 1.25, `synctest.Wait` replaces sleep-and-hope).
Production code that
"sleeps to let X finish" is finding-worthy as written — it encodes an
ordering assumption with no enforcement.

## Audit sweep order (efficient pass over a codebase)

1. **Topology first** (30 min): entry points, spawns, shared state, queues,
   locks. Write down the lock graph and channel graph.
2. **Greps, in descending hit-value:** blocking-in-async list (rule 04) →
   fire-and-forget signatures (#1) → unbounded queue/channel constructors
   (#8, rule 06) → lock-across-await (#4) → `context.Background`/missing
   AbortSignal (rule 05) → check-then-act on the shared state found in step 1
   (rule 02) → retry loops (#6).
3. **Read the shutdown path end-to-end** — it exercises cancellation,
   draining, and task ownership all at once; most codebases fail here.
4. **Read the hottest handler end-to-end** and prove its timeout chain.
5. Report with interleavings, not vibes.

## Audit checklist (meta — the whole catalog)

- [ ] #1 fire-and-forget: every spawn owned, errors observed
- [ ] #2 missing await: truthy-promise checks, forEach(async), return-in-try
- [ ] #3 shared mutable state: cross-request bleed, loop-var capture, Go maps
- [ ] #4 locks across await; Rust std guards across `.await`
- [ ] #5 blocking calls in async (rule 04 grep list); `block_on` in async
- [ ] #6 retries: backoff+jitter+budget+classification+ctx
- [ ] #7 herd: cache stampede single-flight; jittered periodics/reconnects
- [ ] #8 unbounded fan-out: gather/Promise.all/go-in-loop over external input
- [ ] #9 leaks: unsettleable promises, sentinel-less workers, listener/timer
      teardown
- [ ] #10 duplicates & ordering: idempotency keys, per-key partitioning,
      ack-after-commit
- [ ] #11 generator/stream cleanup on early exit
- [ ] #12 sleep-based synchronization anywhere
- [ ] Shutdown path read end-to-end; hottest path timeout chain proven