# 01 — Architecture Styles & Decision-Making

Rules for choosing an architecture style, recording decisions, and keeping the
architecture evolvable. Apply before writing the first service; re-apply at every
major boundary change.

## 1. Default to a modular monolith

**Rule:** Start every new system as a modular monolith: one deployable, internal
modules with explicit boundaries, separate schemas-per-module inside one database.
Extract services only when a *measured* force demands it.

**Rationale:** Network boundaries are the most expensive boundaries you can buy.
They convert function calls into failure modes (latency, partial failure, retries,
versioning). A modular monolith gives you boundary discipline at refactor cost,
not distributed-systems cost.

**Forces that justify extraction (need at least one, measured):**
- Independent scaling: one module needs 50x the replicas of the rest.
- Independent deployment cadence blocked by org structure (multiple teams stepping on one release train).
- Divergent runtime needs (GPU inference vs CRUD; different language/runtime).
- Hard fault isolation requirements (a crash in module A must not take down module B).
- Regulatory isolation (data residency, PCI scope reduction).

**Never extract because:** "microservices are best practice", résumé pressure,
"we might need to scale later", or to fix a tangled codebase (you'll get a
tangled distributed system — see rules/07, distributed monolith).

```text
GOOD: one repo, one deploy
  app/
    billing/      (public API: billing/api.*, everything else internal)
    catalog/
    shipping/
  Module imports only other modules' api/ surface. CI fails on deep imports.

BAD: 14 services, one team of 5, shared DB, lockstep deploys.
```

## 2. Enforce module boundaries mechanically

**Rule:** Boundaries that aren't enforced by tooling don't exist. Use import
linting / architecture tests (dependency-cruiser, ArchUnit, deptrac, Nx tags,
Go internal/ packages, or equivalent) in CI. A human code-review rule is not
enforcement.

**Rule:** Each module exposes exactly one public surface (an `api`/facade
package and/or published events). Cross-module calls go through that surface.
Cross-module data access goes through that surface — never through the other
module's tables.

## 3. Choose style by problem shape, not fashion

| Style | Wins when | Loses when |
|---|---|---|
| Modular monolith | Small/medium team, evolving domain, unknown load profile | Genuinely independent scaling/deploy needs across teams |
| Microservices | Many teams, independent deploy cadence, per-service scaling, mature platform (CI/CD, observability, on-call) | Team < ~3 squads, no platform engineering, chatty domain |
| Serverless (FaaS) | Spiky/bursty load, event glue, low ops budget, embarrassingly stateless handlers | Long-lived connections, latency-critical p99 (cold starts), heavy local state, cost at sustained high throughput |
| Event-driven backbone | Many consumers per fact, audit/replay needs, temporal decoupling | Request/response semantics forced through events (see rules/03) |

**Rule:** Serverless is an operational model, not an architecture. You still owe
module boundaries, idempotency, and observability. A pile of 200 lambdas sharing
a database is a distributed monolith with cold starts.

**Rule:** Never mix request/response and event-driven semantics blindly. Decide
per interaction: does the caller need an answer now (sync), or is it publishing
a fact (async)? Commands that need answers are sync or async-with-correlation;
facts are events.

## 4. Record every significant decision as an ADR

**Rule:** Any decision that is expensive to reverse (datastore, message broker,
service boundary, auth model, multi-tenancy model, sync vs async for a flow)
gets an Architecture Decision Record before implementation. Store ADRs in the
repo (`docs/adr/NNNN-title.md`), immutable once accepted; supersede, never edit
history.

**Minimum ADR format:**

```text
# NNNN: Use outbox pattern for order events
Status: Accepted (2026-03-02)  Supersedes: 0007
Context: Orders must emit events; dual-write to DB + broker loses events on crash.
Decision: Write events to outbox table in the same TX; relay publishes async.
Consequences: + atomicity, + replay; - eventual consistency (~1s lag), - relay to operate.
Alternatives considered: CDC (Debezium) — rejected: no ops capacity for Kafka Connect.
```

**Rationale:** ADRs are the only durable defense against re-litigating decisions
and against cargo-culting old constraints after they expire. "Why is it like
this?" must have a greppable answer.

**Rule:** An ADR without a *Consequences* section listing at least one downside
is marketing, not a decision record. Reject it in review.

**Directory discipline.** Sequential kebab-case filenames
(`0012-use-outbox-for-order-events.md`) plus an `index.md` holding one row per
ADR — number, title, status, date. The index is what makes the practice
auditable at a glance: a status column that is all `proposed` is a stalled
process, and a superseded chain is legible without opening a file. Commit the
ADR **in the PR that implements the decision** where one exists, so rationale
and code land together; a pre-implementation decision lands on its own.

## 4a. A control's cost profile must be checked against decisions already in force

A new control that consumes a **scarce resource on every use** — a hardware touch, a human
approval, an interactive prompt, a rate-limited or paid API call — can silently reintroduce
the exact friction an earlier decision removed. Both artifacts are locally correct; the
contradiction lives in the gap between them, and no test sees it.

Field-reported: a repository decided, with the reasoning written into `CONTRIBUTING`, to
stop signing every commit because the key sat on a touch-required token and the tap per
commit blocked automated work. About an hour later, in the same session, a gate ledger was
built whose `anchor` command signed its chain head with that same key — and anchoring was
recommended after every push. Every gate passed, every test passed, and the user found it.

**At the point of adding the control, write one line: *this costs X, Y times per Z*. Then
grep the repo's own decision records (§4) for X.** If a decision already rejected X at that
frequency, either the new control is wrong or the old decision needs revisiting — the two
cannot both stand unexamined. The tell that you missed it is a user asking *"why am I being
asked for this again?"*, which is late and expensive.

This is a **coherence** failure, not a code defect, so it is invisible per-artifact and to
per-file review. It is worst in agent-authored work: an agent can hold both decisions in
one context and still miss the interaction, because attention is on the artifact being
built, not on the policy it lands inside.

## 4b. A deferral is a standing question — give it somewhere to accumulate answers

**Rule:** When a decision record defers an idea behind a condition ("revisit if…",
"reconsider when we see…", "not yet — wait for a second case"), the record must also say
**where the evidence toward that condition accrues, and who reads it**. A trigger with no
evidence store is not a decision that will be revisited; it is a decision that will be
re-derived from scratch by whoever next asks, at full cost, with no memory of the last
answer.

**Rationale:** The deferral itself is usually right — one instance is a trade-off, not a
pattern, and generalising from it produces rules that fit one project. What fails is the
return path. The evidence arrives incrementally, months apart, in places the record does
not watch; each near-miss is recognised, judged not-quite-enough, and forgotten. The next
person repeats the whole search to reach the same verdict, so the deferral never converges
either way. Cost is asymmetric and invisible: writing the trigger takes a sentence, and
answering it can take a full sweep of everything logged since.

**What the record must carry**, next to the trigger:

- **The trigger in falsifiable terms.** "A second implementation *ships* this design" is
  checkable; "if it becomes a problem" is not, and cannot ever be closed.
- **A running list of candidates checked, with dates and verdicts** — including the ones
  that *did not* count. A near-miss and a deliberate refusal are both evidence about the
  class, and both are lost by default.
- **What the trigger would buy.** If part of the idea is already covered elsewhere, say
  which part, so a future instance is judged on the remainder rather than re-argued whole.

**The same rule covers a deliberately-unfixed state kept as evidence** — a known-bad left
in place to prove a control fires, a pin left stale so an automated bump proves the
automation runs. That is a good technique and an experiment, so it carries an experiment's
obligation: write down where the result will show up and who looks. **An experiment with
no scheduled read-back is indistinguishable from a note**, and it decays the same way —
the state resolves, nobody returns, and the record still describes the world before the
answer arrived. Reviewers reading it then act on a question that is already closed.

**Smell:** a decisions log where several entries say "revisit when…" and none of them has
ever been revisited. Check the dates: if the oldest trigger predates the last two people
who joined, the log is recording intent, not process.

## 5. Practice evolutionary architecture with fitness functions

**Rule:** Encode architectural qualities as automated, continuously-run checks
(fitness functions). If a quality matters, test it; if you can't test it, you
can't claim it.

**Examples of fitness functions (run in CI or scheduled):**
- Dependency direction: "no module imports `billing/internal`" — architecture test.
- Coupling budget: cyclic dependencies between modules = build failure.
- Latency: p99 of checkout API < 300 ms under k6 load profile — perf gate on main.
- Resilience: weekly chaos run kills one replica of each service; SLOs must hold (see rules/04).
- Cost: per-tenant infra cost stays under $X — scheduled report with alert.
- Schema safety: migration linter forbids destructive DDL without expand/contract.

**Rule:** When two qualities conflict (e.g., consistency vs availability), the
ADR picks the winner per context; the fitness function enforces the chosen
trade-off, not both.

## 6. Plan for reversibility; classify decisions by exit cost

**Rule:** Classify every decision: **Type 1** (hard to reverse: datastore,
broker, cloud provider, tenancy model, public API contract) vs **Type 2**
(cheap to reverse: library, internal interface, queue topology detail).
Spend design effort proportionally. Type 2 decisions get minutes and a code
comment; Type 1 decisions get an ADR, a spike, and an exit strategy.

**Rule:** For every Type 1 dependency, write down the exit strategy in the ADR
("we wrap the broker behind `EventBus` port; migration = reimplement adapter +
dual-publish for N days"). Don't build a full abstraction layer preemptively —
a thin port is enough (see rules/02 on ports).

## 7. Conway's law: design team and system boundaries together

**Rule:** Service boundaries that cross team boundaries will erode. Align one
service (or module) to one owning team; shared ownership means no ownership.
If the org chart and the architecture disagree, change one of them deliberately
(inverse Conway maneuver) — don't let the disagreement fester.

**Rule:** Each module/service has a single on-call/owning team recorded in a
machine-readable catalog (`catalog-info.yaml`, CODEOWNERS, or equivalent).
"Orphaned service" is a Critical audit finding.

## 8. Extraction playbook (monolith → service)

**Rule:** Extract via strangler fig, never big-bang rewrite:
1. Harden the module boundary in-process (own schema, api-only access, events).
2. Add an anti-corruption layer at the seam if models differ.
3. Route a slice of traffic to the new service behind a flag; compare outputs (shadow/dark launch).
4. Migrate data with expand/contract: dual-write or CDC, backfill, verify, cut over, contract.
5. Delete the old path. Extraction isn't done until the old code is deleted.

**Rule:** Never extract two things at once (e.g., new service AND new datastore
AND new language). One variable per migration.

## 9. Buy/adopt vs build

**Rule:** Build only what differentiates the business. Auth, payments, search,
feature flags, observability, workflow engines: adopt proven solutions and wrap
them behind a thin port. Building commodity infrastructure is a Type 1 decision
disguised as a weekend project.

**Rule:** Adopted dependencies still get an ADR (lock-in is a consequence) and
a fitness function (e.g., "all calls to vendor X go through adapter Y" —
architecture test).

## 10. Edge composition: gateways and BFFs

**Rule:** Clients never call internal services directly. Put an API gateway at
the edge for cross-cutting concerns (authn, rate limiting, TLS, routing) and —
when client needs diverge (mobile vs web vs partner API) — a Backend-for-
Frontend per client class that composes internal calls into client-shaped
responses.

**Rule:** Keep business logic out of the gateway. A gateway that transforms
payloads, enforces domain rules, or orchestrates workflows is an unowned god
service (rules/07 §3) written in YAML. Gateways route and protect; BFFs compose;
services decide.

**Rule:** Each BFF is owned by the client team it serves. A shared BFF for all
clients recreates the coupling BFFs exist to remove.

## 11. Diagrams and design docs as code

**Rule:** Maintain a current C4-style picture: one system-context diagram and
one container diagram minimum, stored in the repo as text (Mermaid, Structurizr
DSL, PlantUML) so diffs are reviewable and rot is visible in PRs. A diagram in a
wiki dies in a quarter; a diagram in the repo dies in review.

**Rule:** Significant designs get a short RFC/design doc *before* implementation
(problem, constraints, options, recommendation), circulated to affected teams
with a comment deadline. The ADR (§4) records the outcome; the RFC records the
debate. Skip the RFC for Type 2 decisions — process must be proportional too.

## 12. Sacrificial architecture and rewrite discipline

**Rule:** Accept that successful systems outgrow their architecture (~10x scale
changes the right design). Write code expecting parts to be replaced: boundaries
and contracts are the durable assets, implementations are sacrificial. Optimize
boundary quality over implementation polish.

**Rule:** Never green-light a full rewrite while the old system keeps taking
features ("second-system" trap). A rewrite must be: scoped to one bounded
context at a time, strangler-style (§8), with a feature freeze on the replaced
slice and a kill date for the old path. "Big rewrite, both evolve in parallel"
fails at a rate that rounds to always.

## 13. Architecture review cadence

**Rule:** Review the architecture on triggers, not just calendars: 10x traffic
growth, new compliance regime, team doubling, p99 SLO erosion two quarters
running, or a third incident with the same structural cause. Each review:
re-validate prior ADRs' assumptions (load numbers, team size, vendor constraints)
and explicitly supersede the ones whose context expired.

**Rule:** Track architecture debt in the same backlog as features, each item
tied to a measurable symptom (incident class, lead-time drag, cost line).
"Refactor someday" items without symptoms get deleted, not hoarded.

## Audit checklist

- [ ] **Every deferral with a "revisit if…" trigger names where the evidence accrues and
      who reads it** (§4b), and the trigger is stated so that some observation could close
      it. Probe: list every deferred entry, then ask when each was last checked and what
      was found — a trigger nobody has evaluated since it was written, or one no evidence
      could ever satisfy, is a finding. Same probe for a known-bad deliberately left in
      place as proof a control fires: if no one can say where the result appears, it is a
      note, not an experiment.

- [ ] **Every control that spends a scarce resource per use states its cost profile**
      (*what, how often*), and that frequency was checked against the ADRs and contributor
      docs (§4). A control whose per-use cost contradicts a decision already in force is a
      finding even though it works.

- Is there a written rationale (ADR) for the current architecture style, with consequences and alternatives?
- Could this system be a modular monolith? If it's microservices, can the team name the measured force that justified each extraction?
- Are module/service boundaries enforced by CI tooling (import rules, architecture tests), not just convention?
- Does any module reach into another module's internals or tables directly?
- Do services deploy independently in practice (check release history), or do they ship in lockstep?
- Is every Type 1 decision (datastore, broker, tenancy, public contracts) covered by an ADR with an exit strategy?
- Are ADRs immutable and superseded rather than edited? Is the most recent ADR less than ~3 months old (i.e., is the practice alive)?
- Are architectural qualities (latency, coupling, resilience, cost) encoded as automated fitness functions that run in CI or on a schedule?
- Does every service/module have exactly one owning team, recorded machine-readably?
- Were recent extractions done strangler-style with a deleted old path, or do zombie code paths remain?
- Is any commodity capability (auth, flags, queues, search) hand-built without an ADR justifying it?
- Do sync vs async interaction choices match the semantics (answers vs facts), or are request/response flows tunneled through events?
- Do clients reach internal services directly, or through a gateway/BFF? Does the gateway contain business logic?
- Are context/container diagrams stored as text in the repo and current (spot-check three services against the diagram)?
- Is any rewrite running big-bang style with parallel feature development on old and new?
- Do superseded ADRs exist (evidence assumptions get re-validated), or has nothing been revisited since launch?

# 02 — Domain Modeling & Boundaries (DDD, Hexagonal, Clean Layering)

Rules for carving the domain into bounded contexts, modeling aggregates, and
keeping dependencies pointed the right way. These rules apply identically inside
a monolith module and across services.

## 1. Bounded contexts first, services second

**Rule:** Identify bounded contexts (areas where a model and its language are
internally consistent) before drawing any service or module boundary. A service
boundary that splits a bounded context, or fuses two, will generate chatty calls
and translation bugs forever.

**Rule:** One term, one meaning per context. If "Customer" means payer in
Billing and recipient in Shipping, those are two models in two contexts —
never one shared `Customer` class with 40 nullable fields.

```text
BAD:  shared `Customer` entity used by Billing, Shipping, Support
      → every change fans out to all three; fields nobody owns.
GOOD: Billing.Payer, Shipping.Recipient, Support.Contact
      Each context maps from `CustomerRegistered` events into its own model.
```

**Rule:** Maintain a context map: which contexts exist, who owns each, and the
relationship type at every seam (customer/supplier, conformist, anti-corruption
layer, published language, separate ways). Unlabeled seams default to accidental
conformist coupling.

## 2. Use ubiquitous language end to end

**Rule:** Code, schema, APIs, and events use the domain experts' vocabulary
exactly. If the business says "quote expires," the code says `quote.expire()`,
not `setStatus(3)`. Translation between business language and code language is
where requirements bugs breed.

**Rule:** When the business distinguishes two things, the model distinguishes
them (no boolean flags standing in for missing concepts: `isReturn && isExchange`
means you're missing a `ResolutionType`).

## 3. Aggregates: small, consistent, the only write path

**Rule:** An aggregate is a consistency boundary: the set of objects that must
be transactionally consistent with each other. Keep aggregates small — usually
one entity plus value objects. Big aggregates serialize writes (lock contention)
and bloat loads.

**Rule:** One transaction modifies one aggregate. Cross-aggregate consistency is
eventual, coordinated by domain events or sagas (rules/03). If you "need" to
update two aggregates atomically, your boundary is wrong or the invariant is
weaker than you think — interrogate the invariant first.

**Rule:** Reference other aggregates by ID, never by object pointer. Loading an
order must not drag the customer, the product catalog, and the warehouse into
memory.

**Rule:** All writes go through aggregate methods that enforce invariants.
No service-layer code mutating aggregate fields directly; no "anemic" model
where entities are bags of getters and the rules live in 12 service classes.

```text
GOOD: order.addLine(product_id, qty)   # enforces max-lines, status checks inside
BAD:  order.lines.append(line); order.total += price   # invariants enforced nowhere
```

**Rule:** Enforce uniqueness and cross-aggregate invariants at the right layer:
DB unique constraints for identity invariants; reservation patterns or eventual
checks for the rest. Don't pretend an aggregate can guard an invariant spanning
millions of rows.

## 4. Hexagonal architecture: ports and adapters

**Rule:** The domain core depends on nothing but itself. Define **ports**
(interfaces the core needs: `OrderRepository`, `PaymentGateway`, `Clock`,
`EventPublisher`) inside the core; implement **adapters** (Postgres, Stripe,
Kafka, system clock) outside it. Frameworks, ORMs, and SDK types never appear
in core signatures.

**Rationale:** This is what makes the core testable without infrastructure and
makes Type 1 dependencies (rules/01 §6) swappable. It is also the cheapest
insurance against vendor lock-in you can buy.

**Rule:** Ports are defined by the consumer's need, not the provider's API.
`PaymentGateway.charge(order_id, amount) -> ChargeResult`, not a passthrough of
Stripe's 40-field request object. A port that mirrors the vendor SDK is a leaky
abstraction (rules/07).

```text
        inbound adapters              core                outbound adapters
  HTTP handler ──► CommandHandler ──► Domain ──► OrderRepository (port)
  Kafka consumer ─► (use case)        model      └─► PostgresOrderRepo (adapter)
                                          └─► PaymentGateway (port)
                                               └─► StripeAdapter
```

## 5. Layering and the dependency rule

**Rule:** Dependencies point inward only: `adapters → application (use cases) →
domain`. Nothing in domain imports application; nothing in application imports
adapters. Enforce with architecture tests (rules/01 §2), not convention.

**Rule:** Use cases (application services) orchestrate: load aggregate, call
domain method, persist, publish events. They contain no business rules and no
I/O details. If a use case has an `if` encoding a business policy, push it into
the domain; if it builds SQL, push it into an adapter.

**Rule:** Don't over-layer. A CRUD-only context doesn't need aggregates, ports,
and four layers — a transaction script over a table is honest and cheaper.
Match modeling rigor to domain complexity: core domains get full DDD; supporting
and generic subdomains get the simplest thing that works. Applying ceremony
uniformly is itself an anti-pattern.

## 6. Anti-corruption layers at external seams

**Rule:** Every integration with an external system or legacy model goes through
an anti-corruption layer (ACL): a translator that maps their model to yours at
the boundary. External DTOs, vendor enums, and legacy field names stop at the
ACL; they never propagate into the domain.

**Rule:** When consuming another context's events, translate to your own types
in the consumer adapter. Storing someone else's event payload as your domain
state couples your model to their release schedule.

## 7. Domain events as first-class outputs

**Rule:** Aggregates record domain events (`OrderPlaced`, `QuoteExpired`) as
part of their state changes; the application layer publishes them after commit
(via outbox — rules/03 §4). Events are named in past tense, in ubiquitous
language, and carry the data consumers need without forcing a callback query
for every field ("event-carried state transfer" for hot fields, IDs for the rest).

**Rule:** Events published across context boundaries are a public contract:
versioned, schema-checked, additive-only changes by default (see rules/03 §8).
Internal events can change freely; published ones cannot.

## 8. Value objects and invariant-encoding types

**Rule:** Model domain values as types that cannot exist in invalid states:
`Money(amount, currency)`, `EmailAddress`, `DateRange(start <= end)`. Parse,
don't validate — convert raw input into rich types once at the boundary, then
trust the types everywhere inside.

**Rule:** Never represent money as a float, identity as a bare string/int passed
through five layers, or a state machine as a string column with stringly-typed
transitions. Encode the state machine: explicit states, explicit allowed
transitions, transition methods on the aggregate.

## 8a. Absence is not a value — the in-band sentinel

**Rule:** Encode "absent", "unknown" and "failed" **outside** the value's domain —
`Option`/`Maybe`, a nullable type, a second return value, `NULL`, an omitted
property, or an exception. Never a member of the domain itself: `-1`, `0`, `""`,
`9999-12-31`, `0.0.0.0`, `MAX_INT`.

It is the same defect as a stringly-typed state machine (§8), and it is harder to
see because it **type-checks**. Three consequences, none of which any compiler
reports:

- **Distinct causes collapse into one value.** *Absent* and *malformed* returned as
  the same constant can never be told apart, counted, or alerted on downstream — so
  a parser regression upstream is indistinguishable from ordinary sparse data.
- **Presence checks silently pass.** `if x:` / `if (x)` reads as "do I have a
  value?" and admits `-1`, which is truthy in every language that has a truthiness
  rule. `0` fails the same check on a legitimate value. Neither spelling can work,
  because the value carries no absence information to test.
- **It has an ordering, so it changes answers rather than crashing.** A sentinel
  **loses** every `<` against a real value and **wins** every `>`. Wherever a
  number is compared as a proxy for sequence — line/offset numbers for "happens
  before", version numbers, timestamps-as-ints, retry counts — one missing operand
  flips the predicate, and *which way* depends on which side went missing:

  ```text
  use(-1) > alloc(20)  -> False   # fails closed: the real relation is not reported
  use(20) > alloc(-1)  -> True    # fails OPEN: reported, on an unknown operand
  ```

  Both from one input, in one comparison. (Measured, Python 3.14; the arithmetic is
  the same in every language whose sentinel is an ordinary number.)

**Rule:** When a wire format, a fixed-width store, or a stdlib you don't own forces
a sentinel, it is **per field with its domain written down** — not one constant
applied across a set of fields whose domains differ. That last shape is the common
one and the unrecoverable one: on a field where the producer *also* emits the
sentinel legitimately, a stored value is ambiguous forever and no downstream care
recovers it.

**Auditing it.** The constant is a poor signal — it is everywhere and usually fine.
Three probes, decreasing precision:

1. **Producer:** one function returning the *same* constant from a not-found branch
   and from an error/`except` branch. Near-zero false positives; fix it once and
   every caller is fixed.
2. **Asymmetric guard** — the highest-yield tell. A comparison where **one operand
   is filtered against the sentinel and the other is not**
   (`min(x for x in xs if x > 0) < line_num`, where `line_num` is unfiltered).
   Sentinel-filtering is applied per site, so it lands wherever the author was
   thinking about it and is omitted everywhere else. Visible in a diff.
3. **Truthiness-as-presence:** `if n:` guarding a number whose domain includes the
   sentinel. High recall, high false-positive rate — use it to build a reading
   list, not a gate.

**A value-based lint is only sound on a field with a stated non-negative (or
otherwise sentinel-free) domain.** Measure the per-field distribution before
linting; a field where the sentinel is 99% of rows and one where it is 0% are
different problems, and a field whose producer emits it legitimately makes such a
lint 100% false-positive. Language-specific stdlib traps and the idiomatic
alternative: each `sota-<language>` skill. Persistence: `sota-databases` rules/01.

## 9. Repositories and persistence

**Rule:** One repository per aggregate (not per table). Repositories return
whole aggregates, accept whole aggregates, speak domain language
(`findOverdueInvoices()`, not `query(sql)`).

**Rule:** The database schema serves the aggregate, not the other way around.
ORM convenience (lazy loading across aggregates, shared base entities, bi-
directional mappings spanning contexts) must not redraw your consistency
boundaries. If the ORM fights the aggregate shape, map manually at the adapter.

**Rule:** Use optimistic concurrency (version column) on aggregates by default.
Lost updates are a modeling failure, not a tuning detail.

## 10. Explicit state machines

**Rule:** Any entity with a lifecycle (order, subscription, claim, deployment)
gets an explicit state machine: enumerated states, enumerated transitions,
transition methods that reject invalid moves, and a domain event per transition.

```text
GOOD:
  states: Draft -> Submitted -> Approved -> Fulfilled
                      └-> Rejected (terminal)
  order.approve(approver)   # raises if state != Submitted; emits OrderApproved

BAD:
  order.status = "approved"   # any code can set any string at any time
  if order.status in ("approved", "aproved", "APPROVED"): ...
```

**Rule:** Persist the state machine honestly: a `state` column with a CHECK/enum
constraint, plus a transition history table (who/when/why) for anything audited
or money-touching. Reconstructing "how did it get into this state" from logs is
not a design.

## 11. Domain services, factories, and where logic goes

**Rule:** Logic placement, in priority order: (1) value object, (2) aggregate
method, (3) domain service — only for rules genuinely spanning aggregates
(`PricingPolicy`, `TransferService`), (4) application use case — orchestration
only. Each step down loses cohesion; justify the descent.

**Rule:** Complex aggregate creation goes through a factory (function or class)
that returns only valid instances; "create empty then set fields" construction
makes invalid intermediate states representable and they will escape.

**Rule:** Resist `Helper`, `Manager`, `Util`, `Processor` classes in the domain.
Each is usually an aggregate method or value object that hasn't found its home;
the name is the smell.

## 12. Queries and read models inside the architecture

**Rule:** Don't force reads through aggregates. Queries that cross aggregates
(dashboards, lists, search) bypass the domain model via dedicated read models /
query services that return DTOs — read side may join freely against tables or
projections (lightweight CQRS, see rules/03 §6). Loading 500 aggregates to
render a list is the standard self-inflicted performance bug of strict-DDD
codebases.

**Rule:** Read models never feed writes. A write decision is made by loading the
aggregate (current, invariant-protected state), not by trusting a possibly-stale
projection that a list view rendered.

## 13. Sizing modules and avoiding entity-service decomposition

**Rule:** Cut boundaries around *capabilities and change-reasons*
("Pricing", "Fulfillment", "Risk"), not around nouns ("UserService",
"OrderService", "ProductService"). Noun-services force every real use case to
choreograph three services and own no invariant (rules/07 §13).

**Rule:** A bounded context should be explainable in one sentence naming its
responsibility and its key invariants. If the sentence needs "and", consider
splitting; if two contexts' sentences keep mentioning each other, consider
merging or formalizing the seam (customer/supplier with a contract).

## 14. The testability boundary (humble object)

**Rule:** Every system has a shell that automated tests cannot meaningfully
drive — GUI and rendering, external devices, real clocks and networks, process
spawning, anything that opens a window, needs a display, or hangs without one.
Draw that boundary **explicitly** and keep the shell **thin**: all decisions
move inward to the testable core, and the shell degenerates into a translator
with no branches worth asserting (the "humble object"). This is §4's adapter
rule stated as a testability constraint rather than a dependency one.

**Rule:** No business `if` in the shell. A conditional, loop, or calculation in
a UI callback, device driver wrapper, or CLI entrypoint is logic that has
escaped the core — move it in and leave a call behind.

**Rule:** Make the boundary **machine-readable** — a package/module split the
build understands, not a convention in someone's head. Coverage, mutation, and
complexity tooling then measure the core only. Measuring the untestable shell
distorts every number in both directions and buries the signal (see
`sota-testing` rules/07 §7.2 and rules/06 §6.3).

**Rationale:** this is the precondition for any test metric meaning anything.
"62% coverage" over a codebase where a third is an untestable UI shell is not
a number, it's noise; the same 62% over an explicitly bounded core is a signal
you can act on. Teams that skip this step end up either gaming the metric or
disbelieving it — and disbelieving it is the more expensive outcome.

```text
BAD:  render_invoice_row(inv):          # in the view layer
        if inv.total > credit_limit and not inv.customer.is_exempt:
          badge = "OVER LIMIT"          # a business rule, untestable without a UI

GOOD: core:  invoice.limit_status(customer) -> OverLimit | Ok   # unit-tested
      shell: render_invoice_row(inv) -> badge_for(inv.limit_status(cust))
             # no branch of its own; one smoke test proves it is wired
```

**Rule:** The shell still needs *some* verification — a small number of
end-to-end or smoke tests that prove it is wired up (`sota-testing` rules/05),
not unit metrics. And treat shell growth as an **architecture** finding: if the
untestable region keeps expanding, logic is leaking outward, and no amount of
test discipline will reach it.

## Audit checklist

- [ ] Absence/unknown/failure encoded **outside** the value domain (option type,
      nullable, second return value, `NULL`, omitted property, exception) — no
      `-1`/`0`/`""`/`9999-12-31` sentinels. Where one is forced: per-field domain
      documented, and the **comparison** sites audited first (§8a).
- Can the team produce a context map? Are seam relationship types (ACL, conformist, published language) explicit?
- Does any domain term ("customer", "order", "account") have different meanings sharing one class/table across contexts?
- Are aggregates small, referenced by ID, and modified one-per-transaction? Search for transactions touching multiple aggregates.
- Are business invariants enforced inside aggregate methods, or scattered across service classes (anemic model)?
- Does domain code import frameworks, ORM types, vendor SDKs, or transport types (HTTP requests, Kafka records)?
- Are ports defined by consumer need, or do they mirror vendor APIs one-to-one?
- Is the dependency rule (adapters → application → domain) enforced by an automated architecture test?
- Do external/legacy models cross into the domain without an anti-corruption layer?
- Are cross-context events versioned, past-tense, schema-validated public contracts?
- Is money a float anywhere? Are state machines stringly-typed with unguarded transitions?
- Is modeling rigor proportional: full DDD on core domains, simple transaction scripts on CRUD subdomains — or uniform ceremony/uniform mud?
- Do repositories use optimistic concurrency, or can concurrent writers silently lose updates?
- Are entity lifecycles explicit state machines with guarded transitions and history, or free-form status strings?
- Do `Helper`/`Manager`/`Util` classes in the domain hold logic that belongs on aggregates or value objects?
- Do list/dashboard reads go through dedicated read models, or do they load aggregates in bulk?
- Do any write decisions consume read-model/projection data instead of loading the aggregate?
- Are services/modules named after capabilities or after entities (noun-services)?
- Is the boundary between the testable core and the environment-bound shell (GUI, device, IO, process spawn) explicit in the build, or only in convention? Does business branching live in that shell?
- Does the shell keep growing release over release (logic leaking outward), and do coverage/mutation/complexity tools measure the core or the whole tree?

# 03 — Distributed Systems, Messaging & Eventing

Rules for any system where two components are separated by a network. The
network will partition, messages will duplicate and reorder, clocks will lie.
Design for that on day one; it cannot be patched in later.

## 1. CAP/PACELC: pick trade-offs per operation, not per system

**Rule:** During a partition you choose consistency or availability; even
without a partition you choose latency or consistency (PACELC). Make the choice
*per operation*, write it down (ADR), and verify the datastore's defaults match.
"We use a CP database" is not a design; "balance reads are linearizable,
product-view reads are eventually consistent with ≤5 s staleness" is.

**Rule:** Never assume read-your-writes across replicas or caches. If a flow
needs it (user saves, next page shows the save), engineer it explicitly: sticky
session to primary, version-pinned reads, or write-through cache.

**Rule:** Distrust distributed transactions (2PC/XA) across heterogeneous
systems. Coordinators block on failure and availability collapses to the worst
participant. Prefer sagas (§5) and the outbox (§4).

## 2. Idempotency is mandatory, everywhere

**Rule:** Every message handler, every retried API call, and every job must be
safe to execute at least twice. Delivery guarantees are at-least-once in
practice (§3), so non-idempotent consumers corrupt data — it's a when, not an if.

**Mechanisms, in order of preference:**
1. Naturally idempotent operations (`set status = 'shipped'`, upsert by key).
2. Idempotency key: caller supplies a key; handler records `(key → result)` and
   replays the stored result on duplicates. Store key + result *in the same
   transaction* as the state change.
3. Version/sequence checks: reject events older than current aggregate version.

```text
GOOD (dedupe inside the TX):
  BEGIN
    INSERT INTO processed(msg_id) VALUES ($id)   -- unique constraint
      ON CONFLICT DO NOTHING; IF no row inserted: ROLLBACK, ACK, return
    ...apply state change...
  COMMIT; ACK

BAD: if redis.exists(msg_id): skip      -- check and write not atomic;
     apply(); redis.set(msg_id)         -- crash between = duplicate apply
```

**Rule:** Public write APIs accept an `Idempotency-Key` header (or equivalent)
and document retention. Payment-ish operations without idempotency keys are a
Critical finding.

## 3. Exactly-once delivery is a myth; exactly-once *processing* is yours to build

**Rule:** No broker gives you end-to-end exactly-once across your database and
side effects. "Exactly-once" broker features (Kafka transactions) cover
broker-internal read-process-write only. Your contract is: at-least-once
delivery + idempotent processing = effectively-once outcomes. Any design doc
that depends on exactly-once delivery is wrong; fix the design, not the broker.

**Rule:** Acknowledge messages only after the state change is durably committed.
Ack-then-process loses messages on crash; process-then-ack duplicates them —
which is fine, because §2.

## 4. Outbox pattern: never dual-write

**Rule:** Never write to your database and publish to a broker as two separate
operations — a crash between them either loses the event or emits a phantom.
Write the event into an `outbox` table in the same transaction as the state
change; a relay (poller or CDC e.g. Debezium) publishes from the outbox.

```text
BEGIN
  UPDATE orders SET status='placed' WHERE id=$1;
  INSERT INTO outbox(event_id, type, payload, created_at) VALUES (...);
COMMIT
-- relay: SELECT unpublished → publish → mark published (at-least-once; consumers dedupe)
```

**Rule:** The same applies inbound: to atomically consume and act, use an inbox
table (record msg_id + effects in one TX), then ack.

## 5. Sagas for cross-service workflows

**Rule:** A business transaction spanning services is a saga: a sequence of
local transactions, each with a **compensating action** for rollback. Define
compensations at design time; a saga step without a compensation (or an explicit
"pivot — no return past this point" marker) is undesigned failure handling.

**Orchestration vs choreography:**
- **Orchestrate** (explicit coordinator / workflow engine: Temporal, Step
  Functions, or a state-machine table) when steps ≥ 3, ordering matters, or you
  need to answer "where is order 123 stuck?". Default choice.
- **Choreograph** (each service reacts to events) only for short, stable, 2–3
  step flows. Beyond that, the workflow exists only in engineers' heads —
  unauditable and undebuggable.

**Rule:** Compensations are not undo. `cancelReservation` after `reserveStock`
must handle "stock already shipped". Compensations are themselves idempotent,
retried, and may need their own compensations escalated to humans (incident
queue), which must be designed, not improvised.

**Rule:** Every saga has a timeout and a terminal failure state visible in
monitoring. Sagas stuck "in progress" for hours are a High finding.

**Rule:** Order the steps so a partial completion leaves the *conservative*
state — take value out of the source first, put it into the destination last.
Debit then credit; consume the coupon then grant the discount; decrement stock
then confirm the order; revoke the old grant then issue the new one. A crash
between legs then leaves value **missing**, which compensation or retry can
recover, instead of value **duplicated**, which nothing recovers once the
destination has spent it. Credit-first is defensible only when the credited
resource provably cannot be consumed before the saga reaches its terminal state
— and that must be *enforced* (a pending/held balance excluded from the
spendable one), never assumed. The cost is real: debit-first strands the payer's
value while the saga is in flight, which is why the timeout-to-terminal-state
above and a user-visible pending status are part of this rule, not extras.

## 5b. Reconciliation: the completeness check on an external integration

**Rule:** Where an external system holds authoritative state that you also
record — payment processor, exchange, carrier, billing vendor, another team's
service — the two records *will* diverge: a webhook is dropped, a call times out
after the far side committed, a retry posts twice. Idempotency and retries bound
the damage; neither tells you a divergence happened. A periodic reconciliation
does. On a money-, inventory-, or entitlement-bearing integration, its absence
is a High finding. Note what this is: §2–§4 give you *integrity* — each record
you wrote is correct. Reconciliation is the only thing that gives you
**completeness** — that no record is missing.

**Rule:** Reconcile against the counterparty's own extract, not against your own
records. Their settlement file, statement, or list endpoint is an independent
observation; recomputing your database from your own event log proves internal
consistency and says nothing about the seam. Where they publish no extract, the
paginated read API over the period is the substitute — the same call the webhook
was meant to save you (webhook senders owe consumers exactly this, see
`sota-api-design` rules/06).

**Rule:** Classify breaks; a count alone is not a result. Three buckets with
three different actions: **(a)** known shape, safe to adjust by a written rule —
automate it, and log every adjustment; **(b)** known shape, needs a human —
queue it; **(c)** unclassified — page someone, because an unclassified break is
indistinguishable from a bug still producing them. An automated (a) adjustment
on a money path posts as a reversing ledger entry, never an in-place correction
(`sota-databases` rules/01). Breaks also **age**: alert on the oldest unresolved
break, not only on the count, or one permanently stuck break becomes background
noise that hides the next twenty.

**Rule:** A reconciliation that has never reported a break is not evidence of
correctness — it is the inert-control signature. Print the denominator (rows
compared on *each* side, and the period covered), fail closed when either side
reads zero, and prove it can fire by seeding a known break. Diagnostics:
`sota-code-security` rules/11 §2.2 and rules/15.

## 6. Event-driven architecture: events are facts

**Rule:** An event states a fact that happened (`PaymentCaptured`), in past
tense, owned by the producer. A command requests work (`CapturePayment`), owned
by the consumer. Don't disguise commands as events ("EmailShouldBeSent") — that
inverts ownership and creates hidden coupling.

**Rule:** Producers don't know consumers. If producer code must change when a
consumer is added, you have RPC over a message bus, not eventing.

**Rule:** Consumers must tolerate: duplicates (§2), reordering (use aggregate
version numbers or partition-key ordering, never wall-clock timestamps), and
unknown fields (forward-compatible deserialization).

**Rule:** Use CQRS only where read and write shapes genuinely diverge (complex
domains, heavy read fan-out, projections per consumer). CQRS does not require
event sourcing; start with "write model + projected read tables". Event sourcing
is a Type 1 decision — adopt only with replay/audit requirements and ops
capacity for snapshotting, upcasting, and GDPR-compliant deletion (crypto-
shredding), each of which must be designed before adoption.

**Rule (the precondition replay quietly assumes):** rebuilding state means re-running the
**apply** step over old events, so that step must be a **pure function of (state, event)**.
Every non-deterministic input it touches — the wall clock, a random or UUID generator, a
config lookup, a call to another service — makes the rebuilt state differ from the
original, silently and without an error. The classic case is an external query: *"if I ask
for an exchange rate on December 5th and replay that event on December 20th, I will need
the exchange rate on Dec 5"* (Fowler, *Event Sourcing*). Two remedies, and you need both:
**capture the answer in the event** (the rate, the generated id, the timestamp — resolved
once, at command time), and **gate outbound effects during replay** so a rebuild does not
re-send emails or re-charge cards. A gateway that cannot be switched into replay mode is a
replay you can only run in a scratch environment, which is the same as not having one.

**Rule:** the **event log is the durable record; commands are not.** A command is a
*request* that may be rejected, deduplicated, or rewritten; the event is what actually
happened. So the durability, retention and backup budget belongs to the event store, and a
lost command is a retry while a lost event is unrecoverable state. If your design gives
commands the stronger guarantee, the roles have been swapped.

## 7. Queues: DLQs, ordering, backpressure

**Rule:** Every queue/subscription has: a max-retry policy with backoff, a
dead-letter queue, an alert on DLQ depth > 0, and a documented + tested
redrive procedure. A DLQ nobody drains is a data-loss buffer with extra steps.

**Rule:** Distinguish poison messages (malformed/bug — will never succeed; DLQ
immediately after schema validation fails) from transient failures (dependency
down; retry with backoff). Retrying poison messages 10 times just delays the DLQ
and burns ordering.

**Rule:** Ordering is per-partition-key only. Pick the partition key = the
entity whose events must be ordered (order_id, account_id). Global ordering
doesn't scale; don't design flows that require it.

**Rule:** Backpressure is designed, not discovered:
- Bounded queues everywhere (unbounded queues turn overload into OOM + huge latency).
- Consumers pull at their own rate; producers get pushback (block, shed, or buffer-with-bound).
- Define the overload policy per queue: shed lowest-priority work first (see rules/04 §6).
- Monitor consumer lag with alerts; lag growing monotonically = under-provisioned consumer or poison loop.

## 7b. Broker queues (AMQP/RabbitMQ): semantics the log model misses

AMQP-style brokers delete on ack and *push* to consumers — different failure
modes than Kafka-style logs:

**Rule:** Manual ack only after the state change is durably committed (§3
applied to AMQP) — never auto-ack: messages count as delivered once written to
the TCP socket, so a consumer crash under auto-ack loses them silently. With
manual acks, unacked deliveries requeue on channel/connection close — which is
why §2 idempotency is mandatory here too.

**Rule:** Nack/requeue is a decision, not a reflex. Requeue only transient
failures; every consumer requeueing the same delivery is a redelivery storm
(CPU + bandwidth burn). Poison messages are rejected with `requeue=false` to a
dead-letter exchange after validation fails (§7), with redelivery count bounded.

**Rule:** Consumer prefetch (`basic.qos`) is the backpressure knob. AMQP pushes,
so the cap on unacked deliveries is your only bound; unset/0 = unlimited
prefetch = an unbounded buffer in the consumer (and memory growth on the node).
Set it explicitly: ~100–300 for fast handlers, low single digits for heavy
ones (sota-async-concurrency rules/06).

**Rule:** Claim-check pattern: messages carry references (entity IDs,
object-store URLs), never large blobs and never secrets — the broker is a
shared failure domain, not storage. Payloads stay small, schema'd (§8), with
correlation IDs (§9).

**Rule:** Quorum queues are the default for anything that must survive node
loss. Classic queue mirroring was removed in RabbitMQ 4.0; classic queues are
single-replica now. Quorum queues default to delivery-limit 20 — messages
exceeding it are dead-lettered or *dropped*, so configure the DLX target; and
default dead-lettering is at-most-once — where DLQ loss is unacceptable, set
`dead-letter-strategy: at-least-once` + `overflow: reject-publish`.

**Rule:** Broker hardening in one line: per-service users, vhost isolation,
least-privilege configure/write/read permissions, TLS, and no default `guest`
account beyond localhost dev — details in sota-cloud-infrastructure and
sota-secrets-management.

## 8. Contracts and schema evolution

**Rule:** Every cross-service message and API has an explicit, versioned schema
(OpenAPI, protobuf, Avro/JSON Schema in a registry) with compatibility checks in
CI. Changes are additive by default (expand/contract): add optional field →
migrate consumers → remove old field. Breaking changes require a new version
published alongside the old, with a deprecation window.

**Rule:** Use consumer-driven contract tests (Pact or equivalent) or a schema
registry with compatibility mode — one of the two, enforced in CI. "We'll
coordinate releases in Slack" is the distributed monolith (rules/07).

**Rule:** Tolerant reader: consumers ignore unknown fields and validate only
what they use. Strict full-payload validation on consume turns every producer
addition into your outage.

## 9. Time, IDs, and causality

**Rule:** Never use wall-clock timestamps for ordering or uniqueness across
machines. Use per-aggregate version numbers for ordering, UUIDv7/ULID/KSUID
for IDs (sortable, collision-free), and the database for authoritative time
where one exists.

**Rule:** Propagate correlation/trace context (W3C traceparent) through every
hop — HTTP, queue message headers, scheduled jobs. A distributed flow you can't
trace end-to-end is undebuggable; this is an audit-blocking gap, not polish.

## 10. Async request/response done right

**Rule:** When a caller needs an answer but the work is async, use correlation
explicitly: caller sends command with `correlation_id` + reply channel (reply
queue, callback URL, or polling endpoint with job status), worker replies with
the same `correlation_id`. Define the no-reply timeout and its user-facing
behavior up front.

**Rule:** Job-status resources beat held connections: `POST /exports → 202 +
/jobs/123`, then poll or push notification. States exposed: `queued → running →
succeeded(result_url) | failed(error, retryable?)`. Every async API the system
offers should converge on one such job pattern, not five bespoke ones.

## 11. Competing consumers and partition assignment

**Rule:** Scale consumers via the competing-consumers pattern, but know your
broker's unit of parallelism (Kafka: partitions; SQS: messages; AMQP: prefetch).
Max useful consumers = partition count in partitioned brokers; provision
partitions for target parallelism *at topic creation* — repartitioning later
breaks key→partition ordering during the transition.

**Rule:** Keep per-message work small and uniform. One 10-minute message behind
500 fast ones on the same partition is head-of-line blocking; split heavy work
into a separate queue/topic with its own consumers (bulkheads, rules/04 §4).

**Rule:** Handle rebalances: consumers must expect partition reassignment
mid-stream (commit offsets only after processing; idempotency §2 covers the
overlap window).

## 12. Distributed coordination: avoid it, then do it right

**Rule:** The best lock is no lock: partition the work (each worker owns a key
range), or let the database arbitrate via constraints and conditional updates
(`UPDATE ... WHERE version = $expected`). Reach for distributed locks only when
neither applies.

**Rule:** If you must lock distributed-ly: use a lease (lock with TTL +
fencing token), never an unbounded lock — holders crash. Verify the fencing
token at the resource, not just at acquisition; a paused process can wake up
believing it still holds an expired lock.

```text
GOOD: token = lock.acquire(ttl=30s)        # token = monotonically increasing
      storage.write(data, fence=token)      # storage rejects stale tokens
BAD:  if redis.setnx(key): do_work()        # GC pause > TTL → two holders, no fence
```

**Rule:** Singleton jobs (schedulers, relays) use leader election (lease in
etcd/DB/k8s Lease) with the same fencing discipline, and must be idempotent
anyway because elections overlap.

## 13. The fallacies, applied

Design reviews must not contain these assumptions; each appears in real designs
weekly:
- "The call will succeed" — every remote call needs a failure branch with a decision (retry? degrade? surface?), not just a log line.
- "Latency is negligible" — N sequential cross-service calls = N × RTT floor; budget it (rules/04 §1).
- "Bandwidth is infinite" — fat events/payloads (full entity snapshots on every change) saturate brokers; send deltas + version, or IDs + hot fields.
- "Topology doesn't change" — pin nothing to instance identity; discovery via the platform (DNS/service registry), connections re-resolve on failure.
- "The clock is right" — clock skew between machines is unbounded for ordering purposes (§9); never compare timestamps from two machines to decide order or expiry of anything critical.

## Audit checklist

- [ ] **Event sourcing: is replay actually runnable, or only theoretically?** Does the
      `apply` step read a wall clock, a random/UUID generator, config, or another service?
      Each one makes a rebuild differ from the original **silently** (§6). The values must
      be resolved at command time and **captured in the event**.
- [ ] Can outbound effects be **gated during replay** — a gateway with a replay mode? If
      not, replay only runs in a scratch environment, which is not a rebuild capability
      (§6).
- [ ] Do **events** get the durability/retention/backup budget rather than commands? A
      lost command is a retry; a lost event is unrecoverable state (§6).

- Is the consistency model (linearizable vs eventual, staleness bound) written down per critical operation?
- Is every message handler and retried endpoint provably idempotent? Is dedupe state committed atomically with the state change?
- Do public write APIs accept idempotency keys?
- Does anything depend on exactly-once *delivery*? Any naked dual-writes (DB write + publish without outbox/CDC)?
- Does every multi-service workflow have defined, idempotent compensations and a timeout-to-terminal-failure path? Can you answer "where is workflow X stuck" from a dashboard?
- Do multi-leg money/inventory/entitlement workflows take value from the source *before* granting it at the destination? Where a credit lands first, is that balance provably held out of the spendable one — enforced, not assumed?
- Does every integration where an external party holds authoritative state have a periodic reconciliation against **their** extract (not your own event log), with breaks classified into auto-adjustable / manual / unclassified, aged, and alerted? Has that reconciliation ever been observed reporting a break, and does it print how many rows it compared on each side?
- Are events past-tense facts with producer ownership, or commands in disguise?
- Does every queue have bounded size, retry-with-backoff, DLQ, DLQ alerting, and a tested redrive runbook?
- Are poison messages separated from transient failures, or retried identically?
- AMQP consumers: manual ack after durable commit (no auto-ack)? Requeue limited to transient failures, with poison messages rejected (`requeue=false`) to a DLX?
- Is prefetch (`basic.qos`) set explicitly on every AMQP consumer, or is any consumer running with unlimited prefetch?
- Durable RabbitMQ queues: quorum (not classic), delivery-limit dead-lettering configured (not silently dropping)? Messages carry references rather than blobs/secrets? Broker locked down (per-service users, vhosts, least privilege, TLS)?
- Is ordering guaranteed only where a partition key provides it? Any logic assuming global order or wall-clock ordering?
- Are message/API schemas versioned with CI compatibility checks or consumer-driven contract tests?
- Are consumers tolerant readers (ignore unknown fields)?
- Is trace/correlation context propagated across every async hop?
- If event sourcing is used: are snapshotting, upcasting, and deletion (crypto-shredding) designed and tested?
- Do async request/response flows use correlation IDs and a standard job-status pattern with defined no-reply timeouts?
- Is consumer parallelism aligned with broker partitioning? Any head-of-line blocking from mixed-weight messages on one queue?
- Are distributed locks lease-based with fencing tokens verified at the resource? Any bare `setnx`-style locks guarding critical sections?
- Are singleton/scheduled jobs leader-elected and idempotent under overlapping elections?
- Do any designs assume reliable network, ordered clocks, or stable topology (check failure branches on every remote call)?

# 04 — Resilience & Failure Design

Rules for surviving the failures that will happen: slow dependencies, dead
dependencies, overload, and your own retries. Resilience is configuration +
code + tests; any leg missing means it doesn't exist.

## 1. Timeouts: every remote call, no exceptions

**Rule:** Every network call (HTTP, DB, cache, DNS, queue publish) has an
explicit timeout. Library defaults are usually infinite or absurd (e.g., many
HTTP clients default to no timeout) — a missing timeout converts a slow
dependency into thread-pool exhaustion and a full outage. Missing timeout on a
critical path is a Critical finding.

**Rule:** Set timeouts from the callee's measured p99 plus margin (e.g., p99 ×
1.5), not folklore. A 30 s timeout on a 50 ms-p99 service doesn't protect
anything; it queues doomed work for 30 s.

**Rule:** Enforce a deadline budget across hops: the caller's total deadline is
distributed down the chain (propagate remaining-deadline in context/headers).
Inner calls whose timeouts sum to more than the outer deadline do useless work
after the client has already given up.

```text
Client deadline 1000ms
  → gateway (deadline left: 950) → svc A (timeout 400) → svc B (timeout 300)
BAD: A and B each configured 5000ms while the edge gives up at 1000ms.
```

## 2. Retries: bounded, jittered, idempotent-only

**Rule:** Retry only idempotent operations (rules/03 §2), only on retryable
errors (timeouts, 503, connection reset — never 400/401/422), with **bounded
attempts** (2–3), **exponential backoff + full jitter**, and a **retry budget**
(e.g., retries ≤ 10% of requests) so retries can't melt a struggling dependency.

```text
delay = random(0, min(cap, base * 2^attempt))   # full jitter
```

**Rationale:** Synchronized un-jittered retries create thundering herds; the
dependency recovers, gets hit by the synchronized wave, dies again.

**Rule:** Retry at ONE layer. Client retries × mesh retries × queue redelivery
multiply: 3 layers of 3 attempts = up to 27 calls per request — a self-inflicted
DDoS. Decide which layer owns retries per edge and disable the rest.

**Rule:** Never retry on the user's synchronous critical path more than once;
prefer failing fast and letting the user/job-queue retry. Latency added by
retries must fit the deadline budget (§1).

## 3. Circuit breakers: stop calling the dead

**Rule:** Wrap dependencies that can fail-slow with a circuit breaker: after a
failure-rate threshold, open the circuit (fail instantly), then half-open with
probe requests before closing. Failing fast preserves your threads and gives the
dependency air to recover.

**Rule:** Breakers are per-dependency (and ideally per-endpoint), never global.
One breaker shared across all dependencies means a dead recommendation service
opens the circuit to your payment provider.

**Rule:** Every breaker open/close event is logged and alerted; breaker state is
a dashboard metric. A breaker that silently opens turns "degraded" into
"mysteriously missing data".

**Rule:** Define what happens when the breaker is open *as part of the design*:
fallback value, cached last-known-good, degraded UX, or explicit error. An open
breaker with no defined fallback is just a faster outage.

## 4. Bulkheads: isolate the blast radius

**Rule:** Partition resources per dependency and per workload class: separate
connection pools, thread pools/semaphores, and queue consumers so one slow
dependency or one greedy tenant can't starve everything else.

```text
BAD:  one 100-conn pool shared by /checkout and /export-report
      → slow report queries consume all 100; checkout dies.
GOOD: checkout pool: 60, reports pool: 10, admin: 5 — reports saturate alone.
```

**Rule:** Apply bulkheads at infra level too: critical and batch workloads on
separate node pools / autoscaling groups; noisy-neighbor isolation for
multi-tenant systems (rules/05 §7).

## 5. Graceful degradation: rank your features

**Rule:** Classify every dependency of each user flow as *required* or
*optional*. Optional-dependency failure degrades (skip recommendations, show
cached prices with a staleness badge, queue the email) — it never 500s the flow.
This classification is a design artifact, reviewed like code.

**Rule:** Fallbacks must be tested under real failure (chaos, §8) and must be
cheap. A fallback that calls another remote service can fail too; a fallback
that recomputes expensively turns partial failure into overload.

**Rule:** Fail closed for security and money (authz unavailable → deny;
fraud-check down → hold the order or apply strict limits). Fail open only for
genuinely optional features — and record the decision per dependency.

## 6. Load shedding and admission control

**Rule:** Decide what you drop *before* you're overloaded. Under saturation,
reject early (HTTP 429/503 + `Retry-After`) at the edge, cheapest-first:
rejecting a request at admission costs microseconds; timing it out after queuing
costs seconds of capacity.

**Rule:** Shed by priority: health checks and payments last; analytics,
prefetch, and crawlers first. Requires a request-priority signal (header,
route class) plumbed to the shedder.

**Rule:** Bound every in-process queue and use deadline-aware queue draining:
if a request has already exceeded its deadline while queued, drop it without
processing. Serving dead requests is how overload becomes collapse (congestion
collapse / metastable failure).

**Rule:** Protect against retry storms after recovery: combine load shedding
with client retry budgets (§2) and slow-start (gradually re-admit traffic).

## 7. Health checks: liveness ≠ readiness ≠ dependency health

**Rule:** Implement both: **liveness** ("process is not wedged" — never checks
dependencies) and **readiness** ("can serve traffic now" — may check critical
local state, warm caches, config loaded). A liveness probe that pings the
database restarts every replica when the DB blips — converting a dependency
outage into a fleet restart. That's a Critical finding.

**Rule:** Readiness should reflect *this instance's* ability to serve, not
shared dependencies' health: if all replicas mark unready when the DB is down,
you remove all capacity and serve connection errors instead of useful 503s with
degradation. Prefer degrading (§5) over going unready for shared-dependency
failure.

**Rule:** Health endpoints are cheap (<10 ms, no fan-out), unauthenticated only
on internal interfaces, and excluded from load shedding last.

## 8. Test failure on purpose (chaos engineering)

**Rule:** Resilience claims require evidence. Regularly and deliberately inject:
dependency latency (+500 ms), dependency errors (10% 503s), instance kills, AZ
loss, and full dependency outage — in staging always, in production once SLOs
and rollback are in place. Verify SLOs hold and fallbacks fire. Untested
fallbacks fail when needed; this is the most reliable finding in chaos history.

**Rule:** Start with game days (hypothesis → inject → observe → fix), automate
the validated experiments into a recurring suite (fitness functions, rules/01
§5). Every incident's failure mode becomes a permanent chaos experiment.

**Rule:** Define SLOs (availability, latency) per critical user journey with
error budgets. Chaos results, degradation policies, and shedding priorities all
derive from SLOs; without SLOs, "resilient" is an opinion.

## 9. Recovery and operability

**Rule:** Design for fast rollback over fast fix: every deploy is reversible in
minutes (previous artifact kept warm), schema changes are expand/contract so
code rollback never requires schema rollback.

**Rule:** Make restarts safe and boring: graceful shutdown (stop accepting,
drain in-flight within deadline, then exit), startup that tolerates dependency
unavailability (retry with backoff, don't crash-loop the fleet), and idempotent
startup migrations guarded by locks.

**Rule:** Avoid synchronized fleet behavior: jitter cron jobs, cache TTLs, and
token refreshes. Thousands of instances doing anything at the same second is a
self-inflicted spike (cache stampede: use TTL jitter + request coalescing /
single-flight).

## 10. Rate limiting: protect yourself and your dependencies

**Rule:** Rate-limit at every trust boundary: per-client at the public edge
(token bucket, keyed by API key/user, with `429 + Retry-After` and documented
limits), per-tenant inside pooled systems (rules/05 §7), and *outbound* toward
third parties whose limits you must respect — a client-side limiter beats
discovering their limit via a ban.

**Rule:** Prefer adaptive concurrency limits (AIMD/gradient on observed latency)
over fixed RPS numbers for internal hops; fixed numbers go stale the week after
the next deploy changes the cost per request.

**Rule:** Burst handling is a policy decision: token bucket (allows bursts) for
user-facing APIs, leaky bucket / shaping for downstream protection. Pick per
edge and write it down.

## 11. Hedging and tail-latency control

**Rule:** For idempotent reads with bad tail latency, consider hedged requests:
send a second attempt after the p95 mark, take the first response, cancel the
loser. Cap hedging (≤5% extra load) and never hedge writes or non-idempotent
calls.

**Rule:** Attack tail latency at the source before hedging: eliminate
synchronized pauses (GC tuning, connection re-establishment storms), avoid
cross-AZ hops on hot paths, and precompute/coalesce instead of fanning out.
Hedging is a patch over variance, not a substitute for removing it.

## 12. Failure-mode analysis as a design step

**Rule:** For every critical flow, run a lightweight FMEA before launch: list
each dependency hop; for each, ask "what happens if it's slow / erroring /
returning garbage?"; record the designed response (timeout value, retry policy,
breaker, fallback, shed, alert). The output is a table in the design doc;
gaps in the table are gaps in the system.

```text
Checkout flow — failure table (excerpt)
dep          slow                 down                    garbage
payments     timeout 2s, 1 retry  breaker→ hold order,    schema-validate,
             then hold-order      notify user, alert      reject + alert
inventory    timeout 300ms        fallback: optimistic    treat as down
             no retry             reserve, reconcile async
recs         timeout 150ms        skip section            skip section
```

**Rule:** Classify dependencies into tiers (T0: flow fails without it; T1:
degrade; T2: invisible loss) and let the tier dictate the minimum machinery:
T0 = timeout + breaker + tested fallback-or-fail-closed + page; T2 = timeout +
silent skip + ticket-level alert.

## 13. Disaster recovery is resilience at the largest blast radius

**Rule:** Define RTO/RPO per datastore *with the business*, then verify the
architecture delivers it: backup restore is **tested by actually restoring**
on a schedule (an unrestored backup is a hope, not a backup), and cross-region
posture (backup-restore / pilot-light / active-passive / active-active) is an
ADR with cost attached.

**Rule:** Active-active across regions is a Type 1 decision that drags
consistency design with it (conflict resolution, data residency, ID generation).
Don't back into it via "we just added a second region for latency".

## Audit checklist

- Does every remote call (HTTP, DB, cache, queue) have an explicit timeout? Grep for client construction sites and verify.
- Are timeouts derived from measured callee latency, and do nested timeouts fit within the edge deadline?
- Are retries bounded, jittered, restricted to idempotent operations and retryable errors, and owned by exactly one layer per edge?
- Is there a retry budget or equivalent guard against retry storms?
- Are circuit breakers per-dependency, with alerting on state change and a defined fallback per open circuit?
- Are connection/thread pools bulkheaded per dependency and per workload class, or is there one shared pool?
- Is each dependency of each critical flow classified required/optional, with degradation behavior implemented and tested?
- Do security- and money-touching paths fail closed?
- Is there admission control / load shedding with priority ordering, bounded queues, and deadline-aware dropping?
- Do liveness probes avoid dependency checks? Does readiness avoid mass-unready on shared-dependency failure?
- Are SLOs defined per user journey, and are chaos experiments (latency, errors, instance/AZ kill) run on a schedule with results tracked?
- Can every service roll back in minutes, shut down gracefully, and start while its dependencies are down?
- Are cache TTLs/crons jittered, and is stampede protection (single-flight) in place for hot keys?
- Is rate limiting present at the public edge (per client), per tenant internally, and outbound toward third-party limits?
- Is hedging, if used, restricted to idempotent reads with a load cap?
- Does each critical flow have a failure-mode table (slow/down/garbage per dependency) with designed responses, and are dependencies tiered T0/T1/T2?
- Are RTO/RPO defined per datastore, backups restore-tested on a schedule, and the cross-region posture an explicit ADR?

# 05 — Scalability, State & Data Architecture

Rules for scaling horizontally, placing state deliberately, caching without
lying, partitioning data, and isolating tenants. Scaling problems are state
problems; everything stateless is trivially scalable.

## 1. Stateless services by default

**Rule:** Service instances hold no request-scoped state between requests: no
sticky sessions, no local user files, no in-memory state another request depends
on. Session state goes to a shared store (Redis/DB) or signed tokens; files go
to object storage. Any instance can serve any request; any instance can die
mid-flight without data loss.

**Test:** Can you kill any instance at any moment and scale 1→N→1 with zero
correctness impact? If not, find the hidden state and evict it.

**Permissible local state:** caches (rebuildable, with bounded staleness) and
buffers already made durable elsewhere. Local state that is the *only* copy of
anything is a Critical finding.

**Rule:** If state must live in the service (stateful stream processing,
websocket hubs, game servers), be explicit: consistent-hash routing, replication,
and a rebalancing story. Accidental statefulness is the problem; deliberate
statefulness is a design.

## 2. Scale out, not up; know your bottleneck first

**Rule:** Design for horizontal scaling (more replicas behind a load balancer),
but **measure before scaling**: profile and load-test to find the actual
bottleneck. Scaling app replicas when the database is the bottleneck adds
connections and makes it worse.

**Rule:** Autoscale on the constraining signal (queue depth, p95 latency,
concurrent requests) — CPU only when CPU is genuinely the constraint. Set
sane min/max, and verify scale-*down* behaves (connection draining, no flapping).

**Rule:** Every shared resource downstream of a scalable tier needs a guard:
connection poolers (e.g., pgbouncer) in front of databases, concurrency limits
per instance, rate limits per client. N autoscaled replicas × M connections
each is the classic way to murder a database.

## 3. The database is the hard part

**Rule:** Scale reads first via replicas + caching; scale writes via partitioning
(§5) only when measured write throughput or data volume demands it. In between,
exhaust the boring options: indexes, query tuning, fewer round-trips, bigger box.
Vertical scaling is honest and cheap up to a surprisingly high ceiling.

**Rule:** Read replicas are eventually consistent. Audit every read-after-write
flow: route them to the primary, pin by session, or use version tokens
(read-your-writes, rules/03 §1). Random replica reads after writes produce
"my save disappeared" bugs.

**Rule:** Long-running queries, analytics, and reporting never run against the
OLTP primary. Use a replica, CDC into an analytical store, or scheduled
extracts. One BI query table-scanning production is a recurring outage pattern.

## 4. Caching: every cache is a consistency decision

**Rule:** For each cache, write down: key shape, TTL, max staleness tolerated,
invalidation trigger, stampede protection, and fallback when cold/down. A cache
without an invalidation story is a bug factory with good latency.

**Tiering (apply outside-in; each tier only if measured need):**
1. CDN/edge — static assets, anonymous pages.
2. Gateway/HTTP cache — cacheable GETs with correct `Cache-Control`/ETags.
3. Distributed cache (Redis/Memcached) — hot entities, computed views, sessions.
4. In-process cache — tiny, hottest keys, short TTL (it multiplies staleness by replica count).

**Patterns:**
- Default: **cache-aside** with TTL + explicit invalidation on write (delete, don't update, the key — update races produce stale-forever entries).
- Stampede protection on hot keys: single-flight/request coalescing + TTL jitter + optionally serve-stale-while-revalidate.
- Negative caching with short TTL for "not found" to stop miss-storms.

**Rule:** The system must be *correct* (if slow) with the cache completely cold
or down. If cache loss = outage or wrong answers, you've built an unmanaged
datastore; either make it a real datastore (durable, replicated) or fix the
dependency.

**Rule:** Never cache authorization decisions or feature-flag evaluations beyond
their tolerated staleness (seconds, not hours), and never share cached responses
across users/tenants without the user/tenant in the key. Missing tenant in cache
key = cross-tenant data leak = Critical.

## 5. Partitioning (sharding): defer, then commit properly

**Rule:** Partition when a single primary measurably can't hold the write load,
working set, or data volume — not before. Sharding multiplies every operational
task (migrations, backups, queries, rebalancing).

**Rule:** Choose the partition key by access pattern: the key that appears in
~all hot queries (tenant_id for B2B SaaS, user_id for consumer). Getting it
wrong means scatter-gather on every request; changing it later is a full data
migration. This is a Type 1 decision — ADR required.

**Rule:** Design for resharding from day one: use many logical partitions
mapped to few physical nodes (consistent hashing or a directory/lookup service),
so adding nodes moves logical partitions instead of rehashing the world.

**Rule:** Accept and design around the losses: no cross-shard transactions
(use sagas, rules/03 §5), no cross-shard joins (denormalize or query a
projection), hot-partition monitoring (one celebrity tenant can melt a shard —
have an isolation/relocation plan).

## 6. Data lifecycle and growth

**Rule:** Every table/collection/topic has a growth model and a lifecycle:
retention policy, archival path (object storage), and deletion (legal +
GDPR/erasure). "Keep everything forever in the OLTP store" is a slow-motion
incident: backups, migrations, and queries all degrade.

**Rule:** Time-series and append-mostly data (events, logs, audit) goes to
stores built for it (or native table partitioning by time with partition
dropping), not into the same tables as hot transactional rows.

## 7. Multi-tenancy: pick the model consciously

**Models (per tier of customer, not necessarily one global choice):**

| Model | Isolation | Cost/tenant | Use when |
|---|---|---|---|
| Pooled: shared schema, `tenant_id` column | Lowest (logical only) | Lowest | Long tail of small tenants |
| Schema-per-tenant / DB-per-tenant | Medium–high | Medium | Hundreds of tenants, compliance asks |
| Silo: dedicated stack per tenant | Highest | Highest | Regulated/enterprise tier, residency requirements |

**Rule:** In pooled models, tenant isolation must be enforced *below* the
application's good intentions: row-level security in the DB, or a mandatory
tenant-scoped repository layer that makes it impossible to build a query without
`tenant_id` — verified by tests that attempt cross-tenant access. One forgotten
`WHERE tenant_id = ?` is a breach.

**Rule:** Tenant context flows from authenticated identity at the edge through
every hop (headers/context), is never accepted from request bodies or query
params, and appears in: every query, every cache key (§4), every queue message,
every log line, and every metric label (for noisy-neighbor attribution).

**Rule:** Apply fairness controls in pooled tiers: per-tenant rate limits,
per-tenant concurrency caps, per-tenant queue quotas (bulkheads, rules/04 §4).
Without them, your biggest tenant defines everyone's worst day.

**Rule:** Plan tenant mobility: onboarding, offboarding (export + verified
deletion), and *migration between models* (a pooled tenant grows into a silo).
If moving one tenant's data out is impossible, you've built a roach motel.

## 8. Workload separation

**Rule:** Separate latency-sensitive request serving from throughput-oriented
batch/async work: different deployments, pools, and scaling policies (bulkheads,
rules/04 §4). Background jobs run on workers consuming queues — never as fire-
and-forget threads inside request-serving instances, where deploys and
autoscaling silently kill them.

**Rule:** All heavy work triggered by users (exports, imports, report
generation) is async: enqueue, return a job handle, notify on completion.
Holding an HTTP connection open for a 5-minute job fails at every proxy,
timeout, and deploy between you and the user.

## 9. Capacity: utilization math you can't ignore

**Rule:** Queueing theory is not optional: at high utilization, wait time
explodes nonlinearly (M/M/1: wait ∝ ρ/(1−ρ); 80%→90% utilization roughly
doubles queueing delay). Plan steady-state utilization of latency-sensitive
tiers at ~50–70%, not 90% — the "wasted" headroom is what absorbs bursts,
deploys, and AZ loss.

**Rule:** Capacity-plan against peak + failure: the fleet must serve peak
traffic with one AZ down and one deploy in flight. N+1 at the *peak*, not the
average.

**Rule:** Load-test with production-shaped traffic (real key skew, real
read/write mix, real payload sizes) before major launches. Uniform-random
synthetic load hides hot keys and lock contention — the two things that
actually fall over.

## 10. Hot keys and skew

**Rule:** Assume skew: some tenant, product, or celebrity will be 1000x median.
Detect it (top-K key metrics on caches, partitions, and rate limiters) and have
a playbook: replicate hot read keys (key suffixing: `item:42#1..N`), isolate
hot tenants to dedicated capacity, collapse duplicate in-flight work
(single-flight), precompute hot aggregations.

**Rule:** Counters and append-heavy rows (likes, view counts, "current balance"
rows updated by everything) serialize on row locks. Shard the counter
(N sub-rows summed on read) or buffer increments through a queue and apply in
batches.

## 11. Derived data and projections

**Rule:** Treat search indexes, materialized views, denormalized read tables,
and analytics copies as *derived data*: rebuildable from the source of truth by
a documented, tested backfill job. If you can't rebuild a projection, it's
secretly a primary store with no backup discipline.

**Rule:** Keep derivation pipelines idempotent and ordered-per-key (rules/03
§2, §7); track and alert on projection lag the same way as replica lag.
Consumers of a projection must know its staleness contract.

**Rule:** Push denormalization to the read side, never the write side: the
write model stays normalized around invariants (rules/02 §3); projections
denormalize per consumer. Denormalizing the write model "for performance"
trades correctness machinery for a cache you can't invalidate.

## 12. Large objects and blob handling

**Rule:** Binary/large content (images, exports, documents) lives in object
storage with the database holding metadata + key. Uploads and downloads go
direct-to-storage via presigned URLs — streaming gigabytes through your API
tier burns its memory and connection slots for zero value.

**Rule:** Lifecycle-manage blobs like rows (§6): retention classes, orphan
sweeps (DB row deleted but blob remains, or vice versa — reconcile on a
schedule), and per-tenant prefixes so tenant offboarding (§7) can actually
delete their data.

## Audit checklist

- Can every service instance be killed at any moment without data loss? Any sticky sessions, local files, or solo in-memory state?
- Is session/user state in a shared store or token, not instance memory?
- Is autoscaling driven by the constraining metric, with verified scale-down/drain behavior?
- Are database connections guarded (pooler, per-instance caps) against replica multiplication?
- Are read-after-write flows protected against replica/cache staleness?
- Do analytics or long-running queries run against the OLTP primary?
- Does every cache have documented TTL, invalidation trigger, stampede protection, and a correct cold-cache fallback? Is anything cache-as-only-copy?
- Do cache keys include user/tenant scope everywhere responses differ by user/tenant?
- If sharded: was the partition key chosen from access patterns (ADR)? Are logical→physical partitions resharding-friendly? Any cross-shard transactions or joins?
- Does every large table/topic have retention, archival, and erasure paths?
- Is the multi-tenancy model explicit per tenant tier? Is tenant isolation enforced below application code (RLS or mandatory scoped layer) with cross-tenant access tests?
- Is tenant context derived from identity (not request params) and present in queries, cache keys, messages, logs, metrics?
- Are per-tenant rate/concurrency/queue quotas in place in pooled tiers?
- Is all user-triggered heavy work async with job handles, and are background jobs on dedicated workers?
- Is steady-state utilization of latency-sensitive tiers planned with headroom (~50–70%), and does capacity cover peak with an AZ down?
- Are load tests production-shaped (key skew, payload sizes), and is hot-key detection (top-K metrics) with a mitigation playbook in place?
- Are high-contention counters sharded or queue-buffered rather than single-row hot spots?
- Is every projection/search index/denormalized table rebuildable via a tested backfill, with lag monitored and staleness contracts known to consumers?
- Are blobs in object storage with presigned direct transfer, orphan reconciliation, and tenant-scoped prefixes?

# 06 — Cloud-Native Operations, Config & Delivery (12-Factor, 2026 Edition)

The 12-factor principles, updated for containers, orchestrators, and modern
delivery. These are the operational contract every service must satisfy
regardless of architecture style.

## 1. Build/release/run: one artifact, promoted everywhere

**Rule:** Build one immutable artifact (container image, digest-pinned) per
commit; promote that exact artifact through dev → staging → prod. Environment
differences live entirely in config (§2). Rebuilding per environment means you
test one binary and ship another.

**Rule:** Releases are versioned and rollback-able: previous release = previous
(artifact, config) pair, redeployable in minutes (rules/04 §9). Tag images with
commit SHA; `latest` in any deployment manifest is a High finding.

**Rule:** Declare all dependencies explicitly with lockfiles; build in clean
environments. Anything "installed on the host" that the app needs is an
undeclared dependency and will bite during scale-out or migration.

## 2. Config: environment-injected, schema-validated, secrets separate

**Rule:** Config that varies by environment (URLs, pool sizes, flags defaults,
credentials) is injected at runtime (env vars, mounted files, config service) —
never compiled in, never `if env == "prod"` branches in code. Code with
environment-name conditionals can't be tested for prod behavior outside prod.

**Rule:** Validate all config at startup against a typed schema and **fail fast**
with a message naming the bad key. A service that boots with a missing config
and fails on first use turns a deploy-time error into a 3 a.m. incident.

**Rule:** Secrets are not config: they come from a secret manager (Vault, cloud
secret stores) or orchestrator secrets with rotation support — never in env-var
dumps in CI logs, never in the image, never in git (enforce with secret-scanning
in CI). Support rotation without redeploy where feasible (file-mounted, reloaded).

**Rule:** Keep declarative deployment config (manifests, IaC) in git, reviewed
like code (GitOps). Live-mutated infrastructure ("someone changed it in the
console") is configuration drift; drift detection should alarm.

## 3. Processes, disposability, and concurrency

**Rule:** Processes are stateless and share-nothing (rules/05 §1); scale via
process/replica count, not threads-per-giant-instance alone. Start fast
(seconds), shut down gracefully on SIGTERM (drain, then exit; rules/04 §9),
and tolerate sudden death (orchestrators kill freely).

**Rule:** Run one concern per process: web serving, queue workers, and schedulers
are separate deployments with separate scaling (rules/05 §8). Cron-in-a-web-pod
runs N times on N replicas — use the platform scheduler or a leader-elected/
distributed-locked runner, designed idempotent (rules/03 §2).

**Rule:** Logs go to stdout/stderr as structured JSON events; the platform
ships them. Apps never manage log files, rotation, or destinations.

## 4. Observability is a launch requirement (the 13th factor)

**Rule:** Every service ships with, before first prod deploy:
- **Structured logs** with correlation/trace IDs and tenant/user context (no PII/secrets — enforce with log scrubbing).
- **Metrics**: RED (rate, errors, duration) per endpoint/consumer + key business metrics + saturation (pools, queues).
- **Distributed traces** (OpenTelemetry), context propagated across HTTP *and* queues (rules/03 §9).
- **SLOs with alerts on burn rate** — alert on user-impacting symptoms (SLO burn), page on those; cause-based alerts are tickets, not pages.

**Rule:** If you can't answer "what did request X do across services, and why was
it slow?" from telemetry alone, observability is insufficient — that's a High
finding regardless of how the dashboards look.

## 5. Backing services are attached, swappable resources

**Rule:** Treat every backing service (DB, cache, broker, email, third-party
API) as an attached resource addressed by config and accessed through a port/
adapter (rules/02 §4). Swapping prod Postgres for a different instance — or a
local container in dev — must require only a config change.

**Rule:** Dev/prod parity: develop and CI-test against the same *kind* of
backing services as production (Postgres in a container, real broker via
testcontainers), not in-memory fakes of databases. SQLite-in-dev/Postgres-in-
prod ships query bugs straight to production.

## 6. Feature flags: decouple deploy from release

**Rule:** Deploy continuously; release via flags. Every risky or user-visible
change ships dark behind a flag, then rolls out progressively (1% → 10% → 50% →
100%) with metrics watched per cohort and instant kill-switch rollback. This —
not deploy rollback — is your primary mitigation lever.

**Rule:** Flag discipline, enforced:
- Every flag has an owner, a description, and an **expiry/removal ticket** at creation. Permanent "flags" (ops kill switches, entitlements) are explicitly marked as such; everything else is temporary.
- Remove flags within weeks of 100% rollout. A codebase with hundreds of stale flags has 2^N untested configuration states — that's the anti-pattern, not a capability.
- Flag evaluation is local/SDK-cached with a hard default if the flag service is down (flag service must never be a request-path single point of failure).
- Flag checks live at the smallest number of choke points (one branch at the use-case boundary), not sprinkled through every layer.
- Flag changes are audited (who flipped what when) — they are production changes.

**Rule:** Use flags for operational levers too: load-shedding tiers (rules/04
§6), degradation switches (rules/04 §5), and migration cutovers (rules/01 §8).
Test both states of every active flag in CI for critical paths.

**Rule:** A progressive rollout means **two versions of the application are running at
once**, so both flag states must be compatible with the schema *as it is right now* —
not with the schema either version was written against. The flag rule and the migration
rule interact and neither is sufficient alone: expand/contract makes the *schema* safe
for two readers (`sota-databases` rules/02), and the flag makes the *behaviour*
switchable, but nothing checks that the off-state still works after the contract step
has run. Order it: expand → deploy both-compatible code → roll the flag → **remove the
flag** → contract. Contracting while a flag can still route traffic to the old path is
the version-skew outage, and it presents as "the rollback made it worse" because the
rollback target no longer matches the data.

## 7. Delivery pipeline and environments

**Rule:** Trunk-based development with small PRs; every merge produces a
deployable artifact passing: unit + contract tests (rules/03 §8), architecture
fitness functions (rules/01 §5), security and secret scans, and migration lint.
Long-lived feature branches are inventory and merge risk; flags (§6) replace them.

**Rule:** Deploys are progressive (canary or rolling with automated rollback on
SLO regression) and boring: no scheduled "deployment windows," no manual steps.
Deploy frequency is a health metric; if deploys are scary, fix the pipeline, not
the calendar.

**Rule:** Database migrations are decoupled from code deploys and always
expand/contract (additive change → deploy code reading both → backfill →
contract). Any migration that would break the *previous* code version blocks
rollback and is a High finding.

## 8. Admin processes and operational surface

**Rule:** One-off admin tasks (backfills, fixes, replays) run as versioned,
reviewed code (jobs/scripts in the repo) in the same environment/config as the
app — never as ad-hoc SQL in a prod console. Make the dangerous path inconvenient
and the safe path easy.

**Rule:** Every service exposes a minimal operational surface: health endpoints
(rules/04 §7), build/version info endpoint, and runtime-adjustable log level.
Every service has a runbook covering: dashboards, common failure modes, DLQ
redrive, scaling levers, and rollback.

## 9. Cost and sustainability as architectural signals

**Rule:** Tag every resource with service + team + (where applicable) tenant;
review cost per service monthly. Cost anomalies are architecture smells: an
unexpectedly expensive service usually has a chatty integration, a missing
cache, unbounded retention (rules/05 §6), or over-provisioned idle capacity.

**Rule:** Set cost fitness functions for the big levers (egress, storage growth,
per-request compute) with alerts — treat a 2x cost regression like a 2x latency
regression.

## 10. Supply-chain integrity

**Rule:** Pin and verify everything that enters the artifact: lockfiles for app
dependencies, digest-pinned base images, dependency update automation
(Renovate/Dependabot) with CI gates rather than hand-rolled upgrades twice a
year. Generate an SBOM per build and scan images for known vulnerabilities as a
pipeline gate (severity threshold, with an expiring-exception process — not a
permanent ignore file).

**Rule:** Sign artifacts (e.g., Sigstore/cosign) and verify signatures at
deploy/admission time, so "what's running" provably equals "what CI built".
Build provenance (SLSA-style) matters most for anything handling money or PII.

**Rule:** Third-party base images and Helm charts are dependencies too: mirror
them into your registry; never deploy straight from a public registry that can
change or vanish under you.

## 11. Environments: ephemeral over snowflake

**Rule:** Environments are created and destroyed from code (IaC + seeded data),
not curated by hand. Prefer ephemeral per-PR preview environments for
integration verification over one perpetually-broken shared "staging" that
serializes every team.

**Rule:** Staging-like environments earn their cost only if they're
production-shaped where it matters: same topology (real broker, real DB engine,
same proxy chain), scaled down. A staging that differs in kind, not just size,
validates nothing — pair it with progressive prod delivery (§7) and testing in
production behind flags (§6) instead of pretending.

**Rule:** Never let environments share backing state (queues, buckets, third-
party sandboxes) without namespacing — cross-environment bleed produces the
least reproducible bug class there is.

## 12. Telemetry cost and cardinality discipline

**Rule:** Metrics labels are a contract with your bill and your query latency:
never label by unbounded dimensions (user ID, request ID, full URL). Tenant ID
as a label is acceptable only below a known tenant-count ceiling; beyond it,
use exemplars or logs.

**Rule:** Trace-sample deliberately: head-sample a base rate, tail-sample
errors and slow requests at 100%. Log levels are runtime-adjustable (§8);
DEBUG-in-prod-by-default is a cost incident and occasionally a PII incident.

**Rule:** Telemetry pipelines are backpressure-aware and lossy-by-design for
the app: dropping a span must never block or crash request serving (bounded
buffers, drop-and-count).

## Audit checklist

- [ ] For any flag rolled out across a schema change: is the **contract step ordered after
      flag removal**, and does the off-state still work against the current schema? Two app
      versions run during a rollout, and contracting while the old path is still reachable is
      the version-skew outage that presents as "the rollback made it worse" (§6).

- Is exactly one immutable, digest-pinned artifact built per commit and promoted across environments? Any `latest` tags or per-env rebuilds?
- Is all environment-varying config injected at runtime, schema-validated at startup with fail-fast? Any `if env == "prod"` branches?
- Are secrets sourced from a secret manager with rotation, absent from git/images/CI logs, with scanning enforced in CI?
- Are deployment manifests/IaC in git with drift detection?
- Do processes start in seconds, drain on SIGTERM, and survive sudden kill? Are web/workers/schedulers separate deployments?
- Are scheduled jobs single-run-safe (leader election/locks) and idempotent?
- Do logs go to stdout as structured events with trace and tenant context, scrubbed of secrets/PII?
- Are RED metrics, OTel traces (propagated through queues), and burn-rate SLO alerts in place? Do pages fire on symptoms, not causes?
- Does dev/CI use production-kind backing services (real DB/broker in containers)?
- Does every flag have an owner and expiry? Are stale flags (>100% rollout for months) present? Does flag evaluation have local defaults if the flag service dies?
- Are releases progressive (canary + auto-rollback) and decoupled from deploys via flags?
- Are all migrations expand/contract such that the previous code version still runs (rollback-safe)?
- Do admin/backfill tasks run as reviewed, versioned jobs rather than console surgery?
- Are resources cost-tagged per service/team with anomaly alerting?
- Are base images digest-pinned and mirrored, SBOMs generated, image scans gating CI with expiring exceptions, and artifacts signed + verified at admission?
- Can a full environment be recreated from code? Are preview/ephemeral environments available, and is shared backing state namespaced per environment?
- Do metric labels avoid unbounded cardinality, is trace sampling tail-biased toward errors/slow requests, and can telemetry loss never block request serving?

# 07 — Anti-Patterns Catalog

Named failure modes with detection signals and remediation. In AUDIT mode, use
the detection signals as grep/inspection targets; in BUILD mode, treat each as a
"never do this" with the listed alternative.

## 1. Distributed monolith

**What:** Microservices' costs (network, ops, versioning) with a monolith's
coupling (lockstep deploys, shared data, synchronous chains).

**Detection signals:**
- Release notes/pipelines deploy multiple services together "because they have to".
- Service A's PR requires a matching PR in service B to not break (no contract versioning, rules/03 §8).
- Synchronous call chains ≥ 3 services deep on the request path; availability = product of every link.
- Shared libraries containing *domain* logic that force coordinated upgrades.

**Fix:** Either merge them back (modular monolith — usually right) or make them
actually independent: versioned contracts + consumer tests, async where the
semantics allow, kill deep sync chains via composition at the edge (BFF/gateway)
or data replication into the caller.

**Severity default:** Critical (it's all of the cost, none of the benefit).

## 2. Shared database integration

**What:** Two or more services/modules read or write the same tables.

**Why it's fatal:** The schema becomes an unversionable public API; no one can
migrate without breaking unknown others; invariants are enforced nowhere;
ownership is fiction.

**Detection signals:** multiple services with credentials to the same schema;
cross-service foreign keys; "read-only access to their DB for reporting";
views shared as integration interfaces.

**Fix:** One writer-owner per table/schema. Other consumers integrate via the
owner's API, published events + their own projection, or CDC into an analytics
store (for reporting). Strangle existing shared access: inventory consumers,
give each a sanctioned path, then revoke credentials.

**Severity default:** Critical for shared writes; High for shared reads.

## 3. God service / god module / god aggregate

**What:** One component that accretes every new feature (often named `core`,
`common`, `platform`, `UserService`, or the original monolith in a "micro-
services" system).

**Detection signals:** disproportionate size/churn (one service receives 60% of
all commits); every cross-team change touches it; its test suite dominates CI
time; every other service depends on it synchronously; aggregates loading
hundreds of rows to change one field.

**Fix:** Split along bounded contexts (rules/02 §1) using strangler extraction
(rules/01 §8). For god aggregates: re-interrogate true invariants and split into
smaller aggregates linked by events (rules/02 §3).

**Severity default:** High.

## 4. Premature microservices

**What:** Decomposing into services before the domain is understood or the team
can operate them. Wrong boundaries in a distributed system are 10x costlier to
move than module boundaries.

**Detection signals:** more services than engineers; "we'll need it at scale"
with no load data; nano-services (an entity-per-service, services that only
CRUD one table); local dev requires running 12 containers; no platform
engineering yet plenty of services.

**Fix:** Consolidate into a modular monolith with enforced boundaries
(rules/01 §1–2); keep events/contracts at module seams so future extraction
stays cheap.

**Severity default:** High.

## 5. Leaky abstraction / vendor bleed-through

**What:** Infrastructure or vendor details escaping their boundary: ORM
entities as API DTOs, vendor SDK types in domain signatures, HTTP status logic
in domain code, broker-specific headers in business logic.

**Detection signals:** importing the database/ORM/cloud SDK package in domain
or use-case layers (architecture test catches this, rules/02 §5); API responses
that change when an internal column is renamed; "port" interfaces that mirror a
vendor API one-to-one (rules/02 §4).

**Fix:** DTOs at every boundary, consumer-shaped ports, mapping in adapters.
Cheapest enforced as an import-rule fitness function.

**Severity default:** Medium (High when public API = DB schema, since clients
now pin your internals).

## 6. Sync chain of doom / temporal coupling

**What:** Request-path workflows requiring N services up simultaneously;
availability multiplies (0.99^5 ≈ 0.95) and tail latency compounds.

**Detection signals:** traces showing deep sequential fan-out per user request;
a "simple" page making 30 internal calls; timeout budgets that can't fit
(rules/04 §1); incident reports where an unrelated service's outage broke checkout.

**Fix:** Replicate the data you need (event-carried state transfer), accept
staleness; collapse hops (merge services or compose at the edge); make non-
essential steps async (queue, saga). Set a fitness function: max sync depth ≤ 2
on critical paths.

**Severity default:** High on revenue-critical paths.

## 7. Event spaghetti / hidden workflow

**What:** Choreographed events forming a workflow nobody can see: cycles
(A's event triggers B, whose event re-triggers A), side-effect cascades, and
"why did this run?" archaeology.

**Detection signals:** no diagram or registry of who consumes what; event
cycles; an innocuous event causing a 9-hop cascade; debugging requires grepping
all consumers for a topic name; commands disguised as events (rules/03 §6).

**Fix:** Event catalog (producer, consumers, schema, purpose) generated from
code/registry; orchestrate multi-step workflows explicitly (rules/03 §5);
detect and break cycles.

**Severity default:** Medium; High once a cycle exists.

## 8. Cache as source of truth (accidental database)

**What:** Data that exists only in Redis/Memcached, or correctness depending on
cache content (rules/05 §4).

**Detection signals:** writes that go to cache without a durable backing write;
"don't restart Redis, we'll lose X"; missing or infinite TTLs on entries nobody
can rebuild; cache-down = wrong answers rather than slow answers.

**Fix:** Make the durable store authoritative and the cache rebuildable, or
promote the data to a real durable store (Redis with AOF/replication *declared*
as a database, with backup/DR treated accordingly).

**Severity default:** Critical when business data is cache-only.

## 9. Distributed big ball of mud via shared "common" libraries

**What:** A `common`/`shared-utils` library hoarding domain types, clients, and
helpers, version-pinned by every service — coupling all services through the
dependency graph instead of the network.

**Detection signals:** bumping `common` requires releasing everything; domain
entities defined in the shared lib; teams blocked on another team's lib release.

**Fix:** Share only truly generic, stable code (logging, telemetry, auth
middleware) with strict semver. Domain types are duplicated per context
(rules/02 §1) or shared via *schemas* (protobuf/OpenAPI), not via code libraries.

**Severity default:** Medium; High when domain logic lives in the shared lib.

## 10. Resume-driven and dogma-driven architecture

**What:** Technology chosen for novelty or ideology, not problem fit: Kafka for
10 msg/s, Kubernetes for one container, event sourcing for a CRUD app, "no
foreign keys because microservices" inside a single schema.

**Detection signals:** infrastructure whose capacity exceeds need by 100x; no
ADR with rejected alternatives (rules/01 §4); ops burden dominated by tools
serving no measured requirement; the only justification on record is an analogy
to a FAANG blog post.

**Fix:** Demand the ADR with load numbers and consequences; downgrade to boring
technology (managed Postgres, a simple queue, one deployable) where the numbers
say so.

**Severity default:** Medium (High when ops burden causes incidents).

## 11. Lava layer / strangler that never strangles

**What:** Successive half-finished migrations sedimented in the codebase: three
HTTP clients, two ORMs, "old auth" and "new auth" both live, a strangler fig
whose old path was never deleted. Every new engineer adds a fourth pattern
because no one can tell which is canonical.

**Detection signals:** multiple frameworks/libraries serving the same purpose
with no deprecation markers; migration ADRs/tickets open > 2 quarters with both
paths in prod; `_v2`/`_new`/`_legacy` suffixes older than a year; nobody can
answer "which one do I use?" without asking in chat.

**Fix:** Every migration gets a kill date and a completion definition ("old path
deleted") tracked like a feature; freeze new usages of the deprecated pattern
mechanically (lint rule failing on new imports of the old module); finish or
explicitly abandon — a documented "we keep both because X" beats sediment.

**Severity default:** Medium; High when the duplicated layer is security-
relevant (two auth paths = the attacker picks the weaker one).

## 12. Snowflake environments and config sprawl

**What:** Production works because of hand-applied settings nobody recorded;
hundreds of config keys with unknown consumers; per-environment behavior that
exists nowhere in git.

**Detection signals:** "don't touch that box"; IaC plan/apply shows permanent
diff (drift); config keys grep to zero readers; staging incident playbooks
differ from prod's; restoring an environment from code has never been done.

**Fix:** Import live state into IaC, enable drift detection alarms (rules/06
§2, §11); delete config keys with no readers (after a deprecation log-on-read
period); rebuild one environment from code per quarter as a fitness function.

**Severity default:** High (it's unrecoverability in disguise — DR depends on
rebuildability).

## 13. Smaller but deadly (quick list)

- **Fan-out N+1 over the network:** per-item remote calls in a loop → batch APIs / data replication. High on hot paths.
- **Chatty two-way coupling:** A calls B and B calls A → merge them or invert one direction with events. High.
- **Timeout-free integration / retry-everywhere:** see rules/04 §1–2. Critical/High.
- **Config-in-code, env conditionals, secrets in git:** see rules/06 §2. Critical for secrets.
- **Anemic domain + transaction-script-everywhere in a complex core domain:** rules/02 §3. Medium.
- **Entity services (`UserService`, `OrderService` as bags of CRUD)** instead of capability-oriented boundaries → re-cut along use cases/contexts. Medium.
- **Queue as database / years of retention in the broker** for operational reads → project into a store; brokers are transport. Medium.
- **Unowned components:** no team on-call for a prod service (rules/01 §7). Critical.

## Audit checklist

- Do any two services deploy in lockstep or require synchronized PRs (distributed monolith)?
- Does more than one service write — or read without a sanctioned contract — the same tables (shared database)?
- Is there a god component (top of churn + dependency in-degree + size by a wide margin)?
- Are there more services than the team can independently deploy, operate, and debug (premature microservices)? Could the system collapse to fewer deployables?
- Do domain/use-case layers import ORM, transport, or vendor SDK types (leaky abstraction)? Are API DTOs the same classes as DB entities?
- What is the maximum synchronous call depth on revenue-critical paths? Is it > 2?
- Is there an event catalog? Are there event cycles or multi-hop cascades nobody documented?
- Is any business data stored only in a cache? Would a cache flush cause wrong answers?
- Does a shared `common` library contain domain types or force coordinated releases?
- Is every heavyweight technology (broker, orchestrator, event sourcing) justified by an ADR with measured load, or by fashion?
- Are there per-item remote calls in loops on hot paths (network N+1)?
- Does every production component have an owning, on-call team?
- Are there sedimented half-migrations (duplicate clients/ORMs/auth paths, `_v2`/`_legacy` older than a year) without kill dates?
- Can every environment be rebuilt from git, or does production depend on hand-applied, unrecorded state (drift in IaC plans)?

# 08 — NATS JetStream

Scope: building on or auditing NATS JetStream as a primary message bus (Go
services). The general distributed-messaging doctrine of **rules/03** —
exactly-once is a myth, ack after durable commit, outbox/never-dual-write, DLQs,
ordering, schema evolution, claim-check, idempotent consumers — applies in full
and is **not repeated here**. This file maps that doctrine onto JetStream's
concrete mechanisms and flags JetStream-specific failure modes. Consumer-loop
and backpressure mechanics live in **sota-async-concurrency**; TLS/mesh/leafnode
transport security in **sota-network-security**; NKEY/JWT/account auth in
**sota-identity-access** and **sota-secrets-management**.

Version: written against **NATS Server 2.11/2.12**; 2.14 has since superseded
2.12 (2.13 was skipped; 2.12.x continues as a maintenance line) — run the
latest stable release and verify the current line with a quick web search at
time of use. KV and Object stores are GA and JetStream-backed. The `jetstream`
Go package is the current API; the older `nc.JetStream()` JetStreamContext is
legacy. Pin and track your server line — per-message TTL, KV limit markers, and
the 2.12+ additions (atomic batch publish, distributed counters, message
scheduling — recurring/cron schedules from 2.14) are gated on server API level.
Upgrade note: the v2 ack-subject format becomes the default in 2.15 — accounts
with granular `$JS.ACK.<stream>.>`/`$JS.FC.<stream>.>` permissions or
imports/exports must update their ACLs before upgrading (a catch-all
`$JS.ACK.>` needs no change).

## Core NATS vs JetStream: choose per subject, not per cluster

### Rule: Use core NATS for fire-and-forget; reach for JetStream only when you need persistence, replay, or dedup.
- **Core NATS** (plain `nc.Publish`/`Subscribe`, request-reply, queue groups) is
  at-most-once: no subscriber online means the message is gone. Correct for RPC,
  health pings, cache invalidations, ephemeral fan-out — anything where loss is
  acceptable and latency is king.
- **JetStream** adds a persistent log with acks, replay, and publish-dedup. Use
  it when a missed message is a bug: orders, state changes, work queues, audit.
- Request-reply (`nc.Request`) and **queue groups** (load-balanced delivery to a
  named group of subscribers) are core-NATS primitives — you do not need
  JetStream for competing consumers on transient work. Putting RPC traffic
  through a stream is a common over-reach: it pays the storage/ack cost for
  no benefit.
- "Exactly-once" on JetStream is **dedup-on-publish + idempotent consumption**,
  not a delivery guarantee (rules/03 §3). Do not design as if it were one.

## Subject design: the schema of your whole bus

### Rule: Design the subject hierarchy deliberately; tokens are your routing and authz surface.
- Dot-separated tokens, hierarchy from general to specific:
  `orders.eu.created`, `orders.eu.shipped`. Tokens are the unit of wildcard
  matching and of account permissions — design them so a single subject pattern
  grants exactly the access a service needs.
- Wildcards: `*` matches one token, `>` matches one-or-more trailing tokens.
  `orders.*.created` = any region's created events; `orders.>` = everything
  under orders. A consumer filtering on `>` at the top of a busy stream is a
  firehose — filter as specifically as the work requires.
- One stream binds a set of subjects; a subject belongs to **one stream** (two
  streams capturing the same subject double-store and confuse consumers). Map
  subjects→streams on paper before creating either.
- **Subject transforms** (`SubjectTransform`, and per-source transforms) rewrite
  subjects on ingest or when sourcing — use for namespacing aggregated streams
  (`orders.>` → `agg.orders.>`), not as a substitute for getting the producer's
  subjects right.

## Streams: an unbounded stream is a latent outage

### Rule: Every stream has explicit limits. No max-bytes and no max-age = a disk-fill incident waiting to happen.
- Set `MaxBytes` and `MaxAge` on every stream, sized to retention need and disk.
  `MaxMsgs`, `MaxMsgsPerSubject`, and `MaxMsgSize` as the workload requires.
  A stream with all limits unset grows until the file store fills and the node —
  and its RAFT peers — degrade. This is the JetStream form of rules/03's
  "bounded queues everywhere"; treat unbounded as **High**.
- **Storage**: `FileStorage` (default, durable, survives restart) for anything
  that matters; `MemoryStorage` only for fast, loss-tolerant, regenerable data.
- **Retention policy** — pick by consumption shape:
  - `LimitsPolicy` (default): messages kept until a limit evicts them;
    many independent consumers replay the same log. Use for event streams.
  - `WorkQueuePolicy`: each message consumable exactly once, removed on ack;
    a true job queue. Requires non-overlapping consumer filter subjects.
  - `InterestPolicy`: message retained only while a bound consumer hasn't acked
    it; storage tracks consumer interest. Use when you want event semantics but
    automatic cleanup once all known consumers have processed.
- **Discard policy**: `DiscardOld` (default — evict oldest to make room) for
  logs where newest matters; `DiscardNew` (reject the publish) when losing old
  data is unacceptable and you want the producer to feel backpressure. Pair
  `DiscardNew` with `DiscardNewPerSubject` + `MaxMsgsPerSubject` to bound each
  subject independently.

```text
GOOD (bounded, durable, work queue):
  nats stream add ORDERS \
    --subjects 'orders.>' --storage file --retention work \
    --discard new --max-bytes 10GB --max-age 720h \
    --max-msgs-per-subject 1000 --replicas 3 --dupe-window 2m

BAD: nats stream add ORDERS --subjects 'orders.>' --storage file
     # no max-bytes, no max-age, no replicas, R1 → unbounded + no HA
```

### Rule: R3 for anything you can't afford to lose; R1 is dev-only. Mirrors/sources for aggregation, not HA.
- `Replicas: 3` (R3) gives a per-stream RAFT group that survives one node loss.
  R1 has no redundancy — a single disk failure loses the stream. Production
  durable streams are R3 (or R5 across failure zones); `--replicas 1` in prod is
  **High**. Use `Placement` (cluster + tags) to pin streams to the right
  hardware/zone.
- **Mirrors** (`Mirror`) are a read-only 1:1 copy of one origin stream — for
  fan-out reads, backup, or cross-region read replicas. **Sources** (`Sources`)
  aggregate many streams into one (fan-in, cross-region roll-up), with optional
  per-source subject transforms and filters. Neither is a substitute for R3:
  they copy asynchronously and lag; HA is replicas, aggregation is mirrors/sources.
- **Mirror promotion** (2.12+) converts a mirror into a writable stream for DR
  failover — a recovery lever, not HA: the mirror is async and may lag behind
  the origin at promotion time, so reconcile for the tail you may have lost.

## Publish & idempotent dedup: the duplicate window

### Rule: Publishers set `Nats-Msg-Id`; the stream's duplicate window dedups. This is the publish half of effectively-once.
- JetStream publish is at-least-once: a publish ack can be lost and the client
  retries, producing a duplicate. Set the **`Nats-Msg-Id`** header to a stable
  business id; the stream rejects a second message with the same id seen within
  its **duplicate window** (`Duplicates`/`--dupe-window`, default 2 minutes).
- Size the window to your **maximum publish-retry horizon**, not larger — the
  window is held in memory; large windows cost RAM. If a retry can arrive 10
  minutes later, a 2-minute window won't dedup it; if retries resolve in seconds,
  don't set hours.
- **Always check the publish ack.** Use `js.Publish` (sync — blocks for the ack)
  or `js.PublishAsync` with a **bounded** in-flight window (`PublishAsyncMaxPending`)
  and drain/await before considering messages durable. Ignoring the ack is silent
  message loss — the rules/03 "no dual-write / confirm the write" rule applied to
  the publish path; treat fire-and-forget JetStream publishing as **High**.
- This is only half of effectively-once. The other half is **idempotent /
  de-duplicated consumption** (rules/03 §2) — the duplicate window does nothing
  for duplicates a consumer sees from redelivery.
- **Atomic batch publish** (2.12+) writes a multi-message batch to one stream
  all-or-nothing — use it where multi-message atomicity previously forced an
  outbox-style workaround; 2.14 adds a fast flow-controlled variant
  (`AllowBatchPublish`). It does **not** replace the outbox for DB-write +
  publish dual-writes (rules/03) — that atomicity spans two systems.

```go
js, _ := jetstream.New(nc)
ack, err := js.Publish(ctx, "orders.eu.created", payload,
    jetstream.WithMsgID(order.ID)) // dedup key; checked vs the stream's window
if err != nil { return err }       // NOT durable until ack returns without error
_ = ack.Sequence
```

## Consumers: pull is the modern default; explicit ack after processing

### Rule: Use durable pull consumers via the `jetstream` package. Push and JetStreamContext are legacy.
- The `github.com/nats-io/nats.go/jetstream` package is the current API. Create a
  consumer, then pull with `Consume` (callback, continuous), `Messages`
  (iterator), or `Fetch` (explicit batch). The older `nc.JetStream()`
  JetStreamContext + `Subscribe` is legacy — don't write new code against it.
- **Durable** (named, `Durable`/explicit name) for work that must resume where it
  left off after restart; **ephemeral** for throwaway tail-follows. An ephemeral
  consumer for durable work loses its position on disconnect — **High** for any
  at-least-once job (rules/03 anti-pattern).
- **`AckExplicit`** (default and correct): ack **after** the state change is
  durably committed (rules/03 §3). `AckNone` and `AckAll` discard the
  per-message safety net; ack-on-receipt before processing loses messages on
  crash — never do it.
- For stronger guarantees use **double-ack** (`msg.DoubleAck(ctx)`/`AckSync`):
  the client waits for the server to confirm the ack landed, closing the window
  where a lost ack causes redelivery of an already-processed message.

### Rule: `MaxAckPending` is your primary backpressure knob; `MaxDeliver` bounds poison redelivery.
- **`MaxAckPending`**: the cap on un-acked in-flight messages per consumer — the
  JetStream analogue of AMQP prefetch (rules/03 §7b). Unbounded (`-1`) lets a
  slow consumer pull more than it can process; set it explicitly, sized to
  handler throughput (sota-async-concurrency rules/06). Unbounded `MaxAckPending`
  on a real consumer is **High**.
- **`AckWait`**: how long the server waits for an ack before redelivering. Size
  it above your worst-case processing time, or in-flight work gets redelivered
  while still running. Send `msg.InProgress()` to extend it for long handlers.
- **`MaxDeliver`**: cap redelivery attempts. **Unset (infinite) means a poison
  message redelivers forever**, burning a consumer slot and CPU — rules/03's
  poison-vs-transient rule, and **High** here. Set `MaxDeliver` and a `BackOff`
  array (per-attempt delays) so transient failures retry with growing spacing.
- **`FilterSubjects`** (multi-filter supported) narrows a consumer to the
  subjects it handles — essential for `WorkQueuePolicy` streams, where consumer
  filters must not overlap.
- **`DeliverPolicy`**: `DeliverAll` (replay from start), `DeliverNew` (only new),
  `DeliverByStartSequence`/`DeliverByStartTime` (replay from a point),
  `DeliverLastPerSubject` (latest per subject — current-state bootstrap).
  **`ReplayPolicy`**: `ReplayInstant` (default) or `ReplayOriginal` (reproduce
  original inter-message timing).
- **Ordered consumers** (`OrderedConsumer`) give a single client an in-order,
  gap-detecting replay that auto-recreates on error — use for single-consumer
  projections/replay, not for scaled competing-consumer work.

```go
cons, _ := stream.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
    Durable:       "orders-projector",
    AckPolicy:     jetstream.AckExplicitPolicy,
    AckWait:       30 * time.Second,
    MaxDeliver:    5,
    MaxAckPending: 256,                       // backpressure
    FilterSubjects: []string{"orders.eu.>"},
    BackOff:       []time.Duration{1*time.Second, 5*time.Second, 30*time.Second},
})
cc, _ := cons.Consume(func(msg jetstream.Msg) {
    if err := process(ctx, msg); err != nil { // do the work first
        _ = msg.NakWithDelay(5 * time.Second) // transient: retry later
        return
    }
    _ = msg.DoubleAck(ctx)                     // commit only after success
})
defer cc.Stop()
// BAD: msg.Ack() before process()  → crash = silent loss (rules/03 §3)
```

## Error handling: ack/nak/term/inprogress, and the DLQ you must build

### Rule: JetStream has no built-in DLQ. Route poison messages out yourself after `MaxDeliver`.
- The four dispositions:
  - **Ack** — processed successfully and committed; remove from redelivery.
  - **Nak** (`Nak`/`NakWithDelay`) — transient failure; redeliver (with delay).
  - **Term** (`Term`) — permanently unprocessable (poison); stop redelivery
    immediately, do **not** wait out `MaxDeliver`. Use it the moment you know a
    message will never succeed (schema-invalid, references a deleted entity).
  - **InProgress** — still working; reset the `AckWait` timer.
- Distinguish transient from poison (rules/03 §7): `Nak` transient, `Term`
  poison. Don't `Nak` a poison message `MaxDeliver` times — that's the redelivery
  storm rule.
- Because there's no native dead-letter, build one: subscribe to the advisory
  **`$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>`** and republish the exhausted
  message to a dedicated **DLQ stream**, or have the handler republish on its
  final attempt. Then apply rules/03 §7 in full: alert on DLQ depth > 0, and have
  a tested redrive runbook. A DLQ nobody drains is a data-loss buffer.
- Consumers must be **idempotent** regardless (rules/03 §2): redelivery after a
  lost ack, double-ack timeout, or `AckWait` expiry will hand you the same
  message again.

## KV & Object stores: config, coordination, and the claim-check

### Rule: Use JetStream KV for config/locks/dedup-state; Object store for large blobs (claim-check).
- **KV** is a JetStream-backed bucket (GA). Each key is versioned with a
  monotonic **revision**; do optimistic concurrency with **compare-and-set**
  (`Update` against an expected revision — rejects if the key moved under you),
  the rules/03 §12 "let the store arbitrate" pattern without a separate lock
  service. **Watchers** stream changes (config hot-reload, coordination);
  **history** keeps N prior revisions (bucket `History`, default 1).
- NATS KV is a good fit for **token revocation** lists and similar hot lookups:
  fast reads, watch-driven propagation, CAS for safe updates.
- KV TTL: bucket-level TTL is long-standing; **per-key TTL** is newer (NATS 2.11+,
  via per-message TTL / `Nats-TTL` and KV limit markers) — verify your server
  supports it before relying on it, and note the stream's `MaxAge` still takes
  precedence over a per-key TTL. Don't assume per-key expiry on an older server.
- **Object store** holds large payloads. Apply rules/03's **claim-check**: keep
  big blobs and secrets out of messages; publish a reference (object id/bucket),
  let the consumer fetch from the object store. The bus is a shared failure
  domain, not a file server — fat messages saturate streams and replication.

## Clustering, HA & multi-tenancy: accounts are the isolation boundary

### Rule: Accounts isolate tenants; domains/leafnodes bridge edge↔hub; per-account quotas bound blast radius.
- JetStream runs a cluster-wide **meta RAFT** group plus a per-stream/per-consumer
  RAFT group; leaders are elected per group. R3 streams tolerate one node loss.
- **Accounts** are the hard multi-tenant boundary: separate subject space, separate
  JetStream assets, no cross-account visibility except via explicit subject
  exports/imports. Multi-tenant systems isolate tenants by **account**, not by
  subject-prefix convention within one account (rules/05 tenancy discipline) —
  subject-prefix-only "isolation" is a cross-tenant leak risk: **High**.
- Set **per-account JetStream limits/quotas** (max memory, max file store, max
  streams/consumers) so one tenant can't exhaust the cluster — the rules/04
  bulkhead applied to storage.
- **Leafnodes** extend a cluster to the edge; **gateways/superclusters** connect
  clusters across regions; **JetStream domains** name an isolated JetStream
  instance (e.g. an edge leafnode hub) so edge and hub streams don't collide and
  can be addressed/mirrored explicitly. Use domains for edge↔hub topologies.
- Auth and transport are out of scope here: NKEY/JWT and account design live in
  **sota-identity-access**/**sota-secrets-management**; TLS, leafnode/gateway
  transport security, and mesh in **sota-network-security**. JetStream over a
  cluster with no TLS and a shared account is not production-ready.

## Operations: monitor lag, snapshot, drain

### Rule: Watch consumer lag and ack-pending; back up streams; drain on shutdown.
- Monitor per consumer: **`num_pending`** (unprocessed backlog),
  **`num_ack_pending`** (in-flight un-acked), **`redelivered`** count, and stream
  bytes vs `MaxBytes`. `num_pending` climbing monotonically = under-provisioned
  consumer or a poison loop (rules/03 §7). Alert on it.
- `nats stream report` and `nats consumer report` are the first-line operability
  tools; advisories on `$JS.EVENT.ADVISORY.>` surface max-deliveries, terminated
  messages, and leader elections. Export metrics vendor-neutrally (the server's
  monitoring endpoints / `nats-surveyor`) into whatever scrapes them — keep the
  collection stack out of the design.
- **Snapshot/restore**: streams support snapshot/backup and restore — schedule
  them, store off-cluster, and rehearse restore (rules/03 / sota-databases
  backup discipline). Replicas are HA, not backup.
- **Graceful drain**: on shutdown call `Stop`/`Drain` on consume contexts and
  `nc.Drain()` so in-flight messages finish and acks flush rather than being cut
  mid-process and redelivered (sota-async-concurrency graceful-shutdown).
- Don't block the consume callback: offload slow work and let `MaxAckPending`
  bound concurrency (sota-async-concurrency rules/06). A blocking callback stalls
  delivery and inflates ack-pending.

## Anti-patterns (JetStream-specific)

- **Unbounded stream** — no `MaxBytes`/`MaxAge`: disk fills, RAFT peers degrade.
- **No `MaxDeliver`** — a poison message redelivers forever (rules/03 §7).
- **Ack before processing** / `AckNone` on durable work — silent loss on crash.
- **One giant catch-all stream** (`>` over everything) — couples unrelated
  workloads, breaks per-subject limits and per-consumer reasoning; the rules/07
  "god service" smell in stream form.
- **At-least-once publisher with no dedup** — no `Nats-Msg-Id`/duplicate window,
  so publish retries duplicate silently.
- **Ephemeral consumer for durable work** — loses position on disconnect.
- **Unbounded `MaxAckPending`** — no consumer backpressure; slow consumer OOMs.
- **Ignored publish acks** — fire-and-forget JetStream publish = silent loss.
- **Subject-prefix "tenancy"** in one account instead of per-account isolation.
- **Blocking the consume callback** — stalls delivery (sota-async-concurrency).

## Audit checklist

- [ ] Is core NATS used for fire-and-forget/RPC and JetStream reserved for
      persistence/replay/dedup, rather than RPC pushed through streams?
      (`nats stream ls`; look for streams capturing request-reply subjects.)
- [ ] Does **every** stream set `MaxBytes` and `MaxAge` (and per-subject limits
      where relevant)? (`nats stream info <s>` → Limits; grep IaC/config for
      stream definitions missing `max_bytes`/`max_age`.)
- [ ] Are durable production streams **R3+**? Any `--replicas 1` / `Replicas: 1`
      outside dev? (`nats stream report` shows replica counts.)
- [ ] Is retention policy correct for the access pattern (work queue vs limits vs
      interest), and `DiscardNew` used where old-data loss is unacceptable?
- [ ] Do publishers set **`Nats-Msg-Id`** and is the **duplicate window** sized to
      the retry horizon? (grep for `WithMsgID`/`Nats-Msg-Id`; `nats stream info`
      → Duplicate Window.)
- [ ] Are publish acks checked — `js.Publish` sync, or `PublishAsync` with bounded
      pending and a drain/await? Any fire-and-forget publishes? (grep `PublishAsync`
      without `PublishAsyncMaxPending`/await.)
- [ ] Are consumers **durable pull** via the `jetstream` package, not ephemeral
      and not legacy `JetStreamContext`/`Subscribe` for durable work? (grep
      `nc.JetStream(`, `.Subscribe(` in JS paths.)
- [ ] Is `AckPolicy` **explicit**, with ack/DoubleAck **after** the commit, never
      before processing and never `AckNone` on durable work? (read the consume
      callback; ack should be the last successful step.)
- [ ] Is `MaxAckPending` set (not `-1`) and sized to handler throughput?
      (`nats consumer info` → Max Ack Pending.)
- [ ] Is `MaxDeliver` bounded with a `BackOff`, and poison `Term`ed (not `Nak`ed
      to exhaustion)? (grep `MaxDeliver`; consumer config; advisory consumers.)
- [ ] Is there a **DLQ stream** fed by `MAX_DELIVERIES` advisories or a final-attempt
      republish, with depth alerting and a redrive runbook? (`nats stream ls` for a
      DLQ/dead-letter stream; subscriber on `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>`.)
- [ ] Are consumers idempotent under redelivery (rules/03 §2)?
- [ ] KV: optimistic concurrency via revision CAS (`Update`), not blind `Put`,
      for contended keys (token revocation, locks)? Per-key TTL only relied on if
      the server supports it? (`nats kv ls`; grep `kv.Update` vs `kv.Put`.)
- [ ] Large payloads via Object store claim-check, not inline in messages?
      (`nats object ls`; check message sizes / `MaxMsgSize`.)
- [ ] Multi-tenant isolation by **account** (with per-account JetStream quotas),
      not subject-prefix convention in a shared account? (server/account config.)
- [ ] Edge↔hub topologies use leafnodes + **JetStream domains**; TLS and
      NKEY/JWT auth in place (defer to network-security / identity-access)?
- [ ] Monitoring on `num_pending`/`num_ack_pending`/`redelivered` with lag
      alerts; stream snapshots scheduled and restore-rehearsed; graceful
      `Drain` on shutdown? (`nats consumer report`; check shutdown path.)