# 01 — Tooling & Project Setup

Modern Python (2026 baseline): `uv` for everything package/env related, `pyproject.toml` as the
single source of truth, `ruff` for lint+format, one strict type checker, `src/` layout,
Python ≥3.12 target. Anything else needs a written justification.

## 1. uv is the default toolchain

Use `uv` for environments, dependency resolution, lockfiles, Python version management, and
tool running. It replaces pip, pip-tools, pipx, virtualenv, and most of poetry.

```bash
uv init --lib mypkg            # or --app; creates src/ layout + pyproject.toml
uv add httpx 'pydantic>=2.7'   # adds to pyproject + updates uv.lock
uv add --dev pytest ruff       # dev dependency group
uv sync --locked               # CI: install exactly the lockfile, fail if stale
uv run pytest                  # run inside the project env, no manual activation
uv python pin 3.13             # writes .python-version
uvx ruff check .               # ephemeral tool run (pipx replacement)
```

Rules:
- **Commit `uv.lock`.** Applications MUST commit it. Libraries commit it for dev reproducibility
  even though it isn't published.
- **CI installs with `uv sync --locked`** (or `--frozen`). Never bare `pip install -r requirements.txt`
  in new projects; if a legacy `requirements.txt` must exist, generate it:
  `uv export --format requirements-txt --output-file requirements.txt`.
- **Never `sudo pip install`, never install into the system interpreter.** Every project gets its
  own venv; `uv` makes this automatic.
- One-off scripts use **PEP 723 inline metadata** instead of polluting an env:

```python
# /// script
# requires-python = ">=3.12"
# dependencies = ["httpx", "rich"]
# ///
import httpx
```

Run with `uv run script.py` — uv resolves and caches the deps. This is the correct form for
repo maintenance scripts; reject scripts that assume "whatever is installed globally".

Governance note: Astral (uv/ruff/ty) announced its acquisition by OpenAI in March 2026; the
tools remain permissively licensed and developed in the open. The technical recommendation
stands — weigh the ownership change like any vendor dependency when standardizing.

## 2. pyproject.toml — single source of truth

All metadata, dependencies, and tool config live in `pyproject.toml`. No `setup.py`, no
`setup.cfg`, no `.flake8`, no `pytest.ini`, no `mypy.ini` unless a tool genuinely cannot read
pyproject (rare in 2026).

```toml
[project]
name = "mypkg"
requires-python = ">=3.12"
dependencies = ["httpx>=0.27", "pydantic>=2.7"]

[dependency-groups]                 # PEP 735 — not extras; dev-only groups
dev = ["pytest>=9", "ruff", "mypy"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

- Pin **lower bounds** on runtime deps; let the lockfile pin exact versions. Upper-bound caps
  (`<3`) only for deps with a history of breaking (pydantic-style major bumps) — blanket caps
  cause unsolvable resolutions downstream.
- Dev-only tooling goes in `[dependency-groups]`, not `[project.optional-dependencies]`.
  Extras are for users; groups are for developers.
- `requires-python` must match what CI actually tests. Claiming `>=3.9` while using
  `match` statements or `type` aliases is a release-blocking bug.

## 3. ruff replaces flake8 + black + isort + pyupgrade

One tool, one config block, two commands: `ruff check --fix` and `ruff format`.

```toml
[tool.ruff]
target-version = "py312"
line-length = 100
src = ["src", "tests"]

[tool.ruff.lint]
select = [
  "E", "W", "F",      # pycodestyle/pyflakes
  "I",                # isort
  "UP",               # pyupgrade — keeps syntax modern
  "B",                # flake8-bugbear — real bug catchers (B006 mutable defaults, B023 loop closures)
  "S",                # flake8-bandit — security
  "C4",               # comprehensions
  "SIM",              # simplify
  "RUF",              # ruff-specific
  "ASYNC",            # blocking calls in async
  "DTZ",              # naive datetimes
  "PTH",              # pathlib over os.path
  "T20",              # stray print()
  "PERF", "G",        # perf antipatterns, logging format
]
ignore = ["E501"]      # formatter owns line length

[tool.ruff.lint.per-file-ignores]
"tests/**" = ["S101"]  # assert is fine in tests
```

- If you find black/flake8/isort/pylint configs in a repo: that's a MEDIUM audit finding —
  consolidate to ruff. Mixed formatters cause churn diffs.
- `ruff format` is the formatter; do not also run black.
- Treat `B`, `S`, and `ASYNC` violations as real bugs, not style noise.

## 4. Type checking — one strict checker, enforced in CI

Pick **one**: mypy (`strict = true`), pyright/`basedpyright` (`"strict"`), or ty (Astral).
Running zero checkers is unacceptable for non-throwaway code; running two as gates causes
contradictory suppressions — one gates, the other may advise.

ty status (mid-2026): Beta since Dec 2025, 10–60x faster than mypy/pyright, and Astral
recommends it for production to motivated users; stable 1.0 targeted for 2026. First-class
pydantic/Django support is still landing — on those stacks mypy/pyright remain the
conservative default; for new plain-Python services ty is a legitimate primary checker.

```toml
[tool.mypy]
python_version = "3.12"
strict = true
warn_unreachable = true
enable_error_code = ["ignore-without-code", "possibly-undefined"]
```

- Every `# type: ignore` MUST carry a code: `# type: ignore[arg-type]`. Bare ignores rot.
- The checker runs in CI and fails the build. "We run mypy locally sometimes" = not typed.

## 5. src/ layout

```
mypkg/
├── pyproject.toml
├── uv.lock
├── src/mypkg/
│   ├── __init__.py
│   └── py.typed          # ship type info (PEP 561)
└── tests/
```

Why: with flat layout, `import mypkg` silently picks up the working-copy directory instead of
the installed package — tests pass against uninstalled code, broken wheels ship. `src/` forces
an editable install (`uv sync` handles it) and catches packaging bugs. Tests live **outside**
the package; they aren't shipped.

Ship `py.typed` in any annotated library, or downstream checkers see your package as `Any`.

## 6. pre-commit — fast checks only

```yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.15.17
    hooks: [{id: ruff, args: [--fix]}, {id: ruff-format}]
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks: [{id: check-merge-conflict}, {id: detect-private-key}, {id: end-of-file-fixer}]
```

Keep hooks under ~5s; mypy and pytest belong in CI, not pre-commit. CI must re-run the same
checks (`pre-commit run --all-files`) — local hooks are convenience, not enforcement.

## 7. Python 3.12–3.14 features worth using

- **PEP 695 type parameter syntax** (3.12) — default for new generics:
  ```python
  type JSON = dict[str, "JSON"] | list["JSON"] | str | int | float | bool | None

  def first[T](items: Sequence[T]) -> T | None: ...

  class Repo[M: BaseModel]: ...
  ```
  No more `TypeVar("T")` boilerplate, no `Generic[T]` inheritance, correct scoping for free.
- **`match` statements — judiciously.** Use for structural destructuring (AST nodes, parsed
  messages, tagged unions with a `kind` field, sum types). Do NOT use as a fancy if/elif on a
  single scalar — `if x == "a": ... elif x == "b":` is clearer and faster. Always end with
  `case _:` that raises or `assert_never(...)` for exhaustiveness (see rules/02).
- **f-string improvements (3.12, PEP 701):** nested quotes `f"{d["key"]}"`, multi-line
  expressions and comments inside `{}` are legal. Use; don't contort.
- **`itertools.batched(iterable, n)`** (3.12) — replaces hand-rolled chunking.
- **3.13:** improved REPL, clearer error messages, `warnings.deprecated` decorator,
  `copy.replace()`.
- **3.14 — deferred annotations (PEP 649/749):** forward references work without quotes or
  `from __future__ import annotations`; pydantic/FastAPI keep working (see rules/02 §10).
- **3.14 — template strings (PEP 750):** `t"..."` returns a `Template` of static parts +
  `Interpolation` objects instead of a string — for t-string-aware APIs that escape or
  parameterize values (HTML, SQL). Not a drop-in f-string; use only with a consuming library.
- **3.14 — `compression.zstd`** (PEP 784): stdlib Zstandard, also wired into
  `tarfile`/`zipfile`/`shutil`. Drops the third-party `zstandard` dep on a 3.14+ floor.

## 8. Free-threading awareness (3.13t/3.14t)

Free-threaded CPython (PEP 703, no GIL) is **officially supported since 3.14** (PEP 779) —
no longer experimental, though still not the default build; single-threaded overhead is down
to roughly 5–10%. uv installs it via the `t` suffix (`uv python install 3.14t`). Implications:

- **Stop assuming the GIL makes code thread-safe.** `dict`/`list` single ops stay atomic, but
  check-then-act sequences (`if key not in d: d[key] = ...`) were never safe and now break
  observably. Guard shared mutable state with `threading.Lock` or use queues — on every build.
- Library authors: declare support via `Py_mod_gil` / test on `3.13t` if you ship C extensions.
- Don't rewrite multiprocessing pools to threads "because no-GIL" until you've profiled on the
  free-threaded build; single-thread perf differs.
- Decision table for concurrency model is in rules/06.

## 9. Docker packaging with uv

```dockerfile
FROM ghcr.io/astral-sh/uv:python3.14-trixie-slim AS builder
WORKDIR /app
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
# Layer-cache deps separately from source: lockfile changes rarely, code changes often
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv uv sync --locked --no-dev --no-install-project
COPY src/ src/
RUN --mount=type=cache,target=/root/.cache/uv uv sync --locked --no-dev

FROM python:3.14-slim-trixie
RUN useradd --create-home app
USER app
COPY --from=builder --chown=app /app /app
ENV PATH="/app/.venv/bin:$PATH"
CMD ["python", "-m", "mypkg"]
```

Key points: deps installed from the lockfile *before* copying source (cache hit on code-only
changes), `--no-dev` in images, non-root user, no uv/compilers in the final stage. Pin base
images by digest for reproducible rebuilds in regulated environments.

## 10. Repo hygiene quick list

- `.python-version` committed; matches CI matrix floor.
- No `requirements*.txt` as the source of truth (generated-only is fine, mark it as such).
- No committed `.venv/`, `__pycache__/`, `.mypy_cache/` — `.gitignore` covers them.
- Version in exactly one place (`pyproject.toml` or `__init__.py` via dynamic) — not both.
- Entry points via `[project.scripts]`, not instructions to run `python src/mypkg/cli.py`.
- Declared-but-unreached dependencies: `deptry .` reports DEP002 (unused), DEP003
  (imported but only a transitive dep), DEP005 (stdlib shadowed). Treat its output as
  candidates — `importlib`/entry-point/plugin loads read as unused — and prove each by
  removing it in a scratch copy and running the real build and full suite
  (`sota-devsecops` rules/10).

## Audit checklist

Run from repo root. Severity guidance in brackets.

```bash
# Toolchain state
ls pyproject.toml uv.lock 2>/dev/null                      # missing uv.lock in an app [MEDIUM]
ls setup.py setup.cfg Pipfile poetry.lock 2>/dev/null      # legacy/competing toolchains [LOW-MEDIUM]
grep -rn "pip install" --include="*.yml" --include="*.yaml" --include="Dockerfile*" . \
  | grep -v "uv pip"                                       # unlocked installs in CI/images [MEDIUM]
grep -n "sudo pip" -r .                                    # system-interpreter installs [HIGH]

# Competing lint/format config [MEDIUM — consolidate to ruff]
ls .flake8 .isort.cfg .pylintrc 2>/dev/null; grep -n "\[tool.black\]\|\[tool.isort\]" pyproject.toml

# Type checking actually enforced?
grep -n "mypy\|pyright\|basedpyright\| ty " .github/workflows/*.yml .gitlab-ci.yml 2>/dev/null
grep -rn "type: ignore$\|type: ignore " --include="*.py" src/ | grep -v "ignore\["   # bare ignores [LOW]

# requires-python vs syntax reality
grep -n "requires-python" pyproject.toml
grep -rln "match \|type [A-Z].* = \|def .*\[T" --include="*.py" src/ | head  # 3.12 syntax w/ old floor?

# Layout
ls -d src/ 2>/dev/null | grep -q . || echo "flat layout"                  # flat layout in a library [LOW]
find src -name py.typed | head -1                          # annotated lib without py.typed [MEDIUM]

# Ruff coverage
uvx ruff check --statistics .                              # what's currently violated
grep -n "select" pyproject.toml                            # B/S/ASYNC missing from select [LOW]

# Hygiene
git ls-files | grep -E "\.venv/|__pycache__|\.pyc$"        # committed artifacts [LOW]
```

# 02 — Typing & Correctness

Types are executable documentation plus a proof system. The standard: every public API fully
annotated, strict checker green, runtime validation only at trust boundaries, zero `Any` leaks.

## 1. Annotate every public API — fully

Every public function, method, and class attribute gets parameter and return annotations,
including `-> None`. Private helpers should be annotated too; inference inside bodies is the
checker's job, signatures are yours.

```python
# Bad — partial annotation is worse than none: callers get silent Any
def fetch(url, timeout=10.0) -> dict: ...

# Good
def fetch(url: str, timeout: float = 10.0) -> dict[str, object]: ...
```

- Use built-in generics (`list[int]`, `dict[str, X]`) and `X | None` — never `typing.List`,
  `typing.Optional` in new code (ruff `UP` auto-fixes).
- Accept abstract, return concrete: take `Iterable[str]` / `Sequence[str]` / `Mapping[K, V]`,
  return `list[str]` / `dict[K, V]`. Demanding `list` rejects tuples and generators for no reason.
- Annotate module-level constants and class attributes that a checker can't infer precisely
  (`STATUSES: Final = frozenset({"open", "closed"})`).

## 2. Optional is explicit, and `None` is handled

`x: str | None = None` — the `| None` is mandatory; implicit-optional is dead. Every nullable
value must be narrowed before use:

```python
def handler(user: User | None) -> str:
    if user is None:
        raise UnauthenticatedError          # narrow once, early
    return user.name                        # checker knows user: User
```

- Don't smuggle nullability through `or` defaults when `""`/`0`/`[]` are valid values:
  `name = arg or default` is a bug if `arg=""` is meaningful. Use `if arg is None`.
- Return `None` for "absent" only when callers genuinely branch on it; raising a precise
  exception is usually better than `T | None` propagating through five layers.

## 2a. In-band sentinels: `-1` is not `None`

The failure §2 exists to prevent has a second spelling that the type checker cannot
see. The class is language-neutral and stated once in `sota-architecture` rules/02
§8a; this section is the Python instance and the worked example.

Python's own stdlib offers both spellings, which is the cleanest illustration there
is: `"abc".find("z")` returns **`-1`** while `"abc".index("z")` **raises
`ValueError`** (verified, CPython 3.14.6). `find` is safe only because you test it on
the next line; the defect is the same value being stored, returned, or compared later.

Instead of `int | None`, a converter returns a **value from the domain** to mean
"absent":

```python
_SENTINEL = -1                                   # "missing int"

def _convert(key: str, raw: str) -> int:
    if not raw:
        return _SENTINEL                         # absent
    try:
        return int(raw)
    except ValueError:
        return _SENTINEL                         # malformed — same value, different cause
```

Three defects, none of which is a type error:

- **Two conditions collapse into one value.** Absent and malformed are
  indistinguishable downstream, so the one that matters (a parser bug upstream)
  can never be counted, alerted on, or told apart from ordinary sparse data.
- **`-1` is truthy**, so `if line_num:` — which reads as a presence check and is the
  idiom people reach for — **passes** on the sentinel and admits it to the body.
  `0`, the other popular sentinel, fails the same test on a legitimate value. Neither
  spelling of the check is right, because the value carries no absence information.
- **It has an ordering.** The sentinel silently **loses** every `<` comparison against
  a real value and **wins** every `>`. Where line/offset/version numbers are compared
  as a proxy for ordering — `use > alloc`, `check < call`, `end >= start` — one
  missing operand flips the predicate, and *which* direction it flips depends on which
  side went missing. That is a fail-open in one direction and a false positive in the
  other, from one input.

**The tell that this is a class and not a slip: the author usually defends one operand
and not the other.**

```python
# the collection is filtered against the sentinel...
lo = min((x for x in candidates if x and x > 0), default=0)
# ...the scalar on the other side of the same comparison is not
if lo > 0 and lo < line_num:                     # line_num may be -1 → False → guard skipped
    ...
```

Whoever wrote `x > 0` knew the sentinel existed. Filtering is per-site, so it is
applied wherever the author was thinking about it and omitted everywhere else — which
makes an **asymmetric guard** a far better audit signal than the sentinel constant
itself.

**Fix, in order of preference:** return `int | None` and narrow at the call site; or
raise for the malformed case and return `None` only for absent, so the two causes stay
distinct. Reach for a sentinel only when a wire format or a fixed-width store forbids
absence — and then it is **per field**, documented with that field's domain, never one
constant applied across a heterogeneous set (see `sota-databases` rules/01, *Modeling
hygiene*, for the persistence half).

### Detecting it — three probes, decreasing precision

1. **Producer (near-zero false positives, AST-checkable).** One function returning the
   *same constant* from a not-found branch and from an `except` branch. That shape is
   almost never intentional, and it is the declaration — fix it once and every caller
   is fixed.
2. **Asymmetric guard (the one that catches the live bug).** A comparison where one
   operand is filtered against the sentinel and the other is not. Greppable in review
   and visible in a diff; see the example above.
3. **Truthiness-as-presence.** `if x:` / `if x and ...` guarding an `int` whose domain
   includes the sentinel or `0`. High recall, high false-positive rate — use it to
   build a list to read, not a gate.

**A value-based lint ("flag any `-1` in this field") is only sound where the field's
domain is stated non-negative, and you cannot assume that per-field from the converter
— the converter is uniform, the domains are not.** A field where the producer emits
`-1` legitimately (an index meaning "not an argument", an enum's *unset* member, a
`ORDER` that is genuinely signed) makes such a lint 100% false positive, and the code
*cannot recover* which meaning a stored `-1` had. So: key the rule on the
**declaration**, and add a value lint only on the subset that carries an explicit
non-negative domain declaration — which usually does not exist yet, and writing it
down is part of the fix. Measure the per-field distribution before you lint: a field
where the sentinel is 99% of rows and one where it is 0% are different problems.

## 3. No `Any` leaks

`Any` disables checking transitively — one `Any` return poisons every downstream variable.

- Prefer `object` for "truly anything, but I won't touch it": the checker forces a narrow
  before use. `Any` means "trust me"; `object` means "prove it".
- Boundary data (JSON, ORM rows, env vars) enters as `Any` — convert immediately via pydantic /
  TypedDict cast / explicit parsing (see §6). Never pass raw `json.loads` output deep into
  the call graph.
- `cast()` is a last resort and must be locally, obviously true. A `cast` that encodes a
  cross-module assumption is a latent bug.
- Audit signal: `def f(...) -> Any`, `dict[str, Any]` proliferating beyond the parse layer,
  un-parameterized `dict`/`list`/`Callable` in signatures.

## 4. Protocol over ABC when structure is the contract

```python
from typing import Protocol, runtime_checkable

class SupportsClose(Protocol):
    def close(self) -> None: ...

def shutdown(resources: Iterable[SupportsClose]) -> None:
    for r in resources:
        r.close()
```

- **Protocol** when you define the interface for *callers* — third-party and stdlib types
  conform without inheriting. No registration, no import coupling, testable with plain fakes.
- **ABC** when you own a closed hierarchy and want shared implementation + instantiation
  guards (`@abstractmethod` raising at construction).
- Don't write one-method ABCs that exist only for typing — that's a Protocol, or just a
  `Callable[[X], Y]` parameter.
- `@runtime_checkable` only enables `isinstance` checks of method *presence*, not signatures —
  don't rely on it for validation.

## 5. Data-shape decision tree: TypedDict vs dataclass vs pydantic

| Need | Use |
|---|---|
| Annotating dicts you don't construct (JSON you read, kwargs, external API shapes) | `TypedDict` (+ `NotRequired`, `Required`) |
| Internal value objects, domain entities, config you construct in code | `@dataclass(frozen=True, slots=True)` |
| Untrusted/external input needing **runtime validation + coercion** (HTTP bodies, env, files, LLM output) | pydantic v2 `BaseModel` |
| Tiny heterogeneous record, positional, immutable | `NamedTuple` |
| Heavy attrs ecosystem already in place | `attrs` (equivalent to dataclass row) |

Rules of thumb:
- **Pydantic at the boundary, dataclasses inside.** Validating the same data repeatedly in
  inner layers is wasted CPU and muddles trust zones: validate once on entry, then pass typed,
  trusted objects. Inner code relies on the static checker, not re-validation.
- TypedDict performs zero runtime checks — it's a static-only promise. Never "validate" with it.
- Don't use pydantic models as general-purpose internal classes; construction cost and
  validation semantics (coercion!) surprise you. `model_construct()` everywhere is a smell that
  you wanted a dataclass.
- Default to `frozen=True, slots=True` dataclasses; mutability and dynamic attributes are
  opt-in exceptions, not defaults (details in rules/03).

## 6. Runtime validation at boundaries — pydantic v2

```python
from pydantic import BaseModel, Field, TypeAdapter

class CreateUser(BaseModel):
    model_config = {"extra": "forbid", "strict": False}
    email: str = Field(pattern=r"^[^@\s]+@[^@\s]+$")
    age: int = Field(ge=0, le=150)

user = CreateUser.model_validate(request_json)   # raises ValidationError with paths

# Validating non-model shapes:
users = TypeAdapter(list[CreateUser]).validate_python(payload)
```

- `extra="forbid"` on input models — silently dropped unknown fields hide client bugs and
  typo'd field names.
- Know coercion: lax mode turns `"3"` into `3`. For protocol-strict boundaries use
  `strict=True` per-field or model-wide.
- Don't catch `ValidationError` and return a vague 500 — surface field paths (FastAPI does
  this for you; see rules/07).

## 7. Exhaustiveness with `assert_never`

Make the checker fail the build when someone adds an enum member or union arm:

```python
from typing import assert_never

type Event = Created | Updated | Deleted

def apply(e: Event) -> str:
    match e:
        case Created():  return "c"
        case Updated():  return "u"
        case Deleted():  return "d"
        case _:
            assert_never(e)   # checker error if a new arm appears; runtime error if reached
```

Same pattern for `Enum` in if/elif chains and `Literal` unions. Any `match` over a closed type
without `assert_never` (or a raising `case _`) is an audit finding — silent fallthrough returns
`None` and detonates elsewhere.

## 8. Generics that carry information

```python
def first[T](items: Sequence[T]) -> T | None: ...

class Repository[M: HasId]:
    def get(self, id_: int) -> M | None: ...
    def add(self, item: M) -> M: ...
```

- Use a TypeVar only when it appears **at least twice** (linking input to output, or two
  inputs). `def f[T](x: T) -> None` is pointless — that's `object`.
- `Self` (3.11+) for fluent APIs and alternate constructors — not the class name, which breaks
  subclassing:
  ```python
  class Query:
      def where(self, **kw: object) -> Self: ...
  ```
- `ParamSpec` for decorators so wrapped functions keep their signatures:
  ```python
  def retry[**P, R](fn: Callable[P, R]) -> Callable[P, R]: ...
  ```
  A decorator returning `Callable[..., Any]` erases types for every call site under it —
  HIGH-value fix in shared codebases.
- Variance: prefer the PEP 695 inferred variance; if you hand-write `TypeVar`, get
  covariance right for read-only containers (`_T_co`).

## 9. Narrowing tools: TypeGuard, TypeIs, overload

Teach the checker what your runtime checks prove:

```python
from typing import TypeIs, overload

def is_str_list(val: list[object]) -> TypeIs[list[str]]:
    return all(isinstance(x, str) for x in val)

if is_str_list(items):
    items[0].upper()        # checker knows list[str] here — and list[object] in the else
```

- Prefer `TypeIs` (3.13 / typing_extensions) over `TypeGuard`: it narrows in **both**
  branches and requires the narrowed type to be consistent with the input — fewer footguns.
- Write a guard once instead of sprinkling `cast()` after every `isinstance`-ish check.

`@overload` when return type depends on argument types/values — the classic is a
default-vs-None getter:

```python
@overload
def get_setting(key: str) -> str | None: ...
@overload
def get_setting(key: str, default: str) -> str: ...
def get_setting(key: str, default: str | None = None) -> str | None:
    return _settings.get(key, default)
```

Without overloads, every caller with a default still has to handle an impossible `None`.
Keep overload sets small (2–4); a 10-overload function wants a redesign. Typed `**kwargs`:
`def f(**kwargs: Unpack[MoveArgs]) -> None` with a TypedDict — stop typing kwargs as
`object`/`Any` in builder-style APIs.

## 10. Misc correctness rules

- `TYPE_CHECKING` blocks for import-cycle-breaking and heavy import deferral. On a 3.14+
  floor, deferred annotation evaluation (PEP 649/749) is the default: forward references
  work unquoted, runtime-inspected annotations (pydantic/FastAPI) keep working, and
  `from __future__ import annotations` should NOT be added to new code (it forces the old
  string semantics). On older floors, don't blanket-add the future import in
  FastAPI/pydantic modules — they need real annotations there.
- `Literal` for closed string/int sets in signatures (`mode: Literal["r", "w"]`), `Enum` when
  the set needs behavior or iteration.
- `Final` for constants; `@override` (3.12) on every overriding method — catches renamed
  base methods at check time.
- NewType for ids that must not interchange: `UserId = NewType("UserId", int)` prevents
  passing an `OrderId` where a `UserId` is expected at zero runtime cost.

## Audit checklist

```bash
# Any leakage [MEDIUM where it crosses module boundaries]
grep -rn "-> Any\|: Any" --include="*.py" src/ | grep -v "test_" | head -50
grep -rn "dict\[str, Any\]" --include="*.py" src/ | wc -l        # boundary-only is OK; everywhere is not
grep -rn "Callable\[\.\.\., " --include="*.py" src/               # signature-erasing decorators?

# Bare/unscoped ignores [LOW each, MEDIUM in aggregate]
grep -rn "type: ignore$" --include="*.py" .
grep -rn "# noqa$" --include="*.py" .

# Legacy typing forms — ruff UP should be clean
uvx ruff check --select UP,ANN --statistics .
grep -rn "typing.Optional\|typing.List\|typing.Dict\|Optional\[" --include="*.py" src/ | head

# In-band sentinels standing in for None (§2a) [HIGH where the value is compared]
grep -rnE "return -1|return 0[^.0-9]|= *-1 *(#|$)" --include="*.py" src/    # producer: same constant from two branches?
grep -rnB2 -- "-> int:" --include="*.py" src/ | grep -c "except"            # int-returning fn with an except arm
grep -rnE "if [a-z_]+ (and|>) 0.*<|> 0.*if" --include="*.py" src/           # asymmetric guard: one operand filtered
grep -rnE "^\s*if [a-z_]*(num|idx|index|count|offset|line|col)[a-z_]*:" --include="*.py" src/  # truthiness-as-presence

# Unhandled Optionals / implicit None returns
uvx mypy --strict src/ 2>&1 | grep -c "error"                     # any errors = not strict-clean
grep -rn "or \[\]\|or {}\|or ''" --include="*.py" src/            # falsy-default smell [LOW, verify]

# Exhaustiveness
grep -rn "match " --include="*.py" src/ -A1 | grep -B1 "case _" | head   # then check assert_never use
grep -rln "assert_never" --include="*.py" src/ || echo "no exhaustiveness guards"

# Validation layering
grep -rn "model_validate\|TypeAdapter" --include="*.py" src/      # should cluster at edges
grep -rn "model_construct" --include="*.py" src/                  # smell: wanted a dataclass [LOW]
grep -rn 'extra.*allow\|extra="ignore"' --include="*.py" src/     # permissive input models [LOW-MEDIUM]

# Protocol/ABC hygiene
grep -rn "class.*ABC).*:" --include="*.py" src/ -A3 | grep -c abstractmethod  # 1-method ABCs → Protocol?
grep -rn "isinstance.*Protocol" --include="*.py" src/             # runtime_checkable misuse

# cast() density [investigate each]
grep -rn "cast(" --include="*.py" src/ | grep -v test
```

# 03 — Idioms, Pitfalls, Exceptions & Logging

The bugs Python invites, and the idioms that prevent them. Most items here are mechanical to
audit (grep/ruff patterns at the end) and mechanical to fix.

## 1. Mutable default arguments (B006)

Defaults are evaluated **once at definition**. Every call shares the same object.

```python
# Bad — every caller without `tags` shares one list
def create(name: str, tags: list[str] = []) -> Item: ...

# Good
def create(name: str, tags: list[str] | None = None) -> Item:
    tags = [] if tags is None else tags
```

Also applies to `{}`, `set()`, `datetime.now()` (frozen at import time!), and any class
instance default. `dataclass` fields: use `field(default_factory=list)` — Python raises for
bare mutable defaults in dataclasses but not for arbitrary mutable objects.

## 2. Late-binding closures (B023)

Loop variables are looked up when the closure **runs**, not when it's created:

```python
# Bad — all callbacks see the final value of `name`
callbacks = [lambda: greet(name) for name in names]

# Good — bind at definition time
callbacks = [lambda name=name: greet(name) for name in names]
# Better — functools.partial states intent
callbacks = [partial(greet, name) for name in names]
```

Hits hardest with callbacks registered in loops (Qt signals, asyncio callbacks, pytest
parametrization done by hand).

## 3. Comprehensions vs loops

- Comprehension when it fits on 1–2 lines and produces a collection. Loop when there are side
  effects, multiple accumulators, or nested conditionals — a 4-line comprehension with three
  `if`s is worse than a loop.
- **Generator expressions** when the consumer is an aggregator or you iterate once:
  `sum(x.price for x in items)` — no intermediate list.
- Never a comprehension for side effects: `[db.save(x) for x in items]` allocates a list of
  `None`s and hides intent. Use a `for` loop.
- `dict`/`set` comprehensions over `dict(zip(...))` gymnastics. Know
  `{k: f(v) for k, v in d.items()}`.

## 4. Context managers for every resource

Anything with acquire/release semantics goes through `with`: files, locks, connections,
transactions, subprocesses, tempfiles, sockets.

```python
# Bad — leaks on exception, ResourceWarning under -X dev
f = open(path); data = f.read(); f.close()

# Good
with open(path) as f:
    data = f.read()

# Own resources: contextmanager for simple cases
@contextmanager
def acquired(lock: Lock) -> Iterator[None]:
    lock.acquire()
    try:
        yield
    finally:
        lock.release()

# Dynamic numbers of resources
with ExitStack() as stack:
    files = [stack.enter_context(open(p)) for p in paths]
```

Run tests with `python -X dev` occasionally — `ResourceWarning` surfaces unclosed handles.

## 5. pathlib over os.path / string surgery

```python
# Bad
out = os.path.join(os.path.dirname(__file__), "..", "data", name + ".json")

# Good
out = (Path(__file__).parent.parent / "data" / f"{name}.json").resolve()
data = out.read_text(encoding="utf-8")
```

`Path.read_text/write_text/read_bytes`, `.glob`, `.mkdir(parents=True, exist_ok=True)`,
`.with_suffix`, `.relative_to`. Always pass `encoding="utf-8"` to text I/O — the platform
default still bites on Windows until UTF-8 mode is universal. Security note: `Path` does NOT
prevent traversal; see rules/05 §4.

## 6. EAFP vs LBYL

Default to EAFP — ask forgiveness — because LBYL check-then-act races:

```python
# Bad — TOCTOU: file can vanish between check and open
if os.path.exists(path):
    with open(path) as f: ...

# Good
try:
    with open(path) as f: ...
except FileNotFoundError:
    handle_missing()
```

Same for `dict` access (`try/KeyError` or `.get` with a real default, not `if k in d` then
`d[k]` twice), and for any filesystem/network/shared-state precondition. LBYL is fine for pure
in-memory validation of caller arguments where no race exists and the check reads better.

## 7. Dataclass patterns

```python
@dataclass(frozen=True, slots=True)
class Money:
    amount: Decimal
    currency: str

    def __post_init__(self) -> None:
        if self.amount.as_tuple().exponent < -2:
            raise ValueError("sub-cent amount")
```

- `frozen=True` by default: hashable, safe as dict keys, no aliasing surprises.
- `slots=True`: ~40–60% memory reduction, faster attribute access, typo'd attributes raise.
  Skip only when you need `__dict__` (dynamic attrs, some mixin patterns, `cached_property`
  needs `__dict__` — use `slots=True` + a manual cache or drop slots there).
- `kw_only=True` for >3 fields or any boolean field — positional booleans are unreadable.
- `field(default_factory=...)` for mutable defaults; `field(repr=False)` for secrets so they
  don't hit logs.

## 8. Enums

```python
class Status(StrEnum):          # 3.11+: members ARE str — JSON/DB-friendly
    OPEN = "open"
    CLOSED = "closed"
```

- `StrEnum`/`IntEnum` when values serialize; plain `Enum` + `auto()` when values are opaque.
- Compare by identity (`status is Status.OPEN`); never compare enum to raw string except at
  parse boundaries (`Status(raw)` — which also validates and raises `ValueError`).
- `Flag` for bitmask sets instead of int constants.
- Magic strings appearing ≥3 times with branching logic → that's an enum.

## 9. itertools / functools — use the stdlib, don't reimplement

Know cold: `itertools.chain`, `batched` (3.12), `pairwise`, `groupby` (input must be sorted by
the same key — classic bug), `islice`, `product`, `takewhile`;
`functools.cache`/`lru_cache` (caveats in rules/06), `cached_property`, `partial`, `reduce`
(sparingly), `singledispatch` for type-based dispatch without isinstance ladders;
`collections.Counter`, `defaultdict`, `deque` (O(1) popleft — never `list.pop(0)` in a loop).
Hand-rolled chunking/flattening/windowing functions in a utils.py are an audit smell: replace
with itertools and delete.

## 10. Exception design

**Catch narrowly, raise precisely, never silence.**

```python
# Bad — swallows KeyboardInterrupt/SystemExit, hides every bug
try:
    process(item)
except:               # bare
    pass

# Bad — Exception + pass is barely better
except Exception:
    pass

# Good
try:
    process(item)
except (ValidationError, IOError) as e:
    logger.warning("skipping %s: %s", item.id, e)
```

Rules:
- **Never bare `except:`** (E722). `except Exception` is the widest acceptable catch and only
  at top-level boundaries (request handler, worker loop, CLI main) — and it must log with
  traceback (`logger.exception(...)`) and usually re-raise or convert.
- Define a small exception hierarchy per package: `class AppError(Exception)` root, callers
  catch your types, not `Exception`.
- **Chain on translation:** `raise StorageError("save failed") from e` — preserves the cause;
  `from None` only when deliberately hiding (rare). Bare re-wrap without `from` loses the
  trail (ruff B904).
- **No exceptions as cross-layer control flow.** Inside one function/module, `StopIteration`-
  style signaling is idiomatic; raising `NotFoundError` from the DB layer and catching it in
  the HTTP layer is fine *if it's a declared domain exception*. What's banned: using generic
  exceptions to implement branching across layers ("raise ValueError to mean retry"), and
  try/except spanning 50 lines where you can't tell which statement is expected to fail —
  keep `try` bodies minimal.
- `else:` clause on try when code should run only if no exception — keeps the `try` body tight.
- **Exception groups (3.11+):** `TaskGroup` and concurrent code raise `ExceptionGroup`; handle
  with `except*`:
  ```python
  try:
      async with asyncio.TaskGroup() as tg: ...
  except* httpx.HTTPError as eg:
      for e in eg.exceptions: logger.error("fetch failed: %s", e)
  ```
  Code that catches `Exception` around a TaskGroup and inspects nothing loses errors.

## 11. Logging done right

```python
logger = logging.getLogger(__name__)        # module level, never the root logger

# Bad — f-string formats even when DEBUG is off; breaks aggregation grouping
logger.debug(f"user {user_id} fetched {n} rows")

# Good — lazy %-formatting; args interpolated only if the record is emitted
logger.debug("user %s fetched %d rows", user_id, n)

# Errors with traceback
except StorageError:
    logger.exception("save failed for order %s", order_id)   # includes stack automatically
```

- **No f-strings/`.format()`/`+` in log calls** (ruff `G` rules): wasted CPU at high-volume
  call sites, and every message becomes unique — log aggregators can't group them, and
  user-controlled values get interpolated even when not logged (injection surface).
- Libraries: never call `basicConfig()`, never add handlers — configure logging only in the
  application entry point (`logging.config.dictConfig`). A library that configures logging
  hijacks the host app.
- Structured logging for services: `structlog` or stdlib + JSON formatter; bind context
  (request id, user id) once, not per call. Put variable data in fields, not in the message.
- `logger.exception` only inside `except` blocks; `logger.error(..., exc_info=True)` is the
  equivalent elsewhere.
- Never log secrets/tokens/PII — pair with `field(repr=False)` and dedicated redaction.
- No `print()` in library/server code (ruff T20). CLIs print to stdout for *output*, log to
  stderr for *diagnostics*.

## 12. Small-but-deadly grab bag

- `is`/`is not` only for `None`, `True`, `False`, sentinels, enums — never for strings/ints
  (interning makes it *sometimes* work, which is worse).
- Returning `-1`/`0`/`""` to mean "absent" instead of `None`: type-checks, survives a
  truthiness check, and flips ordering comparisons. Producer shape and the three
  detectors in rules/02 §2a.
- Naive datetimes: always `datetime.now(tz=timezone.utc)`; `utcnow()` is deprecated and naive
  (ruff DTZ).
- `zip(a, b, strict=True)` (3.10+) when silent truncation would hide a length-mismatch bug.
- Shadowing builtins (`list`, `id`, `type`, `input`) — rename (ruff A).
- String building in loops: collect + `"".join(parts)` (perf details rules/06).
- `round()` is banker's rounding; money math uses `Decimal` with explicit quantize.
- Don't mutate a list/dict while iterating it — iterate a copy or build a new one.

## Audit checklist

```bash
# Ruff covers most of this file — run first
uvx ruff check --select B006,B008,B023,B904,E722,BLE,G,T20,DTZ,PTH,SIM,A,C4 --statistics .

# Bare/broad excepts [HIGH if swallowing, MEDIUM otherwise]
grep -rn "except:$\|except: " --include="*.py" src/
grep -rn -A1 "except Exception" --include="*.py" src/ | grep -B1 "pass$"

# Exception chaining lost
uvx ruff check --select B904 .                              # raise-without-from in except

# Mutable defaults & late binding
uvx ruff check --select B006,B023 .

# Logging
grep -rn 'logger\.\(debug\|info\|warning\|error\)(f"' --include="*.py" src/   # f-strings in logs [LOW-MED]
grep -rn "basicConfig" --include="*.py" src/ | grep -v "main\|__main__\|cli"  # library configuring logging [MEDIUM]
grep -rn "print(" --include="*.py" src/ | grep -v "cli\|__main__\|test"       # stray prints [LOW]

# Resource handling
grep -rn "= open(" --include="*.py" src/ | grep -v "with "                    # unmanaged file handles [MEDIUM]
grep -rn "\.close()" --include="*.py" src/ | head                             # manual close → with-able?

# datetime & os.path modernization
uvx ruff check --select DTZ,PTH --statistics .
grep -rn "utcnow()" --include="*.py" src/                                     # naive UTC [MEDIUM]

# Identity misuse & list.pop(0)
grep -rn 'is "" \|is "\| is [0-9]' --include="*.py" src/
grep -rn "\.pop(0)" --include="*.py" src/                                     # O(n) dequeue [perf]

# groupby without sort (manual review)
grep -rn "groupby(" --include="*.py" src/
```

# 04 — Async: Structured Concurrency Done Right

asyncio's failure modes are silent: lost exceptions, garbage-collected tasks, a blocked loop
that "works" until production load. The cure is structured concurrency plus a hard ban on
synchronous work inside coroutines.

## 1. TaskGroup is the default; gather is legacy

```python
# Good — structured: scope owns the tasks; one failure cancels siblings;
# all exceptions surface as ExceptionGroup; nothing leaks past the `async with`
async def fetch_all(urls: list[str]) -> list[Response]:
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(fetch(u)) for u in urls]
    return [t.result() for t in tasks]

# Legacy — gather without return_exceptions: first error propagates but siblings
# keep running detached; with return_exceptions=True: errors silently mixed into results
results = await asyncio.gather(*coros)
```

- New code uses `asyncio.TaskGroup` (3.11+). `gather` survives only for "collect results
  including failures" reporting — and then every element MUST be isinstance-checked.
- Handle TaskGroup failures with `except*` (exception groups, see rules/03 §10).
- A coroutine should not outlive the scope that created it. If you can't say which `async with`
  owns a task, the design is wrong.

## 2. Fire-and-forget: the GC eats your tasks

The event loop holds only **weak** references to tasks:

```python
# Bad — task may be garbage-collected mid-flight; exceptions vanish
asyncio.create_task(send_email(user))

# Acceptable when truly detached work is required — keep a strong ref + done callback
_background: set[asyncio.Task[None]] = set()

def _reap(t: asyncio.Task[None]) -> None:
    _background.discard(t)
    if t.cancelled():
        return
    if (exc := t.exception()) is not None:        # NEVER just discard: an unobserved
        log.exception("detached task failed", exc_info=exc)   # exception is a silent failure

def spawn(coro: Coroutine[None, None, None]) -> None:
    t = asyncio.create_task(coro)
    _background.add(t)
    t.add_done_callback(_reap)   # strong ref fixes LIFETIME; this fixes ERROR VISIBILITY
```

Better: don't fire-and-forget. Put background work in a long-lived TaskGroup owned by the app
lifespan (FastAPI lifespan, service main), or a queue + worker. Every unawaited
`create_task(...)` whose return value is discarded is an audit finding.

## 3. Never block the loop

One blocked coroutine freezes **every** request on that loop. Banned inside `async def`:

| Blocking call | Replacement |
|---|---|
| `time.sleep(n)` | `await asyncio.sleep(n)` |
| `requests.get(...)` / `urllib` | `httpx.AsyncClient` / `aiohttp` |
| `subprocess.run` | `await asyncio.create_subprocess_exec(...)` |
| blocking DB driver / ORM call | async driver (asyncpg, SQLAlchemy async) or `to_thread` |
| file I/O on slow media, `Path.read_text` of big files | `await asyncio.to_thread(p.read_text)` |
| CPU-bound work (parsing, crypto, image ops) | `to_thread` (releases GIL?) else ProcessPoolExecutor |

```python
# Sync library you can't replace — push to a thread
data = await asyncio.to_thread(legacy_client.fetch, key)

# CPU-bound — process pool (threads don't help under the GIL)
loop = asyncio.get_running_loop()
result = await loop.run_in_executor(process_pool, crunch, payload)
```

Detection: ruff `ASYNC` rules catch the obvious ones; `loop.slow_callback_duration` + dev mode
(`PYTHONASYNCIODEBUG=1` or `-X dev`) logs callbacks that hog the loop at runtime.

## 4. Timeouts on every external await

Unbounded awaits are how services hang. Use `asyncio.timeout` (3.11+) — composable, cancels
the whole block:

```python
async with asyncio.timeout(5.0):
    conn = await pool.acquire()
    rows = await conn.fetch(query)
```

- Prefer one `asyncio.timeout` around a logical operation over per-call `wait_for` wrappers.
- Library calls with native timeout params (httpx, asyncpg) should set them too — defense in
  depth, and httpx's default timeout exists but DB drivers' often don't.
- **Never swallow `CancelledError`.** Catch it only to clean up, then re-raise:
  ```python
  except asyncio.CancelledError:
      await release_partial()
      raise
  ```
  Swallowing it breaks timeouts, TaskGroup cancellation, and graceful shutdown. `except
  Exception` does NOT catch it (it's `BaseException`) — which is correct; don't "fix" that.
- 3.12 `wait_for` semantics changed; treat `asyncio.timeout` as the only spelling worth
  remembering.

## 5. Async generators & cleanup

Async generators finalize **non-deterministically** unless closed:

```python
# Risky — break exits the loop; generator's finally runs whenever GC gets to it,
# possibly on a dead loop
async for row in stream_rows():
    if found(row):
        break

# Good — deterministic cleanup
from contextlib import aclosing
async with aclosing(stream_rows()) as rows:
    async for row in rows:
        if found(row):
            break
```

- Wrap any async generator that holds resources (cursors, connections, files) in `aclosing`
  when the consumer might exit early.
- Inside the generator, `finally:` must be cancellation-safe — it may run during cancellation
  where further awaits can themselves be cancelled; use `asyncio.shield` only with care.
- Prefer returning an async context manager over a resource-holding async generator in
  public APIs.

## 6. Sharing clients, connections, loops

- One `httpx.AsyncClient` / connection pool per application, created at startup (lifespan),
  closed at shutdown — never per request (`async with AsyncClient()` inside a handler
  costs you connection reuse, TLS session caching, and adds latency).
- Never create objects bound to a loop (`Queue`, `Lock`, clients) at **import time** — they
  bind to whichever loop exists then, or none; instantiate inside async context / lifespan.
- `asyncio.run(main())` exactly once at the program edge. Nested `asyncio.run` or
  `get_event_loop().run_until_complete` inside libraries is a design error.
- asyncio primitives (`asyncio.Lock`, `Queue`) are not thread-safe; crossing threads uses
  `loop.call_soon_threadsafe` / `asyncio.run_coroutine_threadsafe`.

## 7. Common bugs checklist

- **Forgotten `await`:** `client.get(url)` returns an un-awaited coroutine that never runs;
  the only symptom may be a `RuntimeWarning: coroutine ... was never awaited` on stderr.
  Strict type checking flags this (`Coroutine` where `Response` expected) — another reason
  rules/01 mandates a checker. Enable `-W error::RuntimeWarning` in tests.
- **`async def` that never awaits** — either it shouldn't be async (caller pays scheduling
  cost, pretends concurrency) or it's missing the await.
- **Blocking ORM in async views — and the two frameworks fail *differently*:**
  **Django raises**, it does not silently serialize. A sync ORM call from a thread with a
  running event loop gets `SynchronousOnlyOperation`; it only blocks instead if someone set
  `DJANGO_ALLOW_ASYNC_UNSAFE`, which the docs warn risks data loss
  ([Django async safety](https://docs.djangoproject.com/en/5.2/topics/async/), verified
  2026-09-16). So "works in dev, dies in prod" is the wrong symptom to hunt for in Django —
  hunt for the exception, or for the env var that disabled the guard. A sync SQLAlchemy
  `Session` on a FastAPI async path **does** block silently, because nothing is watching.
  Django: `await Model.objects.aget(...)` / `sync_to_async`; FastAPI: see rules/07.
- **Lock-free check-then-act across awaits:** state can change at every `await`. Guard
  multi-step invariants with `asyncio.Lock`, or design single-writer.
- **`time.monotonic` vs loop time** for timing inside coroutines; never `time.time()` deltas.

## 8. anyio — when to consider

anyio runs on asyncio (and trio) with stricter structured-concurrency semantics and level
cancellation. Use it when: writing a **library** that shouldn't dictate the backend, you want
trio-style cancel scopes, or you're already in Starlette/FastAPI internals (they're anyio-
based — `anyio.to_thread.run_sync` is what `def` endpoints use). For applications committed
to asyncio, 3.11+ stdlib (TaskGroup + timeout) covers most of anyio's historical advantage;
don't mix both APIs ad hoc in one codebase — pick one idiom.

## 9. Bounded fan-out and producer/consumer

Unbounded concurrency is a self-inflicted DoS — against your own connection pool, the
remote API's rate limit, or memory.

```python
# Bad — 50_000 simultaneous requests; pool exhaustion, remote 429s, memory spike
async with asyncio.TaskGroup() as tg:
    for url in urls:                      # len(urls) == 50_000
        tg.create_task(fetch(url))

# Good — Semaphore caps in-flight work; TaskGroup still owns lifetimes
sem = asyncio.Semaphore(20)

async def fetch_bounded(url: str) -> Response:
    async with sem:
        return await fetch(url)

async with asyncio.TaskGroup() as tg:
    tasks = [tg.create_task(fetch_bounded(u)) for u in urls]
```

For pipelines, prefer an explicit bounded queue — backpressure for free:

```python
async def pipeline(items: AsyncIterator[Item]) -> None:
    q: asyncio.Queue[Item | None] = asyncio.Queue(maxsize=100)   # maxsize = backpressure

    async def producer() -> None:
        async for item in items:
            await q.put(item)            # blocks when consumers lag — by design
        for _ in range(N_WORKERS):
            await q.put(None)            # sentinel per worker

    async def worker() -> None:
        while (item := await q.get()) is not None:
            await process(item)

    async with asyncio.TaskGroup() as tg:
        tg.create_task(producer())
        for _ in range(N_WORKERS):
            tg.create_task(worker())
```

- `Queue(maxsize=0)` (unbounded) in a pipeline is a memory leak waiting for a slow consumer.
- Rate limiting (req/s) is not the same as concurrency limiting (in-flight); for hard API
  quotas use a token-bucket (`aiolimiter`) in addition to the semaphore.

## 10. Testing async code

- `pytest-asyncio` (or anyio's pytest plugin) with `asyncio_mode = "auto"` in pyproject;
  async tests just work, no decorator noise.
- Fake time with explicit clock injection or `looptime`-style plugins; never `await
  asyncio.sleep(real_seconds)` in tests.
- Test cancellation paths: `task.cancel()` then assert cleanup ran — uncancelled-safe cleanup
  is the most common untested async path.

## Audit checklist

```bash
# Ruff async rules first — blocking calls, sync sleep, etc.
uvx ruff check --select ASYNC --statistics .

# Blocking calls inside async defs [HIGH in servers]
grep -rn "time\.sleep" --include="*.py" src/                 # cross-check: inside async def?
grep -rn "requests\.\(get\|post\|put\|delete\|Session\)" --include="*.py" src/
grep -rn "subprocess\.\(run\|check_output\|call\)" --include="*.py" src/   # in async modules?

# Fire-and-forget tasks [MEDIUM-HIGH]
grep -rn "asyncio.create_task" --include="*.py" src/         # is the return value kept + callback added?
grep -rn "ensure_future" --include="*.py" src/               # legacy spelling, same issue

# gather usage [review each]
grep -rn "asyncio.gather" --include="*.py" src/
grep -rn "return_exceptions=True" --include="*.py" src/      # are results isinstance-checked after?

# Swallowed cancellation [HIGH]
grep -rn -A3 "except asyncio.CancelledError" --include="*.py" src/ | grep -L raise
grep -rn "except BaseException" --include="*.py" src/

# Timeouts
grep -rn "asyncio.timeout\|wait_for" --include="*.py" src/ | wc -l    # zero in a network service = finding
grep -rn "AsyncClient()" --include="*.py" src/               # per-request client construction? [MEDIUM]

# Loop-bound objects at import time [MEDIUM]
grep -rn "^[a-zA-Z_]* = asyncio.\(Queue\|Lock\|Event\)" --include="*.py" src/
grep -rn "get_event_loop" --include="*.py" src/              # legacy API [LOW-MEDIUM]

# Async generators holding resources without aclosing
grep -rln "async def.*->.*AsyncIterator\|AsyncGenerator" --include="*.py" src/
grep -rn "aclosing" --include="*.py" src/                    # compare counts

# Forgotten awaits — runtime + type checker
grep -rn "asyncio_mode" pyproject.toml setup.cfg 2>/dev/null
python -W error::RuntimeWarning -m pytest -x 2>&1 | grep "never awaited"
```

# 05 — Security

Trust-boundary thinking: anything from the network, files, env, DB content, or LLM output is
attacker-controlled until validated. The items below are the Python-specific exploit classes;
each has a grep signature — hunt them all in audits.

## 1. Deserialization & code execution bans

**`pickle` on untrusted data = remote code execution.** `pickle.loads` executes arbitrary
callables during load. Same family: `shelve`, `marshal`, `dill`, `joblib.load`, pandas
`read_pickle`, torch `torch.load` without `weights_only=True`.

```python
# Bad — RCE if attacker controls the bytes (cache poisoning, uploaded model, queue message)
obj = pickle.loads(blob)

# Good — data interchange uses data formats
obj = msgspec.json.decode(blob, type=Job)     # or json + pydantic validation
```

- Pickle is acceptable ONLY for same-trust-domain, integrity-protected data (e.g., local
  multiprocessing, HMAC-signed cache where the key never leaves the service). Document why.
- **`yaml.load` without SafeLoader = code execution.** Always `yaml.safe_load(f)` /
  `yaml.load(f, Loader=yaml.SafeLoader)`.
- **`eval`/`exec` on any string containing external input — banned.** "Sandboxing" with
  `{"__builtins__": {}}` is bypassable; don't review it, reject it. Need expressions? Use
  `ast.literal_eval` (data literals only), a real expression library (simpleeval has caveats
  too), or define a DSL with explicit dispatch.
- Templates: Jinja2 with autoescape on for HTML (`select_autoescape`); never render
  user-controlled **template strings** (SSTI → RCE), only user data into fixed templates.
  Same logic for `str.format` on user-supplied format strings (`"{0.__class__}"` walks objects).

## 2. Subprocess: argv lists, never shell=True

```python
# Bad — shell injection: filename = "x; rm -rf /"
subprocess.run(f"convert {filename} out.png", shell=True)

# Good — argv vector, no shell, timeout, checked
subprocess.run(
    ["convert", "--", filename, "out.png"],
    check=True, capture_output=True, timeout=30,
)
```

- `shell=True` with ANY variable in the string is a HIGH finding. Constant-string
  `shell=True` is still a smell (PATH games, IFS) — rewrite as a list.
- `--` before user-controlled positional args so `-rf`-style values can't become flags
  (argument injection — applies to git, curl, tar, find especially).
- Validate or allowlist executables; never let the user pick the binary. Set `timeout=`,
  handle `CalledProcessError`. `os.system` is banned outright.

## 3. SQL: parameters, never interpolation

```python
# Bad — injection, all variants: f-string, %, +, .format
cur.execute(f"SELECT * FROM users WHERE email = '{email}'")

# Good — driver parameters
cur.execute("SELECT * FROM users WHERE email = %s", (email,))

# Good — SQLAlchemy 2.0 style
stmt = select(User).where(User.email == email)
rows = session.execute(stmt).scalars().all()
# Raw SQL when needed — still bound params:
session.execute(text("SELECT * FROM users WHERE email = :email"), {"email": email})
```

- Identifiers (table/column names) can't be parameterized — allowlist them against a fixed
  set, never interpolate user input.
- `LIKE` patterns: escape `%` and `_` in user input before binding.
- Django ORM is parameterized by default; the dangerous edges are `.raw()`, `.extra()`,
  and `RawSQL` — audit every occurrence.

## 4. Path traversal

`Path` arithmetic does not sandbox: `base / "../../etc/passwd"` escapes, and absolute
user paths replace the base entirely (`Path("/srv") / "/etc/passwd"` → `/etc/passwd`).

```python
def safe_join(base: Path, user_path: str) -> Path:
    candidate = (base / user_path).resolve()
    if not candidate.is_relative_to(base.resolve()):   # 3.9+
        raise ValueError("path escapes base directory")
    return candidate
```

- Apply to every filename from uploads, URLs, archive members, config. Also reject `\0`
  and, for uploads, generate server-side names (uuid) instead of trusting client filenames.
- Symlinks: `resolve()` follows them — decide whether links inside `base` pointing out are
  acceptable (usually not for upload dirs: check `os.path.realpath` containment after write,
  or `O_NOFOLLOW`).

## 5. Archive extraction (zip/tar slip)

Malicious archives contain members named `../../home/user/.bashrc`, absolute paths, links,
or device nodes — and zip bombs (small file → TB of output).

```python
# Bad
tarfile.open(path).extractall(dest)

# Good — 3.12+: filter validates members (rejects traversal, abs paths, devices, bad links)
with tarfile.open(path) as tf:
    tf.extractall(dest, filter="data")
```

- `filter="data"` is mandatory on tar extraction (it became the default in 3.14; be explicit
  anyway). Pre-3.12: validate each member's resolved destination with the §4 containment check.
- The filter itself has had bypasses — CVE-2025-4517: symlink chains pushing the resolved
  path past PATH_MAX escaped the destination even with `filter="data"` (fixed in 3.12.11,
  3.13.4, 3.14+). Keep the interpreter patched and keep the §4 containment check as defense
  in depth; never treat the filter as the sole control for hostile archives.
- `zipfile.extractall` strips leading `/` and dots but still follow up with size limits:
  cap total uncompressed size and member count before extracting (read `ZipInfo.file_size`,
  enforce a budget) — `extractall` has no bomb protection.

## 6. Randomness & secrets

```python
# Bad — Mersenne Twister is predictable from outputs
token = "".join(random.choices(string.ascii_letters, k=32))

# Good
token = secrets.token_urlsafe(32)
code  = f"{secrets.randbelow(1_000_000):06d}"
```

- `random` for simulations only; anything security-relevant (tokens, password resets, session
  ids, OTPs, salts) uses `secrets` or `os.urandom`.
- Compare secrets with `secrets.compare_digest` / `hmac.compare_digest`, never `==` (timing).
- Passwords: argon2 (`argon2-cffi`) or bcrypt — never raw sha256/md5, never homemade salting.
- Secrets come from env/secret manager, not source. `.env` is gitignored; values never appear
  in `repr`/logs (dataclass `field(repr=False)`, pydantic `SecretStr`).
- TLS: never ship `verify=False` (requests/httpx) or `ssl._create_unverified_context`;
  pin an internal CA bundle instead.
- `hashlib.md5/sha1` only for non-security checksums — and mark it:
  `hashlib.md5(data, usedforsecurity=False)`.

## 7. XML & SSRF quickies

- Untrusted XML: use `defusedxml`; stdlib `etree` is OK against entity *expansion* on modern
  versions but external entity and DTD handling across libs (lxml!) still needs hardening:
  `lxml.etree.XMLParser(resolve_entities=False, no_network=True)`.
- SSRF: any user-supplied URL fetched server-side must be validated — scheme allowlist
  (`https` only), resolve and reject private/link-local/metadata ranges (169.254.169.254),
  disable redirects or re-validate per hop. httpx: set `follow_redirects=False` and handle
  explicitly.

## 7a. `assert` is not a control — `-O` deletes it

`python3 -O` and `PYTHONOPTIMIZE=1` strip `assert` statements entirely. Verified:
a function whose `assert x > 0` raised under a normal run printed `passed` under
both. So any validation, authorization, or bounds check written as an `assert`
**does not exist** in an optimized deployment, and the source still reads correct.

- Validation and security checks are `if not ok: raise ...`, never `assert`.
- Keep `assert` for impossible internal states you want loud in development.
- Audit: `grep -rn "assert " --include="*.py"` over request handlers, validators
  and permission code, then check whether the runtime is invoked with `-O` /
  `PYTHONOPTIMIZE` (Dockerfile `CMD`, entrypoint, `uv run` flags).
- Note the sibling trap: lenient numeric parsing. `int(" 12 \n")` is `12` and
  `float("1_0")` is `10.0` — a corrupt field yields a plausible number rather
  than an error. Full class: `sota-code-security` rules/13 §3.

## 8. Input-adjacent denial of service & injection oddities

- **ReDoS:** user input through a regex with nested/ambiguous quantifiers
  (`(a+)+`, `(.*)*`, `(\w+\s?)*`) can run exponentially. Audit every `re.*` whose pattern
  *or* subject is user-controlled; prefer anchored, linear patterns; set a length cap on the
  subject before matching; for hostile-input parsing consider the `regex` module's timeout
  or Rust-backed RE2 bindings.
- **Decompression bombs** beyond archives: `zlib.decompress`, image loading
  (`PIL.Image` — set `Image.MAX_IMAGE_PIXELS`, it defaults to a warning), XML entity
  expansion (§7). Enforce decoded-size budgets, not just encoded-size limits.
- **Log injection:** user strings logged verbatim can forge log lines (`\n` + fake record)
  or smuggle ANSI escapes into terminals. Strip/escape control characters at the logging
  formatter for user-supplied fields; one more reason for structured logging
  (fields are quoted) — see rules/03 §11.
- **`int()`/numeric parsing:** Python ints are unbounded — `int(user_str)` of a 10MB digit
  string allocates happily; 3.11+ caps str→int at 4300 digits by default
  (`sys.set_int_max_str_digits`) — don't raise that limit on request paths. Cap input length
  before parsing.
- **`webbrowser.open()` command injection:** crafted URLs (e.g. containing `%action`)
  passed to `webbrowser.open()` reach the shell with certain browser types —
  CVE-2026-4519 and its incomplete-fix follow-up CVE-2026-4786 (2026). Never pass
  user-influenced URLs; validate scheme/host first, and keep the interpreter patched.
- **Header/CRLF injection:** never place raw user input into HTTP headers, email headers
  (`email.message` does folding — still validate), or redis/SMTP protocol lines; reject
  `\r`/`\n` in any value destined for a protocol line.

## 9. Dependency & supply-chain hygiene

- **Audit continuously:** `uv run pip-audit` or `osv-scanner --lockfile uv.lock` in CI on a
  schedule, not just on PRs (new CVEs land against old lockfiles).
- **Hash-pinned, locked installs everywhere:** `uv.lock` records hashes; CI/containers use
  `uv sync --locked`. Exporting for pip: `uv export --format requirements-txt` includes
  `--hash` entries — keep them.
- **Typosquatting:** verify package names on first add (`requests` not `request`, `pillow`
  not `PIL` on PyPI, `python-dateutil` not `dateutil`). New transitive deps in a lockfile
  diff deserve a glance — lockfile diffs are security-relevant code review.
- **No `pip install` from URLs/git in prod paths** without commit pinning
  (`package @ git+https://...@<full-sha>`).
- **Publish via PyPI Trusted Publishing (OIDC), not long-lived API tokens.** The GhostAction
  campaign (Sept 2025) exfiltrated thousands of CI secrets including PyPI tokens via
  injected GitHub Actions workflows; PyPI invalidated the stolen tokens and recommends
  Trusted Publishers (short-lived, repo-scoped). With `pypa/gh-action-pypi-publish` ≥v1.11
  under a Trusted Publisher, PEP 740 attestations (build provenance) are generated by
  default — don't disable them. Pin third-party Actions by commit SHA, not tag.
- Don't run `setup.py`-era installs of unvetted sdists in CI with secrets in env — build
  scripts execute arbitrary code; prefer wheels, isolate builders.
- Each project in its own venv (uv default); never share one env across trust levels, never
  install into the interpreter that runs your OS tooling.
- Containers: multi-stage build, `uv sync --locked --no-dev`, run as non-root, no compiler
  toolchain in the final image.

## 10. Static analysis gates

- Ruff `S` ruleset (bandit port) in the standard select (rules/01) — covers most greps below
  natively: S301 pickle, S602 shell=True, S608 SQL strings, S324 weak hashes...
- `bandit -r src/ -ll` as a CI job if you want bandit's full set, plus
  `opengrep scan --error --config <your-python-rules>` for taint-style findings
  (`sota-devsecops` rules/05 §5.1 — the CLI is `scan`, there is no `ci` subcommand,
  and prefer vendored or `git+`-cloned rulesets over a registry you do not control).
- Suppressions (`# noqa: S...`, `# nosec`) require a justification comment; bare `# nosec`
  is itself a finding.

## Audit checklist

```bash
# One-shot scanners
uvx ruff check --select S --statistics .
uvx bandit -r src/ -ll -q
uv run pip-audit 2>/dev/null || uvx pip-audit
osv-scanner --lockfile uv.lock 2>/dev/null

# Code execution / deserialization [CRITICAL on untrusted data]
grep -rn "pickle.loads\|pickle.load\|read_pickle\|joblib.load\|marshal.loads\|dill" --include="*.py" src/
grep -rn "torch.load" --include="*.py" src/ | grep -v "weights_only=True"
grep -rn "yaml.load(" --include="*.py" src/ | grep -v "SafeLoader\|safe_load"
grep -rn "\beval(\|\bexec(" --include="*.py" src/ | grep -v "literal_eval\|model.eval()"
grep -rn "\.format(.*request\|f\".*{.*request" --include="*.py" src/ | head   # format-string gadgets

# Subprocess [HIGH]
grep -rn "shell=True" --include="*.py" src/
grep -rn "os.system\|os.popen" --include="*.py" src/

# SQL [CRITICAL]
grep -rn 'execute(f"\|execute(".*%s" *%\|execute(.*+ ' --include="*.py" src/
grep -rn "\.raw(\|\.extra(\|RawSQL" --include="*.py" src/                     # Django edges
grep -rn 'text(f"' --include="*.py" src/                                      # SQLAlchemy text+f-string

# Path traversal & archives [HIGH]
grep -rn "extractall\|extract(" --include="*.py" src/ | grep -v 'filter='
grep -rn "request.*filename\|\.filename" --include="*.py" src/                # then check containment
grep -rn "is_relative_to\|realpath" --include="*.py" src/                     # mitigations present?

# Randomness & secrets [HIGH]
grep -rn "random\.\(choice\|choices\|randint\|random\)" --include="*.py" src/ # security context?
grep -rn "== .*token\|token.* ==" --include="*.py" src/ | head                # timing-unsafe compare
grep -rn "verify=False\|_create_unverified" --include="*.py" src/
grep -rnE "(api_key|secret|password|token) *= *['\"][A-Za-z0-9_\-]{12,}" --include="*.py" .  # hardcoded
grep -rn "md5(\|sha1(" --include="*.py" src/ | grep -v usedforsecurity

# XML / SSRF
grep -rn "lxml.etree\|xml.etree\|xml.dom\|xml.sax" --include="*.py" src/      # defused? entities off?
grep -rn "get(url\|get(request\.\|urlopen(" --include="*.py" src/ | head      # user-controlled URL fetch?

# ReDoS / DoS surfaces
grep -rnE "re\.(match|search|fullmatch|findall|sub)\(" --include="*.py" src/ | head -30   # user-controlled subject?
grep -rnE "\((\.\*|\\\\w\+|\[\^?[^]]*\]\+)\)[\*\+]" --include="*.py" src/                 # nested quantifiers
grep -rn "zlib.decompress\|Image.open" --include="*.py" src/                              # size budgets present?
grep -rn "set_int_max_str_digits" --include="*.py" src/
grep -rn "webbrowser.open" --include="*.py" src/                               # user-influenced URL? [HIGH] CVE-2026-4519/4786

# Supply chain
grep -rn "git+http" pyproject.toml uv.lock 2>/dev/null | grep -v "@[0-9a-f]\{40\}"
grep -rn "nosec\|noqa: S" --include="*.py" src/                               # justified suppressions?
```

# 06 — Performance

Order of operations: **measure, fix the algorithm, fix the data structure, vectorize or move
to native, only then micro-optimize.** Every optimization PR cites a profile; "should be
faster" without numbers is rejected.

## 1. Profile first — tool selection

| Question | Tool |
|---|---|
| Where does CPU time go (dev, deterministic) | `cProfile` + `snakeviz` / `python -m cProfile -o out.prof` |
| What is prod doing *right now*, no restart, low overhead | `py-spy top --pid N`, `py-spy record -o flame.svg --pid N` |
| CPU vs memory vs copy time, line-level, GPU | `scalene mypkg/main.py` |
| Memory growth / leaks | `tracemalloc` snapshots, `memray` |
| Micro-benchmarks of one expression | `python -m timeit`, `pytest-benchmark`, `pyperf` (statistically sound) |

Rules:
- py-spy is safe on production (samples out-of-process; add `--native` for C extensions,
  `--gil` to see GIL contention). cProfile distorts hot loops — fine for *finding* hotspots,
  not for before/after numbers; use pyperf/pytest-benchmark for comparisons.
- Profile realistic data sizes. O(n²) is invisible at n=100.
- Keep a benchmark in the repo for any code with a perf SLA (`pytest-benchmark` with
  `--benchmark-autosave` to track regressions).

## 2. The usual suspects (hot-loop killers)

**String concat in loops — O(n²):**
```python
# Bad
out = ""
for part in parts:
    out += render(part)

# Good — O(n)
out = "".join(render(p) for p in parts)
# Many formatted pieces: io.StringIO or a list + join
```

**Membership tests against a list — O(n) each:**
```python
# Bad — O(len(allowed)) per check, O(n*m) total
if user.role in allowed_roles_list: ...

# Good — build once, O(1) per check
allowed = frozenset(allowed_roles_list)
if user.role in allowed: ...
```
Same for dedupe: `seen: set[str]` not a list. `list.pop(0)`/`insert(0, ...)` → `deque`.

**Repeated lookups in hot loops** — attribute/global/method resolution costs per iteration:
```python
# Good — hoist invariants out of the loop
append = results.append          # method lookup once
pattern = re.compile(r"...")     # NEVER re.compile inside the loop; module level
for line in lines:
    m = pattern.match(line)
    if m:
        append(m.group(1))
```
Also hoist `len()`, dotted constants (`self.config.threshold` → local), and dict `.get`
bound methods. Only do this in *measured* hot loops — it hurts readability.

**List vs generator:**
- Generator when you iterate once or might stop early (`any`, `next`, streaming) — O(1) memory.
- List when you iterate multiple times, need `len`/indexing, or the consumer is `str.join`
  (join materializes anyway — a list is marginally faster there).
- Never `list(...)` just to loop over it once.

**Other classics:** `sort(key=...)` not `functools.cmp_to_key`; `Counter` not manual dict
counting; `dict.setdefault`/`defaultdict` not check-then-insert; exception setup is cheap but
raising in a hot loop is not — restructure if a "miss" is the common case.

## 3. Vectorize: numpy / polars over Python loops

A Python-level loop over a million floats is ~100× slower than the vectorized equivalent.

```python
# Bad — interpreter-bound
total = 0.0
for row in rows:
    if row["qty"] > 0:
        total += row["qty"] * row["price"]

# Good — polars (multi-threaded, lazy, no GIL contention)
total = (
    df.lazy()
      .filter(pl.col("qty") > 0)
      .select((pl.col("qty") * pl.col("price")).sum())
      .collect()
      .item()
)

# numpy equivalent for array math
mask = qty > 0
total = float(np.dot(qty[mask], price[mask]))
```

- Tabular pipelines in 2026: **polars** by default (lazy frames, predicate pushdown, Arrow
  interop); pandas where ecosystem demands it — then prefer Arrow-backed dtypes and avoid
  `DataFrame.apply` with Python lambdas (it's a loop in disguise) and `iterrows` (worst case).
- The cardinal sin is *mixed* mode: a vectorized frame iterated row-by-row in Python.
  If you must apply a Python function elementwise, you've left the fast path — reconsider
  (expression API, `np.vectorize` is NOT faster, numba/cython for genuine custom kernels).
- Crossing the boundary costs: batch conversions (`tolist()` once, not `float(x)` per element).

## 4. Caching with functools — and the caveats

```python
@functools.cache                      # unbounded — only for small, finite domains
def parse_spec(spec: str) -> Spec: ...

@functools.lru_cache(maxsize=1024)    # bounded — default choice for hot pure functions
def resolve(name: str) -> Target: ...
```

Caveats that bite:
- **Unbounded `@cache` on user-influenced arguments = memory leak / DoS.** If callers control
  the argument space, use `lru_cache(maxsize=N)` or an external TTL cache (`cachetools.TTLCache`).
- **Methods:** `@cache` on an instance method keys on `self` → the cache keeps every instance
  alive forever. Use `@cached_property` for per-instance memoization, or cache a module-level
  function taking explicit args.
- **Invalidation:** functools caches never expire. Anything whose answer can change (config,
  DNS, feature flags, files) needs TTL or explicit `cache_clear()` wired to the change event —
  and `cache_clear()` in tests (autouse fixture) or tests pollute each other.
- Arguments must be hashable; `lru_cache` keys distinguish `f(1)` from `f(x=1)`.
- Caching wraps exceptions? No — exceptions are not cached; a failing call re-executes (can
  stampede). For expensive fallible calls add single-flight locking.

## 5. Concurrency model: threads vs processes vs asyncio

| Workload | Choice | Why |
|---|---|---|
| Many slow network calls, one service | **asyncio** | thousands of concurrent ops, single thread, no sync overhead |
| Blocking-library I/O, moderate fan-out | **ThreadPoolExecutor** | GIL released during I/O; simplest retrofit |
| CPU-bound pure Python | **ProcessPoolExecutor** / multiprocessing | GIL serializes threads; processes get real parallelism |
| CPU-bound numpy/polars | often **none needed** | native code releases the GIL / is internally parallel |
| CPU-bound on a free-threaded build (officially supported since 3.14, PEP 779) | threads become viable | measure first; see rules/01 §8 |

- GIL reality (default build): threads never speed up pure-Python CPU work; they *do* help
  I/O and GIL-releasing extensions. Don't "add threads" to a CPU loop and call it done.
- Process pools: pickling costs dominate small tasks — chunk work; pass paths/ids, not fat
  objects; prefer `concurrent.futures` API over raw `multiprocessing`. 3.14 changed the
  Unix default start method from `fork` to `forkserver` (Windows/macOS stay `spawn`) —
  module-global state is no longer inherited; guard with `if __name__ == "__main__"` and
  request `fork` explicitly via `get_context("fork")` only if you must.
- Subinterpreters shipped in 3.14 (PEP 734): `concurrent.interpreters` +
  `concurrent.futures.InterpreterPoolExecutor` — share a process, isolated GILs. A real
  middle ground now, but ecosystem support is young; benchmark before betting on it.
- Mixing: asyncio app with CPU spikes → `loop.run_in_executor(process_pool, ...)` (rules/04 §3).

## 6. Startup latency: lazy imports

CLI tools and serverless functions pay import cost on every invocation. `import pandas`
alone can be 500ms+.

```python
# Bad — every `mycli --help` pays for pandas
import pandas as pd

def report_cmd(path: str) -> None: ...

# Good — defer heavy imports into the command that needs them
def report_cmd(path: str) -> None:
    import pandas as pd            # local import: only this command pays
    ...
```

- Measure with `python -X importtime -c "import mypkg" 2>&1 | sort -t'|' -k2 -rn | head` or
  `tuna` for visualization.
- Combine with `TYPE_CHECKING` imports for annotation-only deps (rules/02 §9).
- PEP 810 explicit lazy imports (`lazy import x`, `-X lazy_imports`) ships in 3.15
  (currently in beta); until your floor is 3.15, function-local imports are the idiom.
  Don't lazy-import inside hot loops (lookup cost per call is small but real —
  module-level once the function is hot path).

## 7. Memory

- `slots=True` dataclasses (rules/03 §7) for objects allocated in volume.
- Generators/streaming over materializing: read files with iteration, not `.read()` of
  multi-GB files; `json` → `ijson`/msgspec streaming for huge payloads.
- `sys.intern()` for millions of repeated strings (parser tokens, column values).
- numpy/Arrow buffers instead of lists-of-floats: 24 bytes/float object vs 8 bytes flat.
- Watch accidental retention: caches on methods (§4), default mutable args, closures over
  large frames, `lru_cache` on functions taking DataFrames.

## 8. I/O and serialization throughput

CPU profiles often blame the interpreter when the real cost is chatty I/O or slow codecs:

- **Batch the round trips.** One query returning 1,000 rows beats 1,000 queries (rules/07
  N+1); one `executemany`/`COPY`/`bulk_create` beats row-at-a-time inserts; pipeline redis
  commands; batch S3/HTTP calls behind a concurrency-bounded TaskGroup (rules/04 §9).
- **JSON:** stdlib `json` is the slow path. For hot serialization use `msgspec` (fastest,
  typed decode in one step) or `orjson`. Typed `msgspec.Struct` decode replaces
  json.loads + pydantic validation at a fraction of the cost when the schema is fixed —
  keep pydantic where you need its coercion/error UX, not on the hot path of an internal
  service mesh.
- **File reading:** iterate (`for line in f`) instead of `.read().splitlines()` on big
  files; `Path.read_bytes()` once instead of many small reads; `mmap` for random access to
  large read-only files.
- **Compression trade:** zstd dominates gzip on both axes for service-to-service payloads
  and cache entries; gzip survives only for compatibility. On 3.14+ it's stdlib
  (`compression.zstd`, PEP 784); older floors use the `zstandard` package.
- Buffered writes: thousands of small `f.write()` calls are fine (buffered), but thousands
  of `open()/close()` cycles are not — hoist the file handle out of the loop.

## 9. Don'ts

- Don't micro-optimize cold code; don't trade clarity for unmeasured wins.
- Don't catch the "optimization" of removing logging — fix lazy formatting instead (rules/03 §11).
- Don't hand-roll C extensions before trying polars/numpy/numba/Rust-via-pyo3 — maintenance
  cost dwarfs the speedup.
- Don't benchmark with `time.time()` — `time.perf_counter()` or pyperf.

## Audit checklist

```bash
# Ruff perf rules
uvx ruff check --select PERF,C4,SIM --statistics .

# String building & containers [hot-loop suspects]
grep -rn "+= .*str(\|+= f\"\|+= \"" --include="*.py" src/        # concat in loops? check context
grep -rn "\.pop(0)\|insert(0," --include="*.py" src/             # O(n) deque ops
grep -rn "in \[" --include="*.py" src/ | head                    # list membership in conditions

# re.compile / invariant work inside loops [manual: confirm loop context]
grep -rn "re.compile\|re.match\|re.search" --include="*.py" src/ | head -30

# DataFrame antipatterns [MEDIUM in data code]
grep -rn "iterrows\|itertuples\|\.apply(lambda" --include="*.py" src/
grep -rn "for .* in df\[" --include="*.py" src/

# Cache hygiene
grep -rn "@cache$\|@functools.cache" --include="*.py" src/       # unbounded — user-controlled args? on methods?
grep -rn "@lru_cache" --include="*.py" src/ -A2 | grep "def .*(self"   # instance leak [MEDIUM]
grep -rn "cache_clear" --include="*.py" tests/ src/              # invalidation/test isolation present?

# Concurrency model sanity
grep -rn "ThreadPoolExecutor" --include="*.py" src/              # used for CPU-bound work? [MEDIUM]
grep -rn "multiprocessing\|ProcessPoolExecutor" --include="*.py" src/ | head
grep -rln "if __name__" $(grep -rln ProcessPoolExecutor --include="*.py" src/ 2>/dev/null)

# Import-time cost (CLIs/lambdas)
python -X importtime -c "import mypkg" 2>&1 | sort -t'|' -k2 -rn | head -15
grep -rn "^import pandas\|^import numpy\|^import torch" --include="*.py" src/*cli* src/*/cli* 2>/dev/null

# Benchmarks exist for perf-critical code?
grep -rln "pytest-benchmark\|pyperf" pyproject.toml tests/ 2>/dev/null
```

# 07 — FastAPI, Django, and pytest Mastery

Framework-specific traps plus the testing discipline that applies everywhere. Deep general
rules live elsewhere (async → rules/04, validation → rules/02, SQL safety → rules/05).

## 1. FastAPI

### Pydantic models at the boundary — separate in/out

```python
class UserIn(BaseModel):
    model_config = {"extra": "forbid"}
    email: EmailStr
    password: SecretStr

class UserOut(BaseModel):
    id: int
    email: EmailStr            # no password field — response_model filters output

@router.post("/users", response_model=UserOut, status_code=201)
async def create_user(payload: UserIn, svc: UserService = Depends(get_user_service)) -> UserOut: ...
```

- Distinct request/response models. Returning ORM objects without `response_model` leaks
  columns you forgot existed (password hashes, internal flags) — recurring HIGH finding.
- `extra="forbid"` on inputs; `SecretStr` for credentials (won't repr into logs).
- Validation errors are FastAPI's job — don't pre-validate manually then re-wrap.

### Dependency injection done right

- Dependencies are the composition mechanism: DB sessions, auth, settings, clients all come
  in via `Depends`, never module-level globals — this is what makes handlers testable via
  `app.dependency_overrides`.
- `yield`-dependencies for resources (session per request, commit/rollback in the
  finally/except of the dependency, not in handlers).
- Singletons (settings, http client, pools) live on the **lifespan**:
  ```python
  @asynccontextmanager
  async def lifespan(app: FastAPI):
      app.state.http = httpx.AsyncClient(timeout=10)
      yield
      await app.state.http.aclose()
  ```
- `Annotated[UserService, Depends(get_user_service)]` aliases kill signature noise.
- Heavy pure-validation dependencies: `use_cache=True` is the default — don't re-resolve per
  sub-dependency; do not hide business logic in dependencies.

### The sync-in-async trap (most common FastAPI perf bug)

- `async def` endpoint + blocking call (requests, sync SQLAlchemy session, `time.sleep`)
  → blocks the **single** event loop; whole service serializes. See rules/04 §3.
- `def` (sync) endpoint → FastAPI runs it in the anyio threadpool (default ~40 threads) —
  blocking is *safe* there but throughput caps at pool size.
- Rule: endpoint is `async def` **only if everything it awaits is truly async** (asyncpg /
  SQLAlchemy async session / httpx.AsyncClient). Mixed stack? Make the endpoint `def` and
  stay sync, or fix the stack. An `async def` endpoint with zero `await` inside is a bug
  marker — it gains nothing and risks someone adding blocking calls later.
- **Exception: the liveness probe.** A liveness endpoint is *supposed* to await nothing —
  its job is "is the event loop responsive", so `sota-observability` rules/05 §1 specifies
  process-internal-only, "usually just return 200". Written as `def` it runs in the anyio
  threadpool, where saturation by slow sync handlers delays the probe and the orchestrator
  restarts a process whose event loop was fine — the restart storm rules/05 exists to
  prevent. So a no-`await` `async def` is **correct** here and the marker does not apply.
  Say so in the docstring, or the next reader "fixes" it. Readiness (`/readyz`), which does
  check dependencies, follows the normal rule.
- Background work: `BackgroundTasks` for small post-response work; real job queue
  (arq/celery/temporal) for anything that must survive a process restart.

## 2. Django

### ORM: N+1 is the default — defeat it explicitly

```python
# Bad — 1 query for orders + 1 per order for .customer + 1 per order for items
for order in Order.objects.all():
    print(order.customer.name, [i.sku for i in order.items.all()])

# Good
orders = (
    Order.objects
    .select_related("customer")            # FK/O2O → SQL JOIN
    .prefetch_related("items")             # M2M/reverse FK → 2nd query + join in Python
)
```

- `select_related` for forward FK/OneToOne; `prefetch_related` for M2M and reverse relations;
  `Prefetch(queryset=...)` to filter/order the prefetched set.
- Make N+1 a test failure: `django-assert-num-queries` /
  `self.assertNumQueries(2)`, or `nplusone`/`django-zen-queries` in dev.
- Other ORM rules: `.only()/.defer()` for wide tables on hot paths; `exists()` not
  `count() > 0` not `len(qs)`; `bulk_create/bulk_update` for batch writes; `update()`
  for field bumps instead of load-modify-save races — or `F()` expressions for atomic
  increments; `iterator(chunk_size=...)` for large scans; aggregate in the DB
  (`annotate/aggregate`), not in Python.
- Querysets are lazy and **cached per object** — slicing/re-filtering re-queries; assigning
  `qs = qs.filter(...)` builds SQL, `if qs:` executes it. Know which line hits the DB.

### Migrations discipline

- One logical change per migration; **never edit an applied migration** — write a new one.
- `python manage.py makemigrations --check` in CI: fails when models drifted from migrations.
- Zero-downtime ordering: additive first (nullable column / new table), deploy code that
  writes both, backfill in batches (separate data migration with `RunPython` + reverse func),
  then constrain/drop in a later release. Never `NOT NULL` + default on a huge table in one
  step (lock).
- Data migrations use `apps.get_model("app", "Model")`, never direct model imports (the
  model's current code may not match the schema at that point in history).
- Squash periodically; name migrations (`0042_order_add_status_index`, not `auto_...`).

### Django misc

- Django 6.0 (Dec 2025): built-in **Tasks framework** for background work — prefer it over
  bolting on celery for simple deferred jobs (it still needs a worker/backend in prod);
  native **Content Security Policy** support (replaces `django-csp`); template partials.
  Django 5.2 remains the LTS — don't flag staying on it as a finding.
- `async def` views must not call the sync ORM directly — use `aget/afirst/acount` async
  ORM methods or `sync_to_async` wrappers; a blocking ORM call in an async view under ASGI
  stalls the loop (rules/04 §7).
- Settings via `django-environ`/env vars; `DEBUG=False`, `ALLOWED_HOSTS`, `SECRET_KEY` from
  secrets store — run `manage.py check --deploy` in CI.
- Keep `.raw()`, `.extra()`, `RawSQL` out of the codebase or parameterized + reviewed
  (rules/05 §3).

## 3. pytest mastery

### Fixtures over setup, composition over inheritance

```python
@pytest.fixture
def db_session(engine) -> Iterator[Session]:     # depends on another fixture
    with engine.begin() as conn:
        session = Session(bind=conn)
        yield session
        session.rollback()                       # teardown after yield — always runs

@pytest.fixture
def user(db_session) -> User:
    return UserFactory.create(session=db_session)
```

- No `unittest.TestCase` setUp/tearDown in new code; fixtures compose, are scoped, and are
  request-only-what-you-need.
- Scope deliberately: `session` scope for expensive immutable resources (containers via
  `testcontainers`, compiled artifacts); `function` scope (default) for anything mutable.
  A session-scoped fixture yielding a mutable object is a test-pollution factory.
- `conftest.py` per directory for shared fixtures; no `from tests.helpers import *`.
- `autouse=True` sparingly — invisible dependencies; acceptable for isolation guards
  (clearing caches, freezing time, fake env).

### Parametrize, don't copy-paste

```python
@pytest.mark.parametrize(
    ("raw", "expected"),
    [
        pytest.param("1h30m", 5400, id="hours-minutes"),
        pytest.param("90s", 90, id="seconds"),
        pytest.param("", None, id="empty", marks=pytest.mark.xfail(raises=ParseError)),
    ],
)
def test_parse_duration(raw: str, expected: int | None) -> None:
    assert parse_duration(raw) == expected
```

- `id=` on every param — `test_parse[2]` failures are unreadable.
- Stack parametrize decorators for cartesian products; parametrize fixtures
  (`params=` on the fixture) when the *resource* varies (e.g., each DB backend).

### No test interdependence

- Every test runs alone and in any order: `pytest -p no:randomly` shouldn't be needed —
  install `pytest-randomly` and keep it green.
- Banned: module-level mutable state shared across tests, tests that rely on execution
  order, `lru_cache`d functions tested without `cache_clear` between tests (rules/06 §4),
  writes to shared tmp paths — use the `tmp_path` fixture, env mutation without
  `monkeypatch.setenv` (which auto-reverts).
- Verify independence in CI occasionally: `pytest -x --randomly-seed=last`, and
  `pytest-xdist` (`-n auto`) — parallel-unsafe tests are interdependent tests.

### Property-based testing with hypothesis

```python
from hypothesis import given, strategies as st

@given(st.text())
def test_roundtrip(s: str) -> None:
    assert decode(encode(s)) == s            # invariant, not example

@given(st.lists(st.integers()))
def test_sort_idempotent(xs: list[int]) -> None:
    assert my_sort(my_sort(xs)) == my_sort(xs)
```

- Use for parsers, serializers, codecs, numeric routines, anything with an invariant
  (roundtrip, idempotence, commutativity, oracle vs reference impl).
- Persist the example database (`.hypothesis/` in CI cache) so regressions replay;
  failing examples get promoted to `@example(...)` regression pins.

### Markers, selection, and suite layering

```toml
[tool.pytest]                       # pytest 9+ native table — real TOML types (arrays, bools)
addopts = ["-ra", "--strict-markers", "--strict-config"]
markers = [
  "slow: takes >1s, excluded from default run",
  "integration: needs docker services",
]
testpaths = ["tests"]
```

- pytest 9 (Nov 2025): the native `[tool.pytest]` table (or a `pytest.toml` file) replaces
  `[tool.pytest.ini_options]` as the state-of-the-art spelling — `ini_options` still works
  pre- and post-9, but the two tables cannot coexist. 9 also drops Python 3.9 and merges
  `pytest-subtests` into core (`subtests` fixture — no plugin needed for loop-style asserts).
- `--strict-markers` always — a typo'd `@pytest.mark.integratoin` silently creates a marker
  and your "skip integration" filter stops matching.
- Layer the suite: fast unit tests run on every push (`-m "not slow and not integration"`);
  integration tests (testcontainers, real DB) run in CI on a service matrix. A suite that
  takes 20 minutes locally stops being run locally.
- `-ra` in addopts so skipped/xfailed reasons are visible; unexplained skips rot into dead
  tests.

### General pytest hygiene

- Plain `assert` with rich introspection — no `assertEquals` ports, no bare
  `assert response` when you mean `assert response.status_code == 200`.
- `pytest.raises(SpecificError, match=r"...")` — never `pytest.raises(Exception)`.
- Mock at the boundary you own (`mocker.patch.object(svc, "client")`), not deep internals;
  patch where the name is *looked up*, not where it's defined. Over-mocked tests that
  assert call sequences test the mock, not the code — prefer fakes (in-memory repo).
- Async tests: `asyncio_mode = "auto"` (rules/04 §9). Time: `freezegun`/`time-machine`,
  never `sleep`.
- Coverage gate (`--cov --cov-fail-under=N`) measures *executed*, not *asserted* — treat as
  floor, not target; mutation testing (`mutmut`) where correctness is critical.

## Audit checklist

```bash
# FastAPI
grep -rn "async def" $(grep -rln "APIRouter\|FastAPI" --include="*.py" src/) | head   # then check for blocking calls inside
grep -rn "requests\.\|time.sleep\|session.query\|Session(" --include="*.py" src/ | grep -i route  # sync-in-async [HIGH]
grep -rn "@\(app\|router\)\.\(get\|post\|put\|delete\)" --include="*.py" src/ -A3 | grep -L response_model | head  # ORM leak risk
grep -rn "AsyncClient()" --include="*.py" src/ | grep -v lifespan              # per-request clients [MEDIUM]
grep -rn -B2 "livez\|/health\|healthz" --include="*.py" src/ | grep "^.*def "  # liveness: `async def`, no deps (§1 exception)
grep -rn "^[A-Z_]* = .*Session\|^engine = " --include="*.py" src/              # module-global state vs Depends

# Django ORM
grep -rn "\.objects\.all()\|\.objects\.filter" --include="*.py" src/ | wc -l
grep -rln "select_related\|prefetch_related" --include="*.py" src/ | wc -l     # ratio sanity check
grep -rn "for .* in .*\.objects\." --include="*.py" src/ -A2 | grep "\.\(name\|user\|customer\)" | head  # N+1 candidates
grep -rn "count() > 0\|len(.*objects" --include="*.py" src/                    # exists() instead
grep -rn "\.raw(\|\.extra(\|RawSQL" --include="*.py" src/                      # [HIGH if interpolated]
git log --oneline -- '**/migrations/*.py' | head                               # edited-after-merge migrations?
grep -rn "makemigrations --check" .github/ .gitlab-ci.yml 2>/dev/null          # drift gate present?

# pytest
grep -rn "def setUp\|TestCase" --include="*.py" tests/                         # legacy style [LOW]
grep -rn "pytest.raises(Exception)" --include="*.py" tests/                    # too-broad [MEDIUM]
grep -rn "time.sleep" --include="*.py" tests/                                  # flaky timing [MEDIUM]
grep -rn "scope=\"session\"\|scope=\"module\"" --include="*.py" tests/ conftest.py 2>/dev/null  # mutable shared state?
grep -rn "os.environ\[" --include="*.py" tests/ | grep -v monkeypatch          # env pollution
grep -rln "parametrize" --include="*.py" tests/ | wc -l
grep -rln "hypothesis" --include="*.py" tests/ || echo "no property tests"
pytest -q -n auto 2>&1 | tail -3                                               # parallel-safe = independent
pytest -q -p randomly 2>&1 | tail -3                                           # order-independent?
```