Claude Skill

declaring-invariants

Find tests that enumerate a domain by copying it, and declare the invariants a codebase depends on. Reports where a parametrize list, for-loop, or it.each iterates a hand-written subset of a dict/set/tuple/Enum that exists in the source, and names the members nothing covers. Use

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download oaustegard-claude-skills-plugins_code-intelligence_skills_declaring-invariants-e39c726.zip · 35 KB
Part of oaustegard/claude-skills — 39 skills

Install

skills CLI npx skills add https://github.com/oaustegard/claude-skills/tree/main/plugins/code-intelligence/skills/declaring-invariants
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oaustegard-claude-skills@llmmart
Git git clone https://github.com/oaustegard/claude-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole oaustegard/claude-skills collection as a plugin from our marketplace. Git is the plain clone.

README

declaring-invariants

Find tests that enumerate a domain by copying it, and declare the invariants a codebase depends on. Stdlib ast only — no install, no config file, no network. Python.

python3 scripts/totality_lint.py <repo>   # tests that copy a domain
python3 scripts/claims.py <repo>          # what the repo declares, and what backs it

Features

  • Copied-domain detection — a parametrize or for over a literal whose members are a strict subset of a dict/set/tuple/Enum in the source, with the uncovered members named
  • Ratchets, the second claim form — an enumeration loops whatever the domain now holds, so it cannot see the domain narrowing. # totality: ratchet — <why> marks a hand-list as a deliberate floor, and the linter checks the pin: ratchet-broken names a member that left, statically, before any test runs. unratcheted names a registry with an enumeration and no floor
  • Vacuity detection — a live registry iterated with no len(...) >= n assertion, which passes over an emptied collection
  • Membership join — [1, 2, 3, 4, 8] and SUPPORTED_BITS share no token, so containment is the join key; no naming convention is assumed
  • Reachability filter — a literal matches a registry only when the test imports its module, shares its top-level directory, or is its paired tests/test_<mod>.py
  • First-class acknowledgement — # totality: partial — <why> retires a finding, and an acknowledgement on a test that later covers the whole domain is reported as stale-ack
  • Claim inventory — a claim is a test whose docstring opens invariant:, and refuted: records the observed negative control
  • Report by default — --strict opts into a nonzero exit; --json for machine consumption; --selftest runs fixtures with no repo

Why

On oaustegard/remex, adding a fourth member to ROTATION_CODES with no construction behind it left the entire 267-test suite green. Four tests looked total; each parametrized ["haar", "rht"] against a three-member registry.

The other direction is worse. Substituting "xyz" for "none" in both spellings of that domain — cardinality unchanged, both spellings agreeing — left the domain floor, the enumeration and the parity check green, five passing tests over a registry that had stopped supporting a rotation every index on disk was written with. That is what the ratchet form is for.

Adapted from the meta-oracle in daniloc/coherence, which classifies an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST. The check needs none of that harness's spec files, claim grammar, ledger or Node runtime.

See SKILL.md for the full reference, including how to write a refutation you have actually observed.

Skill manifest

declaring-invariants

Two scripts over one idea: a test that enumerates a domain must loop the registry rather than a copy of it. Someone also has to say which domains matter in the first place.

python3 scripts/totality_lint.py <repo>   # tests that copy a domain
python3 scripts/claims.py <repo>          # what the repo declares, and what backs it

Python only, stdlib ast only: no install, no config file, no network.

The failure it catches

A test that loops a hand-written list passes its runner and proves nothing about completeness. When the same members also exist as a registry in the source, the list is a copy, and the copy drifts the moment someone adds a member to the registry and not to the test. Nothing goes red.

Measured on oaustegard/remex, 2026-08-24: adding a fourth member to ROTATION_CODES with no construction behind it left the entire 267-test suite green. Four separate tests looked total; each parametrized ["haar", "rht"] against a three-member registry. Only a test that looped the registry itself caught it.

Adapted from the meta-oracle in daniloc/coherence (src/oracle-domain.ts), which classifies an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST. That harness needs spec files, a claim grammar, a ledger and Node; the check does not.

totality_lint.py — tests that copy a domain

finding meaning
sampled-domain a parametrize or for over a literal whose members are a strict subset of a source registry. The uncovered members are named.
ratchet-broken a hand-list marked ratchet names a member the registry no longer contains. Detected statically, without running anything.
unratcheted a registry enumerated live with nothing pinning its membership. Suppressed when no-floor already claimed the same line.
no-floor a test iterates a live registry with no len(...) >= n assertion in the file, so an emptied registry passes vacuously.
stale-ack an acknowledgement on a test that now covers the whole domain.
python3 scripts/totality_lint.py <repo>            # the report
python3 scripts/totality_lint.py <repo> --json
python3 scripts/totality_lint.py <repo> --strict   # exit 1 if any finding
python3 scripts/totality_lint.py --selftest        # fixtures, no repo

A partial domain is often correct. Say so on the test and it stops being a finding:

# totality: partial — mojo has no construction for "none"
@pytest.mark.parametrize("rotation", ["haar", "rht"])
def test_save_params_accepts_every_mojo_rotation(rotation): ...

The marker also works as totality: partial — <why> inside the docstring. An acknowledgement on a test that later covers the whole domain is reported as stale-ack, so a suppression cannot become a silence.

Why a hand-list is the second form

An enumeration loops whatever the domain currently holds, so it is structurally blind to the domain narrowing. Remove a member and the loop simply ranges over fewer of them, green. Raised by Yep, 2026-08-24; reproduced on oaustegard/remex before being believed:

perturbation of ROTATION_CODES domain floor enumeration parity hand-list
grow — add "hadamard2" pass RED RED pass
shrink — drop "none" RED pass RED* pass
substitute — "none" → "xyz" in both spellings pass pass pass pass

* only because the other spelling of the domain did not shrink, which is incidental to that repository.

In the substitution row cardinality holds, both spellings agree, and five green tests cover a registry that quietly stopped supporting a rotation every index on disk was written with. The floor is a cardinality check; it cannot see a member swapped for another.

The second form is a ratchet: a hand-list asserting the domain keeps containing it. Mark it and the linter checks the pin rather than taking the marker's word for it.

# totality: ratchet — these three shipped; one leaving is a compatibility break
def test_no_shipped_rotation_is_ever_removed():
    for shipped in ("haar", "rht", "none"):
        assert shipped in ROTATION_CODES

Under that same substitution the linter reports ratchet-broken naming 'none' statically, before any test runs, and the test itself goes red while the other five pass.

So: an enumeration proves every current member is handled, and a ratchet proves no member left without a decision. unratcheted names a registry that has the first and not the second. Neither half alone is the answer.

A ratchet covers shrink only. Over a strict subset it still reports sampled-domain for the members it never listed, because pinning two of three says nothing about the third. Suppressing that would make the marker a laundering channel — the escape hatch these notes criticise via guard for being in coherence. It was one: until 2026-08-25, # totality: ratchet over two of three members silenced the report entirely and checked nothing about the third. Caught by an adversarial pass on this skill, not by its own tests.

claims.py — what the repo declares

totality_lint asks whether a test's domain is complete. It presumes a test exists. This asks the prior question, the one coherence's own Known Limits concedes it does not answer: nothing enforces exists ⇒ declared.

A claim is a test whose docstring opens with invariant:. No new file format, and the claim inherits its test's pass/fail:

def test_every_skipped_directory_is_actually_skipped(self):
    """invariant: every name in SKIP_DIRS is excluded from the walk.

    refuted: replaced the walk's `part in SKIP_DIRS` check with
    `part in {"node_modules"}` -> this test went red naming `.coherence`,
    while the other 25 tests in this file stayed green.
    """
finding meaning
unrefuted a claim nobody has watched fail
literal the claiming test iterates a copy of the registry, so the claim cannot see a new member
unanchored a registry no invariant names — a question, not a verdict

A claim whose test carries a ratchet marker is reported as pinning its registries rather than copying them, and a ratcheted registry is not unanchored.

python3 scripts/claims.py <repo> [--json] [--strict] [--selftest]

Write the refutation from what you observed

refuted: is the half that costs something. Break the chokepoint, watch the claim go red by name, restore, and record what you saw. A green test and an unfalsifiable one look identical from outside; the refutation is what separates them.

Never write a refutation you have not run. The first refutation authored for the SKIP_DIRS invariant above asserted a failure that did not occur — the fixture placed the registry outside the test's reachability, so the test passed under perturbation for an unrelated reason. A vacuous claim, written while building the tool that catches vacuous claims, and caught only by running the perturbation instead of trusting the sentence.

Procedure, in order:

  1. Break the chokepoint the claim names, with one edit in the source rather than in the test.
  2. Run the claiming test. Read the failure text. Note the member it named.
  3. Run the rest of the file. Confirm the others stay green — if everything goes red, the claim is not localised and the refutation says nothing.
  4. Restore the source.
  5. Write refuted: <the edit> -> <what went red, by name>.

Wiring it into a commit gate

The reference wiring lives in oaustegard/claude-workspace (scripts/tdd_hook.py): a commit where a registry gained a member and no invariant: test iterates it live is denied, naming the registry and what it gained. Override with no-invariant: <why> in the commit body.

Registry shrink is gated the same way, and needs a ratchet rather than an enumeration to clear it. A new function or a new branch is behavioural growth too, but neither is diffable without guessing, and a gate that guesses stops being consulted. A brand-new registry is not gated. Declaring one is a judgement call; growing one has already made it.

Where the two filters came from

Both filters exist because the unfiltered version was noise. Reproduce either by removing the filter and re-running against a real repo.

  • no-floor fires only on a name independently recognised as a registry. Firing on every for x in <local> produced 27 findings on remex, all noise: numpy arrays, query matrices, loop counters.
  • A literal matches a registry only when reachable — the test imports its module or package, shares its top-level directory, or is its paired tests/test_<mod>.py. Matching against any registry in the tree joined a test in discrepancy/ to registries in kb-k-sweep/ and remex-vs-higgs-ablation/ on a monorepo, because small integer sets collide by chance. Four findings became one, and the survivor was real.

The join key is membership, not names: [1, 2, 3, 4, 8] and SUPPORTED_BITS share no token, so containment is what ties them together. No naming convention is assumed, and none is required.

Limits

  • Python only. The extractors are _registries_py and _domains_py; a tree-sitter pair for another language slots in beside them. The join, the acknowledgements and the reports are all language-independent.
  • Report, not gate, by default. --strict opts into a nonzero exit. The tool this was adapted from gates by default, and its parity arm false-fails a correct oracle that binds its domain to a local name first. A gate that false-fails stops being consulted.
  • A registry is a collection of constants. A dict/set/tuple/list of literals, or an Enum body, with at least three members. A domain assembled at runtime is invisible here.
  • Multi-parameter tables are out of scope. Only single-name parametrize is read.
  • unanchored is a question. Most registries need no invariant. Treat the list as candidates for declaration, never as a backlog to clear.
  • A registry is modelled as a SET OF KEYS, so a value swap is invisible. This is the largest hole in the design, and it undercuts the motivating example. ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2} exists to pin bytes on disk; permuting it to {"haar": 0, "rht": 2, "none": 1} keeps every key, every count and every ratchet intact, and every index already written decodes under the wrong rotation. Verified 2026-08-25 on a fixture: the linter reports nothing and the gate's registry half reports nothing. (The gate denied that fixture, but on the unrelated TDD rule — checked, because claiming otherwise would have been the overclaim this skill exists to catch.) Nothing here checks a name-to-code mapping. Extracting (key, value) pairs as the member set would, at the cost of breaking the subset join against a parametrize list of keys.
  • A co-ordinated rename defeats the ratchet. A global find/replace that renames a member in the registry AND in the hand-list leaves the pin intact, and this reports nothing. Verified 2026-08-25 on a fixture: renaming "none" to "identity" in both files was silent, while every index already on disk still decodes byte 2 as the old name. Both sides of a co-located hand-list move together, so no static check over one working tree can see it. The commit gate can, because it diffs against git history — and only if the ratchet itself is untouched in that commit, which it now requires.
  • A brand-new registry is neither gated nor pinned. Create one with five members and never grow it and no gate ever fires. unanchored and unratcheted surface it in the report; the gate deliberately does not, on the grounds that gating every new module is how a gate stops being consulted. That is a judgement, not a proof, and it is the largest hole a cross-model review found.
  • A decorator-built registry is invisible. @register("name") populating a dict at import time is the common Python registry idiom and is not a literal, so nothing here sees it. Named because it is the shape most likely to be mistaken for coverage.
  • Reachability is path-based, so a disconnected integration test is missed. A test that neither imports the module, shares its top-level directory, nor pairs with it by filename will not be joined to the registry it samples. The filter trades that recall for the cross-project precision it was measured to buy.
  • The precision numbers come from three repositories, and the filters were fitted to two of them. 27 noise findings on one, 3-of-4 cross-project joins on another. Those are the measurements that justified each filter; they are not a false-positive rate on a corpus, and should not be read as one.
  • Split and merge are invisible. The diff is keyed on registry NAME, so renaming a registry, splitting one in two, or merging two into one falls through both the gained and lost paths.
  • A claim that passes is not a claim that is right. This checks that a declared invariant loops the domain it names. Whether it is the right invariant is human judgement, and it is not automatable.

Related

  • verifying-claims covers the prose layer: does the documentation match reality? Agent-judged, non-deterministic, run as a triggered review. This skill is the deterministic half, over code and tests rather than prose.
  • tree-sitting locates the registry or the test before you edit it.
Files (claude-skills)
  • scripts
    • claims.py 15.2 KB
      #!/usr/bin/env python3
      """claims.py — the declared invariants of a repository, and what backs them.
      
      `totality_lint.py` answers "is this test's domain complete?". It presumes a
      test exists. The harder question is the one coherence's own Known Limits
      concedes it does not answer: *nothing enforces `exists ⇒ declared`*. An
      invariant the code depends on but no test names is invisible to every gate.
      
      A claim here is a test whose docstring opens with `invariant:`. No new file
      format, no new syntax, and the claim inherits its test's pass/fail for free:
      
          def test_every_declared_rotation_round_trips(rotation):
              '''invariant: every declared rotation survives the on-disk formats.
      
              refuted: added "hadamard2" to ROTATION_CODES with no construction
              behind it -> the whole suite stayed green at 267 passed while this
              went red on [hadamard2].
              '''
      
      Two lines, two jobs. `invariant:` states what must hold. `refuted:` records
      what was broken on purpose and what went red — the observed negative control,
      and the only thing separating a claim that holds from a claim that cannot
      fail. A green test and an unfalsifiable one are indistinguishable from
      outside; the refutation is what tells them apart, and it is worth writing down
      because it is cheap to produce once and impossible to reconstruct later.
      
      Findings:
      
        unrefuted   an invariant with no `refuted:` line. Advisory, and the whole
                    point: break the chokepoint, watch it go red by name, restore,
                    write down what you saw.
      
        unanchored  a registry in source with three or more members that no
                    invariant names. A candidate for declaration, not a defect —
                    most registries do not need one.
      
        literal     an invariant whose test iterates a copied literal rather than
                    the registry. `totality_lint.py` grades this; it is surfaced
                    here so one report answers "what do we claim, and does the
                    claim mean anything".
      
          python3 scripts/claims.py <path>            # the report
          python3 scripts/claims.py <path> --json
          python3 scripts/claims.py <path> --strict   # exit 1 if any finding
          python3 scripts/claims.py --selftest        # fixtures, no repo
      
      Python only, same as `totality_lint.py`, whose registry and domain extractors
      this reuses rather than reimplementing.
      """
      
      from __future__ import annotations
      
      import argparse
      import ast
      import importlib.util
      import json
      import pathlib
      import re
      import sys
      from dataclasses import dataclass, field
      
      
      def _load_totality_lint():
          """Import the sibling extractor without requiring a package layout."""
          here = pathlib.Path(__file__).resolve().parent / "totality_lint.py"
          spec = importlib.util.spec_from_file_location("totality_lint", here)
          mod = importlib.util.module_from_spec(spec)
          # Register before exec: @dataclass resolves annotations through
          # sys.modules[cls.__module__], which is None for a spec-loaded module.
          sys.modules.setdefault("totality_lint", mod)
          spec.loader.exec_module(mod)
          return mod
      
      
      tl = _load_totality_lint()
      
      INVARIANT_RE = re.compile(r"^\s*invariant:\s*(.+)", re.I)
      REFUTED_RE = re.compile(r"^\s*refuted:\s*(.+)", re.I | re.M)
      
      
      @dataclass
      class Claim:
          statement: str
          test: str
          path: str
          line: int
          refuted: str | None = None
          #: Registries this test iterates live, by name. A claim over a domain is
          #: only as good as the domain it loops.
          anchors: list = field(default_factory=list)
          #: Registries this test iterates as a copied literal.
          copies: list = field(default_factory=list)
          #: Registries this test PINS with a deliberate hand-list. A ratchet is the
          #: complement of an enumeration, not a weaker version of it: an enumeration
          #: loops whatever the domain now holds and is blind to it narrowing.
          ratchets: list = field(default_factory=list)
      
      
      @dataclass
      class Finding:
          kind: str
          detail: str
          path: str = ""
          line: int = 0
          test: str = ""
      
      
      def _docstring_claim(fn: ast.FunctionDef) -> tuple[str, str | None] | None:
          """(statement, refutation) if this test declares an invariant."""
          doc = ast.get_docstring(fn)
          if not doc:
              return None
          first = doc.splitlines()[0] if doc.splitlines() else ""
          m = INVARIANT_RE.match(first)
          if not m:
              return None
          statement = m.group(1).strip().rstrip(".")
          ref = REFUTED_RE.search(doc)
          refutation = None
          if ref:
              # A refutation runs to the end of its paragraph.
              tail = doc[ref.start():]
              para = tail.split("\n\n", 1)[0]
              refutation = " ".join(para.split())
              refutation = REFUTED_RE.sub("", refutation, count=1).strip() or para.strip()
          return (statement, refutation)
      
      
      def inventory(root: pathlib.Path) -> tuple[list[Claim], list[tl.Registry], dict]:
          src_files, test_files = [], []
          for p in sorted(root.rglob("*.py")):
              if any(part in tl.SKIP_DIRS for part in p.parts):
                  continue
              rel = str(p.relative_to(root))
              (test_files if tl._is_test_path(rel) else src_files).append((p, rel))
      
          registries: list[tl.Registry] = []
          for p, rel in src_files:
              try:
                  registries.extend(tl._registries_py(ast.parse(p.read_text(errors="replace")), rel))
              except (SyntaxError, ValueError, OSError):
                  continue
      
          claims: list[Claim] = []
          for p, rel in test_files:
              try:
                  src = p.read_text(errors="replace")
                  tree = ast.parse(src)
              except (SyntaxError, ValueError, OSError):
                  continue
      
              imports = tl._imported_modules(tree)
              visible = [r for r in registries if tl._reachable(r, rel, imports)]
              by_name = {}
              for r in visible:
                  for key in {r.name, r.name.rsplit(".", 1)[-1]}:
                      by_name.setdefault(key, r)
      
              domains = {}
              for d in tl._domains_py(tree, rel, src):
                  domains.setdefault(d.test, []).append(d)
      
              for fn in ast.walk(tree):
                  if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
                      continue
                  decl = _docstring_claim(fn)
                  if decl is None:
                      continue
                  statement, refutation = decl
                  anchors, copies, ratchets = [], [], []
                  for d in domains.get(fn.name, []):
                      if d.ack and d.ack[0] == "ratchet" and d.members is not None:
                          for r in visible:
                              if d.members <= r.members:
                                  ratchets.append(r.name)
                          hit = tl._nearest_registry(d, visible)
                          if hit and hit[1]:
                              ratchets.append(hit[0].name)
                          continue
                      if d.live:
                          reg = next(
                              (by_name[n] for n in d.lookup_names() if n in by_name),
                              None,
                          )
                          if reg:
                              anchors.append(reg.name)
                      elif d.members is not None and len(d.members) >= tl.MIN_LITERAL:
                          reg = tl._best_registry(d.members, visible)
                          if reg:
                              copies.append(reg.name)
                  claims.append(Claim(
                      statement=statement, test=fn.name, path=rel, line=fn.lineno,
                      refuted=refutation, anchors=sorted(set(anchors)),
                      copies=sorted(set(copies)), ratchets=sorted(set(ratchets)),
                  ))
      
          stats = {
              "source_files": len(src_files),
              "test_files": len(test_files),
              "registries": len(registries),
              "claims": len(claims),
              "refuted": sum(1 for c in claims if c.refuted),
          }
          return claims, registries, stats
      
      
      def findings_for(claims: list[Claim], registries: list[tl.Registry]) -> list[Finding]:
          out: list[Finding] = []
          for c in claims:
              if c.copies:
                  out.append(Finding(
                      "literal",
                      f'"{c.statement}" is anchored to a test that iterates a COPY of '
                      f'{", ".join("`" + n + "`" for n in c.copies)} — the claim cannot '
                      f'see a member added to the registry',
                      c.path, c.line, c.test,
                  ))
              if not c.refuted:
                  out.append(Finding(
                      "unrefuted",
                      f'"{c.statement}" has never been observed failing — break its '
                      f'chokepoint, watch it go red by name, restore, and record '
                      f'`refuted: <what was broken> -> <what was seen>`',
                      c.path, c.line, c.test,
                  ))
      
          named = set()
          for c in claims:
              named.update(c.anchors)
              named.update(c.copies)
              named.update(c.ratchets)
          for r in registries:
              if r.name in named or r.name.rsplit(".", 1)[-1] in named:
                  continue
              out.append(Finding(
                  "unanchored",
                  f"`{r.name}` ({r.kind}, {len(r.members)} members) is enumerated in "
                  f"source and no invariant names it",
                  r.path, r.line,
              ))
          return out
      
      
      ORDER = {"literal": 0, "unrefuted": 1, "unanchored": 2}
      
      
      def report(claims: list[Claim], findings: list[Finding], stats: dict) -> str:
          out = [
              "",
              "  CLAIMS — what this repository declares must hold",
              "",
              f"  {stats['claims']} invariant(s) declared · {stats['refuted']} with an "
              f"observed refutation · {stats['registries']} registry/ies in source",
              "",
          ]
          if claims:
              for c in sorted(claims, key=lambda c: (c.path, c.line)):
                  mark = "✓" if c.refuted else "·"
                  out.append(f"  {mark} {c.statement}")
                  over = ""
                  if c.anchors:
                      over = f" over {', '.join(c.anchors)}"
                  elif c.ratchets:
                      over = f" pinning {', '.join(c.ratchets)}"
                  out.append(f"      {c.path}:{c.line}  {c.test}{over}")
              out.append("")
      
          if not findings:
              out += ["  nothing to report.", ""]
              return "\n".join(out)
      
          for f in sorted(findings, key=lambda f: (ORDER.get(f.kind, 9), f.path, f.line)):
              where = f"{f.path}:{f.line}" + (f"  {f.test}" if f.test else "")
              out.append(f"  [{f.kind}] {where}")
              out.append(f"      {f.detail}")
              out.append("")
      
          out += [
              "  Candidates, not defects. Most registries need no invariant; an",
              "  `unanchored` row is a question, not a verdict. An `unrefuted` row is",
              "  the one worth clearing — a claim nobody has watched fail is a claim",
              "  nobody has tested.",
              "",
          ]
          return "\n".join(out)
      
      
      # --------------------------------------------------------------------------
      # selftest
      # --------------------------------------------------------------------------
      
      FIX_SRC = '''
      ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}
      UNWATCHED = {"a": 1, "b": 2, "c": 3}
      '''
      
      FIX_TEST = '''
      import pytest
      from fix_src import ROTATION_CODES
      
      
      def test_round_trips(rotation="haar"):
          """invariant: every declared rotation survives the on-disk formats.
      
          refuted: added "hadamard2" with no construction -> suite stayed green,
          this went red on [hadamard2].
          """
          for name in ROTATION_CODES:
              assert name
      
      
      def test_unrefuted():
          """invariant: the codes are stable across releases."""
          for name in ROTATION_CODES:
              assert name
      
      
      @pytest.mark.parametrize("r", ["haar", "rht"])
      def test_claim_over_a_copy(r):
          """invariant: every rotation is accepted by save_params."""
          assert r
      
      
      def test_not_a_claim():
          """Ordinary test, no declaration."""
          assert True
      '''
      
      
      def selftest() -> int:
          import tempfile
      
          fails = []
      
          def check(label, ok, detail=""):
              print(f"  {'ok  ' if ok else 'FAIL'}  {label}" + ("" if ok else f"  -> {detail}"))
              if not ok:
                  fails.append(label)
      
          with tempfile.TemporaryDirectory() as td:
              root = pathlib.Path(td)
              (root / "fix_src.py").write_text(FIX_SRC)
              (root / "tests").mkdir()
              (root / "tests" / "test_fix.py").write_text(FIX_TEST)
              claims, regs, stats = inventory(root)
              f = findings_for(claims, regs)
              by_test = {c.test: c for c in claims}
              kinds = {(x.kind, x.test) for x in f}
      
              check("three declarations found, the plain test excluded",
                    stats["claims"] == 3 and "test_not_a_claim" not in by_test,
                    str(sorted(by_test)))
              check("the statement is captured without its trailing period",
                    by_test["test_round_trips"].statement.endswith("on-disk formats"),
                    by_test["test_round_trips"].statement)
              check("a refutation is captured",
                    "hadamard2" in (by_test["test_round_trips"].refuted or ""),
                    str(by_test["test_round_trips"].refuted))
              check("a refuted claim is not reported unrefuted",
                    ("unrefuted", "test_round_trips") not in kinds, str(kinds))
              check("an unrefuted claim is reported",
                    ("unrefuted", "test_unrefuted") in kinds, str(kinds))
              check("a claim anchored to a live registry records the anchor",
                    by_test["test_round_trips"].anchors == ["ROTATION_CODES"],
                    str(by_test["test_round_trips"].anchors))
              check("a claim over a copied literal is reported",
                    ("literal", "test_claim_over_a_copy") in kinds, str(kinds))
              check("a registry no invariant names is reported unanchored",
                    any(x.kind == "unanchored" and "UNWATCHED" in x.detail for x in f),
                    str([x.detail for x in f if x.kind == "unanchored"]))
              check("a registry an invariant does name is not reported unanchored",
                    not any(x.kind == "unanchored" and "ROTATION_CODES" in x.detail
                            for x in f),
                    str([x.detail for x in f if x.kind == "unanchored"]))
      
          print()
          if fails:
              print(f"{len(fails)} check(s) failed: {fails}")
              return 1
          print("all checks passed")
          return 0
      
      
      def main(argv: list[str] | None = None) -> int:
          ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
          ap.add_argument("path", nargs="?", default=".", help="repository root")
          ap.add_argument("--json", action="store_true", help="machine-readable")
          ap.add_argument("--strict", action="store_true", help="exit 1 if any finding")
          ap.add_argument("--selftest", action="store_true", help="fixtures, no repo")
          args = ap.parse_args(argv)
      
          if args.selftest:
              return selftest()
      
          root = pathlib.Path(args.path).resolve()
          if not root.is_dir():
              print(f"not a directory: {root}", file=sys.stderr)
              return 2
      
          claims, registries, stats = inventory(root)
          findings = findings_for(claims, registries)
          if args.json:
              print(json.dumps({
                  "stats": stats,
                  "claims": [
                      {"statement": c.statement, "test": c.test, "path": c.path,
                       "line": c.line, "refuted": c.refuted, "anchors": c.anchors,
                       "copies": c.copies}
                      for c in claims
                  ],
                  "findings": [
                      {"kind": f.kind, "detail": f.detail, "path": f.path,
                       "line": f.line, "test": f.test}
                      for f in findings
                  ],
              }, indent=2, default=str))
          else:
              print(report(claims, findings, stats))
          return 1 if (findings and args.strict) else 0
      
      
      if __name__ == "__main__":
          sys.exit(main())
      
    • totality_lint.py 35.2 KB
      #!/usr/bin/env python3
      """totality_lint.py — find tests that enumerate a domain by copying it.
      
      A test that loops a hand-written list passes its runner and proves nothing
      about completeness. When the same members also exist as a registry in the
      source — a dict, a set, a tuple, an Enum — the list is a *copy*, and the copy
      drifts the moment someone adds a member to the registry and not to the test.
      Nothing goes red. The suite still says 267 passed.
      
      Measured on `oaustegard/remex` (2026-08-24): adding a fourth member to
      `ROTATION_CODES` with nothing behind it left the entire 267-test suite green.
      Only a test that looped the registry itself caught it.
      
      This reports two findings:
      
        sampled-domain   a test iterates a literal whose members are a strict subset
                         of a source registry's members. The missing members are
                         named, because those are the ones nothing covers.
      
        no-floor         a test iterates a live registry with no length assertion
                         anywhere in the file. A registry that collapses to zero
                         makes the loop range over nothing and pass vacuously.
      
      Adapted from the meta-oracle in daniloc/coherence (`src/oracle-domain.ts`,
      `analyzeOracle`), which classifies an oracle's iteration root as LIVE or
      LITERAL by parsing the oracle's own AST. Three deliberate differences, each
      from something the trial of that tool turned up:
      
        * **Report, not gate.** Its parity arm refuses a correct oracle that binds
          the domain to a local name first (`d = set(REGISTRY); for x in sorted(d)`),
          against a README claiming the analyzer never false-fails. A gate that
          false-fails gets switched off; an advisory gets read. `--strict` opts in.
        * **Join on membership, not on names.** `[1, 2, 3, 4, 8]` and
          `SUPPORTED_BITS` share no token. What ties them is that the literal's
          members are a subset of the registry's, so that is the join key. No
          naming convention is assumed, and none is required.
        * **No spec files.** Coherence needs a `*.spec.md` declaring the claim
          before it will analyse anything. This needs a path.
      
      Acknowledgement is first-class, on the same principle as `memory_lint.py`:
      a partial domain is often correct (`save_params` legitimately accepts two of
      three rotations). Say so on the test and it stops being a finding:
      
          # totality: partial — mojo has no construction for "none"
          @pytest.mark.parametrize("rotation", ["haar", "rht"])
          def test_save_params_accepts_every_mojo_rotation(...):
      
      And a suppression must not become a silence: an acknowledgement on a test that
      now covers the whole domain is reported in its own right (`stale-ack`).
      
          python3 scripts/totality_lint.py <path>            # the report
          python3 scripts/totality_lint.py <path> --json     # machine-readable
          python3 scripts/totality_lint.py <path> --strict   # exit 1 if any finding
          python3 scripts/totality_lint.py --selftest        # fixtures, no repo
      
      Python only. The registry and literal extractors are the language-specific
      half (`_registries_py`, `_domains_py`); the join, the acknowledgement handling
      and the report are not. TypeScript is not covered — a `_registries_ts` pair
      built on tree-sitter would slot in beside them.
      """
      
      from __future__ import annotations
      
      import argparse
      import ast
      import json
      import pathlib
      import re
      import sys
      from dataclasses import dataclass, field
      from typing import Iterable
      
      SKIP_DIRS = {
          ".git", "node_modules", "__pycache__", ".venv", "venv", "dist", "build",
          ".pytest_cache", ".mypy_cache", ".tox", "site-packages", ".coherence",
          ".ruff_cache",
      }
      
      #: A path is a test file if any of these appear in it. Registries are read from
      #: NON-test files only: a list defined in a test file and looped by that same
      #: test file is a fixture, not a copied domain, and treating it as a registry is
      #: the single largest source of false positives.
      TEST_MARKERS = ("test_", "_test.", "/tests/", "/test/", "conftest.py")
      
      #: Fewer members than this and a "domain" is a pair of examples, not an
      #: enumeration. Raising it trades recall for precision; 3 keeps
      #: `{"haar", "rht", "none"}` in scope, which is the motivating case.
      MIN_REGISTRY = 3
      
      #: A literal below this is not plausibly a copy of anything.
      MIN_LITERAL = 2
      
      #: `partial` excuses a hand-list. `ratchet` ASSERTS one: the members named are a
      #: floor the registry must keep containing, and the linter checks that rather
      #: than taking the marker's word for it. Enumeration and a ratchet are
      #: complementary, not a ladder — see `_ratchet_findings`.
      ACK_KINDS = ("partial", "sampled", "ratchet")
      _ACK_ALT = "|".join(ACK_KINDS)
      ACK_RE = re.compile(rf"#\s*totality:\s*({_ACK_ALT})\b[ \t]*[-—:]?[ \t]*(.*)")
      ACK_DOC_RE = re.compile(rf"^\s*totality:\s*({_ACK_ALT})\b[ \t]*[-—:]?[ \t]*(.*)",
                              re.M)
      
      Member = str | int | float | bool
      
      
      @dataclass(frozen=True)
      class Registry:
          """An enumerated collection defined in source (not test) code."""
      
          name: str
          members: frozenset
          path: str
          line: int
          kind: str  # dict | set | list | tuple | enum
      
          @property
          def module(self) -> str:
              return self.path.replace("\\", "/").removesuffix(".py").replace("/", ".")
      
          @property
          def pkg(self) -> str:
              return self.path.replace("\\", "/").split("/", 1)[0].removesuffix(".py")
      
      
      @dataclass
      class Domain:
          """A collection a test iterates, and how it got hold of it."""
      
          path: str
          line: int
          test: str
          live: bool
          members: frozenset | None  # None when live
          symbol: str | None  # the registry the iteration roots in, when known
          how: str  # parametrize | for | comprehension
          alias: str | None = None  # the name actually written at the loop, if different
          candidates: list = field(default_factory=list)  # every name it could address
      
          def lookup_names(self) -> list:
              """Names to try against the registry table, best first.
      
              Candidates before the resolved root: a test that reaches its subject
              through an importlib-loaded module writes `tl.SKIP_DIRS`, and following
              the local `tl` through the file's own constants lands on `_SPEC` — true,
              useless, and it shadows the name that actually matters.
              """
              return list(dict.fromkeys(
                  [c for c in self.candidates if c]
                  + [n for n in (self.symbol, self.alias) if n]
              ))
          ack: tuple[str, str] | None = None  # (kind, reason)
      
      
      @dataclass
      class Finding:
          kind: str
          path: str
          line: int
          test: str
          detail: str
          missing: list = field(default_factory=list)
          registry: str | None = None
      
      
      # --------------------------------------------------------------------------
      # extraction — the language-specific half
      # --------------------------------------------------------------------------
      
      
      def _const_members(node: ast.AST) -> frozenset | None:
          """Members of a literal collection, or None if it is not one.
      
          A dict contributes its KEYS: a name-to-code table is enumerated by name,
          which is what a test parametrizes over.
          """
          if isinstance(node, ast.Dict):
              items = node.keys
          elif isinstance(node, (ast.List, ast.Set, ast.Tuple)):
              items = node.elts
          elif isinstance(node, ast.Call):
              # set([...]) / frozenset([...]) / tuple([...]) / list([...])
              fn = node.func
              name = fn.id if isinstance(fn, ast.Name) else getattr(fn, "attr", None)
              if name in {"set", "frozenset", "tuple", "list"} and len(node.args) == 1:
                  return _const_members(node.args[0])
              return None
          else:
              return None
      
          out = []
          for it in items:
              if it is None:  # `{**other}` in a dict
                  return None
              if not isinstance(it, ast.Constant):
                  return None
              if not isinstance(it.value, (str, int, float, bool)):
                  return None
              out.append(it.value)
          return frozenset(out) if out else None
      
      
      def _is_test_path(path: str) -> bool:
          p = path.replace("\\", "/")
          return any(m in p for m in TEST_MARKERS)
      
      
      def _registries_py(tree: ast.Module, path: str) -> list[Registry]:
          """Module-level and class-level enumerated collections."""
          found: list[Registry] = []
      
          def enum_bases(cls: ast.ClassDef) -> bool:
              for b in cls.bases:
                  n = b.id if isinstance(b, ast.Name) else getattr(b, "attr", "")
                  if n in {"Enum", "IntEnum", "StrEnum", "Flag", "IntFlag"}:
                      return True
              return False
      
          def scan_assign(node, qualifier: str = ""):
              targets = node.targets if isinstance(node, ast.Assign) else [node.target]
              for t in targets:
                  if not isinstance(t, ast.Name):
                      continue
                  members = _const_members(node.value)
                  if members is None or len(members) < MIN_REGISTRY:
                      continue
                  kind = {
                      ast.Dict: "dict", ast.Set: "set",
                      ast.List: "list", ast.Tuple: "tuple",
                  }.get(type(node.value), "set")
                  found.append(Registry(
                      name=qualifier + t.id, members=members, path=path,
                      line=node.lineno, kind=kind,
                  ))
      
          for node in tree.body:
              if isinstance(node, (ast.Assign, ast.AnnAssign)) and node.value is not None:
                  scan_assign(node)
              elif isinstance(node, ast.ClassDef):
                  if enum_bases(node):
                      names = [
                          t.id
                          for b in node.body if isinstance(b, ast.Assign)
                          for t in b.targets if isinstance(t, ast.Name)
                      ]
                      if len(names) >= MIN_REGISTRY:
                          found.append(Registry(
                              name=node.name, members=frozenset(names), path=path,
                              line=node.lineno, kind="enum",
                          ))
                  for b in node.body:
                      if isinstance(b, (ast.Assign, ast.AnnAssign)) and getattr(b, "value", None):
                          scan_assign(b, qualifier=node.name + ".")
          return found
      
      
      def _module_consts(tree: ast.Module) -> dict[str, ast.AST]:
          out = {}
          for node in tree.body:
              if isinstance(node, ast.Assign) and len(node.targets) == 1:
                  if isinstance(node.targets[0], ast.Name):
                      out[node.targets[0].id] = node.value
          return out
      
      
      def _imported_names(tree: ast.Module) -> set[str]:
          out = set()
          for node in ast.walk(tree):
              if isinstance(node, ast.ImportFrom):
                  for a in node.names:
                      out.add(a.asname or a.name)
              elif isinstance(node, ast.Import):
                  for a in node.names:
                      out.add((a.asname or a.name).split(".")[0])
          return out
      
      
      def _imported_modules(tree: ast.Module) -> set[str]:
          """Dotted module paths this file imports, plus each of their prefixes.
      
          `from remex.packing import SUPPORTED_BITS` contributes `remex.packing` and
          `remex`. This is what ties a test to the registries it could plausibly be
          copying — see `_reachable`.
          """
          out: set[str] = set()
      
          def add(mod: str | None):
              if not mod:
                  return
              parts = mod.split(".")
              for i in range(1, len(parts) + 1):
                  out.add(".".join(parts[:i]))
      
          for node in ast.walk(tree):
              if isinstance(node, ast.ImportFrom):
                  add(node.module)
              elif isinstance(node, ast.Import):
                  for a in node.names:
                      add(a.name)
          return out
      
      
      def _symbol_candidates(node: ast.AST) -> list[str]:
          """Names a domain iteration could be addressing, best first.
      
          `_root_symbol` peels an attribute down to its object, so `tl.SKIP_DIRS`
          roots in `tl` — the module, not the registry. That is right for
          `q.rotations()` and wrong for `mod.REGISTRY`, and this repository's own
          tests take the second shape: they load the module under test through
          `importlib` and reach its registries as attributes. So offer the attribute
          name too and let the caller match whichever it recognises.
          """
          out: list[str] = []
          seen = 0
          cur = node
          while seen < 8:
              seen += 1
              if isinstance(cur, ast.Attribute):
                  out.append(cur.attr)
                  cur = cur.value
                  continue
              if isinstance(cur, ast.Call):
                  cur = cur.args[0] if cur.args else cur.func
                  continue
              if isinstance(cur, (ast.Subscript, ast.Starred)):
                  cur = cur.value
                  continue
              break
          root = _root_symbol(node)
          if root:
              out.append(root)
          # Preserve order, drop duplicates.
          return list(dict.fromkeys(out))
      
      
      def _root_symbol(node: ast.AST) -> str | None:
          """Peel `sorted(X)`, `list(X.keys())`, `X.values()` down to `X`."""
          seen = 0
          while seen < 8:
              seen += 1
              if isinstance(node, ast.Name):
                  return node.id
              if isinstance(node, ast.Attribute):
                  node = node.value
                  continue
              if isinstance(node, ast.Call):
                  if node.args:
                      node = node.args[0]
                      continue
                  node = node.func
                  continue
              if isinstance(node, (ast.Subscript, ast.Starred)):
                  node = node.value
                  continue
              return None
          return None
      
      
      def _ack_for(src_lines: list[str], node: ast.AST) -> tuple[str, str] | None:
          """An acknowledgement comment above the test, or in its docstring."""
          start = min(
              [node.lineno] + [d.lineno for d in getattr(node, "decorator_list", [])]
          )
          for ln in range(max(1, start - 4), start):
              m = ACK_RE.search(src_lines[ln - 1])
              if m:
                  return (m.group(1), m.group(2).strip())
          doc = ast.get_docstring(node) if isinstance(
              node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)
          ) else None
          if doc:
              m = ACK_DOC_RE.search(doc)
              if m:
                  return (m.group(1), m.group(2).strip())
          return None
      
      
      def _domains_py(tree: ast.Module, path: str, src: str) -> list[Domain]:
          """Every collection a test function iterates."""
          consts = _module_consts(tree)
          imported = _imported_names(tree)
          lines = src.splitlines()
          out: list[Domain] = []
      
          def classify(value: ast.AST) -> tuple[bool, frozenset | None, str | None, str | None]:
              """-> (live, members, root symbol, local alias).
      
              The alias matters for the floor check: a test that binds
              `LIVE = sorted(REGISTRY)` and asserts `len(LIVE) >= 3` has a floor,
              even though the iteration roots in `REGISTRY`.
              """
              members = _const_members(value)
              if members is not None:
                  return (False, members, None, None)
              cands = _symbol_candidates(value)
              sym = cands[0] if cands else None
              if sym is None:
                  return (True, None, None, None)  # unknown shape: assume live, never fail
              sym = next((c for c in cands if c in consts), sym)
              if sym in consts:
                  inner = _const_members(consts[sym])
                  if inner is not None:
                      return (False, inner, sym, None)
                  root = _root_symbol(consts[sym])
                  return (True, None, root or sym, sym)
              return (True, None, sym, None)
      
          for fn in ast.walk(tree):
              if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
                  continue
              if not fn.name.startswith("test"):
                  continue
              ack = _ack_for(lines, fn)
      
              for dec in fn.decorator_list:
                  if not isinstance(dec, ast.Call):
                      continue
                  if _root_symbol(dec.func) != "pytest" and getattr(
                      dec.func, "attr", ""
                  ) != "parametrize":
                      continue
                  if getattr(dec.func, "attr", "") != "parametrize" or len(dec.args) < 2:
                      continue
                  argnames = dec.args[0]
                  if not (isinstance(argnames, ast.Constant)
                          and isinstance(argnames.value, str)
                          and "," not in argnames.value):
                      continue  # multi-parameter tables are out of scope
                  live, members, sym, alias = classify(dec.args[1])
                  out.append(Domain(path, dec.lineno, fn.name, live, members, sym,
                                    "parametrize", alias, _symbol_candidates(
                                    dec.args[1]), ack))
      
              for node in ast.walk(fn):
                  if isinstance(node, ast.For):
                      live, members, sym, alias = classify(node.iter)
                      out.append(Domain(path, node.lineno, fn.name, live, members,
                                        sym, "for", alias, _symbol_candidates(
                                    node.iter), ack))
                  elif isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp,
                                         ast.DictComp)):
                      for gen in node.generators:
                          live, members, sym, alias = classify(gen.iter)
                          out.append(Domain(path, node.lineno, fn.name, live, members,
                                            sym, "comprehension", alias, _symbol_candidates(
                                    gen.iter), ack))
          return out
      
      
      def _has_floor(tree: ast.Module, *symbols: str | None) -> bool:
          """Any `len(<symbol>) <op> n` comparison anywhere in the module.
      
          Several names can stand for one domain — the registry and whatever local
          the test bound it to — so a floor on any of them counts.
          """
          wanted = {s for s in symbols if s}
          if not wanted:
              return False
          for node in ast.walk(tree):
              if not isinstance(node, ast.Compare):
                  continue
              left = node.left
              if isinstance(left, ast.Call) and _root_symbol(left.func) == "len":
                  if left.args and _root_symbol(left.args[0]) in wanted:
                      return True
          return False
      
      
      # --------------------------------------------------------------------------
      # the join — language-independent
      # --------------------------------------------------------------------------
      
      
      def _reachable(reg: Registry, test_path: str, imports: set[str]) -> bool:
          """Could this test plausibly be copying THIS registry?
      
          Without this, a numeric literal matches any numeric registry anywhere in the
          tree. Measured on `oaustegard/experiments`, a monorepo of unrelated
          projects: 3 of 4 findings joined a test in `discrepancy/` to registries in
          `kb-k-sweep/` and `remex-vs-higgs-ablation/`, purely because small integer
          sets collide. Two ways to be reachable, either sufficient:
      
            * the test imports the registry's module or its package
            * they sit under the same top-level directory
          """
          if reg.module in imports or reg.pkg in imports:
              return True
          tail = reg.module.rsplit(".", 1)[-1]
          if any(i.split(".")[-1] == tail for i in imports):
              return True
          # tests/test_totality_lint.py <-> scripts/totality_lint.py. The convention
          # this repository actually uses, and the one importlib-loaded tests need:
          # they have no import statement naming the module they exercise.
          stem = pathlib.PurePath(test_path).stem
          for prefix, suffix in (("test_", ""), ("", "_test")):
              if prefix and stem.startswith(prefix):
                  stem = stem[len(prefix):]
              if suffix and stem.endswith(suffix):
                  stem = stem[: -len(suffix)]
          if stem == pathlib.PurePath(reg.path).stem:
              return True
          test_pkg = test_path.replace("\\", "/").split("/", 1)[0].removesuffix(".py")
          return test_pkg == reg.pkg
      
      
      def _best_registry(members: frozenset, regs: Iterable[Registry]) -> Registry | None:
          """The smallest registry that strictly contains these members.
      
          Smallest, because a literal contained by both a 4-member and a 40-member
          registry is far more likely to be a copy of the 4.
          """
          cands = [r for r in regs if members < r.members]
          if not cands:
              return None
          return min(cands, key=lambda r: (len(r.members), r.path, r.line))
      
      
      def _nearest_registry(d, regs) -> tuple | None:
          """(registry, members of the ratchet it no longer holds).
      
          An empty second element means the ratchet is intact — it pins the whole
          domain rather than a strict subset of it.
      
          A ratchet is a hand-list asserting the domain keeps containing it. That is
          the direction an enumeration cannot see: a totality oracle loops whatever
          the domain currently is, so a member LEAVING is trivially green. Measured on
          `oaustegard/remex`: substituting one member for another, keeping cardinality
          and keeping a second spelling of the domain in agreement, passed the domain
          floor, the enumeration and the parity check — five green tests while a
          supported rotation silently left.
      
          Match on overlap rather than containment, because containment is exactly
          what broke.
          """
          best, best_key = None, None
          for r in regs:
              overlap = len(d.members & r.members)
              if overlap < MIN_LITERAL:
                  continue
              # Most overlap wins; ties break on the smaller registry, then on a
              # stable address, so one report does not reorder between runs.
              key = (-overlap, len(r.members), r.path, r.line)
              if best_key is None or key < best_key:
                  best, best_key = r, key
          if best is None:
              return None
          return (best, sorted(d.members - best.members, key=repr))
      
      
      def scan(root: pathlib.Path) -> tuple[list[Finding], dict]:
          src_files, test_files = [], []
          for p in sorted(root.rglob("*.py")):
              if any(part in SKIP_DIRS for part in p.parts):
                  continue
              rel = str(p.relative_to(root))
              (test_files if _is_test_path(rel) else src_files).append((p, rel))
      
          registries: list[Registry] = []
          for p, rel in src_files:
              try:
                  registries.extend(_registries_py(ast.parse(p.read_text(errors="replace")), rel))
              except (SyntaxError, ValueError, OSError):
                  continue
      
          findings: list[Finding] = []
          #: registry name -> the ratchet domains pinning it
          ratcheted: dict[str, list] = {}
          #: registry name -> the domains enumerating it live
          enumerated: dict[str, list] = {}
          domains_seen = 0
          for p, rel in test_files:
              try:
                  src = p.read_text(errors="replace")
                  tree = ast.parse(src)
              except (SyntaxError, ValueError, OSError):
                  continue
              imports = _imported_modules(tree)
              visible = [r for r in registries if _reachable(r, rel, imports)]
              vis_by_name: dict[str, Registry] = {}
              for r in visible:
                  for key in {r.name, r.name.rsplit(".", 1)[-1]}:
                      vis_by_name.setdefault(key, r)
      
              for d in _domains_py(tree, rel, src):
                  domains_seen += 1
                  if d.live:
                      # Only a name we independently recognised as a registry earns a
                      # no-floor finding. Firing on every `for x in <local>` produced 27
                      # findings on remex, all noise — numpy arrays, query matrices, loop
                      # counters. A wall of candidates is worse than silence.
                      reg = next(
                          (vis_by_name[c] for c in d.lookup_names() if c in vis_by_name),
                          None,
                      )
                      if reg and not _has_floor(tree, d.symbol, d.alias, reg.name,
                                                reg.name.rsplit(".", 1)[-1]):
                          findings.append(Finding(
                              "no-floor", d.path, d.line, d.test,
                              f"iterates `{reg.name}` ({reg.kind}, {reg.path}:"
                              f"{reg.line}, {len(reg.members)} members) with no "
                              f"`len(...) >= n` assertion in this file — vacuous if "
                              f"it empties",
                              registry=reg.name,
                          ))
                      if reg:
                          enumerated.setdefault(reg.name, []).append(d)
                      if d.ack and reg:
                          findings.append(Finding(
                              "stale-ack", d.path, d.line, d.test,
                              f"acknowledged as {d.ack[0]} ({d.ack[1] or 'no reason given'}), "
                              f"but it iterates a live domain — the ack no longer suppresses "
                              f"anything",
                          ))
                      continue
      
                  if d.members is None or len(d.members) < MIN_LITERAL:
                      continue
                  reg = _best_registry(d.members, visible)
                  if reg is None:
                      if d.ack and d.ack[0] == "ratchet":
                          # No strict superset, so either the ratchet pins the whole
                          # domain (intact) or a member has left (broken).
                          hit = _nearest_registry(d, visible)
                          if hit:
                              reg, left = hit
                              for r in visible:
                                  if d.members <= r.members:
                                      ratcheted.setdefault(r.name, []).append(d)
                              ratcheted.setdefault(reg.name, []).append(d)
                              if left:
                                  findings.append(Finding(
                                      "ratchet-broken", d.path, d.line, d.test,
                                      f"pins {len(d.members)} member(s) of "
                                      f"`{reg.name}`, and the registry no longer "
                                      f"contains all of them — a member left without "
                                      f"the ratchet being retired",
                                      missing=left, registry=reg.name,
                                  ))
                      continue
                  missing = sorted(reg.members - d.members, key=repr)
                  if d.ack and d.ack[0] == "ratchet":
                      # Pin EVERY registry the hand-list is a floor for. One domain is
                      # often spelled twice (`ROTATION_CODES` for the on-disk byte,
                      # `Quantizer.ROTATIONS` for the constructor), and a ratchet over
                      # its members holds each of them.
                      for r in visible:
                          if d.members <= r.members:
                              ratcheted.setdefault(r.name, []).append(d)
                      # And FALL THROUGH. A ratchet is a statement about the domain
                      # SHRINKING; it says nothing about the members it never listed,
                      # so a partial ratchet leaves those exactly as uncovered as an
                      # unmarked hand-list does. Suppressing the finding here would
                      # make the marker a laundering channel — the same escape hatch
                      # this tool's own notes criticise `via guard` for being in
                      # `daniloc/coherence`, reproduced one day later. Verified 2026-08-25:
                      # `# totality: ratchet` over 2 of 3 members silenced the report
                      # entirely and checked nothing about the third.
                  elif d.ack:
                      continue
                  others = sum(
                      1 for r in visible
                      if d.members < r.members and len(r.members) == len(reg.members)
                  ) - 1
                  also = f" (+{others} registry/ies of the same size also contain it)" if others else ""
                  pinned = " (pinned as a ratchet, which covers only shrink)" \
                      if d.ack and d.ack[0] == "ratchet" else ""
                  findings.append(Finding(
                      "sampled-domain", d.path, d.line, d.test,
                      f"{d.how} over a literal of {len(d.members)}{pinned}, but "
                      f"`{reg.name}` ({reg.kind}, {reg.path}:{reg.line}) has "
                      f"{len(reg.members)}{also}",
                      missing=missing, registry=reg.name,
                  ))
      
          # The complement Yep named: enumeration proves every CURRENT member is
          # handled and is structurally blind to the domain narrowing, because it
          # loops whatever the domain now is. A ratchet is the other direction.
          # A registry with no floor AND no ratchet produced two findings on the same
          # line saying overlapping things — 3 of 3 on `claude-workspace`, which a
          # cross-model review named as finding fatigue. `no-floor` is the narrower
          # statement and names the same fix first, so it wins the line.
          floored_out = {
              (f.path, f.line) for f in findings if f.kind == "no-floor"
          }
          for name, doms in sorted(enumerated.items()):
              if name in ratcheted:
                  continue
              d = doms[0]
              if (d.path, d.line) in floored_out:
                  continue
              findings.append(Finding(
                  "unratcheted", d.path, d.line, d.test,
                  f"enumerates `{name}` live, and nothing pins its membership — a "
                  f"member REMOVED from the registry keeps this green, because the "
                  f"loop ranges over whatever the registry now holds. Pin a floor "
                  f"with `# totality: ratchet — <why>` on a hand-list test",
                  registry=name,
              ))
      
          stats = {
              "source_files": len(src_files),
              "test_files": len(test_files),
              "ratcheted": len(ratcheted),
              "enumerated": len(enumerated),
              "registries": len(registries),
              "domains": domains_seen,
          }
          return findings, stats
      
      
      # --------------------------------------------------------------------------
      # report
      # --------------------------------------------------------------------------
      
      ORDER = {"ratchet-broken": 0, "sampled-domain": 1, "stale-ack": 2,
               "no-floor": 3, "unratcheted": 4}
      
      
      def report(findings: list[Finding], stats: dict) -> str:
          out = [
              "",
              "  TOTALITY — a test that enumerates a domain by copying it",
              "",
              f"  {stats['source_files']} source file(s) · {stats['registries']} registry/ies · "
              f"{stats['test_files']} test file(s) · {stats['domains']} iterated domain(s)",
              "",
          ]
          if not findings:
              out += ["  nothing to report.", ""]
              return "\n".join(out)
      
          for f in sorted(findings, key=lambda f: (ORDER.get(f.kind, 9), f.path, f.line)):
              out.append(f"  [{f.kind}] {f.path}:{f.line}  {f.test}")
              out.append(f"      {f.detail}")
              if f.missing:
                  shown = ", ".join(repr(m) for m in f.missing[:8])
                  more = "" if len(f.missing) <= 8 else f" … and {len(f.missing) - 8} more"
                  out.append(f"      never exercised: {shown}{more}")
              out.append("")
      
          out += [
              "  Candidates, not defects. A partial domain is often correct — say so on",
              "  the test and it stops being a finding:",
              "",
              "      # totality: partial — <why this subset is the right domain>",
              "",
              "  Otherwise loop the registry itself, and assert a floor so an emptied",
              "  registry cannot pass vacuously.",
              "",
          ]
          return "\n".join(out)
      
      
      # --------------------------------------------------------------------------
      # selftest
      # --------------------------------------------------------------------------
      
      FIXTURE_SRC = '''
      ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}
      SUPPORTED_BITS = (1, 2, 3, 4, 8)
      PAIR = {"a": 1, "b": 2}
      '''
      
      FIXTURE_TEST = '''
      import pytest
      from fixture_src import ROTATION_CODES, SUPPORTED_BITS
      
      DIMS = [64, 128, 384, 768, 1024]
      LIVE = sorted(ROTATION_CODES)
      
      
      @pytest.mark.parametrize("rotation", ["haar", "rht"])
      def test_copies_the_registry(rotation):
          assert rotation
      
      
      # totality: partial — mojo has no construction for "none"
      @pytest.mark.parametrize("rotation", ["haar", "rht"])
      def test_acknowledged_subset(rotation):
          assert rotation
      
      
      @pytest.mark.parametrize("bits", [1, 2, 3, 4])
      def test_copies_the_widths(bits):
          assert bits
      
      
      @pytest.mark.parametrize("d", DIMS)
      def test_local_fixture_is_not_a_copy(d):
          assert d
      
      
      @pytest.mark.parametrize("rotation", LIVE)
      def test_live_without_floor(rotation):
          assert rotation
      
      
      # totality: partial — stale, this one covers everything
      @pytest.mark.parametrize("rotation", sorted(ROTATION_CODES))
      def test_stale_ack(rotation):
          assert rotation
      
      
      def test_pair_is_below_the_registry_floor():
          for k in ["a"]:
              assert k
      '''
      
      FIXTURE_TEST_FLOORED = '''
      import pytest
      from fixture_src import ROTATION_CODES
      
      LIVE = sorted(ROTATION_CODES)
      
      
      def test_floor():
          assert len(LIVE) >= 3
      
      
      @pytest.mark.parametrize("rotation", LIVE)
      def test_live_with_floor(rotation):
          assert rotation
      '''
      
      
      def selftest() -> int:
          import tempfile
      
          fails = []
      
          def check(label, ok, detail=""):
              print(f"  {'ok  ' if ok else 'FAIL'}  {label}" + ("" if ok else f"  -> {detail}"))
              if not ok:
                  fails.append(label)
      
          with tempfile.TemporaryDirectory() as td:
              root = pathlib.Path(td)
              (root / "fixture_src.py").write_text(FIXTURE_SRC)
              (root / "tests").mkdir()
              (root / "tests" / "test_fixture.py").write_text(FIXTURE_TEST)
              findings, stats = scan(root)
      
              by = {}
              for f in findings:
                  by.setdefault(f.test, []).append(f.kind)
      
              check("registries found (2 of 3; PAIR is below MIN_REGISTRY)",
                    stats["registries"] == 2, str(stats["registries"]))
              check("a literal subset of a registry is reported",
                    "sampled-domain" in by.get("test_copies_the_registry", []), str(by))
              check("the missing member is named",
                    any(f.missing == ["none"] for f in findings
                        if f.test == "test_copies_the_registry"),
                    str([f.missing for f in findings]))
              check("a second registry is matched independently",
                    "sampled-domain" in by.get("test_copies_the_widths", []), str(by))
              check("an acknowledged subset is not a finding",
                    "test_acknowledged_subset" not in by, str(by))
              check("a test-local fixture list is not a copy",
                    "test_local_fixture_is_not_a_copy" not in by, str(by))
              check("a live domain with no floor is reported",
                    "no-floor" in by.get("test_live_without_floor", []), str(by))
              check("an ack on a live domain is reported stale",
                    "stale-ack" in by.get("test_stale_ack", []), str(by))
              check("a 1-member loop is below MIN_LITERAL",
                    "test_pair_is_below_the_registry_floor" not in by, str(by))
      
          with tempfile.TemporaryDirectory() as td:
              root = pathlib.Path(td)
              (root / "fixture_src.py").write_text(FIXTURE_SRC)
              (root / "tests").mkdir()
              (root / "tests" / "test_floored.py").write_text(FIXTURE_TEST_FLOORED)
              findings, _ = scan(root)
              check("a live domain WITH a floor is silent",
                    not [f for f in findings if f.kind == "no-floor"],
                    str([f.detail for f in findings]))
      
          print()
          if fails:
              print(f"{len(fails)} check(s) failed: {fails}")
              return 1
          print("all checks passed")
          return 0
      
      
      def main() -> int:
          ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
          ap.add_argument("path", nargs="?", default=".", help="repository root")
          ap.add_argument("--json", action="store_true", help="machine-readable")
          ap.add_argument("--strict", action="store_true", help="exit 1 if any finding")
          ap.add_argument("--selftest", action="store_true", help="fixtures, no repo")
          args = ap.parse_args()
      
          if args.selftest:
              return selftest()
      
          root = pathlib.Path(args.path).resolve()
          if not root.is_dir():
              print(f"not a directory: {root}", file=sys.stderr)
              return 2
      
          findings, stats = scan(root)
          if args.json:
              print(json.dumps({
                  "stats": stats,
                  "findings": [
                      {"kind": f.kind, "path": f.path, "line": f.line, "test": f.test,
                       "detail": f.detail, "missing": f.missing, "registry": f.registry}
                      for f in findings
                  ],
              }, indent=2, default=str))
          else:
              print(report(findings, stats))
          return 1 if (findings and args.strict) else 0
      
      
      if __name__ == "__main__":
          sys.exit(main())
      
  • tests
    • test_claims.py 11.5 KB
      """Tests for scripts/claims.py — the declared-invariant inventory.
      
      A claim is a test whose docstring opens `invariant:`. The parts that can go
      wrong are the parsing (what counts as a declaration, and how much of the
      docstring the refutation swallows) and the anchoring (which registry, if any,
      the claiming test actually iterates).
      
      The anchoring case that matters is this repository's own shape: a test that
      loads its subject through `importlib` and reaches a registry as
      `tl.SKIP_DIRS`. Peeling that attribute down to its object lands on the local
      module handle, and following THAT through the file's own constants lands on
      `_SPEC` — true, useless, and it shadows the name that matters. Pinned below.
      
          python3 tests/test_claims.py
      """
      
      from __future__ import annotations
      
      import importlib.util
      import sys
      import tempfile
      import textwrap
      import unittest
      from pathlib import Path
      
      _SPEC = importlib.util.spec_from_file_location(
          "claims", Path(__file__).resolve().parent.parent / "scripts" / "claims.py"
      )
      cl = importlib.util.module_from_spec(_SPEC)
      sys.modules["claims"] = cl
      _SPEC.loader.exec_module(cl)
      
      
      def inv(**files):
          """Write {relpath: source} into a temp tree and take its inventory."""
          with tempfile.TemporaryDirectory() as td:
              root = Path(td)
              for rel, src in files.items():
                  p = root / rel
                  p.parent.mkdir(parents=True, exist_ok=True)
                  p.write_text(textwrap.dedent(src))
              claims, registries, stats = cl.inventory(root)
              return claims, registries, stats, cl.findings_for(claims, registries)
      
      
      SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n'
      
      
      class Declaration(unittest.TestCase):
      
          def test_a_docstring_opening_with_invariant_is_a_claim(self):
              claims, _, stats, _ = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          """invariant: every rotation round-trips."""
                          for r in ROTATION_CODES:
                              assert r
                  ''',
              })
              self.assertEqual(stats["claims"], 1)
              self.assertEqual(claims[0].statement, "every rotation round-trips")
      
          def test_an_ordinary_docstring_is_not_a_claim(self):
              claims, _, _, _ = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      def test_x():
                          """Checks the thing."""
                          assert True
                  ''',
              })
              self.assertEqual(claims, [])
      
          def test_the_marker_must_open_the_docstring(self):
              """A mention buried in prose is discussion, not a declaration."""
              claims, _, _, _ = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      def test_x():
                          """Checks the thing.
      
                          invariant: this is a sentence about invariants.
                          """
                          assert True
                  ''',
              })
              self.assertEqual(claims, [])
      
          def test_a_test_with_no_docstring_is_skipped(self):
              claims, _, _, _ = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": "def test_x():\n    assert True\n",
              })
              self.assertEqual(claims, [])
      
      
      class Refutation(unittest.TestCase):
      
          def _claim(self, doc: str):
              """doc is the docstring body; it is indented here, not by the caller."""
              body = "\n".join(
                  ("        " + ln if ln.strip() else "") for ln in doc.splitlines()
              ).strip("\n")
              src = (
                  "from pkg.mod import ROTATION_CODES\n\n\n"
                  "def test_x():\n"
                  '    """' + body.lstrip() + '\n    """\n'
                  "    for r in ROTATION_CODES:\n"
                  "        assert r\n"
              )
              with tempfile.TemporaryDirectory() as td:
                  root = Path(td)
                  (root / "pkg" / "tests").mkdir(parents=True)
                  (root / "pkg" / "mod.py").write_text(SRC)
                  (root / "pkg" / "tests" / "test_it.py").write_text(src)
                  claims, _, _ = cl.inventory(root)
              self.assertEqual(len(claims), 1, src)
              return claims[0]
      
          def test_a_refutation_is_captured(self):
              c = self._claim(
                  'invariant: every rotation round-trips.\n'
                  '\n'
                  'refuted: added "hadamard2" -> suite green, this went red.\n'
              )
              self.assertIsNotNone(c.refuted)
              self.assertIn("hadamard2", c.refuted)
      
          def test_a_refutation_stops_at_its_paragraph(self):
              c = self._claim(
                  'invariant: every rotation round-trips.\n'
                  '\n'
                  'refuted: added "hadamard2", this went red.\n'
                  '\n'
                  'Unrelated closing note that is not part of the refutation.\n'
              )
              self.assertNotIn("Unrelated", c.refuted)
      
          def test_a_claim_with_no_refutation_is_reported(self):
              _, _, _, f = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          """invariant: every rotation round-trips."""
                          for r in ROTATION_CODES:
                              assert r
                  ''',
              })
              self.assertIn("unrefuted", [x.kind for x in f])
      
          def test_a_refuted_claim_is_not(self):
              _, _, _, f = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          """invariant: every rotation round-trips.
      
                          refuted: added a member -> this went red.
                          """
                          for r in ROTATION_CODES:
                              assert r
                  ''',
              })
              self.assertNotIn("unrefuted", [x.kind for x in f])
      
      
      class Anchoring(unittest.TestCase):
      
          def test_a_live_iteration_records_the_registry(self):
              claims, _, _, _ = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          """invariant: every rotation round-trips.
      
                          refuted: yes.
                          """
                          for r in ROTATION_CODES:
                              assert r
                  ''',
              })
              self.assertEqual(claims[0].anchors, ["ROTATION_CODES"])
      
          def test_an_importlib_loaded_module_attribute_still_anchors(self):
              """The shape this repository's own tests take.
      
              `tl.SKIP_DIRS` peels to the module handle `tl`, and following `tl`
              through the test file's constants lands on `_SPEC`. The attribute name
              has to win, or every claim in this repo reads as unanchored.
              """
              claims, _, _, _ = inv(**{
                  "scripts/thing.py": 'SKIP_DIRS = {"a", "b", "c"}\n',
                  "tests/test_thing.py": '''
                      import importlib.util
      
                      _SPEC = importlib.util.spec_from_file_location("t", "scripts/thing.py")
                      tl = importlib.util.module_from_spec(_SPEC)
      
                      def test_x():
                          """invariant: every skipped name is skipped.
      
                          refuted: yes.
                          """
                          for name in sorted(tl.SKIP_DIRS):
                              assert name
                  ''',
              })
              self.assertEqual(claims[0].anchors, ["SKIP_DIRS"])
      
          def test_a_claim_over_a_copied_literal_is_reported(self):
              claims, _, _, f = inv(**{
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r", ["haar", "rht"])
                      def test_x(r):
                          """invariant: every rotation is accepted.
      
                          refuted: yes.
                          """
                          assert r
                  ''',
              })
              self.assertEqual(claims[0].copies, ["ROTATION_CODES"])
              self.assertIn("literal", [x.kind for x in f])
      
      
      class Unanchored(unittest.TestCase):
      
          def test_a_registry_no_invariant_names_is_reported(self):
              _, _, _, f = inv(**{
                  "pkg/mod.py": 'WATCHED = {"a": 1, "b": 2, "c": 3}\n'
                                'IGNORED = {"x": 1, "y": 2, "z": 3}\n',
                  "pkg/tests/test_it.py": '''
                      from pkg.mod import WATCHED
      
                      def test_x():
                          """invariant: watched holds.
      
                          refuted: yes.
                          """
                          for k in WATCHED:
                              assert k
                  ''',
              })
              un = [x.detail for x in f if x.kind == "unanchored"]
              self.assertTrue(any("IGNORED" in d for d in un), str(un))
              self.assertFalse(any("WATCHED" in d for d in un), str(un))
      
      
      class Robustness(unittest.TestCase):
      
          def test_an_unparseable_file_is_skipped_not_fatal(self):
              claims, _, _, _ = inv(**{
                  "pkg/broken.py": "def (((:\n",
                  "pkg/mod.py": SRC,
                  "pkg/tests/test_it.py": '''
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          """invariant: it holds.
      
                          refuted: yes.
                          """
                          for r in ROTATION_CODES:
                              assert r
                  ''',
              })
              self.assertEqual(len(claims), 1)
      
          def test_an_empty_tree_reports_nothing(self):
              claims, _, stats, f = inv(**{"README.md": "nothing\n"})
              self.assertEqual((claims, f), ([], []))
              self.assertEqual(stats["claims"], 0)
      
          def test_the_report_renders_with_no_claims(self):
              out = cl.report([], [], {"claims": 0, "refuted": 0, "registries": 0})
              self.assertIn("nothing to report", out)
      
          def test_strict_exits_nonzero_only_on_a_finding(self):
              with tempfile.TemporaryDirectory() as td:
                  self.assertEqual(cl.main([td, "--strict"]), 0)
      
          def test_selftest_passes(self):
              self.assertEqual(cl.selftest(), 0)
      
      
      class Ratchets(unittest.TestCase):
          """A hand-list marked `ratchet` anchors a claim; it is not a copy."""
      
          SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n'
      
          BODY = """
              from pkg.mod import ROTATION_CODES
      
              MARKER
              def test_x():
                  '''invariant: a shipped rotation never leaves.
      
                  refuted: yes.
                  '''
                  for shipped in MEMBERS:
                      assert shipped in ROTATION_CODES
          """
      
          def _tree(self, marker, members):
              body = self.BODY.replace("MARKER", marker).replace("MEMBERS", members)
              return {"pkg/mod.py": self.SRC, "pkg/tests/test_it.py": body}
      
          def test_a_ratchet_anchors_rather_than_copying(self):
              claims, _, _, f = inv(**self._tree(
                  "# totality: ratchet — these two shipped first", '("haar", "rht")'))
              self.assertEqual(claims[0].ratchets, ["ROTATION_CODES"])
              self.assertEqual(claims[0].copies, [])
              self.assertNotIn("literal", [x.kind for x in f])
      
          def test_an_unmarked_subset_is_still_a_copy(self):
              claims, _, _, f = inv(**self._tree("# ordinary comment", '("haar", "rht")'))
              self.assertEqual(claims[0].copies, ["ROTATION_CODES"])
              self.assertIn("literal", [x.kind for x in f])
      
          def test_a_ratcheted_registry_is_not_unanchored(self):
              _, _, _, f = inv(**self._tree(
                  "# totality: ratchet — all three shipped", '("haar", "rht", "none")'))
              un = [x.detail for x in f if x.kind == "unanchored"]
              self.assertFalse(any("ROTATION_CODES" in d for d in un), str(un))
      
      if __name__ == "__main__":
          unittest.main(verbosity=2)
      
    • test_totality_lint.py 20.3 KB
      """Tests for scripts/totality_lint.py — the copied-domain report.
      
      The lint's whole value is precision: a wall of candidates is worse than
      silence, and its two precision mechanisms both came from a measured false
      positive rather than from taste.
      
        * `no-floor` originally fired on every `for x in <local>`, which produced 27
          findings on `remex` — numpy arrays, query matrices, loop counters, all
          noise. It now fires only on a name independently recognised as a registry.
        * `sampled-domain` originally matched a literal against any registry in the
          tree, which on the `experiments` monorepo joined a test in `discrepancy/`
          to registries in `kb-k-sweep/` and `remex-vs-higgs-ablation/` because
          small integer sets collide. It now requires reachability: an import, or a
          shared top-level directory.
      
      Both are pinned below. `--selftest` carries the fixture suite; this adds the
      regressions those two measurements produced, and asserts the classifier
      directly.
      
          python3 tests/test_totality_lint.py
      """
      
      from __future__ import annotations
      
      import importlib.util
      import sys
      import tempfile
      import textwrap
      import unittest
      from pathlib import Path
      
      _SPEC = importlib.util.spec_from_file_location(
          "totality_lint",
          Path(__file__).resolve().parent.parent / "scripts" / "totality_lint.py",
      )
      tl = importlib.util.module_from_spec(_SPEC)
      # Register before exec: `@dataclass` resolves annotations through
      # sys.modules[cls.__module__], which is None for a spec-loaded module.
      sys.modules["totality_lint"] = tl
      _SPEC.loader.exec_module(tl)
      
      
      def scan_files(**files) -> list:
          """Write {relpath: source} into a temp tree and scan it."""
          with tempfile.TemporaryDirectory() as td:
              root = Path(td)
              for rel, src in files.items():
                  p = root / rel
                  p.parent.mkdir(parents=True, exist_ok=True)
                  p.write_text(textwrap.dedent(src))
              findings, _ = tl.scan(root)
              return findings
      
      
      def kinds(findings, test=None):
          return sorted(
              f.kind for f in findings if test is None or f.test == test
          )
      
      
      class Registries(unittest.TestCase):
          """What counts as an enumerated domain in source."""
      
          def regs(self, src):
              import ast
              return tl._registries_py(ast.parse(textwrap.dedent(src)), "m.py")
      
          def test_dict_contributes_its_keys(self):
              r = self.regs('CODES = {"a": 0, "b": 1, "c": 2}')
              self.assertEqual(len(r), 1)
              self.assertEqual(r[0].members, frozenset({"a", "b", "c"}))
              self.assertEqual(r[0].kind, "dict")
      
          def test_tuple_set_and_list_all_count(self):
              r = self.regs("""
                  T = (1, 2, 3)
                  S = {"x", "y", "z"}
                  L = ["p", "q", "r"]
              """)
              self.assertEqual({x.kind for x in r}, {"tuple", "set", "list"})
      
          def test_enum_members_are_the_domain(self):
              r = self.regs("""
                  from enum import Enum
      
                  class Colour(Enum):
                      RED = 1
                      GREEN = 2
                      BLUE = 3
              """)
              self.assertEqual([x.name for x in r], ["Colour"])
              self.assertEqual(r[0].members, frozenset({"RED", "GREEN", "BLUE"}))
      
          def test_class_attribute_registry_is_qualified(self):
              r = self.regs("""
                  class Q:
                      ROTATIONS = {"haar": 1, "rht": 2, "none": 3}
              """)
              self.assertEqual([x.name for x in r], ["Q.ROTATIONS"])
      
          def test_below_min_registry_is_not_a_domain(self):
              self.assertEqual(self.regs('PAIR = {"a": 1, "b": 2}'), [])
      
          def test_a_non_constant_member_disqualifies_the_whole_collection(self):
              self.assertEqual(self.regs("T = (1, 2, compute())"), [])
      
          def test_a_spread_disqualifies_a_dict(self):
              self.assertEqual(self.regs("D = {**other, 'a': 1, 'b': 2, 'c': 3}"), [])
      
      
      class SampledDomain(unittest.TestCase):
      
          SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n'
      
          def test_a_strict_subset_is_reported_with_what_is_missing(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r", ["haar", "rht"])
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), ["sampled-domain"])
              self.assertEqual(f[0].missing, ["none"])
              self.assertEqual(f[0].registry, "ROTATION_CODES")
      
          def test_the_full_domain_as_a_literal_is_not_a_finding(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r", ["haar", "rht", "none"])
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), [])
      
          def test_a_superset_is_not_a_finding(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r", ["haar", "rht", "none", "extra"])
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), [])
      
          def test_a_multi_parameter_table_is_out_of_scope(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r,n", [("haar", 1), ("rht", 2)])
                      def test_x(r, n): assert r
                  """,
              })
              self.assertEqual(kinds(f), [])
      
          def test_a_same_file_const_is_still_a_literal(self):
              """Binding the copy to a name first does not make it live."""
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      CASES = ["haar", "rht"]
      
                      @pytest.mark.parametrize("r", CASES)
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), ["sampled-domain"])
      
      
      class Reachability(unittest.TestCase):
          """Measured on `experiments`: 3 of 4 findings were cross-project noise."""
      
          NUMERIC = "KS = [1, 2, 3, 4]\n"
      
          def test_an_unrelated_project_is_not_matched(self):
              f = scan_files(**{
                  "kb-k-sweep/srht.py": self.NUMERIC,
                  "discrepancy/tests/test_calibration.py": """
                      def test_x():
                          for k in [1, 2, 3]:
                              assert k
                  """,
              })
              self.assertEqual(kinds(f), [])
      
          def test_an_import_makes_it_reachable(self):
              f = scan_files(**{
                  "kb_k_sweep/srht.py": self.NUMERIC,
                  "discrepancy/tests/test_calibration.py": """
                      from kb_k_sweep.srht import KS
      
                      def test_x():
                          for k in [1, 2, 3]:
                              assert k
                  """,
              })
              self.assertEqual(kinds(f), ["sampled-domain"])
      
          def test_a_shared_top_level_directory_makes_it_reachable(self):
              f = scan_files(**{
                  "caps/conditions.py": "DOSE = [0.0, 0.1, 0.5, 1.0]\n",
                  "caps/test_lib.py": """
                      def test_x():
                          for d in (0.0, 0.5, 1.0):
                              assert d >= 0
                  """,
              })
              self.assertEqual(kinds(f), ["sampled-domain"])
              self.assertEqual(f[0].missing, [0.1])
      
      
      class NoFloor(unittest.TestCase):
          """Measured on `remex`: firing on every local produced 27 noise findings.
      
          A live enumeration with no ratchet would also raise `unratcheted`, but the
          two landed on the same line saying overlapping things (3 of 3 on
          `claude-workspace`), so `no-floor` — the narrower statement, naming the same
          fix first — wins the line. See `Ratchet` below for `unratcheted` on its own.
          """
      
          SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n'
      
          def test_the_finding_names_the_registry_not_the_module_handle(self):
              """`for k in bra.CLASS.items()` is about CLASS, not about `bra`.
      
              The resolved root of an importlib-loaded attribute chain is the local
              module handle, which named `_spec` in this repo's own report before the
              message was keyed on the matched registry instead.
              """
              f = scan_files(**{
                  "scripts/thing.py": 'CLASS = {"a": 1, "b": 2, "c": 3}\n',
                  "tests/test_thing.py": """
                      import importlib.util
      
                      _spec = importlib.util.spec_from_file_location("t", "x.py")
                      bra = importlib.util.module_from_spec(_spec)
      
                      def test_x():
                          for k in bra.CLASS:
                              assert k
                  """,
              })
              self.assertEqual(kinds(f), ["no-floor"])
              self.assertIn("`CLASS`", f[0].detail)
              self.assertNotIn("_spec", f[0].detail)
      
          def test_a_live_registry_without_a_floor_is_reported(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r", sorted(ROTATION_CODES))
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), ["no-floor"])
      
          def test_a_floor_on_the_local_alias_counts(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      LIVE = sorted(ROTATION_CODES)
      
                      def test_floor(): assert len(LIVE) >= 3
      
                      @pytest.mark.parametrize("r", LIVE)
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), ["unratcheted"])
      
          def test_an_ordinary_local_is_not_a_domain(self):
              """`for q in queries` over a numpy array is not an enumeration."""
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          queries = make_queries()
                          for q in queries:
                              assert q
                  """,
              })
              self.assertEqual(kinds(f), [])
      
      
      class Acknowledgement(unittest.TestCase):
      
          SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n'
      
          def test_a_comment_ack_suppresses_the_finding(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      # totality: partial — mojo has no construction for "none"
                      @pytest.mark.parametrize("r", ["haar", "rht"])
                      def test_x(r): assert r
                  """,
              })
              self.assertEqual(kinds(f), [])
      
          def test_a_docstring_ack_suppresses_the_finding(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": '''
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      @pytest.mark.parametrize("r", ["haar", "rht"])
                      def test_x(r):
                          """Compare the two constructions.
      
                          totality: partial — "none" has no fidelity claim.
                          """
                          assert r
                  ''',
              })
              self.assertEqual(kinds(f), [])
      
          def test_an_ack_that_no_longer_suppresses_anything_is_reported(self):
              """A suppression must not become a silence."""
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      import pytest
                      from pkg.mod import ROTATION_CODES
      
                      # totality: partial — stale, this covers everything now
                      @pytest.mark.parametrize("r", sorted(ROTATION_CODES))
                      def test_x(r): assert r
                  """,
              })
              self.assertIn("stale-ack", kinds(f))
      
      
      class Robustness(unittest.TestCase):
      
          def test_an_unparseable_file_is_skipped_not_fatal(self):
              f = scan_files(**{
                  "pkg/broken.py": "def (((:\n",
                  "pkg/mod.py": 'CODES = {"a": 0, "b": 1, "c": 2}\n',
                  "pkg/tests/test_it.py": """
                      from pkg.mod import CODES
      
                      def test_x():
                          for k in ["a", "b"]:
                              assert k
                  """,
              })
              self.assertEqual(kinds(f), ["sampled-domain"])
      
          def test_vendored_trees_are_not_scanned(self):
              f = scan_files(**{
                  "node_modules/pkg/mod.py": 'CODES = {"a": 0, "b": 1, "c": 2}\n',
                  "pkg/tests/test_it.py": """
                      def test_x():
                          for k in ["a", "b"]:
                              assert k
                  """,
              })
              self.assertEqual(kinds(f), [])
      
          def test_an_empty_tree_reports_nothing_and_does_not_raise(self):
              self.assertEqual(scan_files(**{"README.md": "nothing here\n"}), [])
      
          def test_the_report_renders_with_no_findings(self):
              out = tl.report([], {"source_files": 0, "registries": 0,
                                   "test_files": 0, "domains": 0})
              self.assertIn("nothing to report", out)
      
          def test_selftest_passes(self):
              self.assertEqual(tl.selftest(), 0)
      
      
      class SkipDirs(unittest.TestCase):
      
          def test_every_skipped_directory_is_actually_skipped(self):
              """invariant: every name in SKIP_DIRS is excluded from the walk.
      
              The domain is SKIP_DIRS itself. A name added to the set with nothing
              behind it reads as protection the walk does not provide, and the
              failure is silent: the linter scans a vendored tree and reports its
              registries as the project's own.
      
              The fixture keeps the registry and the test under one top-level
              directory on purpose. An earlier version put the registry at
              `<skipdir>/mod.py` and the test at `pkg/tests/`, which passes whether
              or not the skip works — `_reachable` rejects the pair either way. It
              was green under perturbation, which is the whole failure class this
              file exists to catch.
      
              refuted: replaced the walk's `part in SKIP_DIRS` check with
              `part in {"node_modules"}` -> this test went red naming `.coherence`,
              while the other 25 tests in this file stayed green.
              """
              self.assertGreaterEqual(len(tl.SKIP_DIRS), 8)
              for name in sorted(tl.SKIP_DIRS):
                  files = {
                      f"pkg/{name}/mod.py": 'CODES = {"a": 0, "b": 1, "c": 2}\n',
                      "pkg/tests/test_it.py": """
                          def test_x():
                              for k in ["a", "b"]:
                                  assert k
                      """,
                  }
                  self.assertEqual(
                      kinds(scan_files(**files)), [],
                      f"a registry under {name}/ reached the report",
                  )
      
      
      class Ratchet(unittest.TestCase):
          """Enumeration and a hand-list are complementary, not a ladder.
      
          An enumeration loops whatever the domain currently holds, so it is
          structurally blind to the domain NARROWING. Measured on `oaustegard/remex`:
          substituting one member for another, keeping cardinality and keeping a
          second spelling of the domain in agreement, left the domain floor, the
          enumeration and the parity check all green while a supported rotation
          silently left. Raised by Yep, 2026-08-24; reproduced before being believed.
          """
      
          SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n'
          SHRUNK = 'ROTATION_CODES = {"haar": 0, "rht": 1, "xyz": 2}\n'
      
          def _tree(self, src, marker="# totality: ratchet — these three shipped"):
              return {
                  "pkg/mod.py": src,
                  "pkg/tests/test_it.py": f"""
                      from pkg.mod import ROTATION_CODES
      
                      {marker}
                      def test_pinned():
                          for shipped in ("haar", "rht", "none"):
                              assert shipped in ROTATION_CODES
                  """,
              }
      
          def test_an_intact_ratchet_is_silent(self):
              self.assertEqual(kinds(scan_files(**self._tree(self.SRC))), [])
      
          def test_a_member_leaving_breaks_the_ratchet_and_is_named(self):
              f = scan_files(**self._tree(self.SHRUNK))
              self.assertEqual(kinds(f), ["ratchet-broken"])
              self.assertEqual(f[0].missing, ["none"])
              self.assertEqual(f[0].registry, "ROTATION_CODES")
      
          def test_a_full_hand_list_is_not_a_sample_with_or_without_the_marker(self):
              """Pinning the WHOLE domain is not a sampling oracle in either case."""
              self.assertEqual(
                  kinds(scan_files(**self._tree(self.SRC, marker="# ordinary"))), [])
      
          def test_a_partial_ratchet_still_reports_what_it_does_not_cover(self):
              """The marker must not be a laundering channel.
      
              A ratchet is a statement about the domain SHRINKING. It says nothing
              about members it never listed, so a partial ratchet leaves those exactly
              as uncovered as an unmarked hand-list does. Verified 2026-08-25: before
              this, `# totality: ratchet` over 2 of 3 members silenced the report
              entirely — the same escape hatch this tool's notes criticise `via guard`
              for being in `daniloc/coherence`, reproduced one day later.
              """
              tree = {
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      from pkg.mod import ROTATION_CODES
      
                      def test_pinned():
                          for shipped in ("haar", "rht"):
                              assert shipped in ROTATION_CODES
                  """,
              }
              self.assertEqual(kinds(scan_files(**tree)), ["sampled-domain"])
              tree["pkg/tests/test_it.py"] = tree["pkg/tests/test_it.py"].replace(
                  "                def test_pinned",
                  "                # totality: ratchet — the two that shipped first\n"
                  "                def test_pinned")
              marked = scan_files(**tree)
              self.assertEqual(kinds(marked), ["sampled-domain"])
              self.assertEqual(marked[0].missing, ["none"])
              self.assertIn("covers only shrink", marked[0].detail)
      
          def test_a_partial_ack_does_not_earn_ratchet_checking(self):
              """`partial` excuses a hand-list; only `ratchet` asserts one."""
              f = scan_files(**self._tree(self.SHRUNK, marker="# totality: partial — two of three"))
              self.assertEqual(kinds(f), [])
      
          def test_an_enumeration_with_no_ratchet_is_reported(self):
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      from pkg.mod import ROTATION_CODES
      
                      def test_floor(): assert len(ROTATION_CODES) >= 3
      
                      def test_x():
                          for r in ROTATION_CODES:
                              assert r
                  """,
              })
              self.assertEqual(kinds(f), ["unratcheted"])
              self.assertEqual(f[0].registry, "ROTATION_CODES")
      
          def test_no_floor_and_unratcheted_do_not_both_claim_one_line(self):
              """Two findings on one line saying overlapping things is fatigue.
      
              Measured 3 of 3 on `claude-workspace` and named by a cross-model review;
              `no-floor` is the narrower statement and wins.
              """
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      from pkg.mod import ROTATION_CODES
      
                      def test_x():
                          for r in ROTATION_CODES:
                              assert r
                  """,
              })
              self.assertEqual(kinds(f), ["no-floor"])
      
          def test_an_enumeration_WITH_a_ratchet_is_silent(self):
              """The pair is the answer; neither half alone is."""
              f = scan_files(**{
                  "pkg/mod.py": self.SRC,
                  "pkg/tests/test_it.py": """
                      from pkg.mod import ROTATION_CODES
      
                      def test_floor(): assert len(ROTATION_CODES) >= 3
      
                      def test_x():
                          for r in ROTATION_CODES:
                              assert r
      
                      # totality: ratchet — these three shipped
                      def test_pinned():
                          for shipped in ("haar", "rht", "none"):
                              assert shipped in ROTATION_CODES
                  """,
              })
              self.assertEqual(kinds(f), [])
      
      if __name__ == "__main__":
          unittest.main(verbosity=2)
      
  • CHANGELOG.md 6.6 KB
    # Changelog
    
    ## 0.2.3 — 2026-08-25
    
    Four more limits from the second half of the cross-model pass, each reproduced.
    The largest undercuts the motivating example: a registry is modelled as a set of
    KEYS, so permuting `{"haar": 0, "rht": 1, "none": 2}` to
    `{"haar": 0, "rht": 2, "none": 1}` keeps every key, every count and every ratchet
    intact while every index already on disk decodes under the wrong rotation. The
    linter reports nothing; the gate's registry half reports nothing. Also named: a
    decorator-built registry is invisible, reachability misses a disconnected
    integration test, and the precision numbers come from three repositories with
    the filters fitted to two of them.
    
    No behaviour change. Documentation only, which is the honest response to holes
    that need a design decision rather than a patch.
    
    ## 0.2.2 — 2026-08-25
    
    `unratcheted` no longer claims a line `no-floor` already has. They landed
    together on 3 of 3 findings in one repository, saying overlapping things and
    naming the same fix; `no-floor` is the narrower statement and wins.
    
    Three limits added from a cross-model adversarial pass, each reproduced before
    being written down. A co-ordinated rename defeats the ratchet: renaming a member
    in the registry and in the hand-list together leaves the pin intact and this
    reports nothing, because both sides of a co-located hand-list move together.
    A brand-new registry is neither gated nor pinned. Split, merge and registry
    rename are invisible, because the diff is keyed on registry name.
    
    ## 0.2.1 — 2026-08-25
    
    A ratchet over a strict subset silenced `sampled-domain` entirely and checked
    nothing about the members it never listed. Writing `# totality: ratchet` on any
    hand-list therefore retired the finding for free — the same escape hatch these
    notes criticise `via guard` for being in `daniloc/coherence`, reproduced one day
    later. A ratchet is a statement about the domain SHRINKING and says nothing
    about growth, so a partial ratchet now still reports what it does not cover,
    annotated "pinned as a ratchet, which covers only shrink". A ratchet pinning the
    whole domain stays silent, as it should.
    
    Found by an adversarial pass on the skill, not by its own tests, which is the
    uncomfortable part: 55 tests and a `--selftest` all passed over it.
    
    ## 0.2.0 — 2026-08-24
    
    A second claim form, because an enumeration is structurally blind to its own
    domain narrowing. It loops whatever the registry currently holds, so removing a
    member leaves it green over the ones that remain. Raised by Yep; reproduced on
    `oaustegard/remex` before being believed, and the reproduction is worse than the
    report. Substituting `"xyz"` for `"none"` in both spellings of the rotation
    domain — keeping cardinality, keeping the two spellings in agreement — left the
    domain floor, the enumeration and the parity check all green. Five passing tests
    over a registry that had quietly stopped supporting a rotation every index on
    disk was written with. The floor is a cardinality check and cannot see a member
    swapped for another.
    
    `# totality: ratchet — <why>` marks a hand-list as a deliberate floor. Unlike
    `partial`, which excuses a hand-list, `ratchet` asserts one, and the linter
    checks the pin rather than taking the marker's word for it. Two new findings:
    `ratchet-broken` names a pinned member the registry no longer contains, detected
    statically before any test runs, and `unratcheted` names a registry enumerated
    live with nothing pinning its membership. `sampled-domain` no longer fires on a
    marked ratchet — a hand list is sometimes the right answer, on purpose.
    
    A ratchet pins every registry it is a floor for, not just the nearest one, since
    a domain is often spelled twice. `claims.py` reports a ratchet claim as pinning
    its registries rather than copying them, and a ratcheted registry is no longer
    `unanchored`.
    
    Verified on remex under the same substitution: `ratchet-broken` naming `'none'`
    statically, and the ratchet test red while the other five pass.
    
    55 tests, up from 45.
    
    ## 0.1.0 — 2026-08-24
    
    First release. Two scripts, one idea: a test that enumerates a domain must loop
    the registry rather than a copy of it.
    
    `totality_lint.py` reports `sampled-domain` (a `parametrize` or `for` over a
    literal whose members are a strict subset of a dict/set/tuple/Enum in the
    source, naming the members nothing covers), `no-floor` (a live registry
    iterated with no length assertion, so an emptied registry passes vacuously),
    and `stale-ack`. `claims.py` reports what the repository declares: a claim is a
    test whose docstring opens `invariant:`, and `refuted:` records the observed
    negative control. Findings are `unrefuted`, `literal`, `unanchored`.
    
    Adapted from the meta-oracle in `daniloc/coherence` (`src/oracle-domain.ts`),
    which classifies an oracle's iteration root as LIVE or LITERAL by parsing the
    oracle's own AST. Three deliberate differences. The join is on membership
    rather than on names, because `[1, 2, 3, 4, 8]` and `SUPPORTED_BITS` share no
    token and containment is what ties them together. Reporting is the default and
    `--strict` opts into a nonzero exit, because the original's parity arm
    false-fails a correct oracle that binds its domain to a local name first. And
    there are no spec files: these take a path.
    
    Both precision filters came from a measured false positive rather than from
    taste. `no-floor` firing on every `for x in <local>` produced 27 findings on
    `oaustegard/remex`, all noise — numpy arrays, query matrices, loop counters —
    so it now fires only on a name independently recognised as a registry. Matching
    a literal against any registry in the tree joined a test in `discrepancy/` to
    registries in `kb-k-sweep/` and `remex-vs-higgs-ablation/` on a monorepo,
    because small integer sets collide by chance; requiring reachability took four
    findings to one, and the survivor was real.
    
    Two extractor fixes came from the first real use. Tests that load their subject
    through `importlib` reach registries as `tl.SKIP_DIRS`; peeling that attribute
    to its object lands on the module handle, and following the handle through the
    file's own constants lands on `_SPEC`, which shadowed the name that mattered
    and made every claim read as unanchored. Candidates now carry the attribute
    name and win over the resolved root, `_reachable` accepts the
    `tests/test_<mod>.py` to `<mod>.py` pairing, and the `no-floor` message names
    the matched registry rather than whatever the chain rooted in.
    
    Measured at release: on `remex` at main, exactly the four rotation tests that
    parametrize `["haar", "rht"]` against a three-member `ROTATION_CODES`, each
    naming `'none'`. 45 tests, plus a `--selftest` in each script.
    
    ## [0.2.3] - 2026-08-25
    
    ### Other
    
    - declaring-invariants: a skill for domains a test copies instead of loops (#773)
    
  • README.md 2.8 KB
    # declaring-invariants
    
    Find tests that enumerate a domain by copying it, and declare the invariants a
    codebase depends on. Stdlib `ast` only — no install, no config file, no
    network. Python.
    
    ```bash
    python3 scripts/totality_lint.py <repo>   # tests that copy a domain
    python3 scripts/claims.py <repo>          # what the repo declares, and what backs it
    ```
    
    ## Features
    
    - **Copied-domain detection** — a `parametrize` or `for` over a literal whose
      members are a strict subset of a dict/set/tuple/Enum in the source, with the
      uncovered members named
    - **Ratchets, the second claim form** — an enumeration loops whatever the
      domain now holds, so it cannot see the domain narrowing.
      `# totality: ratchet — <why>` marks a hand-list as a deliberate floor, and the
      linter checks the pin: `ratchet-broken` names a member that left, statically,
      before any test runs. `unratcheted` names a registry with an enumeration and
      no floor
    - **Vacuity detection** — a live registry iterated with no `len(...) >= n`
      assertion, which passes over an emptied collection
    - **Membership join** — `[1, 2, 3, 4, 8]` and `SUPPORTED_BITS` share no token,
      so containment is the join key; no naming convention is assumed
    - **Reachability filter** — a literal matches a registry only when the test
      imports its module, shares its top-level directory, or is its paired
      `tests/test_<mod>.py`
    - **First-class acknowledgement** — `# totality: partial — <why>` retires a
      finding, and an acknowledgement on a test that later covers the whole domain
      is reported as `stale-ack`
    - **Claim inventory** — a claim is a test whose docstring opens `invariant:`,
      and `refuted:` records the observed negative control
    - **Report by default** — `--strict` opts into a nonzero exit; `--json` for
      machine consumption; `--selftest` runs fixtures with no repo
    
    ## Why
    
    On `oaustegard/remex`, adding a fourth member to `ROTATION_CODES` with no
    construction behind it left the entire 267-test suite green. Four tests looked
    total; each parametrized `["haar", "rht"]` against a three-member registry.
    
    The other direction is worse. Substituting `"xyz"` for `"none"` in both
    spellings of that domain — cardinality unchanged, both spellings agreeing — left
    the domain floor, the enumeration and the parity check green, five passing tests
    over a registry that had stopped supporting a rotation every index on disk was
    written with. That is what the ratchet form is for.
    
    Adapted from the meta-oracle in
    [`daniloc/coherence`](https://github.com/daniloc/coherence), which classifies
    an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST.
    The check needs none of that harness's spec files, claim grammar, ledger or
    Node runtime.
    
    See [SKILL.md](SKILL.md) for the full reference, including how to write a
    refutation you have actually observed.
    
  • SKILL.md 14.2 KB
    ---
    name: declaring-invariants
    description: Find tests that enumerate a domain by copying it, and declare the invariants a codebase depends on. Reports where a parametrize list, for-loop, or it.each iterates a hand-written subset of a dict/set/tuple/Enum that exists in the source, and names the members nothing covers. Use when reviewing tests, when a module gains a name-to-thing table, registry, enum, or dispatch map, before trusting a green suite as evidence a domain is covered, or when asked "is this test actually total", "does anything cover X", "what does this repo guarantee", "which invariants do we declare". Also for vacuous tests that pass over an empty collection, for a domain that has silently NARROWED (an enumeration cannot see that), and for recording the refutation that proves a claim can fail.
    metadata:
      version: 0.2.3
    ---
    
    # declaring-invariants
    
    Two scripts over one idea: **a test that enumerates a domain must loop the
    registry rather than a copy of it.** Someone also has to say which domains
    matter in the first place.
    
    ```bash
    python3 scripts/totality_lint.py <repo>   # tests that copy a domain
    python3 scripts/claims.py <repo>          # what the repo declares, and what backs it
    ```
    
    Python only, stdlib `ast` only: no install, no config file, no network.
    
    ## The failure it catches
    
    A test that loops a hand-written list passes its runner and proves nothing
    about completeness. When the same members also exist as a registry in the
    source, the list is a copy, and the copy drifts the moment someone adds a
    member to the registry and not to the test. Nothing goes red.
    
    Measured on `oaustegard/remex`, 2026-08-24: adding a fourth member to
    `ROTATION_CODES` with no construction behind it left the **entire 267-test
    suite green**. Four separate tests looked total; each parametrized
    `["haar", "rht"]` against a three-member registry. Only a test that looped the
    registry itself caught it.
    
    Adapted from the meta-oracle in [`daniloc/coherence`](https://github.com/daniloc/coherence)
    (`src/oracle-domain.ts`), which classifies an oracle's iteration root as LIVE
    or LITERAL by parsing the oracle's own AST. That harness needs spec files, a
    claim grammar, a ledger and Node; the check does not.
    
    ## `totality_lint.py` — tests that copy a domain
    
    | finding | meaning |
    |---|---|
    | `sampled-domain` | a `parametrize` or `for` over a literal whose members are a strict subset of a source registry. The uncovered members are named. |
    | `ratchet-broken` | a hand-list marked `ratchet` names a member the registry no longer contains. Detected statically, without running anything. |
    | `unratcheted` | a registry enumerated live with nothing pinning its membership. Suppressed when `no-floor` already claimed the same line. |
    | `no-floor` | a test iterates a live registry with no `len(...) >= n` assertion in the file, so an emptied registry passes vacuously. |
    | `stale-ack` | an acknowledgement on a test that now covers the whole domain. |
    
    ```bash
    python3 scripts/totality_lint.py <repo>            # the report
    python3 scripts/totality_lint.py <repo> --json
    python3 scripts/totality_lint.py <repo> --strict   # exit 1 if any finding
    python3 scripts/totality_lint.py --selftest        # fixtures, no repo
    ```
    
    A partial domain is often correct. Say so on the test and it stops being a
    finding:
    
    ```python
    # totality: partial — mojo has no construction for "none"
    @pytest.mark.parametrize("rotation", ["haar", "rht"])
    def test_save_params_accepts_every_mojo_rotation(rotation): ...
    ```
    
    The marker also works as `totality: partial — <why>` inside the docstring. An
    acknowledgement on a test that later covers the whole domain is reported as
    `stale-ack`, so a suppression cannot become a silence.
    
    ## Why a hand-list is the second form
    
    An enumeration loops **whatever the domain currently holds**, so it is
    structurally blind to the domain narrowing. Remove a member and the loop simply
    ranges over fewer of them, green. Raised by Yep, 2026-08-24; reproduced on
    `oaustegard/remex` before being believed:
    
    | perturbation of `ROTATION_CODES` | domain floor | enumeration | parity | hand-list |
    |---|---|---|---|---|
    | grow — add `"hadamard2"` | pass | **RED** | **RED** | pass |
    | shrink — drop `"none"` | **RED** | pass | RED\* | pass |
    | substitute — `"none"` → `"xyz"` in both spellings | pass | pass | pass | pass |
    
    \* only because the *other* spelling of the domain did not shrink, which is
    incidental to that repository.
    
    In the substitution row cardinality holds, both spellings agree, and five green
    tests cover a registry that quietly stopped supporting a rotation every index on
    disk was written with. The floor is a **cardinality**
    check; it cannot see a member swapped for another.
    
    The second form is a **ratchet**: a hand-list asserting the domain keeps
    containing it. Mark it and the linter checks the pin rather than taking the
    marker's word for it.
    
    ```python
    # totality: ratchet — these three shipped; one leaving is a compatibility break
    def test_no_shipped_rotation_is_ever_removed():
        for shipped in ("haar", "rht", "none"):
            assert shipped in ROTATION_CODES
    ```
    
    Under that same substitution the linter reports `ratchet-broken` naming
    `'none'` **statically, before any test runs**, and the test itself goes red
    while the other five pass.
    
    So: an enumeration proves every current member is handled, and a ratchet proves
    no member left without a decision. `unratcheted` names a registry that has the
    first and not the second. Neither half alone is the answer.
    
    **A ratchet covers shrink only.** Over a strict subset it still reports
    `sampled-domain` for the members it never listed, because pinning two of three
    says nothing about the third. Suppressing that would make the marker a
    laundering channel — the escape hatch these notes criticise `via guard` for
    being in coherence. It was one: until 2026-08-25, `# totality: ratchet` over
    two of three members silenced the report entirely and checked nothing about the
    third. Caught by an adversarial pass on this skill, not by its own tests.
    
    ## `claims.py` — what the repo declares
    
    `totality_lint` asks whether a test's domain is complete. It presumes a test
    exists. This asks the prior question, the one coherence's own Known Limits
    concedes it does not answer: nothing enforces `exists ⇒ declared`.
    
    A claim is a test whose docstring **opens** with `invariant:`. No new file
    format, and the claim inherits its test's pass/fail:
    
    ```python
    def test_every_skipped_directory_is_actually_skipped(self):
        """invariant: every name in SKIP_DIRS is excluded from the walk.
    
        refuted: replaced the walk's `part in SKIP_DIRS` check with
        `part in {"node_modules"}` -> this test went red naming `.coherence`,
        while the other 25 tests in this file stayed green.
        """
    ```
    
    | finding | meaning |
    |---|---|
    | `unrefuted` | a claim nobody has watched fail |
    | `literal` | the claiming test iterates a copy of the registry, so the claim cannot see a new member |
    | `unanchored` | a registry no invariant names — a question, not a verdict |
    
    A claim whose test carries a `ratchet` marker is reported as *pinning* its
    registries rather than copying them, and a ratcheted registry is not
    `unanchored`.
    
    ```bash
    python3 scripts/claims.py <repo> [--json] [--strict] [--selftest]
    ```
    
    ## Write the refutation from what you observed
    
    `refuted:` is the half that costs something. Break the chokepoint, watch the
    claim go red **by name**, restore, and record what you saw. A green test and an
    unfalsifiable one look identical from outside; the refutation is what separates
    them.
    
    **Never write a refutation you have not run.** The first refutation authored
    for the `SKIP_DIRS` invariant above asserted a failure that did not occur — the
    fixture placed the registry outside the test's reachability, so the test passed
    under perturbation for an unrelated reason. A vacuous claim, written while
    building the tool that catches vacuous claims, and caught only by running the
    perturbation instead of trusting the sentence.
    
    Procedure, in order:
    
    1. Break the chokepoint the claim names, with one edit in the source rather
       than in the test.
    2. Run the claiming test. Read the failure text. Note the member it named.
    3. Run the rest of the file. Confirm the others stay green — if everything goes
       red, the claim is not localised and the refutation says nothing.
    4. Restore the source.
    5. Write `refuted: <the edit> -> <what went red, by name>`.
    
    ## Wiring it into a commit gate
    
    The reference wiring lives in `oaustegard/claude-workspace`
    (`scripts/tdd_hook.py`): a commit where a registry gained a member and no
    `invariant:` test iterates it live is denied, naming the registry and what it
    gained. Override with `no-invariant: <why>` in the commit body.
    
    Registry **shrink** is gated the same way, and needs a ratchet rather than an
    enumeration to clear it. A new function or a new branch is behavioural growth
    too, but neither is diffable without guessing, and a gate that guesses stops
    being consulted. A brand-new registry is not gated. Declaring one is a
    judgement call; growing one has already made it.
    
    ## Where the two filters came from
    
    Both filters exist because the unfiltered version was noise. Reproduce either
    by removing the filter and re-running against a real repo.
    
    - **`no-floor` fires only on a name independently recognised as a registry.**
      Firing on every `for x in <local>` produced 27 findings on `remex`, all
      noise: numpy arrays, query matrices, loop counters.
    - **A literal matches a registry only when reachable** — the test imports its
      module or package, shares its top-level directory, or is its paired
      `tests/test_<mod>.py`. Matching against any registry in the tree joined a
      test in `discrepancy/` to registries in `kb-k-sweep/` and
      `remex-vs-higgs-ablation/` on a monorepo, because small integer sets collide
      by chance. Four findings became one, and the survivor was real.
    
    The join key is **membership, not names**: `[1, 2, 3, 4, 8]` and
    `SUPPORTED_BITS` share no token, so containment is what ties them together. No
    naming convention is assumed, and none is required.
    
    ## Limits
    
    - **Python only.** The extractors are `_registries_py` and `_domains_py`; a
      tree-sitter pair for another language slots in beside them. The join, the
      acknowledgements and the reports are all language-independent.
    - **Report, not gate, by default.** `--strict` opts into a nonzero exit. The
      tool this was adapted from gates by default, and its parity arm false-fails a
      correct oracle that binds its domain to a local name first. A gate that
      false-fails stops being consulted.
    - **A registry is a collection of constants.** A dict/set/tuple/list of
      literals, or an Enum body, with at least three members. A domain assembled at
      runtime is invisible here.
    - **Multi-parameter tables are out of scope.** Only single-name
      `parametrize` is read.
    - **`unanchored` is a question.** Most registries need no invariant. Treat the
      list as candidates for declaration, never as a backlog to clear.
    - **A registry is modelled as a SET OF KEYS, so a value swap is invisible.**
      This is the largest hole in the design, and it undercuts the motivating
      example. `ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}` exists to pin
      bytes on disk; permuting it to `{"haar": 0, "rht": 2, "none": 1}` keeps every
      key, every count and every ratchet intact, and every index already written
      decodes under the wrong rotation. Verified 2026-08-25 on a fixture: the
      linter reports nothing and the gate's registry half reports nothing. (The
      gate denied that fixture, but on the unrelated TDD rule — checked, because
      claiming otherwise would have been the overclaim this skill exists to catch.)
      Nothing here checks a name-to-code mapping. Extracting `(key, value)` pairs
      as the member set would, at the cost of breaking the subset join against a
      `parametrize` list of keys.
    - **A co-ordinated rename defeats the ratchet.** A global find/replace that
      renames a member in the registry AND in the hand-list leaves the pin intact,
      and this reports nothing. Verified 2026-08-25 on a fixture: renaming `"none"`
      to `"identity"` in both files was silent, while every index already on disk
      still decodes byte 2 as the old name. Both sides of a co-located hand-list
      move together, so no static check over one working tree can see it. The
      commit gate can, because it diffs against git history — and only if the
      ratchet itself is untouched in that commit, which it now requires.
    - **A brand-new registry is neither gated nor pinned.** Create one with five
      members and never grow it and no gate ever fires. `unanchored` and
      `unratcheted` surface it in the report; the gate deliberately does not, on
      the grounds that gating every new module is how a gate stops being consulted.
      That is a judgement, not a proof, and it is the largest hole a cross-model
      review found.
    - **A decorator-built registry is invisible.** `@register("name")` populating a
      dict at import time is the common Python registry idiom and is not a literal,
      so nothing here sees it. Named because it is the shape most likely to be
      mistaken for coverage.
    - **Reachability is path-based, so a disconnected integration test is missed.**
      A test that neither imports the module, shares its top-level directory, nor
      pairs with it by filename will not be joined to the registry it samples. The
      filter trades that recall for the cross-project precision it was measured to
      buy.
    - **The precision numbers come from three repositories, and the filters were
      fitted to two of them.** 27 noise findings on one, 3-of-4 cross-project joins
      on another. Those are the measurements that justified each filter; they are
      not a false-positive rate on a corpus, and should not be read as one.
    - **Split and merge are invisible.** The diff is keyed on registry NAME, so
      renaming a registry, splitting one in two, or merging two into one falls
      through both the gained and lost paths.
    - **A claim that passes is not a claim that is right.** This checks that a
      declared invariant loops the domain it names. Whether it is the *right*
      invariant is human judgement, and it is not automatable.
    
    ## Related
    
    - `verifying-claims` covers the prose layer: does the documentation match
      reality? Agent-judged, non-deterministic, run as a triggered review. This
      skill is the deterministic half, over code and tests rather than prose.
    - `tree-sitting` locates the registry or the test before you edit it.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related