declaring-invariants
Find tests that enumerate a domain by copying it, and declare the invariants a codebase depends on. Reports where a parametrize list, for-loop, or it.each iterates a hand-written subset of a dict/set/tuple/Enum that exists in the source, and names the members nothing covers. Use
Install
npx skills add https://github.com/oaustegard/claude-skills/tree/main/plugins/code-intelligence/skills/declaring-invariants
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oaustegard-claude-skills@llmmart
git clone https://github.com/oaustegard/claude-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole oaustegard/claude-skills collection as a plugin from our marketplace. Git is the plain clone.
README
declaring-invariants
Find tests that enumerate a domain by copying it, and declare the invariants a
codebase depends on. Stdlib ast only — no install, no config file, no
network. Python.
python3 scripts/totality_lint.py <repo> # tests that copy a domain
python3 scripts/claims.py <repo> # what the repo declares, and what backs it
Features
- Copied-domain detection — a
parametrizeorforover a literal whose members are a strict subset of a dict/set/tuple/Enum in the source, with the uncovered members named - Ratchets, the second claim form — an enumeration loops whatever the
domain now holds, so it cannot see the domain narrowing.
# totality: ratchet — <why>marks a hand-list as a deliberate floor, and the linter checks the pin:ratchet-brokennames a member that left, statically, before any test runs.unratchetednames a registry with an enumeration and no floor - Vacuity detection — a live registry iterated with no
len(...) >= nassertion, which passes over an emptied collection - Membership join —
[1, 2, 3, 4, 8]andSUPPORTED_BITSshare no token, so containment is the join key; no naming convention is assumed - Reachability filter — a literal matches a registry only when the test
imports its module, shares its top-level directory, or is its paired
tests/test_<mod>.py - First-class acknowledgement —
# totality: partial — <why>retires a finding, and an acknowledgement on a test that later covers the whole domain is reported asstale-ack - Claim inventory — a claim is a test whose docstring opens
invariant:, andrefuted:records the observed negative control - Report by default —
--strictopts into a nonzero exit;--jsonfor machine consumption;--selftestruns fixtures with no repo
Why
On oaustegard/remex, adding a fourth member to ROTATION_CODES with no
construction behind it left the entire 267-test suite green. Four tests looked
total; each parametrized ["haar", "rht"] against a three-member registry.
The other direction is worse. Substituting "xyz" for "none" in both
spellings of that domain — cardinality unchanged, both spellings agreeing — left
the domain floor, the enumeration and the parity check green, five passing tests
over a registry that had stopped supporting a rotation every index on disk was
written with. That is what the ratchet form is for.
Adapted from the meta-oracle in
daniloc/coherence, which classifies
an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST.
The check needs none of that harness's spec files, claim grammar, ledger or
Node runtime.
See SKILL.md for the full reference, including how to write a refutation you have actually observed.
Skill manifest
declaring-invariants
Two scripts over one idea: a test that enumerates a domain must loop the registry rather than a copy of it. Someone also has to say which domains matter in the first place.
python3 scripts/totality_lint.py <repo> # tests that copy a domain
python3 scripts/claims.py <repo> # what the repo declares, and what backs it
Python only, stdlib ast only: no install, no config file, no network.
The failure it catches
A test that loops a hand-written list passes its runner and proves nothing about completeness. When the same members also exist as a registry in the source, the list is a copy, and the copy drifts the moment someone adds a member to the registry and not to the test. Nothing goes red.
Measured on oaustegard/remex, 2026-08-24: adding a fourth member to
ROTATION_CODES with no construction behind it left the entire 267-test
suite green. Four separate tests looked total; each parametrized
["haar", "rht"] against a three-member registry. Only a test that looped the
registry itself caught it.
Adapted from the meta-oracle in daniloc/coherence
(src/oracle-domain.ts), which classifies an oracle's iteration root as LIVE
or LITERAL by parsing the oracle's own AST. That harness needs spec files, a
claim grammar, a ledger and Node; the check does not.
totality_lint.py — tests that copy a domain
| finding | meaning |
|---|---|
sampled-domain |
a parametrize or for over a literal whose members are a strict subset of a source registry. The uncovered members are named. |
ratchet-broken |
a hand-list marked ratchet names a member the registry no longer contains. Detected statically, without running anything. |
unratcheted |
a registry enumerated live with nothing pinning its membership. Suppressed when no-floor already claimed the same line. |
no-floor |
a test iterates a live registry with no len(...) >= n assertion in the file, so an emptied registry passes vacuously. |
stale-ack |
an acknowledgement on a test that now covers the whole domain. |
python3 scripts/totality_lint.py <repo> # the report
python3 scripts/totality_lint.py <repo> --json
python3 scripts/totality_lint.py <repo> --strict # exit 1 if any finding
python3 scripts/totality_lint.py --selftest # fixtures, no repo
A partial domain is often correct. Say so on the test and it stops being a finding:
# totality: partial — mojo has no construction for "none"
@pytest.mark.parametrize("rotation", ["haar", "rht"])
def test_save_params_accepts_every_mojo_rotation(rotation): ...
The marker also works as totality: partial — <why> inside the docstring. An
acknowledgement on a test that later covers the whole domain is reported as
stale-ack, so a suppression cannot become a silence.
Why a hand-list is the second form
An enumeration loops whatever the domain currently holds, so it is
structurally blind to the domain narrowing. Remove a member and the loop simply
ranges over fewer of them, green. Raised by Yep, 2026-08-24; reproduced on
oaustegard/remex before being believed:
perturbation of ROTATION_CODES |
domain floor | enumeration | parity | hand-list |
|---|---|---|---|---|
grow — add "hadamard2" |
pass | RED | RED | pass |
shrink — drop "none" |
RED | pass | RED* | pass |
substitute — "none" → "xyz" in both spellings |
pass | pass | pass | pass |
* only because the other spelling of the domain did not shrink, which is incidental to that repository.
In the substitution row cardinality holds, both spellings agree, and five green tests cover a registry that quietly stopped supporting a rotation every index on disk was written with. The floor is a cardinality check; it cannot see a member swapped for another.
The second form is a ratchet: a hand-list asserting the domain keeps containing it. Mark it and the linter checks the pin rather than taking the marker's word for it.
# totality: ratchet — these three shipped; one leaving is a compatibility break
def test_no_shipped_rotation_is_ever_removed():
for shipped in ("haar", "rht", "none"):
assert shipped in ROTATION_CODES
Under that same substitution the linter reports ratchet-broken naming
'none' statically, before any test runs, and the test itself goes red
while the other five pass.
So: an enumeration proves every current member is handled, and a ratchet proves
no member left without a decision. unratcheted names a registry that has the
first and not the second. Neither half alone is the answer.
A ratchet covers shrink only. Over a strict subset it still reports
sampled-domain for the members it never listed, because pinning two of three
says nothing about the third. Suppressing that would make the marker a
laundering channel — the escape hatch these notes criticise via guard for
being in coherence. It was one: until 2026-08-25, # totality: ratchet over
two of three members silenced the report entirely and checked nothing about the
third. Caught by an adversarial pass on this skill, not by its own tests.
claims.py — what the repo declares
totality_lint asks whether a test's domain is complete. It presumes a test
exists. This asks the prior question, the one coherence's own Known Limits
concedes it does not answer: nothing enforces exists ⇒ declared.
A claim is a test whose docstring opens with invariant:. No new file
format, and the claim inherits its test's pass/fail:
def test_every_skipped_directory_is_actually_skipped(self):
"""invariant: every name in SKIP_DIRS is excluded from the walk.
refuted: replaced the walk's `part in SKIP_DIRS` check with
`part in {"node_modules"}` -> this test went red naming `.coherence`,
while the other 25 tests in this file stayed green.
"""
| finding | meaning |
|---|---|
unrefuted |
a claim nobody has watched fail |
literal |
the claiming test iterates a copy of the registry, so the claim cannot see a new member |
unanchored |
a registry no invariant names — a question, not a verdict |
A claim whose test carries a ratchet marker is reported as pinning its
registries rather than copying them, and a ratcheted registry is not
unanchored.
python3 scripts/claims.py <repo> [--json] [--strict] [--selftest]
Write the refutation from what you observed
refuted: is the half that costs something. Break the chokepoint, watch the
claim go red by name, restore, and record what you saw. A green test and an
unfalsifiable one look identical from outside; the refutation is what separates
them.
Never write a refutation you have not run. The first refutation authored
for the SKIP_DIRS invariant above asserted a failure that did not occur — the
fixture placed the registry outside the test's reachability, so the test passed
under perturbation for an unrelated reason. A vacuous claim, written while
building the tool that catches vacuous claims, and caught only by running the
perturbation instead of trusting the sentence.
Procedure, in order:
- Break the chokepoint the claim names, with one edit in the source rather than in the test.
- Run the claiming test. Read the failure text. Note the member it named.
- Run the rest of the file. Confirm the others stay green — if everything goes red, the claim is not localised and the refutation says nothing.
- Restore the source.
- Write
refuted: <the edit> -> <what went red, by name>.
Wiring it into a commit gate
The reference wiring lives in oaustegard/claude-workspace
(scripts/tdd_hook.py): a commit where a registry gained a member and no
invariant: test iterates it live is denied, naming the registry and what it
gained. Override with no-invariant: <why> in the commit body.
Registry shrink is gated the same way, and needs a ratchet rather than an enumeration to clear it. A new function or a new branch is behavioural growth too, but neither is diffable without guessing, and a gate that guesses stops being consulted. A brand-new registry is not gated. Declaring one is a judgement call; growing one has already made it.
Where the two filters came from
Both filters exist because the unfiltered version was noise. Reproduce either by removing the filter and re-running against a real repo.
no-floorfires only on a name independently recognised as a registry. Firing on everyfor x in <local>produced 27 findings onremex, all noise: numpy arrays, query matrices, loop counters.- A literal matches a registry only when reachable — the test imports its
module or package, shares its top-level directory, or is its paired
tests/test_<mod>.py. Matching against any registry in the tree joined a test indiscrepancy/to registries inkb-k-sweep/andremex-vs-higgs-ablation/on a monorepo, because small integer sets collide by chance. Four findings became one, and the survivor was real.
The join key is membership, not names: [1, 2, 3, 4, 8] and
SUPPORTED_BITS share no token, so containment is what ties them together. No
naming convention is assumed, and none is required.
Limits
- Python only. The extractors are
_registries_pyand_domains_py; a tree-sitter pair for another language slots in beside them. The join, the acknowledgements and the reports are all language-independent. - Report, not gate, by default.
--strictopts into a nonzero exit. The tool this was adapted from gates by default, and its parity arm false-fails a correct oracle that binds its domain to a local name first. A gate that false-fails stops being consulted. - A registry is a collection of constants. A dict/set/tuple/list of literals, or an Enum body, with at least three members. A domain assembled at runtime is invisible here.
- Multi-parameter tables are out of scope. Only single-name
parametrizeis read. unanchoredis a question. Most registries need no invariant. Treat the list as candidates for declaration, never as a backlog to clear.- A registry is modelled as a SET OF KEYS, so a value swap is invisible.
This is the largest hole in the design, and it undercuts the motivating
example.
ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}exists to pin bytes on disk; permuting it to{"haar": 0, "rht": 2, "none": 1}keeps every key, every count and every ratchet intact, and every index already written decodes under the wrong rotation. Verified 2026-08-25 on a fixture: the linter reports nothing and the gate's registry half reports nothing. (The gate denied that fixture, but on the unrelated TDD rule — checked, because claiming otherwise would have been the overclaim this skill exists to catch.) Nothing here checks a name-to-code mapping. Extracting(key, value)pairs as the member set would, at the cost of breaking the subset join against aparametrizelist of keys. - A co-ordinated rename defeats the ratchet. A global find/replace that
renames a member in the registry AND in the hand-list leaves the pin intact,
and this reports nothing. Verified 2026-08-25 on a fixture: renaming
"none"to"identity"in both files was silent, while every index already on disk still decodes byte 2 as the old name. Both sides of a co-located hand-list move together, so no static check over one working tree can see it. The commit gate can, because it diffs against git history — and only if the ratchet itself is untouched in that commit, which it now requires. - A brand-new registry is neither gated nor pinned. Create one with five
members and never grow it and no gate ever fires.
unanchoredandunratchetedsurface it in the report; the gate deliberately does not, on the grounds that gating every new module is how a gate stops being consulted. That is a judgement, not a proof, and it is the largest hole a cross-model review found. - A decorator-built registry is invisible.
@register("name")populating a dict at import time is the common Python registry idiom and is not a literal, so nothing here sees it. Named because it is the shape most likely to be mistaken for coverage. - Reachability is path-based, so a disconnected integration test is missed. A test that neither imports the module, shares its top-level directory, nor pairs with it by filename will not be joined to the registry it samples. The filter trades that recall for the cross-project precision it was measured to buy.
- The precision numbers come from three repositories, and the filters were fitted to two of them. 27 noise findings on one, 3-of-4 cross-project joins on another. Those are the measurements that justified each filter; they are not a false-positive rate on a corpus, and should not be read as one.
- Split and merge are invisible. The diff is keyed on registry NAME, so renaming a registry, splitting one in two, or merging two into one falls through both the gained and lost paths.
- A claim that passes is not a claim that is right. This checks that a declared invariant loops the domain it names. Whether it is the right invariant is human judgement, and it is not automatable.
Related
verifying-claimscovers the prose layer: does the documentation match reality? Agent-judged, non-deterministic, run as a triggered review. This skill is the deterministic half, over code and tests rather than prose.tree-sittinglocates the registry or the test before you edit it.
Files (claude-skills)
-
scripts
-
claims.py 15.2 KB
#!/usr/bin/env python3 """claims.py — the declared invariants of a repository, and what backs them. `totality_lint.py` answers "is this test's domain complete?". It presumes a test exists. The harder question is the one coherence's own Known Limits concedes it does not answer: *nothing enforces `exists ⇒ declared`*. An invariant the code depends on but no test names is invisible to every gate. A claim here is a test whose docstring opens with `invariant:`. No new file format, no new syntax, and the claim inherits its test's pass/fail for free: def test_every_declared_rotation_round_trips(rotation): '''invariant: every declared rotation survives the on-disk formats. refuted: added "hadamard2" to ROTATION_CODES with no construction behind it -> the whole suite stayed green at 267 passed while this went red on [hadamard2]. ''' Two lines, two jobs. `invariant:` states what must hold. `refuted:` records what was broken on purpose and what went red — the observed negative control, and the only thing separating a claim that holds from a claim that cannot fail. A green test and an unfalsifiable one are indistinguishable from outside; the refutation is what tells them apart, and it is worth writing down because it is cheap to produce once and impossible to reconstruct later. Findings: unrefuted an invariant with no `refuted:` line. Advisory, and the whole point: break the chokepoint, watch it go red by name, restore, write down what you saw. unanchored a registry in source with three or more members that no invariant names. A candidate for declaration, not a defect — most registries do not need one. literal an invariant whose test iterates a copied literal rather than the registry. `totality_lint.py` grades this; it is surfaced here so one report answers "what do we claim, and does the claim mean anything". python3 scripts/claims.py <path> # the report python3 scripts/claims.py <path> --json python3 scripts/claims.py <path> --strict # exit 1 if any finding python3 scripts/claims.py --selftest # fixtures, no repo Python only, same as `totality_lint.py`, whose registry and domain extractors this reuses rather than reimplementing. """ from __future__ import annotations import argparse import ast import importlib.util import json import pathlib import re import sys from dataclasses import dataclass, field def _load_totality_lint(): """Import the sibling extractor without requiring a package layout.""" here = pathlib.Path(__file__).resolve().parent / "totality_lint.py" spec = importlib.util.spec_from_file_location("totality_lint", here) mod = importlib.util.module_from_spec(spec) # Register before exec: @dataclass resolves annotations through # sys.modules[cls.__module__], which is None for a spec-loaded module. sys.modules.setdefault("totality_lint", mod) spec.loader.exec_module(mod) return mod tl = _load_totality_lint() INVARIANT_RE = re.compile(r"^\s*invariant:\s*(.+)", re.I) REFUTED_RE = re.compile(r"^\s*refuted:\s*(.+)", re.I | re.M) @dataclass class Claim: statement: str test: str path: str line: int refuted: str | None = None #: Registries this test iterates live, by name. A claim over a domain is #: only as good as the domain it loops. anchors: list = field(default_factory=list) #: Registries this test iterates as a copied literal. copies: list = field(default_factory=list) #: Registries this test PINS with a deliberate hand-list. A ratchet is the #: complement of an enumeration, not a weaker version of it: an enumeration #: loops whatever the domain now holds and is blind to it narrowing. ratchets: list = field(default_factory=list) @dataclass class Finding: kind: str detail: str path: str = "" line: int = 0 test: str = "" def _docstring_claim(fn: ast.FunctionDef) -> tuple[str, str | None] | None: """(statement, refutation) if this test declares an invariant.""" doc = ast.get_docstring(fn) if not doc: return None first = doc.splitlines()[0] if doc.splitlines() else "" m = INVARIANT_RE.match(first) if not m: return None statement = m.group(1).strip().rstrip(".") ref = REFUTED_RE.search(doc) refutation = None if ref: # A refutation runs to the end of its paragraph. tail = doc[ref.start():] para = tail.split("\n\n", 1)[0] refutation = " ".join(para.split()) refutation = REFUTED_RE.sub("", refutation, count=1).strip() or para.strip() return (statement, refutation) def inventory(root: pathlib.Path) -> tuple[list[Claim], list[tl.Registry], dict]: src_files, test_files = [], [] for p in sorted(root.rglob("*.py")): if any(part in tl.SKIP_DIRS for part in p.parts): continue rel = str(p.relative_to(root)) (test_files if tl._is_test_path(rel) else src_files).append((p, rel)) registries: list[tl.Registry] = [] for p, rel in src_files: try: registries.extend(tl._registries_py(ast.parse(p.read_text(errors="replace")), rel)) except (SyntaxError, ValueError, OSError): continue claims: list[Claim] = [] for p, rel in test_files: try: src = p.read_text(errors="replace") tree = ast.parse(src) except (SyntaxError, ValueError, OSError): continue imports = tl._imported_modules(tree) visible = [r for r in registries if tl._reachable(r, rel, imports)] by_name = {} for r in visible: for key in {r.name, r.name.rsplit(".", 1)[-1]}: by_name.setdefault(key, r) domains = {} for d in tl._domains_py(tree, rel, src): domains.setdefault(d.test, []).append(d) for fn in ast.walk(tree): if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)): continue decl = _docstring_claim(fn) if decl is None: continue statement, refutation = decl anchors, copies, ratchets = [], [], [] for d in domains.get(fn.name, []): if d.ack and d.ack[0] == "ratchet" and d.members is not None: for r in visible: if d.members <= r.members: ratchets.append(r.name) hit = tl._nearest_registry(d, visible) if hit and hit[1]: ratchets.append(hit[0].name) continue if d.live: reg = next( (by_name[n] for n in d.lookup_names() if n in by_name), None, ) if reg: anchors.append(reg.name) elif d.members is not None and len(d.members) >= tl.MIN_LITERAL: reg = tl._best_registry(d.members, visible) if reg: copies.append(reg.name) claims.append(Claim( statement=statement, test=fn.name, path=rel, line=fn.lineno, refuted=refutation, anchors=sorted(set(anchors)), copies=sorted(set(copies)), ratchets=sorted(set(ratchets)), )) stats = { "source_files": len(src_files), "test_files": len(test_files), "registries": len(registries), "claims": len(claims), "refuted": sum(1 for c in claims if c.refuted), } return claims, registries, stats def findings_for(claims: list[Claim], registries: list[tl.Registry]) -> list[Finding]: out: list[Finding] = [] for c in claims: if c.copies: out.append(Finding( "literal", f'"{c.statement}" is anchored to a test that iterates a COPY of ' f'{", ".join("`" + n + "`" for n in c.copies)} — the claim cannot ' f'see a member added to the registry', c.path, c.line, c.test, )) if not c.refuted: out.append(Finding( "unrefuted", f'"{c.statement}" has never been observed failing — break its ' f'chokepoint, watch it go red by name, restore, and record ' f'`refuted: <what was broken> -> <what was seen>`', c.path, c.line, c.test, )) named = set() for c in claims: named.update(c.anchors) named.update(c.copies) named.update(c.ratchets) for r in registries: if r.name in named or r.name.rsplit(".", 1)[-1] in named: continue out.append(Finding( "unanchored", f"`{r.name}` ({r.kind}, {len(r.members)} members) is enumerated in " f"source and no invariant names it", r.path, r.line, )) return out ORDER = {"literal": 0, "unrefuted": 1, "unanchored": 2} def report(claims: list[Claim], findings: list[Finding], stats: dict) -> str: out = [ "", " CLAIMS — what this repository declares must hold", "", f" {stats['claims']} invariant(s) declared · {stats['refuted']} with an " f"observed refutation · {stats['registries']} registry/ies in source", "", ] if claims: for c in sorted(claims, key=lambda c: (c.path, c.line)): mark = "✓" if c.refuted else "·" out.append(f" {mark} {c.statement}") over = "" if c.anchors: over = f" over {', '.join(c.anchors)}" elif c.ratchets: over = f" pinning {', '.join(c.ratchets)}" out.append(f" {c.path}:{c.line} {c.test}{over}") out.append("") if not findings: out += [" nothing to report.", ""] return "\n".join(out) for f in sorted(findings, key=lambda f: (ORDER.get(f.kind, 9), f.path, f.line)): where = f"{f.path}:{f.line}" + (f" {f.test}" if f.test else "") out.append(f" [{f.kind}] {where}") out.append(f" {f.detail}") out.append("") out += [ " Candidates, not defects. Most registries need no invariant; an", " `unanchored` row is a question, not a verdict. An `unrefuted` row is", " the one worth clearing — a claim nobody has watched fail is a claim", " nobody has tested.", "", ] return "\n".join(out) # -------------------------------------------------------------------------- # selftest # -------------------------------------------------------------------------- FIX_SRC = ''' ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2} UNWATCHED = {"a": 1, "b": 2, "c": 3} ''' FIX_TEST = ''' import pytest from fix_src import ROTATION_CODES def test_round_trips(rotation="haar"): """invariant: every declared rotation survives the on-disk formats. refuted: added "hadamard2" with no construction -> suite stayed green, this went red on [hadamard2]. """ for name in ROTATION_CODES: assert name def test_unrefuted(): """invariant: the codes are stable across releases.""" for name in ROTATION_CODES: assert name @pytest.mark.parametrize("r", ["haar", "rht"]) def test_claim_over_a_copy(r): """invariant: every rotation is accepted by save_params.""" assert r def test_not_a_claim(): """Ordinary test, no declaration.""" assert True ''' def selftest() -> int: import tempfile fails = [] def check(label, ok, detail=""): print(f" {'ok ' if ok else 'FAIL'} {label}" + ("" if ok else f" -> {detail}")) if not ok: fails.append(label) with tempfile.TemporaryDirectory() as td: root = pathlib.Path(td) (root / "fix_src.py").write_text(FIX_SRC) (root / "tests").mkdir() (root / "tests" / "test_fix.py").write_text(FIX_TEST) claims, regs, stats = inventory(root) f = findings_for(claims, regs) by_test = {c.test: c for c in claims} kinds = {(x.kind, x.test) for x in f} check("three declarations found, the plain test excluded", stats["claims"] == 3 and "test_not_a_claim" not in by_test, str(sorted(by_test))) check("the statement is captured without its trailing period", by_test["test_round_trips"].statement.endswith("on-disk formats"), by_test["test_round_trips"].statement) check("a refutation is captured", "hadamard2" in (by_test["test_round_trips"].refuted or ""), str(by_test["test_round_trips"].refuted)) check("a refuted claim is not reported unrefuted", ("unrefuted", "test_round_trips") not in kinds, str(kinds)) check("an unrefuted claim is reported", ("unrefuted", "test_unrefuted") in kinds, str(kinds)) check("a claim anchored to a live registry records the anchor", by_test["test_round_trips"].anchors == ["ROTATION_CODES"], str(by_test["test_round_trips"].anchors)) check("a claim over a copied literal is reported", ("literal", "test_claim_over_a_copy") in kinds, str(kinds)) check("a registry no invariant names is reported unanchored", any(x.kind == "unanchored" and "UNWATCHED" in x.detail for x in f), str([x.detail for x in f if x.kind == "unanchored"])) check("a registry an invariant does name is not reported unanchored", not any(x.kind == "unanchored" and "ROTATION_CODES" in x.detail for x in f), str([x.detail for x in f if x.kind == "unanchored"])) print() if fails: print(f"{len(fails)} check(s) failed: {fails}") return 1 print("all checks passed") return 0 def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("path", nargs="?", default=".", help="repository root") ap.add_argument("--json", action="store_true", help="machine-readable") ap.add_argument("--strict", action="store_true", help="exit 1 if any finding") ap.add_argument("--selftest", action="store_true", help="fixtures, no repo") args = ap.parse_args(argv) if args.selftest: return selftest() root = pathlib.Path(args.path).resolve() if not root.is_dir(): print(f"not a directory: {root}", file=sys.stderr) return 2 claims, registries, stats = inventory(root) findings = findings_for(claims, registries) if args.json: print(json.dumps({ "stats": stats, "claims": [ {"statement": c.statement, "test": c.test, "path": c.path, "line": c.line, "refuted": c.refuted, "anchors": c.anchors, "copies": c.copies} for c in claims ], "findings": [ {"kind": f.kind, "detail": f.detail, "path": f.path, "line": f.line, "test": f.test} for f in findings ], }, indent=2, default=str)) else: print(report(claims, findings, stats)) return 1 if (findings and args.strict) else 0 if __name__ == "__main__": sys.exit(main()) -
totality_lint.py 35.2 KB
#!/usr/bin/env python3 """totality_lint.py — find tests that enumerate a domain by copying it. A test that loops a hand-written list passes its runner and proves nothing about completeness. When the same members also exist as a registry in the source — a dict, a set, a tuple, an Enum — the list is a *copy*, and the copy drifts the moment someone adds a member to the registry and not to the test. Nothing goes red. The suite still says 267 passed. Measured on `oaustegard/remex` (2026-08-24): adding a fourth member to `ROTATION_CODES` with nothing behind it left the entire 267-test suite green. Only a test that looped the registry itself caught it. This reports two findings: sampled-domain a test iterates a literal whose members are a strict subset of a source registry's members. The missing members are named, because those are the ones nothing covers. no-floor a test iterates a live registry with no length assertion anywhere in the file. A registry that collapses to zero makes the loop range over nothing and pass vacuously. Adapted from the meta-oracle in daniloc/coherence (`src/oracle-domain.ts`, `analyzeOracle`), which classifies an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST. Three deliberate differences, each from something the trial of that tool turned up: * **Report, not gate.** Its parity arm refuses a correct oracle that binds the domain to a local name first (`d = set(REGISTRY); for x in sorted(d)`), against a README claiming the analyzer never false-fails. A gate that false-fails gets switched off; an advisory gets read. `--strict` opts in. * **Join on membership, not on names.** `[1, 2, 3, 4, 8]` and `SUPPORTED_BITS` share no token. What ties them is that the literal's members are a subset of the registry's, so that is the join key. No naming convention is assumed, and none is required. * **No spec files.** Coherence needs a `*.spec.md` declaring the claim before it will analyse anything. This needs a path. Acknowledgement is first-class, on the same principle as `memory_lint.py`: a partial domain is often correct (`save_params` legitimately accepts two of three rotations). Say so on the test and it stops being a finding: # totality: partial — mojo has no construction for "none" @pytest.mark.parametrize("rotation", ["haar", "rht"]) def test_save_params_accepts_every_mojo_rotation(...): And a suppression must not become a silence: an acknowledgement on a test that now covers the whole domain is reported in its own right (`stale-ack`). python3 scripts/totality_lint.py <path> # the report python3 scripts/totality_lint.py <path> --json # machine-readable python3 scripts/totality_lint.py <path> --strict # exit 1 if any finding python3 scripts/totality_lint.py --selftest # fixtures, no repo Python only. The registry and literal extractors are the language-specific half (`_registries_py`, `_domains_py`); the join, the acknowledgement handling and the report are not. TypeScript is not covered — a `_registries_ts` pair built on tree-sitter would slot in beside them. """ from __future__ import annotations import argparse import ast import json import pathlib import re import sys from dataclasses import dataclass, field from typing import Iterable SKIP_DIRS = { ".git", "node_modules", "__pycache__", ".venv", "venv", "dist", "build", ".pytest_cache", ".mypy_cache", ".tox", "site-packages", ".coherence", ".ruff_cache", } #: A path is a test file if any of these appear in it. Registries are read from #: NON-test files only: a list defined in a test file and looped by that same #: test file is a fixture, not a copied domain, and treating it as a registry is #: the single largest source of false positives. TEST_MARKERS = ("test_", "_test.", "/tests/", "/test/", "conftest.py") #: Fewer members than this and a "domain" is a pair of examples, not an #: enumeration. Raising it trades recall for precision; 3 keeps #: `{"haar", "rht", "none"}` in scope, which is the motivating case. MIN_REGISTRY = 3 #: A literal below this is not plausibly a copy of anything. MIN_LITERAL = 2 #: `partial` excuses a hand-list. `ratchet` ASSERTS one: the members named are a #: floor the registry must keep containing, and the linter checks that rather #: than taking the marker's word for it. Enumeration and a ratchet are #: complementary, not a ladder — see `_ratchet_findings`. ACK_KINDS = ("partial", "sampled", "ratchet") _ACK_ALT = "|".join(ACK_KINDS) ACK_RE = re.compile(rf"#\s*totality:\s*({_ACK_ALT})\b[ \t]*[-—:]?[ \t]*(.*)") ACK_DOC_RE = re.compile(rf"^\s*totality:\s*({_ACK_ALT})\b[ \t]*[-—:]?[ \t]*(.*)", re.M) Member = str | int | float | bool @dataclass(frozen=True) class Registry: """An enumerated collection defined in source (not test) code.""" name: str members: frozenset path: str line: int kind: str # dict | set | list | tuple | enum @property def module(self) -> str: return self.path.replace("\\", "/").removesuffix(".py").replace("/", ".") @property def pkg(self) -> str: return self.path.replace("\\", "/").split("/", 1)[0].removesuffix(".py") @dataclass class Domain: """A collection a test iterates, and how it got hold of it.""" path: str line: int test: str live: bool members: frozenset | None # None when live symbol: str | None # the registry the iteration roots in, when known how: str # parametrize | for | comprehension alias: str | None = None # the name actually written at the loop, if different candidates: list = field(default_factory=list) # every name it could address def lookup_names(self) -> list: """Names to try against the registry table, best first. Candidates before the resolved root: a test that reaches its subject through an importlib-loaded module writes `tl.SKIP_DIRS`, and following the local `tl` through the file's own constants lands on `_SPEC` — true, useless, and it shadows the name that actually matters. """ return list(dict.fromkeys( [c for c in self.candidates if c] + [n for n in (self.symbol, self.alias) if n] )) ack: tuple[str, str] | None = None # (kind, reason) @dataclass class Finding: kind: str path: str line: int test: str detail: str missing: list = field(default_factory=list) registry: str | None = None # -------------------------------------------------------------------------- # extraction — the language-specific half # -------------------------------------------------------------------------- def _const_members(node: ast.AST) -> frozenset | None: """Members of a literal collection, or None if it is not one. A dict contributes its KEYS: a name-to-code table is enumerated by name, which is what a test parametrizes over. """ if isinstance(node, ast.Dict): items = node.keys elif isinstance(node, (ast.List, ast.Set, ast.Tuple)): items = node.elts elif isinstance(node, ast.Call): # set([...]) / frozenset([...]) / tuple([...]) / list([...]) fn = node.func name = fn.id if isinstance(fn, ast.Name) else getattr(fn, "attr", None) if name in {"set", "frozenset", "tuple", "list"} and len(node.args) == 1: return _const_members(node.args[0]) return None else: return None out = [] for it in items: if it is None: # `{**other}` in a dict return None if not isinstance(it, ast.Constant): return None if not isinstance(it.value, (str, int, float, bool)): return None out.append(it.value) return frozenset(out) if out else None def _is_test_path(path: str) -> bool: p = path.replace("\\", "/") return any(m in p for m in TEST_MARKERS) def _registries_py(tree: ast.Module, path: str) -> list[Registry]: """Module-level and class-level enumerated collections.""" found: list[Registry] = [] def enum_bases(cls: ast.ClassDef) -> bool: for b in cls.bases: n = b.id if isinstance(b, ast.Name) else getattr(b, "attr", "") if n in {"Enum", "IntEnum", "StrEnum", "Flag", "IntFlag"}: return True return False def scan_assign(node, qualifier: str = ""): targets = node.targets if isinstance(node, ast.Assign) else [node.target] for t in targets: if not isinstance(t, ast.Name): continue members = _const_members(node.value) if members is None or len(members) < MIN_REGISTRY: continue kind = { ast.Dict: "dict", ast.Set: "set", ast.List: "list", ast.Tuple: "tuple", }.get(type(node.value), "set") found.append(Registry( name=qualifier + t.id, members=members, path=path, line=node.lineno, kind=kind, )) for node in tree.body: if isinstance(node, (ast.Assign, ast.AnnAssign)) and node.value is not None: scan_assign(node) elif isinstance(node, ast.ClassDef): if enum_bases(node): names = [ t.id for b in node.body if isinstance(b, ast.Assign) for t in b.targets if isinstance(t, ast.Name) ] if len(names) >= MIN_REGISTRY: found.append(Registry( name=node.name, members=frozenset(names), path=path, line=node.lineno, kind="enum", )) for b in node.body: if isinstance(b, (ast.Assign, ast.AnnAssign)) and getattr(b, "value", None): scan_assign(b, qualifier=node.name + ".") return found def _module_consts(tree: ast.Module) -> dict[str, ast.AST]: out = {} for node in tree.body: if isinstance(node, ast.Assign) and len(node.targets) == 1: if isinstance(node.targets[0], ast.Name): out[node.targets[0].id] = node.value return out def _imported_names(tree: ast.Module) -> set[str]: out = set() for node in ast.walk(tree): if isinstance(node, ast.ImportFrom): for a in node.names: out.add(a.asname or a.name) elif isinstance(node, ast.Import): for a in node.names: out.add((a.asname or a.name).split(".")[0]) return out def _imported_modules(tree: ast.Module) -> set[str]: """Dotted module paths this file imports, plus each of their prefixes. `from remex.packing import SUPPORTED_BITS` contributes `remex.packing` and `remex`. This is what ties a test to the registries it could plausibly be copying — see `_reachable`. """ out: set[str] = set() def add(mod: str | None): if not mod: return parts = mod.split(".") for i in range(1, len(parts) + 1): out.add(".".join(parts[:i])) for node in ast.walk(tree): if isinstance(node, ast.ImportFrom): add(node.module) elif isinstance(node, ast.Import): for a in node.names: add(a.name) return out def _symbol_candidates(node: ast.AST) -> list[str]: """Names a domain iteration could be addressing, best first. `_root_symbol` peels an attribute down to its object, so `tl.SKIP_DIRS` roots in `tl` — the module, not the registry. That is right for `q.rotations()` and wrong for `mod.REGISTRY`, and this repository's own tests take the second shape: they load the module under test through `importlib` and reach its registries as attributes. So offer the attribute name too and let the caller match whichever it recognises. """ out: list[str] = [] seen = 0 cur = node while seen < 8: seen += 1 if isinstance(cur, ast.Attribute): out.append(cur.attr) cur = cur.value continue if isinstance(cur, ast.Call): cur = cur.args[0] if cur.args else cur.func continue if isinstance(cur, (ast.Subscript, ast.Starred)): cur = cur.value continue break root = _root_symbol(node) if root: out.append(root) # Preserve order, drop duplicates. return list(dict.fromkeys(out)) def _root_symbol(node: ast.AST) -> str | None: """Peel `sorted(X)`, `list(X.keys())`, `X.values()` down to `X`.""" seen = 0 while seen < 8: seen += 1 if isinstance(node, ast.Name): return node.id if isinstance(node, ast.Attribute): node = node.value continue if isinstance(node, ast.Call): if node.args: node = node.args[0] continue node = node.func continue if isinstance(node, (ast.Subscript, ast.Starred)): node = node.value continue return None return None def _ack_for(src_lines: list[str], node: ast.AST) -> tuple[str, str] | None: """An acknowledgement comment above the test, or in its docstring.""" start = min( [node.lineno] + [d.lineno for d in getattr(node, "decorator_list", [])] ) for ln in range(max(1, start - 4), start): m = ACK_RE.search(src_lines[ln - 1]) if m: return (m.group(1), m.group(2).strip()) doc = ast.get_docstring(node) if isinstance( node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef) ) else None if doc: m = ACK_DOC_RE.search(doc) if m: return (m.group(1), m.group(2).strip()) return None def _domains_py(tree: ast.Module, path: str, src: str) -> list[Domain]: """Every collection a test function iterates.""" consts = _module_consts(tree) imported = _imported_names(tree) lines = src.splitlines() out: list[Domain] = [] def classify(value: ast.AST) -> tuple[bool, frozenset | None, str | None, str | None]: """-> (live, members, root symbol, local alias). The alias matters for the floor check: a test that binds `LIVE = sorted(REGISTRY)` and asserts `len(LIVE) >= 3` has a floor, even though the iteration roots in `REGISTRY`. """ members = _const_members(value) if members is not None: return (False, members, None, None) cands = _symbol_candidates(value) sym = cands[0] if cands else None if sym is None: return (True, None, None, None) # unknown shape: assume live, never fail sym = next((c for c in cands if c in consts), sym) if sym in consts: inner = _const_members(consts[sym]) if inner is not None: return (False, inner, sym, None) root = _root_symbol(consts[sym]) return (True, None, root or sym, sym) return (True, None, sym, None) for fn in ast.walk(tree): if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)): continue if not fn.name.startswith("test"): continue ack = _ack_for(lines, fn) for dec in fn.decorator_list: if not isinstance(dec, ast.Call): continue if _root_symbol(dec.func) != "pytest" and getattr( dec.func, "attr", "" ) != "parametrize": continue if getattr(dec.func, "attr", "") != "parametrize" or len(dec.args) < 2: continue argnames = dec.args[0] if not (isinstance(argnames, ast.Constant) and isinstance(argnames.value, str) and "," not in argnames.value): continue # multi-parameter tables are out of scope live, members, sym, alias = classify(dec.args[1]) out.append(Domain(path, dec.lineno, fn.name, live, members, sym, "parametrize", alias, _symbol_candidates( dec.args[1]), ack)) for node in ast.walk(fn): if isinstance(node, ast.For): live, members, sym, alias = classify(node.iter) out.append(Domain(path, node.lineno, fn.name, live, members, sym, "for", alias, _symbol_candidates( node.iter), ack)) elif isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp, ast.DictComp)): for gen in node.generators: live, members, sym, alias = classify(gen.iter) out.append(Domain(path, node.lineno, fn.name, live, members, sym, "comprehension", alias, _symbol_candidates( gen.iter), ack)) return out def _has_floor(tree: ast.Module, *symbols: str | None) -> bool: """Any `len(<symbol>) <op> n` comparison anywhere in the module. Several names can stand for one domain — the registry and whatever local the test bound it to — so a floor on any of them counts. """ wanted = {s for s in symbols if s} if not wanted: return False for node in ast.walk(tree): if not isinstance(node, ast.Compare): continue left = node.left if isinstance(left, ast.Call) and _root_symbol(left.func) == "len": if left.args and _root_symbol(left.args[0]) in wanted: return True return False # -------------------------------------------------------------------------- # the join — language-independent # -------------------------------------------------------------------------- def _reachable(reg: Registry, test_path: str, imports: set[str]) -> bool: """Could this test plausibly be copying THIS registry? Without this, a numeric literal matches any numeric registry anywhere in the tree. Measured on `oaustegard/experiments`, a monorepo of unrelated projects: 3 of 4 findings joined a test in `discrepancy/` to registries in `kb-k-sweep/` and `remex-vs-higgs-ablation/`, purely because small integer sets collide. Two ways to be reachable, either sufficient: * the test imports the registry's module or its package * they sit under the same top-level directory """ if reg.module in imports or reg.pkg in imports: return True tail = reg.module.rsplit(".", 1)[-1] if any(i.split(".")[-1] == tail for i in imports): return True # tests/test_totality_lint.py <-> scripts/totality_lint.py. The convention # this repository actually uses, and the one importlib-loaded tests need: # they have no import statement naming the module they exercise. stem = pathlib.PurePath(test_path).stem for prefix, suffix in (("test_", ""), ("", "_test")): if prefix and stem.startswith(prefix): stem = stem[len(prefix):] if suffix and stem.endswith(suffix): stem = stem[: -len(suffix)] if stem == pathlib.PurePath(reg.path).stem: return True test_pkg = test_path.replace("\\", "/").split("/", 1)[0].removesuffix(".py") return test_pkg == reg.pkg def _best_registry(members: frozenset, regs: Iterable[Registry]) -> Registry | None: """The smallest registry that strictly contains these members. Smallest, because a literal contained by both a 4-member and a 40-member registry is far more likely to be a copy of the 4. """ cands = [r for r in regs if members < r.members] if not cands: return None return min(cands, key=lambda r: (len(r.members), r.path, r.line)) def _nearest_registry(d, regs) -> tuple | None: """(registry, members of the ratchet it no longer holds). An empty second element means the ratchet is intact — it pins the whole domain rather than a strict subset of it. A ratchet is a hand-list asserting the domain keeps containing it. That is the direction an enumeration cannot see: a totality oracle loops whatever the domain currently is, so a member LEAVING is trivially green. Measured on `oaustegard/remex`: substituting one member for another, keeping cardinality and keeping a second spelling of the domain in agreement, passed the domain floor, the enumeration and the parity check — five green tests while a supported rotation silently left. Match on overlap rather than containment, because containment is exactly what broke. """ best, best_key = None, None for r in regs: overlap = len(d.members & r.members) if overlap < MIN_LITERAL: continue # Most overlap wins; ties break on the smaller registry, then on a # stable address, so one report does not reorder between runs. key = (-overlap, len(r.members), r.path, r.line) if best_key is None or key < best_key: best, best_key = r, key if best is None: return None return (best, sorted(d.members - best.members, key=repr)) def scan(root: pathlib.Path) -> tuple[list[Finding], dict]: src_files, test_files = [], [] for p in sorted(root.rglob("*.py")): if any(part in SKIP_DIRS for part in p.parts): continue rel = str(p.relative_to(root)) (test_files if _is_test_path(rel) else src_files).append((p, rel)) registries: list[Registry] = [] for p, rel in src_files: try: registries.extend(_registries_py(ast.parse(p.read_text(errors="replace")), rel)) except (SyntaxError, ValueError, OSError): continue findings: list[Finding] = [] #: registry name -> the ratchet domains pinning it ratcheted: dict[str, list] = {} #: registry name -> the domains enumerating it live enumerated: dict[str, list] = {} domains_seen = 0 for p, rel in test_files: try: src = p.read_text(errors="replace") tree = ast.parse(src) except (SyntaxError, ValueError, OSError): continue imports = _imported_modules(tree) visible = [r for r in registries if _reachable(r, rel, imports)] vis_by_name: dict[str, Registry] = {} for r in visible: for key in {r.name, r.name.rsplit(".", 1)[-1]}: vis_by_name.setdefault(key, r) for d in _domains_py(tree, rel, src): domains_seen += 1 if d.live: # Only a name we independently recognised as a registry earns a # no-floor finding. Firing on every `for x in <local>` produced 27 # findings on remex, all noise — numpy arrays, query matrices, loop # counters. A wall of candidates is worse than silence. reg = next( (vis_by_name[c] for c in d.lookup_names() if c in vis_by_name), None, ) if reg and not _has_floor(tree, d.symbol, d.alias, reg.name, reg.name.rsplit(".", 1)[-1]): findings.append(Finding( "no-floor", d.path, d.line, d.test, f"iterates `{reg.name}` ({reg.kind}, {reg.path}:" f"{reg.line}, {len(reg.members)} members) with no " f"`len(...) >= n` assertion in this file — vacuous if " f"it empties", registry=reg.name, )) if reg: enumerated.setdefault(reg.name, []).append(d) if d.ack and reg: findings.append(Finding( "stale-ack", d.path, d.line, d.test, f"acknowledged as {d.ack[0]} ({d.ack[1] or 'no reason given'}), " f"but it iterates a live domain — the ack no longer suppresses " f"anything", )) continue if d.members is None or len(d.members) < MIN_LITERAL: continue reg = _best_registry(d.members, visible) if reg is None: if d.ack and d.ack[0] == "ratchet": # No strict superset, so either the ratchet pins the whole # domain (intact) or a member has left (broken). hit = _nearest_registry(d, visible) if hit: reg, left = hit for r in visible: if d.members <= r.members: ratcheted.setdefault(r.name, []).append(d) ratcheted.setdefault(reg.name, []).append(d) if left: findings.append(Finding( "ratchet-broken", d.path, d.line, d.test, f"pins {len(d.members)} member(s) of " f"`{reg.name}`, and the registry no longer " f"contains all of them — a member left without " f"the ratchet being retired", missing=left, registry=reg.name, )) continue missing = sorted(reg.members - d.members, key=repr) if d.ack and d.ack[0] == "ratchet": # Pin EVERY registry the hand-list is a floor for. One domain is # often spelled twice (`ROTATION_CODES` for the on-disk byte, # `Quantizer.ROTATIONS` for the constructor), and a ratchet over # its members holds each of them. for r in visible: if d.members <= r.members: ratcheted.setdefault(r.name, []).append(d) # And FALL THROUGH. A ratchet is a statement about the domain # SHRINKING; it says nothing about the members it never listed, # so a partial ratchet leaves those exactly as uncovered as an # unmarked hand-list does. Suppressing the finding here would # make the marker a laundering channel — the same escape hatch # this tool's own notes criticise `via guard` for being in # `daniloc/coherence`, reproduced one day later. Verified 2026-08-25: # `# totality: ratchet` over 2 of 3 members silenced the report # entirely and checked nothing about the third. elif d.ack: continue others = sum( 1 for r in visible if d.members < r.members and len(r.members) == len(reg.members) ) - 1 also = f" (+{others} registry/ies of the same size also contain it)" if others else "" pinned = " (pinned as a ratchet, which covers only shrink)" \ if d.ack and d.ack[0] == "ratchet" else "" findings.append(Finding( "sampled-domain", d.path, d.line, d.test, f"{d.how} over a literal of {len(d.members)}{pinned}, but " f"`{reg.name}` ({reg.kind}, {reg.path}:{reg.line}) has " f"{len(reg.members)}{also}", missing=missing, registry=reg.name, )) # The complement Yep named: enumeration proves every CURRENT member is # handled and is structurally blind to the domain narrowing, because it # loops whatever the domain now is. A ratchet is the other direction. # A registry with no floor AND no ratchet produced two findings on the same # line saying overlapping things — 3 of 3 on `claude-workspace`, which a # cross-model review named as finding fatigue. `no-floor` is the narrower # statement and names the same fix first, so it wins the line. floored_out = { (f.path, f.line) for f in findings if f.kind == "no-floor" } for name, doms in sorted(enumerated.items()): if name in ratcheted: continue d = doms[0] if (d.path, d.line) in floored_out: continue findings.append(Finding( "unratcheted", d.path, d.line, d.test, f"enumerates `{name}` live, and nothing pins its membership — a " f"member REMOVED from the registry keeps this green, because the " f"loop ranges over whatever the registry now holds. Pin a floor " f"with `# totality: ratchet — <why>` on a hand-list test", registry=name, )) stats = { "source_files": len(src_files), "test_files": len(test_files), "ratcheted": len(ratcheted), "enumerated": len(enumerated), "registries": len(registries), "domains": domains_seen, } return findings, stats # -------------------------------------------------------------------------- # report # -------------------------------------------------------------------------- ORDER = {"ratchet-broken": 0, "sampled-domain": 1, "stale-ack": 2, "no-floor": 3, "unratcheted": 4} def report(findings: list[Finding], stats: dict) -> str: out = [ "", " TOTALITY — a test that enumerates a domain by copying it", "", f" {stats['source_files']} source file(s) · {stats['registries']} registry/ies · " f"{stats['test_files']} test file(s) · {stats['domains']} iterated domain(s)", "", ] if not findings: out += [" nothing to report.", ""] return "\n".join(out) for f in sorted(findings, key=lambda f: (ORDER.get(f.kind, 9), f.path, f.line)): out.append(f" [{f.kind}] {f.path}:{f.line} {f.test}") out.append(f" {f.detail}") if f.missing: shown = ", ".join(repr(m) for m in f.missing[:8]) more = "" if len(f.missing) <= 8 else f" … and {len(f.missing) - 8} more" out.append(f" never exercised: {shown}{more}") out.append("") out += [ " Candidates, not defects. A partial domain is often correct — say so on", " the test and it stops being a finding:", "", " # totality: partial — <why this subset is the right domain>", "", " Otherwise loop the registry itself, and assert a floor so an emptied", " registry cannot pass vacuously.", "", ] return "\n".join(out) # -------------------------------------------------------------------------- # selftest # -------------------------------------------------------------------------- FIXTURE_SRC = ''' ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2} SUPPORTED_BITS = (1, 2, 3, 4, 8) PAIR = {"a": 1, "b": 2} ''' FIXTURE_TEST = ''' import pytest from fixture_src import ROTATION_CODES, SUPPORTED_BITS DIMS = [64, 128, 384, 768, 1024] LIVE = sorted(ROTATION_CODES) @pytest.mark.parametrize("rotation", ["haar", "rht"]) def test_copies_the_registry(rotation): assert rotation # totality: partial — mojo has no construction for "none" @pytest.mark.parametrize("rotation", ["haar", "rht"]) def test_acknowledged_subset(rotation): assert rotation @pytest.mark.parametrize("bits", [1, 2, 3, 4]) def test_copies_the_widths(bits): assert bits @pytest.mark.parametrize("d", DIMS) def test_local_fixture_is_not_a_copy(d): assert d @pytest.mark.parametrize("rotation", LIVE) def test_live_without_floor(rotation): assert rotation # totality: partial — stale, this one covers everything @pytest.mark.parametrize("rotation", sorted(ROTATION_CODES)) def test_stale_ack(rotation): assert rotation def test_pair_is_below_the_registry_floor(): for k in ["a"]: assert k ''' FIXTURE_TEST_FLOORED = ''' import pytest from fixture_src import ROTATION_CODES LIVE = sorted(ROTATION_CODES) def test_floor(): assert len(LIVE) >= 3 @pytest.mark.parametrize("rotation", LIVE) def test_live_with_floor(rotation): assert rotation ''' def selftest() -> int: import tempfile fails = [] def check(label, ok, detail=""): print(f" {'ok ' if ok else 'FAIL'} {label}" + ("" if ok else f" -> {detail}")) if not ok: fails.append(label) with tempfile.TemporaryDirectory() as td: root = pathlib.Path(td) (root / "fixture_src.py").write_text(FIXTURE_SRC) (root / "tests").mkdir() (root / "tests" / "test_fixture.py").write_text(FIXTURE_TEST) findings, stats = scan(root) by = {} for f in findings: by.setdefault(f.test, []).append(f.kind) check("registries found (2 of 3; PAIR is below MIN_REGISTRY)", stats["registries"] == 2, str(stats["registries"])) check("a literal subset of a registry is reported", "sampled-domain" in by.get("test_copies_the_registry", []), str(by)) check("the missing member is named", any(f.missing == ["none"] for f in findings if f.test == "test_copies_the_registry"), str([f.missing for f in findings])) check("a second registry is matched independently", "sampled-domain" in by.get("test_copies_the_widths", []), str(by)) check("an acknowledged subset is not a finding", "test_acknowledged_subset" not in by, str(by)) check("a test-local fixture list is not a copy", "test_local_fixture_is_not_a_copy" not in by, str(by)) check("a live domain with no floor is reported", "no-floor" in by.get("test_live_without_floor", []), str(by)) check("an ack on a live domain is reported stale", "stale-ack" in by.get("test_stale_ack", []), str(by)) check("a 1-member loop is below MIN_LITERAL", "test_pair_is_below_the_registry_floor" not in by, str(by)) with tempfile.TemporaryDirectory() as td: root = pathlib.Path(td) (root / "fixture_src.py").write_text(FIXTURE_SRC) (root / "tests").mkdir() (root / "tests" / "test_floored.py").write_text(FIXTURE_TEST_FLOORED) findings, _ = scan(root) check("a live domain WITH a floor is silent", not [f for f in findings if f.kind == "no-floor"], str([f.detail for f in findings])) print() if fails: print(f"{len(fails)} check(s) failed: {fails}") return 1 print("all checks passed") return 0 def main() -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("path", nargs="?", default=".", help="repository root") ap.add_argument("--json", action="store_true", help="machine-readable") ap.add_argument("--strict", action="store_true", help="exit 1 if any finding") ap.add_argument("--selftest", action="store_true", help="fixtures, no repo") args = ap.parse_args() if args.selftest: return selftest() root = pathlib.Path(args.path).resolve() if not root.is_dir(): print(f"not a directory: {root}", file=sys.stderr) return 2 findings, stats = scan(root) if args.json: print(json.dumps({ "stats": stats, "findings": [ {"kind": f.kind, "path": f.path, "line": f.line, "test": f.test, "detail": f.detail, "missing": f.missing, "registry": f.registry} for f in findings ], }, indent=2, default=str)) else: print(report(findings, stats)) return 1 if (findings and args.strict) else 0 if __name__ == "__main__": sys.exit(main())
-
-
tests
-
test_claims.py 11.5 KB
"""Tests for scripts/claims.py — the declared-invariant inventory. A claim is a test whose docstring opens `invariant:`. The parts that can go wrong are the parsing (what counts as a declaration, and how much of the docstring the refutation swallows) and the anchoring (which registry, if any, the claiming test actually iterates). The anchoring case that matters is this repository's own shape: a test that loads its subject through `importlib` and reaches a registry as `tl.SKIP_DIRS`. Peeling that attribute down to its object lands on the local module handle, and following THAT through the file's own constants lands on `_SPEC` — true, useless, and it shadows the name that matters. Pinned below. python3 tests/test_claims.py """ from __future__ import annotations import importlib.util import sys import tempfile import textwrap import unittest from pathlib import Path _SPEC = importlib.util.spec_from_file_location( "claims", Path(__file__).resolve().parent.parent / "scripts" / "claims.py" ) cl = importlib.util.module_from_spec(_SPEC) sys.modules["claims"] = cl _SPEC.loader.exec_module(cl) def inv(**files): """Write {relpath: source} into a temp tree and take its inventory.""" with tempfile.TemporaryDirectory() as td: root = Path(td) for rel, src in files.items(): p = root / rel p.parent.mkdir(parents=True, exist_ok=True) p.write_text(textwrap.dedent(src)) claims, registries, stats = cl.inventory(root) return claims, registries, stats, cl.findings_for(claims, registries) SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n' class Declaration(unittest.TestCase): def test_a_docstring_opening_with_invariant_is_a_claim(self): claims, _, stats, _ = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' from pkg.mod import ROTATION_CODES def test_x(): """invariant: every rotation round-trips.""" for r in ROTATION_CODES: assert r ''', }) self.assertEqual(stats["claims"], 1) self.assertEqual(claims[0].statement, "every rotation round-trips") def test_an_ordinary_docstring_is_not_a_claim(self): claims, _, _, _ = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' def test_x(): """Checks the thing.""" assert True ''', }) self.assertEqual(claims, []) def test_the_marker_must_open_the_docstring(self): """A mention buried in prose is discussion, not a declaration.""" claims, _, _, _ = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' def test_x(): """Checks the thing. invariant: this is a sentence about invariants. """ assert True ''', }) self.assertEqual(claims, []) def test_a_test_with_no_docstring_is_skipped(self): claims, _, _, _ = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": "def test_x():\n assert True\n", }) self.assertEqual(claims, []) class Refutation(unittest.TestCase): def _claim(self, doc: str): """doc is the docstring body; it is indented here, not by the caller.""" body = "\n".join( (" " + ln if ln.strip() else "") for ln in doc.splitlines() ).strip("\n") src = ( "from pkg.mod import ROTATION_CODES\n\n\n" "def test_x():\n" ' """' + body.lstrip() + '\n """\n' " for r in ROTATION_CODES:\n" " assert r\n" ) with tempfile.TemporaryDirectory() as td: root = Path(td) (root / "pkg" / "tests").mkdir(parents=True) (root / "pkg" / "mod.py").write_text(SRC) (root / "pkg" / "tests" / "test_it.py").write_text(src) claims, _, _ = cl.inventory(root) self.assertEqual(len(claims), 1, src) return claims[0] def test_a_refutation_is_captured(self): c = self._claim( 'invariant: every rotation round-trips.\n' '\n' 'refuted: added "hadamard2" -> suite green, this went red.\n' ) self.assertIsNotNone(c.refuted) self.assertIn("hadamard2", c.refuted) def test_a_refutation_stops_at_its_paragraph(self): c = self._claim( 'invariant: every rotation round-trips.\n' '\n' 'refuted: added "hadamard2", this went red.\n' '\n' 'Unrelated closing note that is not part of the refutation.\n' ) self.assertNotIn("Unrelated", c.refuted) def test_a_claim_with_no_refutation_is_reported(self): _, _, _, f = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' from pkg.mod import ROTATION_CODES def test_x(): """invariant: every rotation round-trips.""" for r in ROTATION_CODES: assert r ''', }) self.assertIn("unrefuted", [x.kind for x in f]) def test_a_refuted_claim_is_not(self): _, _, _, f = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' from pkg.mod import ROTATION_CODES def test_x(): """invariant: every rotation round-trips. refuted: added a member -> this went red. """ for r in ROTATION_CODES: assert r ''', }) self.assertNotIn("unrefuted", [x.kind for x in f]) class Anchoring(unittest.TestCase): def test_a_live_iteration_records_the_registry(self): claims, _, _, _ = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' from pkg.mod import ROTATION_CODES def test_x(): """invariant: every rotation round-trips. refuted: yes. """ for r in ROTATION_CODES: assert r ''', }) self.assertEqual(claims[0].anchors, ["ROTATION_CODES"]) def test_an_importlib_loaded_module_attribute_still_anchors(self): """The shape this repository's own tests take. `tl.SKIP_DIRS` peels to the module handle `tl`, and following `tl` through the test file's constants lands on `_SPEC`. The attribute name has to win, or every claim in this repo reads as unanchored. """ claims, _, _, _ = inv(**{ "scripts/thing.py": 'SKIP_DIRS = {"a", "b", "c"}\n', "tests/test_thing.py": ''' import importlib.util _SPEC = importlib.util.spec_from_file_location("t", "scripts/thing.py") tl = importlib.util.module_from_spec(_SPEC) def test_x(): """invariant: every skipped name is skipped. refuted: yes. """ for name in sorted(tl.SKIP_DIRS): assert name ''', }) self.assertEqual(claims[0].anchors, ["SKIP_DIRS"]) def test_a_claim_over_a_copied_literal_is_reported(self): claims, _, _, f = inv(**{ "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r", ["haar", "rht"]) def test_x(r): """invariant: every rotation is accepted. refuted: yes. """ assert r ''', }) self.assertEqual(claims[0].copies, ["ROTATION_CODES"]) self.assertIn("literal", [x.kind for x in f]) class Unanchored(unittest.TestCase): def test_a_registry_no_invariant_names_is_reported(self): _, _, _, f = inv(**{ "pkg/mod.py": 'WATCHED = {"a": 1, "b": 2, "c": 3}\n' 'IGNORED = {"x": 1, "y": 2, "z": 3}\n', "pkg/tests/test_it.py": ''' from pkg.mod import WATCHED def test_x(): """invariant: watched holds. refuted: yes. """ for k in WATCHED: assert k ''', }) un = [x.detail for x in f if x.kind == "unanchored"] self.assertTrue(any("IGNORED" in d for d in un), str(un)) self.assertFalse(any("WATCHED" in d for d in un), str(un)) class Robustness(unittest.TestCase): def test_an_unparseable_file_is_skipped_not_fatal(self): claims, _, _, _ = inv(**{ "pkg/broken.py": "def (((:\n", "pkg/mod.py": SRC, "pkg/tests/test_it.py": ''' from pkg.mod import ROTATION_CODES def test_x(): """invariant: it holds. refuted: yes. """ for r in ROTATION_CODES: assert r ''', }) self.assertEqual(len(claims), 1) def test_an_empty_tree_reports_nothing(self): claims, _, stats, f = inv(**{"README.md": "nothing\n"}) self.assertEqual((claims, f), ([], [])) self.assertEqual(stats["claims"], 0) def test_the_report_renders_with_no_claims(self): out = cl.report([], [], {"claims": 0, "refuted": 0, "registries": 0}) self.assertIn("nothing to report", out) def test_strict_exits_nonzero_only_on_a_finding(self): with tempfile.TemporaryDirectory() as td: self.assertEqual(cl.main([td, "--strict"]), 0) def test_selftest_passes(self): self.assertEqual(cl.selftest(), 0) class Ratchets(unittest.TestCase): """A hand-list marked `ratchet` anchors a claim; it is not a copy.""" SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n' BODY = """ from pkg.mod import ROTATION_CODES MARKER def test_x(): '''invariant: a shipped rotation never leaves. refuted: yes. ''' for shipped in MEMBERS: assert shipped in ROTATION_CODES """ def _tree(self, marker, members): body = self.BODY.replace("MARKER", marker).replace("MEMBERS", members) return {"pkg/mod.py": self.SRC, "pkg/tests/test_it.py": body} def test_a_ratchet_anchors_rather_than_copying(self): claims, _, _, f = inv(**self._tree( "# totality: ratchet — these two shipped first", '("haar", "rht")')) self.assertEqual(claims[0].ratchets, ["ROTATION_CODES"]) self.assertEqual(claims[0].copies, []) self.assertNotIn("literal", [x.kind for x in f]) def test_an_unmarked_subset_is_still_a_copy(self): claims, _, _, f = inv(**self._tree("# ordinary comment", '("haar", "rht")')) self.assertEqual(claims[0].copies, ["ROTATION_CODES"]) self.assertIn("literal", [x.kind for x in f]) def test_a_ratcheted_registry_is_not_unanchored(self): _, _, _, f = inv(**self._tree( "# totality: ratchet — all three shipped", '("haar", "rht", "none")')) un = [x.detail for x in f if x.kind == "unanchored"] self.assertFalse(any("ROTATION_CODES" in d for d in un), str(un)) if __name__ == "__main__": unittest.main(verbosity=2) -
test_totality_lint.py 20.3 KB
"""Tests for scripts/totality_lint.py — the copied-domain report. The lint's whole value is precision: a wall of candidates is worse than silence, and its two precision mechanisms both came from a measured false positive rather than from taste. * `no-floor` originally fired on every `for x in <local>`, which produced 27 findings on `remex` — numpy arrays, query matrices, loop counters, all noise. It now fires only on a name independently recognised as a registry. * `sampled-domain` originally matched a literal against any registry in the tree, which on the `experiments` monorepo joined a test in `discrepancy/` to registries in `kb-k-sweep/` and `remex-vs-higgs-ablation/` because small integer sets collide. It now requires reachability: an import, or a shared top-level directory. Both are pinned below. `--selftest` carries the fixture suite; this adds the regressions those two measurements produced, and asserts the classifier directly. python3 tests/test_totality_lint.py """ from __future__ import annotations import importlib.util import sys import tempfile import textwrap import unittest from pathlib import Path _SPEC = importlib.util.spec_from_file_location( "totality_lint", Path(__file__).resolve().parent.parent / "scripts" / "totality_lint.py", ) tl = importlib.util.module_from_spec(_SPEC) # Register before exec: `@dataclass` resolves annotations through # sys.modules[cls.__module__], which is None for a spec-loaded module. sys.modules["totality_lint"] = tl _SPEC.loader.exec_module(tl) def scan_files(**files) -> list: """Write {relpath: source} into a temp tree and scan it.""" with tempfile.TemporaryDirectory() as td: root = Path(td) for rel, src in files.items(): p = root / rel p.parent.mkdir(parents=True, exist_ok=True) p.write_text(textwrap.dedent(src)) findings, _ = tl.scan(root) return findings def kinds(findings, test=None): return sorted( f.kind for f in findings if test is None or f.test == test ) class Registries(unittest.TestCase): """What counts as an enumerated domain in source.""" def regs(self, src): import ast return tl._registries_py(ast.parse(textwrap.dedent(src)), "m.py") def test_dict_contributes_its_keys(self): r = self.regs('CODES = {"a": 0, "b": 1, "c": 2}') self.assertEqual(len(r), 1) self.assertEqual(r[0].members, frozenset({"a", "b", "c"})) self.assertEqual(r[0].kind, "dict") def test_tuple_set_and_list_all_count(self): r = self.regs(""" T = (1, 2, 3) S = {"x", "y", "z"} L = ["p", "q", "r"] """) self.assertEqual({x.kind for x in r}, {"tuple", "set", "list"}) def test_enum_members_are_the_domain(self): r = self.regs(""" from enum import Enum class Colour(Enum): RED = 1 GREEN = 2 BLUE = 3 """) self.assertEqual([x.name for x in r], ["Colour"]) self.assertEqual(r[0].members, frozenset({"RED", "GREEN", "BLUE"})) def test_class_attribute_registry_is_qualified(self): r = self.regs(""" class Q: ROTATIONS = {"haar": 1, "rht": 2, "none": 3} """) self.assertEqual([x.name for x in r], ["Q.ROTATIONS"]) def test_below_min_registry_is_not_a_domain(self): self.assertEqual(self.regs('PAIR = {"a": 1, "b": 2}'), []) def test_a_non_constant_member_disqualifies_the_whole_collection(self): self.assertEqual(self.regs("T = (1, 2, compute())"), []) def test_a_spread_disqualifies_a_dict(self): self.assertEqual(self.regs("D = {**other, 'a': 1, 'b': 2, 'c': 3}"), []) class SampledDomain(unittest.TestCase): SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n' def test_a_strict_subset_is_reported_with_what_is_missing(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r", ["haar", "rht"]) def test_x(r): assert r """, }) self.assertEqual(kinds(f), ["sampled-domain"]) self.assertEqual(f[0].missing, ["none"]) self.assertEqual(f[0].registry, "ROTATION_CODES") def test_the_full_domain_as_a_literal_is_not_a_finding(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r", ["haar", "rht", "none"]) def test_x(r): assert r """, }) self.assertEqual(kinds(f), []) def test_a_superset_is_not_a_finding(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r", ["haar", "rht", "none", "extra"]) def test_x(r): assert r """, }) self.assertEqual(kinds(f), []) def test_a_multi_parameter_table_is_out_of_scope(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r,n", [("haar", 1), ("rht", 2)]) def test_x(r, n): assert r """, }) self.assertEqual(kinds(f), []) def test_a_same_file_const_is_still_a_literal(self): """Binding the copy to a name first does not make it live.""" f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES CASES = ["haar", "rht"] @pytest.mark.parametrize("r", CASES) def test_x(r): assert r """, }) self.assertEqual(kinds(f), ["sampled-domain"]) class Reachability(unittest.TestCase): """Measured on `experiments`: 3 of 4 findings were cross-project noise.""" NUMERIC = "KS = [1, 2, 3, 4]\n" def test_an_unrelated_project_is_not_matched(self): f = scan_files(**{ "kb-k-sweep/srht.py": self.NUMERIC, "discrepancy/tests/test_calibration.py": """ def test_x(): for k in [1, 2, 3]: assert k """, }) self.assertEqual(kinds(f), []) def test_an_import_makes_it_reachable(self): f = scan_files(**{ "kb_k_sweep/srht.py": self.NUMERIC, "discrepancy/tests/test_calibration.py": """ from kb_k_sweep.srht import KS def test_x(): for k in [1, 2, 3]: assert k """, }) self.assertEqual(kinds(f), ["sampled-domain"]) def test_a_shared_top_level_directory_makes_it_reachable(self): f = scan_files(**{ "caps/conditions.py": "DOSE = [0.0, 0.1, 0.5, 1.0]\n", "caps/test_lib.py": """ def test_x(): for d in (0.0, 0.5, 1.0): assert d >= 0 """, }) self.assertEqual(kinds(f), ["sampled-domain"]) self.assertEqual(f[0].missing, [0.1]) class NoFloor(unittest.TestCase): """Measured on `remex`: firing on every local produced 27 noise findings. A live enumeration with no ratchet would also raise `unratcheted`, but the two landed on the same line saying overlapping things (3 of 3 on `claude-workspace`), so `no-floor` — the narrower statement, naming the same fix first — wins the line. See `Ratchet` below for `unratcheted` on its own. """ SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n' def test_the_finding_names_the_registry_not_the_module_handle(self): """`for k in bra.CLASS.items()` is about CLASS, not about `bra`. The resolved root of an importlib-loaded attribute chain is the local module handle, which named `_spec` in this repo's own report before the message was keyed on the matched registry instead. """ f = scan_files(**{ "scripts/thing.py": 'CLASS = {"a": 1, "b": 2, "c": 3}\n', "tests/test_thing.py": """ import importlib.util _spec = importlib.util.spec_from_file_location("t", "x.py") bra = importlib.util.module_from_spec(_spec) def test_x(): for k in bra.CLASS: assert k """, }) self.assertEqual(kinds(f), ["no-floor"]) self.assertIn("`CLASS`", f[0].detail) self.assertNotIn("_spec", f[0].detail) def test_a_live_registry_without_a_floor_is_reported(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r", sorted(ROTATION_CODES)) def test_x(r): assert r """, }) self.assertEqual(kinds(f), ["no-floor"]) def test_a_floor_on_the_local_alias_counts(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES LIVE = sorted(ROTATION_CODES) def test_floor(): assert len(LIVE) >= 3 @pytest.mark.parametrize("r", LIVE) def test_x(r): assert r """, }) self.assertEqual(kinds(f), ["unratcheted"]) def test_an_ordinary_local_is_not_a_domain(self): """`for q in queries` over a numpy array is not an enumeration.""" f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ from pkg.mod import ROTATION_CODES def test_x(): queries = make_queries() for q in queries: assert q """, }) self.assertEqual(kinds(f), []) class Acknowledgement(unittest.TestCase): SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n' def test_a_comment_ack_suppresses_the_finding(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES # totality: partial — mojo has no construction for "none" @pytest.mark.parametrize("r", ["haar", "rht"]) def test_x(r): assert r """, }) self.assertEqual(kinds(f), []) def test_a_docstring_ack_suppresses_the_finding(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": ''' import pytest from pkg.mod import ROTATION_CODES @pytest.mark.parametrize("r", ["haar", "rht"]) def test_x(r): """Compare the two constructions. totality: partial — "none" has no fidelity claim. """ assert r ''', }) self.assertEqual(kinds(f), []) def test_an_ack_that_no_longer_suppresses_anything_is_reported(self): """A suppression must not become a silence.""" f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ import pytest from pkg.mod import ROTATION_CODES # totality: partial — stale, this covers everything now @pytest.mark.parametrize("r", sorted(ROTATION_CODES)) def test_x(r): assert r """, }) self.assertIn("stale-ack", kinds(f)) class Robustness(unittest.TestCase): def test_an_unparseable_file_is_skipped_not_fatal(self): f = scan_files(**{ "pkg/broken.py": "def (((:\n", "pkg/mod.py": 'CODES = {"a": 0, "b": 1, "c": 2}\n', "pkg/tests/test_it.py": """ from pkg.mod import CODES def test_x(): for k in ["a", "b"]: assert k """, }) self.assertEqual(kinds(f), ["sampled-domain"]) def test_vendored_trees_are_not_scanned(self): f = scan_files(**{ "node_modules/pkg/mod.py": 'CODES = {"a": 0, "b": 1, "c": 2}\n', "pkg/tests/test_it.py": """ def test_x(): for k in ["a", "b"]: assert k """, }) self.assertEqual(kinds(f), []) def test_an_empty_tree_reports_nothing_and_does_not_raise(self): self.assertEqual(scan_files(**{"README.md": "nothing here\n"}), []) def test_the_report_renders_with_no_findings(self): out = tl.report([], {"source_files": 0, "registries": 0, "test_files": 0, "domains": 0}) self.assertIn("nothing to report", out) def test_selftest_passes(self): self.assertEqual(tl.selftest(), 0) class SkipDirs(unittest.TestCase): def test_every_skipped_directory_is_actually_skipped(self): """invariant: every name in SKIP_DIRS is excluded from the walk. The domain is SKIP_DIRS itself. A name added to the set with nothing behind it reads as protection the walk does not provide, and the failure is silent: the linter scans a vendored tree and reports its registries as the project's own. The fixture keeps the registry and the test under one top-level directory on purpose. An earlier version put the registry at `<skipdir>/mod.py` and the test at `pkg/tests/`, which passes whether or not the skip works — `_reachable` rejects the pair either way. It was green under perturbation, which is the whole failure class this file exists to catch. refuted: replaced the walk's `part in SKIP_DIRS` check with `part in {"node_modules"}` -> this test went red naming `.coherence`, while the other 25 tests in this file stayed green. """ self.assertGreaterEqual(len(tl.SKIP_DIRS), 8) for name in sorted(tl.SKIP_DIRS): files = { f"pkg/{name}/mod.py": 'CODES = {"a": 0, "b": 1, "c": 2}\n', "pkg/tests/test_it.py": """ def test_x(): for k in ["a", "b"]: assert k """, } self.assertEqual( kinds(scan_files(**files)), [], f"a registry under {name}/ reached the report", ) class Ratchet(unittest.TestCase): """Enumeration and a hand-list are complementary, not a ladder. An enumeration loops whatever the domain currently holds, so it is structurally blind to the domain NARROWING. Measured on `oaustegard/remex`: substituting one member for another, keeping cardinality and keeping a second spelling of the domain in agreement, left the domain floor, the enumeration and the parity check all green while a supported rotation silently left. Raised by Yep, 2026-08-24; reproduced before being believed. """ SRC = 'ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}\n' SHRUNK = 'ROTATION_CODES = {"haar": 0, "rht": 1, "xyz": 2}\n' def _tree(self, src, marker="# totality: ratchet — these three shipped"): return { "pkg/mod.py": src, "pkg/tests/test_it.py": f""" from pkg.mod import ROTATION_CODES {marker} def test_pinned(): for shipped in ("haar", "rht", "none"): assert shipped in ROTATION_CODES """, } def test_an_intact_ratchet_is_silent(self): self.assertEqual(kinds(scan_files(**self._tree(self.SRC))), []) def test_a_member_leaving_breaks_the_ratchet_and_is_named(self): f = scan_files(**self._tree(self.SHRUNK)) self.assertEqual(kinds(f), ["ratchet-broken"]) self.assertEqual(f[0].missing, ["none"]) self.assertEqual(f[0].registry, "ROTATION_CODES") def test_a_full_hand_list_is_not_a_sample_with_or_without_the_marker(self): """Pinning the WHOLE domain is not a sampling oracle in either case.""" self.assertEqual( kinds(scan_files(**self._tree(self.SRC, marker="# ordinary"))), []) def test_a_partial_ratchet_still_reports_what_it_does_not_cover(self): """The marker must not be a laundering channel. A ratchet is a statement about the domain SHRINKING. It says nothing about members it never listed, so a partial ratchet leaves those exactly as uncovered as an unmarked hand-list does. Verified 2026-08-25: before this, `# totality: ratchet` over 2 of 3 members silenced the report entirely — the same escape hatch this tool's notes criticise `via guard` for being in `daniloc/coherence`, reproduced one day later. """ tree = { "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ from pkg.mod import ROTATION_CODES def test_pinned(): for shipped in ("haar", "rht"): assert shipped in ROTATION_CODES """, } self.assertEqual(kinds(scan_files(**tree)), ["sampled-domain"]) tree["pkg/tests/test_it.py"] = tree["pkg/tests/test_it.py"].replace( " def test_pinned", " # totality: ratchet — the two that shipped first\n" " def test_pinned") marked = scan_files(**tree) self.assertEqual(kinds(marked), ["sampled-domain"]) self.assertEqual(marked[0].missing, ["none"]) self.assertIn("covers only shrink", marked[0].detail) def test_a_partial_ack_does_not_earn_ratchet_checking(self): """`partial` excuses a hand-list; only `ratchet` asserts one.""" f = scan_files(**self._tree(self.SHRUNK, marker="# totality: partial — two of three")) self.assertEqual(kinds(f), []) def test_an_enumeration_with_no_ratchet_is_reported(self): f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ from pkg.mod import ROTATION_CODES def test_floor(): assert len(ROTATION_CODES) >= 3 def test_x(): for r in ROTATION_CODES: assert r """, }) self.assertEqual(kinds(f), ["unratcheted"]) self.assertEqual(f[0].registry, "ROTATION_CODES") def test_no_floor_and_unratcheted_do_not_both_claim_one_line(self): """Two findings on one line saying overlapping things is fatigue. Measured 3 of 3 on `claude-workspace` and named by a cross-model review; `no-floor` is the narrower statement and wins. """ f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ from pkg.mod import ROTATION_CODES def test_x(): for r in ROTATION_CODES: assert r """, }) self.assertEqual(kinds(f), ["no-floor"]) def test_an_enumeration_WITH_a_ratchet_is_silent(self): """The pair is the answer; neither half alone is.""" f = scan_files(**{ "pkg/mod.py": self.SRC, "pkg/tests/test_it.py": """ from pkg.mod import ROTATION_CODES def test_floor(): assert len(ROTATION_CODES) >= 3 def test_x(): for r in ROTATION_CODES: assert r # totality: ratchet — these three shipped def test_pinned(): for shipped in ("haar", "rht", "none"): assert shipped in ROTATION_CODES """, }) self.assertEqual(kinds(f), []) if __name__ == "__main__": unittest.main(verbosity=2)
-
-
CHANGELOG.md 6.6 KB
# Changelog ## 0.2.3 — 2026-08-25 Four more limits from the second half of the cross-model pass, each reproduced. The largest undercuts the motivating example: a registry is modelled as a set of KEYS, so permuting `{"haar": 0, "rht": 1, "none": 2}` to `{"haar": 0, "rht": 2, "none": 1}` keeps every key, every count and every ratchet intact while every index already on disk decodes under the wrong rotation. The linter reports nothing; the gate's registry half reports nothing. Also named: a decorator-built registry is invisible, reachability misses a disconnected integration test, and the precision numbers come from three repositories with the filters fitted to two of them. No behaviour change. Documentation only, which is the honest response to holes that need a design decision rather than a patch. ## 0.2.2 — 2026-08-25 `unratcheted` no longer claims a line `no-floor` already has. They landed together on 3 of 3 findings in one repository, saying overlapping things and naming the same fix; `no-floor` is the narrower statement and wins. Three limits added from a cross-model adversarial pass, each reproduced before being written down. A co-ordinated rename defeats the ratchet: renaming a member in the registry and in the hand-list together leaves the pin intact and this reports nothing, because both sides of a co-located hand-list move together. A brand-new registry is neither gated nor pinned. Split, merge and registry rename are invisible, because the diff is keyed on registry name. ## 0.2.1 — 2026-08-25 A ratchet over a strict subset silenced `sampled-domain` entirely and checked nothing about the members it never listed. Writing `# totality: ratchet` on any hand-list therefore retired the finding for free — the same escape hatch these notes criticise `via guard` for being in `daniloc/coherence`, reproduced one day later. A ratchet is a statement about the domain SHRINKING and says nothing about growth, so a partial ratchet now still reports what it does not cover, annotated "pinned as a ratchet, which covers only shrink". A ratchet pinning the whole domain stays silent, as it should. Found by an adversarial pass on the skill, not by its own tests, which is the uncomfortable part: 55 tests and a `--selftest` all passed over it. ## 0.2.0 — 2026-08-24 A second claim form, because an enumeration is structurally blind to its own domain narrowing. It loops whatever the registry currently holds, so removing a member leaves it green over the ones that remain. Raised by Yep; reproduced on `oaustegard/remex` before being believed, and the reproduction is worse than the report. Substituting `"xyz"` for `"none"` in both spellings of the rotation domain — keeping cardinality, keeping the two spellings in agreement — left the domain floor, the enumeration and the parity check all green. Five passing tests over a registry that had quietly stopped supporting a rotation every index on disk was written with. The floor is a cardinality check and cannot see a member swapped for another. `# totality: ratchet — <why>` marks a hand-list as a deliberate floor. Unlike `partial`, which excuses a hand-list, `ratchet` asserts one, and the linter checks the pin rather than taking the marker's word for it. Two new findings: `ratchet-broken` names a pinned member the registry no longer contains, detected statically before any test runs, and `unratcheted` names a registry enumerated live with nothing pinning its membership. `sampled-domain` no longer fires on a marked ratchet — a hand list is sometimes the right answer, on purpose. A ratchet pins every registry it is a floor for, not just the nearest one, since a domain is often spelled twice. `claims.py` reports a ratchet claim as pinning its registries rather than copying them, and a ratcheted registry is no longer `unanchored`. Verified on remex under the same substitution: `ratchet-broken` naming `'none'` statically, and the ratchet test red while the other five pass. 55 tests, up from 45. ## 0.1.0 — 2026-08-24 First release. Two scripts, one idea: a test that enumerates a domain must loop the registry rather than a copy of it. `totality_lint.py` reports `sampled-domain` (a `parametrize` or `for` over a literal whose members are a strict subset of a dict/set/tuple/Enum in the source, naming the members nothing covers), `no-floor` (a live registry iterated with no length assertion, so an emptied registry passes vacuously), and `stale-ack`. `claims.py` reports what the repository declares: a claim is a test whose docstring opens `invariant:`, and `refuted:` records the observed negative control. Findings are `unrefuted`, `literal`, `unanchored`. Adapted from the meta-oracle in `daniloc/coherence` (`src/oracle-domain.ts`), which classifies an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST. Three deliberate differences. The join is on membership rather than on names, because `[1, 2, 3, 4, 8]` and `SUPPORTED_BITS` share no token and containment is what ties them together. Reporting is the default and `--strict` opts into a nonzero exit, because the original's parity arm false-fails a correct oracle that binds its domain to a local name first. And there are no spec files: these take a path. Both precision filters came from a measured false positive rather than from taste. `no-floor` firing on every `for x in <local>` produced 27 findings on `oaustegard/remex`, all noise — numpy arrays, query matrices, loop counters — so it now fires only on a name independently recognised as a registry. Matching a literal against any registry in the tree joined a test in `discrepancy/` to registries in `kb-k-sweep/` and `remex-vs-higgs-ablation/` on a monorepo, because small integer sets collide by chance; requiring reachability took four findings to one, and the survivor was real. Two extractor fixes came from the first real use. Tests that load their subject through `importlib` reach registries as `tl.SKIP_DIRS`; peeling that attribute to its object lands on the module handle, and following the handle through the file's own constants lands on `_SPEC`, which shadowed the name that mattered and made every claim read as unanchored. Candidates now carry the attribute name and win over the resolved root, `_reachable` accepts the `tests/test_<mod>.py` to `<mod>.py` pairing, and the `no-floor` message names the matched registry rather than whatever the chain rooted in. Measured at release: on `remex` at main, exactly the four rotation tests that parametrize `["haar", "rht"]` against a three-member `ROTATION_CODES`, each naming `'none'`. 45 tests, plus a `--selftest` in each script. ## [0.2.3] - 2026-08-25 ### Other - declaring-invariants: a skill for domains a test copies instead of loops (#773) -
README.md 2.8 KB
# declaring-invariants Find tests that enumerate a domain by copying it, and declare the invariants a codebase depends on. Stdlib `ast` only — no install, no config file, no network. Python. ```bash python3 scripts/totality_lint.py <repo> # tests that copy a domain python3 scripts/claims.py <repo> # what the repo declares, and what backs it ``` ## Features - **Copied-domain detection** — a `parametrize` or `for` over a literal whose members are a strict subset of a dict/set/tuple/Enum in the source, with the uncovered members named - **Ratchets, the second claim form** — an enumeration loops whatever the domain now holds, so it cannot see the domain narrowing. `# totality: ratchet — <why>` marks a hand-list as a deliberate floor, and the linter checks the pin: `ratchet-broken` names a member that left, statically, before any test runs. `unratcheted` names a registry with an enumeration and no floor - **Vacuity detection** — a live registry iterated with no `len(...) >= n` assertion, which passes over an emptied collection - **Membership join** — `[1, 2, 3, 4, 8]` and `SUPPORTED_BITS` share no token, so containment is the join key; no naming convention is assumed - **Reachability filter** — a literal matches a registry only when the test imports its module, shares its top-level directory, or is its paired `tests/test_<mod>.py` - **First-class acknowledgement** — `# totality: partial — <why>` retires a finding, and an acknowledgement on a test that later covers the whole domain is reported as `stale-ack` - **Claim inventory** — a claim is a test whose docstring opens `invariant:`, and `refuted:` records the observed negative control - **Report by default** — `--strict` opts into a nonzero exit; `--json` for machine consumption; `--selftest` runs fixtures with no repo ## Why On `oaustegard/remex`, adding a fourth member to `ROTATION_CODES` with no construction behind it left the entire 267-test suite green. Four tests looked total; each parametrized `["haar", "rht"]` against a three-member registry. The other direction is worse. Substituting `"xyz"` for `"none"` in both spellings of that domain — cardinality unchanged, both spellings agreeing — left the domain floor, the enumeration and the parity check green, five passing tests over a registry that had stopped supporting a rotation every index on disk was written with. That is what the ratchet form is for. Adapted from the meta-oracle in [`daniloc/coherence`](https://github.com/daniloc/coherence), which classifies an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST. The check needs none of that harness's spec files, claim grammar, ledger or Node runtime. See [SKILL.md](SKILL.md) for the full reference, including how to write a refutation you have actually observed. -
SKILL.md 14.2 KB
--- name: declaring-invariants description: Find tests that enumerate a domain by copying it, and declare the invariants a codebase depends on. Reports where a parametrize list, for-loop, or it.each iterates a hand-written subset of a dict/set/tuple/Enum that exists in the source, and names the members nothing covers. Use when reviewing tests, when a module gains a name-to-thing table, registry, enum, or dispatch map, before trusting a green suite as evidence a domain is covered, or when asked "is this test actually total", "does anything cover X", "what does this repo guarantee", "which invariants do we declare". Also for vacuous tests that pass over an empty collection, for a domain that has silently NARROWED (an enumeration cannot see that), and for recording the refutation that proves a claim can fail. metadata: version: 0.2.3 --- # declaring-invariants Two scripts over one idea: **a test that enumerates a domain must loop the registry rather than a copy of it.** Someone also has to say which domains matter in the first place. ```bash python3 scripts/totality_lint.py <repo> # tests that copy a domain python3 scripts/claims.py <repo> # what the repo declares, and what backs it ``` Python only, stdlib `ast` only: no install, no config file, no network. ## The failure it catches A test that loops a hand-written list passes its runner and proves nothing about completeness. When the same members also exist as a registry in the source, the list is a copy, and the copy drifts the moment someone adds a member to the registry and not to the test. Nothing goes red. Measured on `oaustegard/remex`, 2026-08-24: adding a fourth member to `ROTATION_CODES` with no construction behind it left the **entire 267-test suite green**. Four separate tests looked total; each parametrized `["haar", "rht"]` against a three-member registry. Only a test that looped the registry itself caught it. Adapted from the meta-oracle in [`daniloc/coherence`](https://github.com/daniloc/coherence) (`src/oracle-domain.ts`), which classifies an oracle's iteration root as LIVE or LITERAL by parsing the oracle's own AST. That harness needs spec files, a claim grammar, a ledger and Node; the check does not. ## `totality_lint.py` — tests that copy a domain | finding | meaning | |---|---| | `sampled-domain` | a `parametrize` or `for` over a literal whose members are a strict subset of a source registry. The uncovered members are named. | | `ratchet-broken` | a hand-list marked `ratchet` names a member the registry no longer contains. Detected statically, without running anything. | | `unratcheted` | a registry enumerated live with nothing pinning its membership. Suppressed when `no-floor` already claimed the same line. | | `no-floor` | a test iterates a live registry with no `len(...) >= n` assertion in the file, so an emptied registry passes vacuously. | | `stale-ack` | an acknowledgement on a test that now covers the whole domain. | ```bash python3 scripts/totality_lint.py <repo> # the report python3 scripts/totality_lint.py <repo> --json python3 scripts/totality_lint.py <repo> --strict # exit 1 if any finding python3 scripts/totality_lint.py --selftest # fixtures, no repo ``` A partial domain is often correct. Say so on the test and it stops being a finding: ```python # totality: partial — mojo has no construction for "none" @pytest.mark.parametrize("rotation", ["haar", "rht"]) def test_save_params_accepts_every_mojo_rotation(rotation): ... ``` The marker also works as `totality: partial — <why>` inside the docstring. An acknowledgement on a test that later covers the whole domain is reported as `stale-ack`, so a suppression cannot become a silence. ## Why a hand-list is the second form An enumeration loops **whatever the domain currently holds**, so it is structurally blind to the domain narrowing. Remove a member and the loop simply ranges over fewer of them, green. Raised by Yep, 2026-08-24; reproduced on `oaustegard/remex` before being believed: | perturbation of `ROTATION_CODES` | domain floor | enumeration | parity | hand-list | |---|---|---|---|---| | grow — add `"hadamard2"` | pass | **RED** | **RED** | pass | | shrink — drop `"none"` | **RED** | pass | RED\* | pass | | substitute — `"none"` → `"xyz"` in both spellings | pass | pass | pass | pass | \* only because the *other* spelling of the domain did not shrink, which is incidental to that repository. In the substitution row cardinality holds, both spellings agree, and five green tests cover a registry that quietly stopped supporting a rotation every index on disk was written with. The floor is a **cardinality** check; it cannot see a member swapped for another. The second form is a **ratchet**: a hand-list asserting the domain keeps containing it. Mark it and the linter checks the pin rather than taking the marker's word for it. ```python # totality: ratchet — these three shipped; one leaving is a compatibility break def test_no_shipped_rotation_is_ever_removed(): for shipped in ("haar", "rht", "none"): assert shipped in ROTATION_CODES ``` Under that same substitution the linter reports `ratchet-broken` naming `'none'` **statically, before any test runs**, and the test itself goes red while the other five pass. So: an enumeration proves every current member is handled, and a ratchet proves no member left without a decision. `unratcheted` names a registry that has the first and not the second. Neither half alone is the answer. **A ratchet covers shrink only.** Over a strict subset it still reports `sampled-domain` for the members it never listed, because pinning two of three says nothing about the third. Suppressing that would make the marker a laundering channel — the escape hatch these notes criticise `via guard` for being in coherence. It was one: until 2026-08-25, `# totality: ratchet` over two of three members silenced the report entirely and checked nothing about the third. Caught by an adversarial pass on this skill, not by its own tests. ## `claims.py` — what the repo declares `totality_lint` asks whether a test's domain is complete. It presumes a test exists. This asks the prior question, the one coherence's own Known Limits concedes it does not answer: nothing enforces `exists ⇒ declared`. A claim is a test whose docstring **opens** with `invariant:`. No new file format, and the claim inherits its test's pass/fail: ```python def test_every_skipped_directory_is_actually_skipped(self): """invariant: every name in SKIP_DIRS is excluded from the walk. refuted: replaced the walk's `part in SKIP_DIRS` check with `part in {"node_modules"}` -> this test went red naming `.coherence`, while the other 25 tests in this file stayed green. """ ``` | finding | meaning | |---|---| | `unrefuted` | a claim nobody has watched fail | | `literal` | the claiming test iterates a copy of the registry, so the claim cannot see a new member | | `unanchored` | a registry no invariant names — a question, not a verdict | A claim whose test carries a `ratchet` marker is reported as *pinning* its registries rather than copying them, and a ratcheted registry is not `unanchored`. ```bash python3 scripts/claims.py <repo> [--json] [--strict] [--selftest] ``` ## Write the refutation from what you observed `refuted:` is the half that costs something. Break the chokepoint, watch the claim go red **by name**, restore, and record what you saw. A green test and an unfalsifiable one look identical from outside; the refutation is what separates them. **Never write a refutation you have not run.** The first refutation authored for the `SKIP_DIRS` invariant above asserted a failure that did not occur — the fixture placed the registry outside the test's reachability, so the test passed under perturbation for an unrelated reason. A vacuous claim, written while building the tool that catches vacuous claims, and caught only by running the perturbation instead of trusting the sentence. Procedure, in order: 1. Break the chokepoint the claim names, with one edit in the source rather than in the test. 2. Run the claiming test. Read the failure text. Note the member it named. 3. Run the rest of the file. Confirm the others stay green — if everything goes red, the claim is not localised and the refutation says nothing. 4. Restore the source. 5. Write `refuted: <the edit> -> <what went red, by name>`. ## Wiring it into a commit gate The reference wiring lives in `oaustegard/claude-workspace` (`scripts/tdd_hook.py`): a commit where a registry gained a member and no `invariant:` test iterates it live is denied, naming the registry and what it gained. Override with `no-invariant: <why>` in the commit body. Registry **shrink** is gated the same way, and needs a ratchet rather than an enumeration to clear it. A new function or a new branch is behavioural growth too, but neither is diffable without guessing, and a gate that guesses stops being consulted. A brand-new registry is not gated. Declaring one is a judgement call; growing one has already made it. ## Where the two filters came from Both filters exist because the unfiltered version was noise. Reproduce either by removing the filter and re-running against a real repo. - **`no-floor` fires only on a name independently recognised as a registry.** Firing on every `for x in <local>` produced 27 findings on `remex`, all noise: numpy arrays, query matrices, loop counters. - **A literal matches a registry only when reachable** — the test imports its module or package, shares its top-level directory, or is its paired `tests/test_<mod>.py`. Matching against any registry in the tree joined a test in `discrepancy/` to registries in `kb-k-sweep/` and `remex-vs-higgs-ablation/` on a monorepo, because small integer sets collide by chance. Four findings became one, and the survivor was real. The join key is **membership, not names**: `[1, 2, 3, 4, 8]` and `SUPPORTED_BITS` share no token, so containment is what ties them together. No naming convention is assumed, and none is required. ## Limits - **Python only.** The extractors are `_registries_py` and `_domains_py`; a tree-sitter pair for another language slots in beside them. The join, the acknowledgements and the reports are all language-independent. - **Report, not gate, by default.** `--strict` opts into a nonzero exit. The tool this was adapted from gates by default, and its parity arm false-fails a correct oracle that binds its domain to a local name first. A gate that false-fails stops being consulted. - **A registry is a collection of constants.** A dict/set/tuple/list of literals, or an Enum body, with at least three members. A domain assembled at runtime is invisible here. - **Multi-parameter tables are out of scope.** Only single-name `parametrize` is read. - **`unanchored` is a question.** Most registries need no invariant. Treat the list as candidates for declaration, never as a backlog to clear. - **A registry is modelled as a SET OF KEYS, so a value swap is invisible.** This is the largest hole in the design, and it undercuts the motivating example. `ROTATION_CODES = {"haar": 0, "rht": 1, "none": 2}` exists to pin bytes on disk; permuting it to `{"haar": 0, "rht": 2, "none": 1}` keeps every key, every count and every ratchet intact, and every index already written decodes under the wrong rotation. Verified 2026-08-25 on a fixture: the linter reports nothing and the gate's registry half reports nothing. (The gate denied that fixture, but on the unrelated TDD rule — checked, because claiming otherwise would have been the overclaim this skill exists to catch.) Nothing here checks a name-to-code mapping. Extracting `(key, value)` pairs as the member set would, at the cost of breaking the subset join against a `parametrize` list of keys. - **A co-ordinated rename defeats the ratchet.** A global find/replace that renames a member in the registry AND in the hand-list leaves the pin intact, and this reports nothing. Verified 2026-08-25 on a fixture: renaming `"none"` to `"identity"` in both files was silent, while every index already on disk still decodes byte 2 as the old name. Both sides of a co-located hand-list move together, so no static check over one working tree can see it. The commit gate can, because it diffs against git history — and only if the ratchet itself is untouched in that commit, which it now requires. - **A brand-new registry is neither gated nor pinned.** Create one with five members and never grow it and no gate ever fires. `unanchored` and `unratcheted` surface it in the report; the gate deliberately does not, on the grounds that gating every new module is how a gate stops being consulted. That is a judgement, not a proof, and it is the largest hole a cross-model review found. - **A decorator-built registry is invisible.** `@register("name")` populating a dict at import time is the common Python registry idiom and is not a literal, so nothing here sees it. Named because it is the shape most likely to be mistaken for coverage. - **Reachability is path-based, so a disconnected integration test is missed.** A test that neither imports the module, shares its top-level directory, nor pairs with it by filename will not be joined to the registry it samples. The filter trades that recall for the cross-project precision it was measured to buy. - **The precision numbers come from three repositories, and the filters were fitted to two of them.** 27 noise findings on one, 3-of-4 cross-project joins on another. Those are the measurements that justified each filter; they are not a false-positive rate on a corpus, and should not be read as one. - **Split and merge are invisible.** The diff is keyed on registry NAME, so renaming a registry, splitting one in two, or merging two into one falls through both the gained and lost paths. - **A claim that passes is not a claim that is right.** This checks that a declared invariant loops the domain it names. Whether it is the *right* invariant is human judgement, and it is not automatable. ## Related - `verifying-claims` covers the prose layer: does the documentation match reality? Agent-judged, non-deterministic, run as a triggered review. This skill is the deterministic half, over code and tests rather than prose. - `tree-sitting` locates the registry or the test before you edit it.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.