red-first-review
Review someone else's pull request by measuring which of its own guards its own test suite actually catches, via a small mutation matrix. Use when reviewing a PR, checking whether a regression test is real, or asked 'is this change actually tested?', 'does this test do anything',
Install
npx skills add https://github.com/tonydzi/second-brain-starter-kit/tree/master/skills/red-first-review
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tonydzi-second-brain-starter-kit@llmmart
git clone https://github.com/tonydzi/second-brain-starter-kit.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tonydzi/second-brain-starter-kit collection as a plugin from our marketplace. Git is the plain clone.
README
red-first-review
A test that has never been shown failing is a claim, not a check.
This is a Claude Code skill plus a small standalone tool for reviewing someone else's pull request by measuring the one thing you cannot get by reading it: which of the guards this PR adds would survive being deleted?
You neutralise each guard in turn, run the project's own test suite, and record what nothing notices. Survivors are the finding. Everything else in the review — the suggested test, the argument, the ranking — hangs off that table.
Standard library only. Python 3.8+. Nothing to install.
python3 mutmatrix.py \
--repo . \
--test "pytest tests/unit -q -p no:randomly" \
--file src/pkg/api.py \
--call-identity redact_secrets
What that produces, from the hand-rolled run this repository was built out of (google/adk-python#6957, a PR redacting OAuth secrets at eight call sites):
| mutant | verdict | killed by |
|---|---|---|
/run |
SURVIVED | nothing |
/run_sse |
killed | test_agent_run_sse_redacts_oauth2_client_secret |
/run_live (websocket) |
SURVIVED | nothing |
_redacted_session_response |
killed | test_get_session_redacts_oauth2_client_secret |
_redacted_sessions_response (list) |
SURVIVED | nothing |
dev get_eval_result_legacy |
SURVIVED | nothing |
dev get_eval |
killed | test_get_eval_redacts_oauth2_client_secret |
dev get_eval_result |
SURVIVED | nothing |
The PR's title named three endpoints. One of them was actually guarded by a test.
Reverting the /run line alone left the entire suite green.
Why it refuses to guess
Most of this tool is not the mutation loop — that part is twenty lines. Most of it is the checks that stop it printing a table it cannot stand behind.
| gate | what it catches | exit |
|---|---|---|
| baseline | suite already red → nothing is attributable to the PR | 3 |
| canary | test command never imports the file → every "killed" is another test's | 3 |
| compile check | mutant does not parse → every test fails, which looks exactly like perfect coverage | 3 |
| no-canary | you skipped the canary → the result is UNPROVEN, never clean |
3 |
Exit 0 means no survivors. Exit 1 means survivors — that is your review.
Exit 3 means the run measured nothing and you should not quote it.
The compile check exists because of a measurement made building this: a
delimiter collision produced two unparseable mutants on
anthropic-sdk-python#1906,
and both were scored killed. The review would have certified coverage nobody
had. Now:
| `sync delete-root guard` | INVALID | mutant does not compile -- not a coverage result |
0 killed, 0 SURVIVED, 1 invalid.
A real run, end to end
anthropics/anthropic-sdk-python#1906 — "fix(memory): compare resolved paths in
the delete root guard". The memory tool guarded its root directory with
if command.path == "/memories", comparing the raw string the model sent, so
/memories/, /memories/. and /memories/subdir/.. all reached shutil.rmtree.
Step 1, baseline.
$ .venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q
79 passed in 5.66s
Step 3, neutralise both guards (sync and async), one at a time:
python3 mutmatrix.py --repo . \
--test ".venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q -p no:randomly" \
--canary-file src/anthropic/lib/tools/_beta_builtin_memory_tool.py --sep '@@' \
--mutant 'sync delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if full_path == self.memory_root.resolve():@@if False:' \
--mutant 'async delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if Path(str(full_path)) == Path(str(self.memory_root)).resolve():@@if False:'
BASELINE: green (7s) -- 79 passed in 5.29s
CANARY : red (1s) -- the test command does reach _beta_builtin_memory_tool.py
[1/2] sync delete-root guard killed (7s)
[2/2] async delete-root guard killed (7s)
2 killed, 0 SURVIVED, 0 invalid.
Both guards are real. Six named tests die when the sync guard goes:
test_delete_not_allow_deleting_memories_directory
test_delete_not_allow_deleting_memories_directory_via_symlink
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories//]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/.]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/subdir/..]
A clean table is a result, not a wasted run. "I removed your guard and your tests caught it, here are the six by name" is worth more to a maintainer than another opinion about style.
Step 0 is where the finding came from. Reading the whole file rather than the
diff turned up a sibling the PR's own reasoning applies to and the diff never
touches — rename, whose old_path can also be the root, and which has no guard
at all:
rename("/memories", "/memories/backup")
-> OSError(22, 'Invalid argument') # not a ToolError; logged at ERROR with a traceback
Every other refusal in that class is a clean ToolError the model can act on.
That became the review's one actionable finding, and it exists because of step 0,
not because of the mutation table.
Where the method comes from
The table at the top of this README is that run. Five of eight redaction sites survived, including two of the three endpoints the PR's own title claimed to fix. The author's reply:
Thanks for this - genuinely one of the most useful reviews I've gotten on this PR. All three findings confirmed and fixed, verified the same way you found them (neutering each call site and checking the suite).
That review was hand-rolled. This repository is that procedure made repeatable.
Install
As a Claude Code skill:
git clone https://github.com/tonydzi/red-first-review-skill \
~/.claude/skills/red-first-review
Then ask Claude to review a PR; the skill triggers on "review this PR", "is this change actually tested?", "mutation test this".
As a plain tool, mutmatrix.py is one file with no dependencies. Copy it in.
Targeting
| flag | use |
|---|---|
--call-identity NAME --file PATH |
one mutant per call site of NAME; rewrites NAME(x) → x |
--mutant 'LABEL::FILE::FIND::REPLACE' |
an exact textual edit; FIND must be unique in the file |
--line 'LABEL::FILE::N' |
comment out line N |
--sep '@@' |
change the separator when your code contains :: (Python usually does) |
--canary-file PATH |
the file the test command must import (default: first mutated file) |
--no-canary |
skip the reach check; the run is then reported UNPROVEN, never clean |
Gotchas paid for in advance
- Bytecode caching will lie to you. CPython validates a cached
.pycby source size and mtime in whole seconds. Neutralising the same guard at two call sites usually yields two files of identical length — so two mutants inside one second run the first one's bytecode and both scoreSURVIVED. Measured on the self-test before the fix:1 killedbecame0 killed.mutmatrixgives every write its own mtime second. - Random test ordering makes runs incomparable. Pass
-p no:randomly. gh pr commentis not a review. It creates no review object, so the work lands in no contribution graph and noreviewed-by:filter — while still looking like noise to spam heuristics. Usegh pr review --comment.- Do not mutate a repository you have uncommitted work in. The tool restores
from an in-memory copy and a backup directory, and refuses to start if a
previous run left one behind, but
git statusbefore you start is free.
Does the tool's own test suite pass this bar?
It has to, or the whole thing is a joke. _test_mutmatrix.py builds a throwaway
project whose right answer is known by construction (three call sites, one
covered), and it ships with a kill-list — the mutations of itself it claims to
catch:
$ python3 _test_mutmatrix.py
C1 PASS C2 PASS C3 PASS C4 PASS C5 PASS C7 PASS C6 PASS
all cases behaved as expected.
$ python3 _test_mutmatrix.py --red M3 # killed/survived inverted
C3 FAIL C4 FAIL C5 FAIL # exactly the cases it declared
| mutation | what it breaks | must fail |
|---|---|---|
| M1 | baseline check disabled | C1 |
| M2 | canary verdict hardcoded green | C2 |
| M3 | killed/survived inverted | C3, C4, C5 |
| M4 | compile check on the mutant removed | C7 |
C6 is the control: green from the start and under every mutation above. Without it, a mutation that merely broke the tool would look like a working red mode.
Licence
MIT. Built for one person's fleet — adapt the paths and the test commands to yours.
Skill manifest
Red-first review
A test that has never been shown failing is a claim, not a check. This skill turns that claim into a measurement: break the thing the test is supposed to protect, and see whether anything goes red.
Applied to someone else's pull request it answers one question a human reviewer cannot answer by reading: which of the guards this PR adds would survive being deleted? Those are the ones the author will regress next month without noticing.
When to use it
- reviewing a PR that adds a guard, a validation, a sanitiser, a redaction, a permission check — anything that says "no" on some paths
- someone claims a fix is covered by tests, and you want that claim measured
- you wrote a regression test yourself and want to prove it fails without the fix
- auditing a test file that has never failed in CI
Not for: pure refactors with no behavioural claim, docs, dependency bumps.
The procedure
0. Read the whole files, not the diff
The diff shows what changed. The bug is usually in what did not. Open every
touched source file end to end, and specifically look for sibling call sites:
if the PR guards delete, does rename reach the same object? If it guards one
endpoint, how many endpoints are there?
Write down the full list of places the PR's own reasoning applies to. That list, not the diff, is what you are about to measure.
1. Get the suite green before you touch anything
Check out the PR branch, install, run the tests it claims to affect.
A red baseline makes everything after it unattributable. If the suite is already failing, narrow the command until it is green, and check the same failures on the base branch by name — pre-existing failures reported as PR damage is the fastest way to lose a maintainer's trust.
2. Prove your test command reaches the code
Break the file under review on purpose (append raise RuntimeError("canary"))
and re-run. If the suite stays green, that command never imports the file, and
every "covered" verdict you are about to produce is an artefact of some other
test. mutmatrix.py does this automatically and refuses to continue.
3. Neutralise each guard in turn, one at a time
python3 mutmatrix.py \
--repo . \
--test "pytest tests/unit -q -p no:randomly" \
--file src/pkg/api.py \
--call-identity redact_secrets
--call-identity NAME makes one mutant per call site of NAME, rewriting
NAME(x) to x. For guards that are branches rather than calls, spell the edit
out:
python3 mutmatrix.py --repo . --test "pytest tests/unit -q" --sep '@@' \
--mutant 'root guard@@src/pkg/store.py@@if full_path == self.root.resolve():@@if False:'
Read the exit code, not just the table: 0 no survivors · 1 survivors found ·
3 undecidable (baseline, canary, or a mutant that did not compile) · 4 the
tool broke. Only 0 and 1 are results.
Disable random test ordering (-p no:randomly) so runs are comparable.
4. Write the finding, one per survivor
A survivor is not "missing coverage" in the abstract. It is a specific sentence: reverting this one line leaves the whole suite green. Say which line, and say what the mutant table showed.
Then supply the test. Not a description of a test — the code, using the project's existing fixtures and naming, so it can be pasted in.
5. Prove the new test is red for the right reason
Re-run the same command with your test added. The proof has three parts and you report all three:
- the row for that mutant flips from
SURVIVEDtokilled - every other row stays exactly as it was — a test that kills every mutant is not specific, it is just noisy
- the test passes on the PR as it stands
Without part 2 you have written a test that fails whenever anything changes.
6. Say what you did not check
Name the paths you never exercised — the UI, a live provider, an endpoint you skipped. A review that implies total coverage is a review the maintainer has to re-verify from scratch. Publish the hypotheses that came out clean too: it tells them which doors are already closed.
7. Post it as a review object, not a comment
gh pr review <N> -R <owner/repo> --comment --body-file review.md
gh pr comment writes to the issue thread. It does not create a review object,
so it appears in no contribution graph and no reviewed-by: filter, while still
looking exactly like noise to spam heuristics. Reviews get gh pr review.
Do not use --approve or --request-changes on a repo you do not maintain.
If an AI agent produced the review, say so in the first line, and say whether a human read it before it posted.
Judgement rules
- A syntactically broken mutant is not a covered site. Every test fails on a
SyntaxError, which looks identical to perfect coverage.mutmatrixcompiles each mutant first and reportsINVALID; if you mutate by hand, check it parses. - Never mutate a file you have not backed up, and never leave a mutant behind.
mutmatrixkeeps a backup directory and refuses to start if a previous run left one. - Bytecode caching will lie to you. CPython validates a
.pycby source size and mtime in whole seconds; two mutants of the same length in the same second run identical bytecode. Measured here: a killed site scored as SURVIVED.mutmatrixbumps mtime on every write. - Rank the findings and say which one matters. Three findings with no order is a list. "Finding 1 is the one worth acting on" is a review.
- Do not report a known issue as a discovery. Search the repo's open PRs and issues for the adjacent problem first, and cite it if it exists.
What ships with this skill
mutmatrix.py— the engine. Standard library only, Python 3.8+, no install._test_mutmatrix.py— its self-test, including--red M1..M4, which runs the suite against deliberately broken copies of the engine to prove the tests themselves can fail. A test suite that has never been shown red does not get to certify anyone else's.
Like this skill? It is part of second-brain-starter-kit: the second brain we built for ourselves and run every day at Palo Alto AI Research Lab. Install the whole set with npx skills add tonydzi/second-brain-starter-kit. Everything is open source and free, so take what you need.
Flagships worth a look on their own: secondop-panel (a second opinion from a panel of external models), claude-memory-tidy (stop your agent's memory from rotting), telegram-mcp-kit (your own Telegram over MCP in about 15 minutes).
Author: Anton Dziatkovskii, Palo Alto AI Research Lab. Telegram @tonydzi - WhatsApp +1 341 222 9178 - X @Tony_Stef_
Engineers: want to test-drive this setup? Message me. I hand out free starter seeds to engineers who test and report back, and custom skill requests are welcome.
Files (second-brain-starter-kit)
-
LICENSE 1 KB · in bundle
-
mutmatrix.py 21 KB
Loading… -
README.md 8.9 KB
# red-first-review **A test that has never been shown failing is a claim, not a check.** This is a Claude Code skill plus a small standalone tool for reviewing someone else's pull request by measuring the one thing you cannot get by reading it: *which of the guards this PR adds would survive being deleted?* You neutralise each guard in turn, run the project's own test suite, and record what nothing notices. Survivors are the finding. Everything else in the review — the suggested test, the argument, the ranking — hangs off that table. Standard library only. Python 3.8+. Nothing to install. ```bash python3 mutmatrix.py \ --repo . \ --test "pytest tests/unit -q -p no:randomly" \ --file src/pkg/api.py \ --call-identity redact_secrets ``` What that produces, from the hand-rolled run this repository was built out of ([google/adk-python#6957](https://github.com/google/adk-python/pull/6957), a PR redacting OAuth secrets at eight call sites): | mutant | verdict | killed by | | --- | --- | --- | | `/run` | **SURVIVED** | nothing | | `/run_sse` | killed | `test_agent_run_sse_redacts_oauth2_client_secret` | | `/run_live` (websocket) | **SURVIVED** | nothing | | `_redacted_session_response` | killed | `test_get_session_redacts_oauth2_client_secret` | | `_redacted_sessions_response` (list) | **SURVIVED** | nothing | | dev `get_eval_result_legacy` | **SURVIVED** | nothing | | dev `get_eval` | killed | `test_get_eval_redacts_oauth2_client_secret` | | dev `get_eval_result` | **SURVIVED** | nothing | The PR's title named three endpoints. One of them was actually guarded by a test. Reverting the `/run` line alone left the entire suite green. --- ## Why it refuses to guess Most of this tool is not the mutation loop — that part is twenty lines. Most of it is the checks that stop it printing a table it cannot stand behind. | gate | what it catches | exit | | --- | --- | --- | | **baseline** | suite already red → nothing is attributable to the PR | 3 | | **canary** | test command never imports the file → every "killed" is another test's | 3 | | **compile check** | mutant does not parse → every test fails, which looks exactly like perfect coverage | 3 | | **no-canary** | you skipped the canary → the result is `UNPROVEN`, never `clean` | 3 | Exit `0` means no survivors. Exit `1` means survivors — that is your review. Exit `3` means the run measured nothing and you should not quote it. The compile check exists because of a measurement made building this: a delimiter collision produced two unparseable mutants on [anthropic-sdk-python#1906](https://github.com/anthropics/anthropic-sdk-python/pull/1906), and both were scored `killed`. The review would have certified coverage nobody had. Now: ``` | `sync delete-root guard` | INVALID | mutant does not compile -- not a coverage result | 0 killed, 0 SURVIVED, 1 invalid. ``` --- ## A real run, end to end `anthropics/anthropic-sdk-python#1906` — *"fix(memory): compare resolved paths in the delete root guard"*. The memory tool guarded its root directory with `if command.path == "/memories"`, comparing the raw string the model sent, so `/memories/`, `/memories/.` and `/memories/subdir/..` all reached `shutil.rmtree`. **Step 1, baseline.** ``` $ .venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q 79 passed in 5.66s ``` **Step 3, neutralise both guards** (sync and async), one at a time: ```bash python3 mutmatrix.py --repo . \ --test ".venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q -p no:randomly" \ --canary-file src/anthropic/lib/tools/_beta_builtin_memory_tool.py --sep '@@' \ --mutant 'sync delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if full_path == self.memory_root.resolve():@@if False:' \ --mutant 'async delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if Path(str(full_path)) == Path(str(self.memory_root)).resolve():@@if False:' ``` ``` BASELINE: green (7s) -- 79 passed in 5.29s CANARY : red (1s) -- the test command does reach _beta_builtin_memory_tool.py [1/2] sync delete-root guard killed (7s) [2/2] async delete-root guard killed (7s) 2 killed, 0 SURVIVED, 0 invalid. ``` Both guards are real. Six named tests die when the sync guard goes: ``` test_delete_not_allow_deleting_memories_directory test_delete_not_allow_deleting_memories_directory_via_symlink test_delete_not_allow_deleting_memories_directory_via_alias[/memories/] test_delete_not_allow_deleting_memories_directory_via_alias[/memories//] test_delete_not_allow_deleting_memories_directory_via_alias[/memories/.] test_delete_not_allow_deleting_memories_directory_via_alias[/memories/subdir/..] ``` **A clean table is a result, not a wasted run.** "I removed your guard and your tests caught it, here are the six by name" is worth more to a maintainer than another opinion about style. **Step 0 is where the finding came from.** Reading the whole file rather than the diff turned up a sibling the PR's own reasoning applies to and the diff never touches — `rename`, whose `old_path` can also be the root, and which has no guard at all: ``` rename("/memories", "/memories/backup") -> OSError(22, 'Invalid argument') # not a ToolError; logged at ERROR with a traceback ``` Every other refusal in that class is a clean `ToolError` the model can act on. That became the review's one actionable finding, and it exists because of step 0, not because of the mutation table. ## Where the method comes from The table at the top of this README is that run. Five of eight redaction sites survived, including two of the three endpoints the PR's own title claimed to fix. The author's reply: > Thanks for this - genuinely one of the most useful reviews I've gotten on this > PR. All three findings confirmed and fixed, verified the same way you found > them (neutering each call site and checking the suite). — [prasanna8585, 1 Sep 2026](https://github.com/google/adk-python/pull/6957#issuecomment-5489421299) That review was hand-rolled. This repository is that procedure made repeatable. --- ## Install As a Claude Code skill: ```bash git clone https://github.com/tonydzi/red-first-review-skill \ ~/.claude/skills/red-first-review ``` Then ask Claude to review a PR; the skill triggers on "review this PR", "is this change actually tested?", "mutation test this". As a plain tool, `mutmatrix.py` is one file with no dependencies. Copy it in. ## Targeting | flag | use | | --- | --- | | `--call-identity NAME --file PATH` | one mutant per call site of `NAME`; rewrites `NAME(x)` → `x` | | `--mutant 'LABEL::FILE::FIND::REPLACE'` | an exact textual edit; `FIND` must be unique in the file | | `--line 'LABEL::FILE::N'` | comment out line N | | `--sep '@@'` | change the separator when your code contains `::` (Python usually does) | | `--canary-file PATH` | the file the test command must import (default: first mutated file) | | `--no-canary` | skip the reach check; the run is then reported `UNPROVEN`, never clean | ## Gotchas paid for in advance - **Bytecode caching will lie to you.** CPython validates a cached `.pyc` by source size and mtime *in whole seconds*. Neutralising the same guard at two call sites usually yields two files of identical length — so two mutants inside one second run the *first* one's bytecode and both score `SURVIVED`. Measured on the self-test before the fix: `1 killed` became `0 killed`. `mutmatrix` gives every write its own mtime second. - **Random test ordering** makes runs incomparable. Pass `-p no:randomly`. - **`gh pr comment` is not a review.** It creates no review object, so the work lands in no contribution graph and no `reviewed-by:` filter — while still looking like noise to spam heuristics. Use `gh pr review --comment`. - **Do not mutate a repository you have uncommitted work in.** The tool restores from an in-memory copy and a backup directory, and refuses to start if a previous run left one behind, but `git status` before you start is free. ## Does the tool's own test suite pass this bar? It has to, or the whole thing is a joke. `_test_mutmatrix.py` builds a throwaway project whose right answer is known by construction (three call sites, one covered), and it ships with a kill-list — the mutations of *itself* it claims to catch: ```bash $ python3 _test_mutmatrix.py C1 PASS C2 PASS C3 PASS C4 PASS C5 PASS C7 PASS C6 PASS all cases behaved as expected. $ python3 _test_mutmatrix.py --red M3 # killed/survived inverted C3 FAIL C4 FAIL C5 FAIL # exactly the cases it declared ``` | mutation | what it breaks | must fail | | --- | --- | --- | | M1 | baseline check disabled | C1 | | M2 | canary verdict hardcoded green | C2 | | M3 | killed/survived inverted | C3, C4, C5 | | M4 | compile check on the mutant removed | C7 | C6 is the control: green from the start and under every mutation above. Without it, a mutation that merely broke the tool would look like a working red mode. ## Licence MIT. Built for one person's fleet — adapt the paths and the test commands to yours. -
SKILL.md 7.4 KB
--- name: red-first-review description: "Review someone else's pull request by measuring which of its own guards its own test suite actually catches, via a small mutation matrix. Use when reviewing a PR, checking whether a regression test is real, or asked 'is this change actually tested?', 'does this test do anything', 'mutation test this'." license: MIT version: 1.0.0 --- # Red-first review A test that has never been shown failing is a claim, not a check. This skill turns that claim into a measurement: break the thing the test is supposed to protect, and see whether anything goes red. Applied to someone else's pull request it answers one question a human reviewer cannot answer by reading: **which of the guards this PR adds would survive being deleted?** Those are the ones the author will regress next month without noticing. ## When to use it - reviewing a PR that adds a guard, a validation, a sanitiser, a redaction, a permission check — anything that says "no" on some paths - someone claims a fix is covered by tests, and you want that claim measured - you wrote a regression test yourself and want to prove it fails without the fix - auditing a test file that has never failed in CI Not for: pure refactors with no behavioural claim, docs, dependency bumps. ## The procedure ### 0. Read the whole files, not the diff The diff shows what changed. The bug is usually in what did not. Open every touched source file end to end, and specifically look for **sibling call sites**: if the PR guards `delete`, does `rename` reach the same object? If it guards one endpoint, how many endpoints are there? Write down the full list of places the PR's own reasoning applies to. That list, not the diff, is what you are about to measure. ### 1. Get the suite green before you touch anything Check out the PR branch, install, run the tests it claims to affect. A red baseline makes everything after it unattributable. If the suite is already failing, narrow the command until it is green, **and check the same failures on the base branch by name** — pre-existing failures reported as PR damage is the fastest way to lose a maintainer's trust. ### 2. Prove your test command reaches the code Break the file under review on purpose (append `raise RuntimeError("canary")`) and re-run. If the suite stays green, that command never imports the file, and every "covered" verdict you are about to produce is an artefact of some other test. `mutmatrix.py` does this automatically and refuses to continue. ### 3. Neutralise each guard in turn, one at a time ```bash python3 mutmatrix.py \ --repo . \ --test "pytest tests/unit -q -p no:randomly" \ --file src/pkg/api.py \ --call-identity redact_secrets ``` `--call-identity NAME` makes one mutant per call site of `NAME`, rewriting `NAME(x)` to `x`. For guards that are branches rather than calls, spell the edit out: ```bash python3 mutmatrix.py --repo . --test "pytest tests/unit -q" --sep '@@' \ --mutant 'root guard@@src/pkg/store.py@@if full_path == self.root.resolve():@@if False:' ``` Read the exit code, not just the table: `0` no survivors · `1` survivors found · `3` undecidable (baseline, canary, or a mutant that did not compile) · `4` the tool broke. **Only `0` and `1` are results.** Disable random test ordering (`-p no:randomly`) so runs are comparable. ### 4. Write the finding, one per survivor A survivor is not "missing coverage" in the abstract. It is a specific sentence: *reverting this one line leaves the whole suite green.* Say which line, and say what the mutant table showed. Then supply the test. Not a description of a test — the code, using the project's existing fixtures and naming, so it can be pasted in. ### 5. Prove the new test is red for the right reason Re-run the same command with your test added. The proof has three parts and you report all three: 1. the row for that mutant flips from `SURVIVED` to `killed` 2. **every other row stays exactly as it was** — a test that kills every mutant is not specific, it is just noisy 3. the test passes on the PR as it stands Without part 2 you have written a test that fails whenever anything changes. ### 6. Say what you did not check Name the paths you never exercised — the UI, a live provider, an endpoint you skipped. A review that implies total coverage is a review the maintainer has to re-verify from scratch. Publish the hypotheses that came out clean too: it tells them which doors are already closed. ### 7. Post it as a review object, not a comment ```bash gh pr review <N> -R <owner/repo> --comment --body-file review.md ``` `gh pr comment` writes to the issue thread. It does not create a review object, so it appears in no contribution graph and no `reviewed-by:` filter, while still looking exactly like noise to spam heuristics. Reviews get `gh pr review`. Do not use `--approve` or `--request-changes` on a repo you do not maintain. If an AI agent produced the review, say so in the first line, and say whether a human read it before it posted. ## Judgement rules - **A syntactically broken mutant is not a covered site.** Every test fails on a `SyntaxError`, which looks identical to perfect coverage. `mutmatrix` compiles each mutant first and reports `INVALID`; if you mutate by hand, check it parses. - **Never mutate a file you have not backed up**, and never leave a mutant behind. `mutmatrix` keeps a backup directory and refuses to start if a previous run left one. - **Bytecode caching will lie to you.** CPython validates a `.pyc` by source size and mtime *in whole seconds*; two mutants of the same length in the same second run identical bytecode. Measured here: a killed site scored as SURVIVED. `mutmatrix` bumps mtime on every write. - **Rank the findings and say which one matters.** Three findings with no order is a list. "Finding 1 is the one worth acting on" is a review. - **Do not report a known issue as a discovery.** Search the repo's open PRs and issues for the adjacent problem first, and cite it if it exists. ## What ships with this skill - `mutmatrix.py` — the engine. Standard library only, Python 3.8+, no install. - `_test_mutmatrix.py` — its self-test, including `--red M1..M4`, which runs the suite against deliberately broken copies of the engine to prove the tests themselves can fail. A test suite that has never been shown red does not get to certify anyone else's. <!--kit-footer--> --- **Like this skill?** It is part of [second-brain-starter-kit](https://github.com/tonydzi/second-brain-starter-kit): the second brain we built for ourselves and run every day at Palo Alto AI Research Lab. Install the whole set with `npx skills add tonydzi/second-brain-starter-kit`. Everything is open source and free, so take what you need. Flagships worth a look on their own: [secondop-panel](https://github.com/tonydzi/secondop-panel) (a second opinion from a panel of external models), [claude-memory-tidy](https://github.com/tonydzi/claude-memory-tidy) (stop your agent's memory from rotting), [telegram-mcp-kit](https://github.com/tonydzi/telegram-mcp-kit) (your own Telegram over MCP in about 15 minutes). Author: **Anton Dziatkovskii**, Palo Alto AI Research Lab. Telegram [@tonydzi](https://t.me/tonydzi) - WhatsApp [+1 341 222 9178](https://wa.me/13412229178) - X [@Tony_Stef_](https://x.com/Tony_Stef_) **Engineers: want to test-drive this setup?** Message me. I hand out free starter seeds to engineers who test and report back, and custom skill requests are welcome. -
_test_mutmatrix.py 8.9 KB
Loading…
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.