Claude Skill

red-first-review

Review someone else's pull request by measuring which of its own guards its own test suite actually catches, via a small mutation matrix. Use when reviewing a PR, checking whether a regression test is real, or asked 'is this change actually tested?', 'does this test do anything',

LLM Mart · 0 points · 1 views 0 listing impressions 0 install-command copies
Download tonydzi-second-brain-starter-kit-skills_red-first-review-fdb53d7.zip · 18 KB

Install

skills CLI npx skills add https://github.com/tonydzi/second-brain-starter-kit/tree/master/skills/red-first-review
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tonydzi-second-brain-starter-kit@llmmart
Git git clone https://github.com/tonydzi/second-brain-starter-kit.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tonydzi/second-brain-starter-kit collection as a plugin from our marketplace. Git is the plain clone.

README

red-first-review

A test that has never been shown failing is a claim, not a check.

This is a Claude Code skill plus a small standalone tool for reviewing someone else's pull request by measuring the one thing you cannot get by reading it: which of the guards this PR adds would survive being deleted?

You neutralise each guard in turn, run the project's own test suite, and record what nothing notices. Survivors are the finding. Everything else in the review — the suggested test, the argument, the ranking — hangs off that table.

Standard library only. Python 3.8+. Nothing to install.

python3 mutmatrix.py \
  --repo . \
  --test "pytest tests/unit -q -p no:randomly" \
  --file src/pkg/api.py \
  --call-identity redact_secrets

What that produces, from the hand-rolled run this repository was built out of (google/adk-python#6957, a PR redacting OAuth secrets at eight call sites):

mutant verdict killed by
/run SURVIVED nothing
/run_sse killed test_agent_run_sse_redacts_oauth2_client_secret
/run_live (websocket) SURVIVED nothing
_redacted_session_response killed test_get_session_redacts_oauth2_client_secret
_redacted_sessions_response (list) SURVIVED nothing
dev get_eval_result_legacy SURVIVED nothing
dev get_eval killed test_get_eval_redacts_oauth2_client_secret
dev get_eval_result SURVIVED nothing

The PR's title named three endpoints. One of them was actually guarded by a test. Reverting the /run line alone left the entire suite green.


Why it refuses to guess

Most of this tool is not the mutation loop — that part is twenty lines. Most of it is the checks that stop it printing a table it cannot stand behind.

gate what it catches exit
baseline suite already red → nothing is attributable to the PR 3
canary test command never imports the file → every "killed" is another test's 3
compile check mutant does not parse → every test fails, which looks exactly like perfect coverage 3
no-canary you skipped the canary → the result is UNPROVEN, never clean 3

Exit 0 means no survivors. Exit 1 means survivors — that is your review. Exit 3 means the run measured nothing and you should not quote it.

The compile check exists because of a measurement made building this: a delimiter collision produced two unparseable mutants on anthropic-sdk-python#1906, and both were scored killed. The review would have certified coverage nobody had. Now:

| `sync delete-root guard`  | INVALID | mutant does not compile -- not a coverage result |
0 killed, 0 SURVIVED, 1 invalid.

A real run, end to end

anthropics/anthropic-sdk-python#1906 — "fix(memory): compare resolved paths in the delete root guard". The memory tool guarded its root directory with if command.path == "/memories", comparing the raw string the model sent, so /memories/, /memories/. and /memories/subdir/.. all reached shutil.rmtree.

Step 1, baseline.

$ .venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q
79 passed in 5.66s

Step 3, neutralise both guards (sync and async), one at a time:

python3 mutmatrix.py --repo . \
  --test ".venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q -p no:randomly" \
  --canary-file src/anthropic/lib/tools/_beta_builtin_memory_tool.py --sep '@@' \
  --mutant 'sync delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if full_path == self.memory_root.resolve():@@if False:' \
  --mutant 'async delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if Path(str(full_path)) == Path(str(self.memory_root)).resolve():@@if False:'
BASELINE: green (7s) -- 79 passed in 5.29s
CANARY  : red (1s) -- the test command does reach _beta_builtin_memory_tool.py

  [1/2] sync delete-root guard    killed   (7s)
  [2/2] async delete-root guard   killed   (7s)

2 killed, 0 SURVIVED, 0 invalid.

Both guards are real. Six named tests die when the sync guard goes:

test_delete_not_allow_deleting_memories_directory
test_delete_not_allow_deleting_memories_directory_via_symlink
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories//]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/.]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/subdir/..]

A clean table is a result, not a wasted run. "I removed your guard and your tests caught it, here are the six by name" is worth more to a maintainer than another opinion about style.

Step 0 is where the finding came from. Reading the whole file rather than the diff turned up a sibling the PR's own reasoning applies to and the diff never touches — rename, whose old_path can also be the root, and which has no guard at all:

rename("/memories", "/memories/backup")
  -> OSError(22, 'Invalid argument')   # not a ToolError; logged at ERROR with a traceback

Every other refusal in that class is a clean ToolError the model can act on. That became the review's one actionable finding, and it exists because of step 0, not because of the mutation table.

Where the method comes from

The table at the top of this README is that run. Five of eight redaction sites survived, including two of the three endpoints the PR's own title claimed to fix. The author's reply:

Thanks for this - genuinely one of the most useful reviews I've gotten on this PR. All three findings confirmed and fixed, verified the same way you found them (neutering each call site and checking the suite).

— prasanna8585, 1 Sep 2026

That review was hand-rolled. This repository is that procedure made repeatable.


Install

As a Claude Code skill:

git clone https://github.com/tonydzi/red-first-review-skill \
  ~/.claude/skills/red-first-review

Then ask Claude to review a PR; the skill triggers on "review this PR", "is this change actually tested?", "mutation test this".

As a plain tool, mutmatrix.py is one file with no dependencies. Copy it in.

Targeting

flag use
--call-identity NAME --file PATH one mutant per call site of NAME; rewrites NAME(x) → x
--mutant 'LABEL::FILE::FIND::REPLACE' an exact textual edit; FIND must be unique in the file
--line 'LABEL::FILE::N' comment out line N
--sep '@@' change the separator when your code contains :: (Python usually does)
--canary-file PATH the file the test command must import (default: first mutated file)
--no-canary skip the reach check; the run is then reported UNPROVEN, never clean

Gotchas paid for in advance

  • Bytecode caching will lie to you. CPython validates a cached .pyc by source size and mtime in whole seconds. Neutralising the same guard at two call sites usually yields two files of identical length — so two mutants inside one second run the first one's bytecode and both score SURVIVED. Measured on the self-test before the fix: 1 killed became 0 killed. mutmatrix gives every write its own mtime second.
  • Random test ordering makes runs incomparable. Pass -p no:randomly.
  • gh pr comment is not a review. It creates no review object, so the work lands in no contribution graph and no reviewed-by: filter — while still looking like noise to spam heuristics. Use gh pr review --comment.
  • Do not mutate a repository you have uncommitted work in. The tool restores from an in-memory copy and a backup directory, and refuses to start if a previous run left one behind, but git status before you start is free.

Does the tool's own test suite pass this bar?

It has to, or the whole thing is a joke. _test_mutmatrix.py builds a throwaway project whose right answer is known by construction (three call sites, one covered), and it ships with a kill-list — the mutations of itself it claims to catch:

$ python3 _test_mutmatrix.py
  C1 PASS  C2 PASS  C3 PASS  C4 PASS  C5 PASS  C7 PASS  C6 PASS
all cases behaved as expected.

$ python3 _test_mutmatrix.py --red M3     # killed/survived inverted
  C3 FAIL  C4 FAIL  C5 FAIL                # exactly the cases it declared
mutation what it breaks must fail
M1 baseline check disabled C1
M2 canary verdict hardcoded green C2
M3 killed/survived inverted C3, C4, C5
M4 compile check on the mutant removed C7

C6 is the control: green from the start and under every mutation above. Without it, a mutation that merely broke the tool would look like a working red mode.

Licence

MIT. Built for one person's fleet — adapt the paths and the test commands to yours.

Skill manifest

Red-first review

A test that has never been shown failing is a claim, not a check. This skill turns that claim into a measurement: break the thing the test is supposed to protect, and see whether anything goes red.

Applied to someone else's pull request it answers one question a human reviewer cannot answer by reading: which of the guards this PR adds would survive being deleted? Those are the ones the author will regress next month without noticing.

When to use it

  • reviewing a PR that adds a guard, a validation, a sanitiser, a redaction, a permission check — anything that says "no" on some paths
  • someone claims a fix is covered by tests, and you want that claim measured
  • you wrote a regression test yourself and want to prove it fails without the fix
  • auditing a test file that has never failed in CI

Not for: pure refactors with no behavioural claim, docs, dependency bumps.

The procedure

0. Read the whole files, not the diff

The diff shows what changed. The bug is usually in what did not. Open every touched source file end to end, and specifically look for sibling call sites: if the PR guards delete, does rename reach the same object? If it guards one endpoint, how many endpoints are there?

Write down the full list of places the PR's own reasoning applies to. That list, not the diff, is what you are about to measure.

1. Get the suite green before you touch anything

Check out the PR branch, install, run the tests it claims to affect.

A red baseline makes everything after it unattributable. If the suite is already failing, narrow the command until it is green, and check the same failures on the base branch by name — pre-existing failures reported as PR damage is the fastest way to lose a maintainer's trust.

2. Prove your test command reaches the code

Break the file under review on purpose (append raise RuntimeError("canary")) and re-run. If the suite stays green, that command never imports the file, and every "covered" verdict you are about to produce is an artefact of some other test. mutmatrix.py does this automatically and refuses to continue.

3. Neutralise each guard in turn, one at a time

python3 mutmatrix.py \
  --repo . \
  --test "pytest tests/unit -q -p no:randomly" \
  --file src/pkg/api.py \
  --call-identity redact_secrets

--call-identity NAME makes one mutant per call site of NAME, rewriting NAME(x) to x. For guards that are branches rather than calls, spell the edit out:

python3 mutmatrix.py --repo . --test "pytest tests/unit -q" --sep '@@' \
  --mutant 'root guard@@src/pkg/store.py@@if full_path == self.root.resolve():@@if False:'

Read the exit code, not just the table: 0 no survivors · 1 survivors found · 3 undecidable (baseline, canary, or a mutant that did not compile) · 4 the tool broke. Only 0 and 1 are results.

Disable random test ordering (-p no:randomly) so runs are comparable.

4. Write the finding, one per survivor

A survivor is not "missing coverage" in the abstract. It is a specific sentence: reverting this one line leaves the whole suite green. Say which line, and say what the mutant table showed.

Then supply the test. Not a description of a test — the code, using the project's existing fixtures and naming, so it can be pasted in.

5. Prove the new test is red for the right reason

Re-run the same command with your test added. The proof has three parts and you report all three:

  1. the row for that mutant flips from SURVIVED to killed
  2. every other row stays exactly as it was — a test that kills every mutant is not specific, it is just noisy
  3. the test passes on the PR as it stands

Without part 2 you have written a test that fails whenever anything changes.

6. Say what you did not check

Name the paths you never exercised — the UI, a live provider, an endpoint you skipped. A review that implies total coverage is a review the maintainer has to re-verify from scratch. Publish the hypotheses that came out clean too: it tells them which doors are already closed.

7. Post it as a review object, not a comment

gh pr review <N> -R <owner/repo> --comment --body-file review.md

gh pr comment writes to the issue thread. It does not create a review object, so it appears in no contribution graph and no reviewed-by: filter, while still looking exactly like noise to spam heuristics. Reviews get gh pr review. Do not use --approve or --request-changes on a repo you do not maintain.

If an AI agent produced the review, say so in the first line, and say whether a human read it before it posted.

Judgement rules

  • A syntactically broken mutant is not a covered site. Every test fails on a SyntaxError, which looks identical to perfect coverage. mutmatrix compiles each mutant first and reports INVALID; if you mutate by hand, check it parses.
  • Never mutate a file you have not backed up, and never leave a mutant behind. mutmatrix keeps a backup directory and refuses to start if a previous run left one.
  • Bytecode caching will lie to you. CPython validates a .pyc by source size and mtime in whole seconds; two mutants of the same length in the same second run identical bytecode. Measured here: a killed site scored as SURVIVED. mutmatrix bumps mtime on every write.
  • Rank the findings and say which one matters. Three findings with no order is a list. "Finding 1 is the one worth acting on" is a review.
  • Do not report a known issue as a discovery. Search the repo's open PRs and issues for the adjacent problem first, and cite it if it exists.

What ships with this skill

  • mutmatrix.py — the engine. Standard library only, Python 3.8+, no install.
  • _test_mutmatrix.py — its self-test, including --red M1..M4, which runs the suite against deliberately broken copies of the engine to prove the tests themselves can fail. A test suite that has never been shown red does not get to certify anyone else's.

Like this skill? It is part of second-brain-starter-kit: the second brain we built for ourselves and run every day at Palo Alto AI Research Lab. Install the whole set with npx skills add tonydzi/second-brain-starter-kit. Everything is open source and free, so take what you need.

Flagships worth a look on their own: secondop-panel (a second opinion from a panel of external models), claude-memory-tidy (stop your agent's memory from rotting), telegram-mcp-kit (your own Telegram over MCP in about 15 minutes).

Author: Anton Dziatkovskii, Palo Alto AI Research Lab. Telegram @tonydzi - WhatsApp +1 341 222 9178 - X @Tony_Stef_

Engineers: want to test-drive this setup? Message me. I hand out free starter seeds to engineers who test and report back, and custom skill requests are welcome.

Files (second-brain-starter-kit)
  • LICENSE 1 KB · in bundle
  • mutmatrix.py 21 KB
    Loading…
  • README.md 8.9 KB
    # red-first-review
    
    **A test that has never been shown failing is a claim, not a check.**
    
    This is a Claude Code skill plus a small standalone tool for reviewing someone
    else's pull request by measuring the one thing you cannot get by reading it:
    *which of the guards this PR adds would survive being deleted?*
    
    You neutralise each guard in turn, run the project's own test suite, and record
    what nothing notices. Survivors are the finding. Everything else in the review —
    the suggested test, the argument, the ranking — hangs off that table.
    
    Standard library only. Python 3.8+. Nothing to install.
    
    ```bash
    python3 mutmatrix.py \
      --repo . \
      --test "pytest tests/unit -q -p no:randomly" \
      --file src/pkg/api.py \
      --call-identity redact_secrets
    ```
    
    What that produces, from the hand-rolled run this repository was built out of
    ([google/adk-python#6957](https://github.com/google/adk-python/pull/6957), a PR
    redacting OAuth secrets at eight call sites):
    
    | mutant | verdict | killed by |
    | --- | --- | --- |
    | `/run` | **SURVIVED** | nothing |
    | `/run_sse` | killed | `test_agent_run_sse_redacts_oauth2_client_secret` |
    | `/run_live` (websocket) | **SURVIVED** | nothing |
    | `_redacted_session_response` | killed | `test_get_session_redacts_oauth2_client_secret` |
    | `_redacted_sessions_response` (list) | **SURVIVED** | nothing |
    | dev `get_eval_result_legacy` | **SURVIVED** | nothing |
    | dev `get_eval` | killed | `test_get_eval_redacts_oauth2_client_secret` |
    | dev `get_eval_result` | **SURVIVED** | nothing |
    
    The PR's title named three endpoints. One of them was actually guarded by a test.
    Reverting the `/run` line alone left the entire suite green.
    
    ---
    
    ## Why it refuses to guess
    
    Most of this tool is not the mutation loop — that part is twenty lines. Most of
    it is the checks that stop it printing a table it cannot stand behind.
    
    | gate | what it catches | exit |
    | --- | --- | --- |
    | **baseline** | suite already red → nothing is attributable to the PR | 3 |
    | **canary** | test command never imports the file → every "killed" is another test's | 3 |
    | **compile check** | mutant does not parse → every test fails, which looks exactly like perfect coverage | 3 |
    | **no-canary** | you skipped the canary → the result is `UNPROVEN`, never `clean` | 3 |
    
    Exit `0` means no survivors. Exit `1` means survivors — that is your review.
    Exit `3` means the run measured nothing and you should not quote it.
    
    The compile check exists because of a measurement made building this: a
    delimiter collision produced two unparseable mutants on
    [anthropic-sdk-python#1906](https://github.com/anthropics/anthropic-sdk-python/pull/1906),
    and both were scored `killed`. The review would have certified coverage nobody
    had. Now:
    
    ```
    | `sync delete-root guard`  | INVALID | mutant does not compile -- not a coverage result |
    0 killed, 0 SURVIVED, 1 invalid.
    ```
    
    ---
    
    ## A real run, end to end
    
    `anthropics/anthropic-sdk-python#1906` — *"fix(memory): compare resolved paths in
    the delete root guard"*. The memory tool guarded its root directory with
    `if command.path == "/memories"`, comparing the raw string the model sent, so
    `/memories/`, `/memories/.` and `/memories/subdir/..` all reached `shutil.rmtree`.
    
    **Step 1, baseline.**
    
    ```
    $ .venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q
    79 passed in 5.66s
    ```
    
    **Step 3, neutralise both guards** (sync and async), one at a time:
    
    ```bash
    python3 mutmatrix.py --repo . \
      --test ".venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q -p no:randomly" \
      --canary-file src/anthropic/lib/tools/_beta_builtin_memory_tool.py --sep '@@' \
      --mutant 'sync delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if full_path == self.memory_root.resolve():@@if False:' \
      --mutant 'async delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if Path(str(full_path)) == Path(str(self.memory_root)).resolve():@@if False:'
    ```
    
    ```
    BASELINE: green (7s) -- 79 passed in 5.29s
    CANARY  : red (1s) -- the test command does reach _beta_builtin_memory_tool.py
    
      [1/2] sync delete-root guard    killed   (7s)
      [2/2] async delete-root guard   killed   (7s)
    
    2 killed, 0 SURVIVED, 0 invalid.
    ```
    
    Both guards are real. Six named tests die when the sync guard goes:
    
    ```
    test_delete_not_allow_deleting_memories_directory
    test_delete_not_allow_deleting_memories_directory_via_symlink
    test_delete_not_allow_deleting_memories_directory_via_alias[/memories/]
    test_delete_not_allow_deleting_memories_directory_via_alias[/memories//]
    test_delete_not_allow_deleting_memories_directory_via_alias[/memories/.]
    test_delete_not_allow_deleting_memories_directory_via_alias[/memories/subdir/..]
    ```
    
    **A clean table is a result, not a wasted run.** "I removed your guard and your
    tests caught it, here are the six by name" is worth more to a maintainer than
    another opinion about style.
    
    **Step 0 is where the finding came from.** Reading the whole file rather than the
    diff turned up a sibling the PR's own reasoning applies to and the diff never
    touches — `rename`, whose `old_path` can also be the root, and which has no guard
    at all:
    
    ```
    rename("/memories", "/memories/backup")
      -> OSError(22, 'Invalid argument')   # not a ToolError; logged at ERROR with a traceback
    ```
    
    Every other refusal in that class is a clean `ToolError` the model can act on.
    That became the review's one actionable finding, and it exists because of step 0,
    not because of the mutation table.
    
    ## Where the method comes from
    
    The table at the top of this README is that run. Five of eight redaction sites
    survived, including two of the three endpoints the PR's own title claimed to fix.
    The author's reply:
    
    > Thanks for this - genuinely one of the most useful reviews I've gotten on this
    > PR. All three findings confirmed and fixed, verified the same way you found
    > them (neutering each call site and checking the suite).
    
    — [prasanna8585, 1 Sep 2026](https://github.com/google/adk-python/pull/6957#issuecomment-5489421299)
    
    That review was hand-rolled. This repository is that procedure made repeatable.
    
    ---
    
    ## Install
    
    As a Claude Code skill:
    
    ```bash
    git clone https://github.com/tonydzi/red-first-review-skill \
      ~/.claude/skills/red-first-review
    ```
    
    Then ask Claude to review a PR; the skill triggers on "review this PR", "is this
    change actually tested?", "mutation test this".
    
    As a plain tool, `mutmatrix.py` is one file with no dependencies. Copy it in.
    
    ## Targeting
    
    | flag | use |
    | --- | --- |
    | `--call-identity NAME --file PATH` | one mutant per call site of `NAME`; rewrites `NAME(x)` → `x` |
    | `--mutant 'LABEL::FILE::FIND::REPLACE'` | an exact textual edit; `FIND` must be unique in the file |
    | `--line 'LABEL::FILE::N'` | comment out line N |
    | `--sep '@@'` | change the separator when your code contains `::` (Python usually does) |
    | `--canary-file PATH` | the file the test command must import (default: first mutated file) |
    | `--no-canary` | skip the reach check; the run is then reported `UNPROVEN`, never clean |
    
    ## Gotchas paid for in advance
    
    - **Bytecode caching will lie to you.** CPython validates a cached `.pyc` by
      source size and mtime *in whole seconds*. Neutralising the same guard at two
      call sites usually yields two files of identical length — so two mutants
      inside one second run the *first* one's bytecode and both score `SURVIVED`.
      Measured on the self-test before the fix: `1 killed` became `0 killed`.
      `mutmatrix` gives every write its own mtime second.
    - **Random test ordering** makes runs incomparable. Pass `-p no:randomly`.
    - **`gh pr comment` is not a review.** It creates no review object, so the work
      lands in no contribution graph and no `reviewed-by:` filter — while still
      looking like noise to spam heuristics. Use `gh pr review --comment`.
    - **Do not mutate a repository you have uncommitted work in.** The tool restores
      from an in-memory copy and a backup directory, and refuses to start if a
      previous run left one behind, but `git status` before you start is free.
    
    ## Does the tool's own test suite pass this bar?
    
    It has to, or the whole thing is a joke. `_test_mutmatrix.py` builds a throwaway
    project whose right answer is known by construction (three call sites, one
    covered), and it ships with a kill-list — the mutations of *itself* it claims to
    catch:
    
    ```bash
    $ python3 _test_mutmatrix.py
      C1 PASS  C2 PASS  C3 PASS  C4 PASS  C5 PASS  C7 PASS  C6 PASS
    all cases behaved as expected.
    
    $ python3 _test_mutmatrix.py --red M3     # killed/survived inverted
      C3 FAIL  C4 FAIL  C5 FAIL                # exactly the cases it declared
    ```
    
    | mutation | what it breaks | must fail |
    | --- | --- | --- |
    | M1 | baseline check disabled | C1 |
    | M2 | canary verdict hardcoded green | C2 |
    | M3 | killed/survived inverted | C3, C4, C5 |
    | M4 | compile check on the mutant removed | C7 |
    
    C6 is the control: green from the start and under every mutation above. Without
    it, a mutation that merely broke the tool would look like a working red mode.
    
    ## Licence
    
    MIT. Built for one person's fleet — adapt the paths and the test commands to
    yours.
    
  • SKILL.md 7.4 KB
    ---
    name: red-first-review
    description: "Review someone else's pull request by measuring which of its own guards its own test suite actually catches, via a small mutation matrix. Use when reviewing a PR, checking whether a regression test is real, or asked 'is this change actually tested?', 'does this test do anything', 'mutation test this'."
    license: MIT
    version: 1.0.0
    ---
    
    # Red-first review
    
    A test that has never been shown failing is a claim, not a check. This skill turns
    that claim into a measurement: break the thing the test is supposed to protect,
    and see whether anything goes red.
    
    Applied to someone else's pull request it answers one question a human reviewer
    cannot answer by reading: **which of the guards this PR adds would survive being
    deleted?** Those are the ones the author will regress next month without noticing.
    
    ## When to use it
    
    - reviewing a PR that adds a guard, a validation, a sanitiser, a redaction, a
      permission check — anything that says "no" on some paths
    - someone claims a fix is covered by tests, and you want that claim measured
    - you wrote a regression test yourself and want to prove it fails without the fix
    - auditing a test file that has never failed in CI
    
    Not for: pure refactors with no behavioural claim, docs, dependency bumps.
    
    ## The procedure
    
    ### 0. Read the whole files, not the diff
    
    The diff shows what changed. The bug is usually in what did not. Open every
    touched source file end to end, and specifically look for **sibling call sites**:
    if the PR guards `delete`, does `rename` reach the same object? If it guards one
    endpoint, how many endpoints are there?
    
    Write down the full list of places the PR's own reasoning applies to. That list,
    not the diff, is what you are about to measure.
    
    ### 1. Get the suite green before you touch anything
    
    Check out the PR branch, install, run the tests it claims to affect.
    
    A red baseline makes everything after it unattributable. If the suite is already
    failing, narrow the command until it is green, **and check the same failures on
    the base branch by name** — pre-existing failures reported as PR damage is the
    fastest way to lose a maintainer's trust.
    
    ### 2. Prove your test command reaches the code
    
    Break the file under review on purpose (append `raise RuntimeError("canary")`)
    and re-run. If the suite stays green, that command never imports the file, and
    every "covered" verdict you are about to produce is an artefact of some other
    test. `mutmatrix.py` does this automatically and refuses to continue.
    
    ### 3. Neutralise each guard in turn, one at a time
    
    ```bash
    python3 mutmatrix.py \
      --repo . \
      --test "pytest tests/unit -q -p no:randomly" \
      --file src/pkg/api.py \
      --call-identity redact_secrets
    ```
    
    `--call-identity NAME` makes one mutant per call site of `NAME`, rewriting
    `NAME(x)` to `x`. For guards that are branches rather than calls, spell the edit
    out:
    
    ```bash
    python3 mutmatrix.py --repo . --test "pytest tests/unit -q" --sep '@@' \
      --mutant 'root guard@@src/pkg/store.py@@if full_path == self.root.resolve():@@if False:'
    ```
    
    Read the exit code, not just the table: `0` no survivors · `1` survivors found ·
    `3` undecidable (baseline, canary, or a mutant that did not compile) · `4` the
    tool broke. **Only `0` and `1` are results.**
    
    Disable random test ordering (`-p no:randomly`) so runs are comparable.
    
    ### 4. Write the finding, one per survivor
    
    A survivor is not "missing coverage" in the abstract. It is a specific sentence:
    *reverting this one line leaves the whole suite green.* Say which line, and say
    what the mutant table showed.
    
    Then supply the test. Not a description of a test — the code, using the
    project's existing fixtures and naming, so it can be pasted in.
    
    ### 5. Prove the new test is red for the right reason
    
    Re-run the same command with your test added. The proof has three parts and you
    report all three:
    
    1. the row for that mutant flips from `SURVIVED` to `killed`
    2. **every other row stays exactly as it was** — a test that kills every mutant
       is not specific, it is just noisy
    3. the test passes on the PR as it stands
    
    Without part 2 you have written a test that fails whenever anything changes.
    
    ### 6. Say what you did not check
    
    Name the paths you never exercised — the UI, a live provider, an endpoint you
    skipped. A review that implies total coverage is a review the maintainer has to
    re-verify from scratch. Publish the hypotheses that came out clean too: it tells
    them which doors are already closed.
    
    ### 7. Post it as a review object, not a comment
    
    ```bash
    gh pr review <N> -R <owner/repo> --comment --body-file review.md
    ```
    
    `gh pr comment` writes to the issue thread. It does not create a review object,
    so it appears in no contribution graph and no `reviewed-by:` filter, while still
    looking exactly like noise to spam heuristics. Reviews get `gh pr review`.
    Do not use `--approve` or `--request-changes` on a repo you do not maintain.
    
    If an AI agent produced the review, say so in the first line, and say whether a
    human read it before it posted.
    
    ## Judgement rules
    
    - **A syntactically broken mutant is not a covered site.** Every test fails on a
      `SyntaxError`, which looks identical to perfect coverage. `mutmatrix` compiles
      each mutant first and reports `INVALID`; if you mutate by hand, check it parses.
    - **Never mutate a file you have not backed up**, and never leave a mutant behind.
      `mutmatrix` keeps a backup directory and refuses to start if a previous run
      left one.
    - **Bytecode caching will lie to you.** CPython validates a `.pyc` by source size
      and mtime *in whole seconds*; two mutants of the same length in the same second
      run identical bytecode. Measured here: a killed site scored as SURVIVED.
      `mutmatrix` bumps mtime on every write.
    - **Rank the findings and say which one matters.** Three findings with no order
      is a list. "Finding 1 is the one worth acting on" is a review.
    - **Do not report a known issue as a discovery.** Search the repo's open PRs and
      issues for the adjacent problem first, and cite it if it exists.
    
    ## What ships with this skill
    
    - `mutmatrix.py` — the engine. Standard library only, Python 3.8+, no install.
    - `_test_mutmatrix.py` — its self-test, including `--red M1..M4`, which runs the
      suite against deliberately broken copies of the engine to prove the tests
      themselves can fail. A test suite that has never been shown red does not get to
      certify anyone else's.
    
    <!--kit-footer-->
    
    ---
    
    **Like this skill?** It is part of [second-brain-starter-kit](https://github.com/tonydzi/second-brain-starter-kit): the second brain we built for ourselves and run every day at Palo Alto AI Research Lab. Install the whole set with `npx skills add tonydzi/second-brain-starter-kit`. Everything is open source and free, so take what you need.
    
    Flagships worth a look on their own: [secondop-panel](https://github.com/tonydzi/secondop-panel) (a second opinion from a panel of external models), [claude-memory-tidy](https://github.com/tonydzi/claude-memory-tidy) (stop your agent's memory from rotting), [telegram-mcp-kit](https://github.com/tonydzi/telegram-mcp-kit) (your own Telegram over MCP in about 15 minutes).
    
    Author: **Anton Dziatkovskii**, Palo Alto AI Research Lab. Telegram [@tonydzi](https://t.me/tonydzi) - WhatsApp [+1 341 222 9178](https://wa.me/13412229178) - X [@Tony_Stef_](https://x.com/Tony_Stef_)
    
    **Engineers: want to test-drive this setup?** Message me. I hand out free starter seeds to engineers who test and report back, and custom skill requests are welcome.
    
  • _test_mutmatrix.py 8.9 KB
    Loading…

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.