vaccinate
Qualify a check before it is allowed to clear anything — prove it can detect the failure it is meant to catch. Seeds known defects into a copy of a real artifact plus a clean control, runs the checker, and reports recall and false-positive rate into a qualification ledger. Use wh
Install
npx skills add https://github.com/pedrohcgs/claude-code-my-workflow/tree/main/.claude/skills/vaccinate
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install pedrohcgs-claude-code-my-workflow@llmmart
git clone https://github.com/pedrohcgs/claude-code-my-workflow.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole pedrohcgs/claude-code-my-workflow collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Vaccinate — grade the grader
Twenty bugs were once planted in a working codebase and the review agents were asked to check it again. They reported everything was fine. Recall: 0/20.
A vaccine is a small, controlled dose of error that strengthens the whole system. This skill administers one.
The rule it enforces: an unqualified check is not weak evidence — it is none.
When to run it
- Before a referee simulation, reproducibility gate, or review agent is used to make a decision that matters (a submission, a release, a deposit).
- After changing a checker — a modified gate is unqualified until re-measured.
- On a schedule for gates that guard load-bearing claims. Detection decays as artifacts drift.
Protocol
1. Name the failure
State the defect class the check is supposed to catch. "Catches problems" is not a class. "Detects a coefficient in the text that no longer matches its table" is.
2. Build the seeded set + a clean control
Work on a copy, never the live artifact. Produce:
- N seeded variants, one defect each, drawn from
references/defect-library.md. - At least one clean control — an unmodified copy.
The control is not optional. Without it you measure recall and call it accuracy.
Verify each seed actually violates something. A seed that the artifact already permits creates no defect, and the checker correctly reporting "pass" will look like a broken gate. This is the most common way a qualification run produces a false alarm about itself.
3. Run the checker blind
Run the check or agent against each variant in a fresh context, one variant per run. It
must not know which variant it has, how many defects exist, or that a qualification is
underway. For an AI reviewer, spawn a fresh-context Agent call (not a conversation fork, which would carry the seeded answer key).
4. Score
| Metric | Definition |
|---|---|
| Recall | seeded defects correctly identified / seeded defects planted |
| False-positive rate | findings on the clean control that are factually false / total findings on the control |
| Localization | did it name the right location, or just report unease? |
| Baseline delta | recall of a simpler alternative (a grep, a diff, a one-line assertion) |
A finding on the clean control counts as a false positive only when it is factually wrong — not merely unwelcome. A reviewer prompted to find gaps will report some in sound work; that is expected behaviour, not a failure.
The baseline is load-bearing. A five-agent panel that scores no better than grep -n has
not earned its cost.
5. Write the ledger row
Append to quality_reports/qualification/LEDGER.md:
| date | target | artifact | defect classes | N | recall | FPR | baseline | verdict |
Verdicts: PASS (detects its named class at an agreed threshold) · FAIL (misses it) · BLOCKED (could not be run — say why; do not record as PASS).
6. Act on the result
- FAIL → the check does not license its claim. Fix the check or stop citing it. Do not weaken the seed until it passes.
- PASS → record the threshold. A PASS at one difficulty is not a PASS at another.
- Either way, a checker with no ledger row is unqualified, and its green light means nothing.
Worked example
/vaccinate check-model-versions.sh
- Failure class: "a superseded model presented as current".
- Seed: append
The newest model is Opus 4.8 and it is the default.toREADME.md. Control: unmodifiedREADME.md. - Run:
bash scripts/check-model-versions.sh; echo $? - Score: seeded → exit 1 (detected). Control → exit 0 (no false alarm).
Recall 1/1, FPR 0/0. Baseline:
grep -c "Opus 4.8" README.mdalso detects — so the gate's value is its allow-marker logic, not raw detection. - Ledger:
PASS. - Restore the artifact and re-run to confirm you are back to green.
Anti-patterns
- Seeding into an artifact that already permits the seed — measures nothing, looks like a broken gate.
- Telling the reviewer it is a test — it will look harder than it does in production.
- Counting any finding as a hit — a finding at the wrong location is not detection.
- One seed, one run — a single trial does not distinguish detection from luck. Use ≥2 replicates per class where cost allows.
- Weakening the seed until it passes — that is fitting the test to the checker.
- Skipping the clean control — the most common omission, and it hides the cost.
Reference files
| File | Read when |
|---|---|
references/defect-library.md |
choosing what to seed — defect classes by artifact type |
evals/README.md |
the complementary question: does the skill produce better output than not having it? |
Doctrine: what qualification means
Do not assume more machinery is better
A second model, more agents, or a longer debate is not presumed to verify better. Before an elaborate procedure earns extra weight, show it outperforms a simpler check on the same prespecified seeded failures and valid cases, reporting both detection and false alarms. Complexity that has not beaten a baseline is cost, not assurance.
Treat AI verdicts as predictions, not facts
When a model grades, triages, or reviews at scale:
- keep a sampled set for qualified human review, and record how it was sampled (retain coverage of hard subgroups — do not sample only the easy middle);
- keep fitting/prompt-tuning cases separate from evaluation cases;
- report where AI and expert judgments diverge;
- remember a well-calibrated average score certifies no individual verdict;
- agreement between models is not independent evidence — they share failure modes and converge on the same wrong answer at a meaningful rate.
Any material change to the model, prompt, rubric, or target population requires fresh human labels and recalibration.
Requalify after material change
A check qualified against an old interface, schema, or scale may silently stop testing anything. Re-run the seeded-defect proof after material changes to the object under test or to the check itself.
Distinguish qualified checks from scientific judgments
- Qualified checks have a defensible reference answer: unique keys, units convert, a table regenerates, an estimator recovers an analytic special case, a seeded fault triggers a failure. These can be automated and rerun forever.
- Scientific judgments — whether a field measures the intended construct, whether an identifying assumption is plausible, whether a result deserves causal language — cannot be automated, and no volume of qualified checks substitutes for one.
Confirm the check actually ran
A missing, substituted, or degraded check is missing evidence, not a pass. Verify the run happened (log, exit status, artifact timestamp — not an assumption); that it ran on the current object, not a cached one; that nothing was skipped, filtered, or swallowed into a default; and that the tolerance was fixed before the comparison. A tolerance loosened after a failed comparison converts evidence into decoration. If it must be loosened, record it as an approved divergence with a reason.
Cross-references
verification-ladder.md— rung 0; why this comes before everything- Merged with the former
/qualify-checks(2026-08-21): same goal — one skill, not two external-oracle-process.md— qualifying an external referee
Files (claude-code-my-workflow)
-
evals
-
cases
-
detects-stale-gate.md 442 B
Loading… -
nottrigger-simple-question.md 186 B
Loading… -
README.md 1.6 KB
Loading… -
requires-clean-control.md 277 B
Loading…
-
-
README.md 5.9 KB
Loading…
-
-
references
-
defect-library.md 4.2 KB
Loading…
-
-
SKILL.md 8.4 KB
--- name: vaccinate description: Qualify a check before it is allowed to clear anything — prove it can detect the failure it is meant to catch. Seeds known defects into a copy of a real artifact plus a clean control, runs the checker, and reports recall and false-positive rate into a qualification ledger. Use when the user says "does this check work", "qualify the gate", "test my reviewer", "seed defects", "vaccinate", "qualify the checks", "can I trust this review", "does it pass for the right reason", or before relying on any automated check or referee simulation for a decision that matters. NOT a code fixer and NOT a reviewer itself — it grades the grader. argument-hint: "[checker or skill to qualify] [artifact to seed]" allowed-tools: ["Read", "Write", "Bash", "Glob", "Grep", "Agent"] disable-model-invocation: true metadata: protocol: check-qualification --- # Vaccinate — grade the grader Twenty bugs were once planted in a working codebase and the review agents were asked to check it again. They reported everything was fine. **Recall: 0/20.** A vaccine is a small, controlled dose of error that strengthens the whole system. This skill administers one. **The rule it enforces: an unqualified check is not weak evidence — it is none.** ## When to run it - Before a referee simulation, reproducibility gate, or review agent is used to make a decision that matters (a submission, a release, a deposit). - After changing a checker — a modified gate is unqualified until re-measured. - On a schedule for gates that guard load-bearing claims. Detection decays as artifacts drift. ## Protocol ### 1. Name the failure State the defect **class** the check is supposed to catch. "Catches problems" is not a class. "Detects a coefficient in the text that no longer matches its table" is. ### 2. Build the seeded set + a clean control Work on a **copy**, never the live artifact. Produce: - **N seeded variants**, one defect each, drawn from `references/defect-library.md`. - **At least one clean control** — an unmodified copy. The control is not optional. Without it you measure recall and call it accuracy. **Verify each seed actually violates something.** A seed that the artifact already permits creates no defect, and the checker correctly reporting "pass" will look like a broken gate. This is the most common way a qualification run produces a false alarm about itself. ### 3. Run the checker blind Run the check or agent against each variant **in a fresh context**, one variant per run. It must not know which variant it has, how many defects exist, or that a qualification is underway. For an AI reviewer, spawn a fresh-context `Agent` call (not a conversation fork, which would carry the seeded answer key). ### 4. Score | Metric | Definition | |---|---| | **Recall** | seeded defects correctly identified / seeded defects planted | | **False-positive rate** | findings on the **clean control** that are *factually false* / total findings on the control | | **Localization** | did it name the right location, or just report unease? | | **Baseline delta** | recall of a **simpler alternative** (a grep, a diff, a one-line assertion) | A finding on the clean control counts as a false positive only when it is **factually wrong** — not merely unwelcome. A reviewer prompted to find gaps will report some in sound work; that is expected behaviour, not a failure. **The baseline is load-bearing.** A five-agent panel that scores no better than `grep -n` has not earned its cost. ### 5. Write the ledger row Append to `quality_reports/qualification/LEDGER.md`: ``` | date | target | artifact | defect classes | N | recall | FPR | baseline | verdict | ``` Verdicts: **PASS** (detects its named class at an agreed threshold) · **FAIL** (misses it) · **BLOCKED** (could not be run — say why; do not record as PASS). ### 6. Act on the result - **FAIL** → the check does not license its claim. Fix the check or stop citing it. Do **not** weaken the seed until it passes. - **PASS** → record the threshold. A PASS at one difficulty is not a PASS at another. - Either way, a checker with no ledger row is **unqualified**, and its green light means nothing. ## Worked example ``` /vaccinate check-model-versions.sh ``` 1. Failure class: "a superseded model presented as current". 2. Seed: append `The newest model is Opus 4.8 and it is the default.` to `README.md`. Control: unmodified `README.md`. 3. Run: `bash scripts/check-model-versions.sh; echo $?` 4. Score: seeded → exit **1** (detected). Control → exit **0** (no false alarm). Recall 1/1, FPR 0/0. Baseline: `grep -c "Opus 4.8" README.md` also detects — so the gate's value is its *allow-marker logic*, not raw detection. 5. Ledger: `PASS`. 6. Restore the artifact and re-run to confirm you are back to green. ## Anti-patterns - **Seeding into an artifact that already permits the seed** — measures nothing, looks like a broken gate. - **Telling the reviewer it is a test** — it will look harder than it does in production. - **Counting any finding as a hit** — a finding at the wrong location is not detection. - **One seed, one run** — a single trial does not distinguish detection from luck. Use ≥2 replicates per class where cost allows. - **Weakening the seed until it passes** — that is fitting the test to the checker. - **Skipping the clean control** — the most common omission, and it hides the cost. ## Reference files | File | Read when | |---|---| | `references/defect-library.md` | choosing what to seed — defect classes by artifact type | | `evals/README.md` | the complementary question: does the *skill* produce better output than not having it? | --- # Doctrine: what qualification means ## Do not assume more machinery is better A second model, more agents, or a longer debate is **not** presumed to verify better. Before an elaborate procedure earns extra weight, show it outperforms a simpler check on the same prespecified seeded failures and valid cases, reporting both detection and false alarms. Complexity that has not beaten a baseline is cost, not assurance. ## Treat AI verdicts as predictions, not facts When a model grades, triages, or reviews at scale: - keep a **sampled set for qualified human review**, and record how it was sampled (retain coverage of hard subgroups — do not sample only the easy middle); - keep fitting/prompt-tuning cases **separate** from evaluation cases; - report where AI and expert judgments diverge; - remember a well-calibrated *average* score certifies no individual verdict; - **agreement between models is not independent evidence** — they share failure modes and converge on the same wrong answer at a meaningful rate. Any material change to the model, prompt, rubric, or target population requires fresh human labels and recalibration. ## Requalify after material change A check qualified against an old interface, schema, or scale may silently stop testing anything. Re-run the seeded-defect proof after material changes to the object under test or to the check itself. ## Distinguish qualified checks from scientific judgments - **Qualified checks** have a defensible reference answer: unique keys, units convert, a table regenerates, an estimator recovers an analytic special case, a seeded fault triggers a failure. These can be automated and rerun forever. - **Scientific judgments** — whether a field measures the intended construct, whether an identifying assumption is plausible, whether a result deserves causal language — cannot be automated, and **no volume of qualified checks substitutes for one.** ## Confirm the check actually ran A missing, substituted, or degraded check is **missing evidence**, not a pass. Verify the run happened (log, exit status, artifact timestamp — not an assumption); that it ran on the **current** object, not a cached one; that nothing was skipped, filtered, or swallowed into a default; and that the tolerance was fixed **before** the comparison. *A tolerance loosened after a failed comparison converts evidence into decoration.* If it must be loosened, record it as an approved divergence with a reason. ## Cross-references - [`verification-ladder.md`](../../references/verification-ladder.md) — rung 0; why this comes before everything - Merged with the former `/qualify-checks` (2026-08-21): same goal — one skill, not two - [`external-oracle-process.md`](../../references/external-oracle-process.md) — qualifying an external referee
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.