Treat a skill like a function under test. Feed it example inputs in a clean\nroom, check the artifacts against what good looks like, and let the failures\ndrive the edits. The eval is only honest if the run is blind: the agent\nexecuting the skill must carry none of this conversation's context and must\nnever see the expected output. Leak either and you are teaching to the test.
\nConfirm all three before spawning anything. If any is missing or\nunresolvable, stop and tell the user exactly which one and what a good\nversion looks like. Do not invent cases, guess intent, or eval against a\nfuzzy wish.
\nSKILL.md. If you can't find it,\nlist the skills you can see and ask which one they mean.Validate inputs and surface first principles. Resolve the skill and\nread its first principles — what it's for and the standard it holds\nitself to; this is what the judge grades against, so if the skill doesn't\nmake them clear, clarify with the user rather than inventing them. Settle\nthe eval mode here too: judgment (a bar the judge applies) vs conformance\n(an exact task hit exactly) — ask the user if it's ambiguous. Then sharpen\neach case's bar — the outcome plus the smells, kept at the altitude the\nuser cares about, never widened into a prescribed parts list unless the\nskill is conformance-style. Done when you can state the skill's first\nprinciples in a sentence and every case has a concrete input and a bar a\ncompetent judge could hold an artifact to.
\nBlind run, one fresh agent per case. Isolate every run so a misbehaving\nskill can't touch the live checkout and each case starts clean. Prefer\ncapturing the artifact from the runner's final message — if the skill's\noutput is a plan or text, ask for it inline and nothing hits disk to leak.\nWhen the skill must write files, give the runner a throwaway sandbox dir as\nits only writable root, not a worktree of the live repo (worktree isolation\nguards git state, not absolute-path or escaped writes). After every run,\nsweep the live checkout (git status) and clean anything the run leaked —\nisolation is best-effort, the sweep is the guarantee. Give the runner\nonly the input and the instruction to use the target skill — never the\nbar, the smells, the other cases, or why you're asking. Done when you hold\none artifact per case, each from a context-free run, and the checkout is\nclean.
Grade with a separate judge that applies judgment. Hand a fresh judge\nthe artifact, the bar, and the skill's first principles — so it grades\nagainst the skill's own intent, not its personal taste — but never the\nexpected output and never \"make this pass.\" Grounded in those principles\nthe judge is a competent practitioner: it decides whether the work clears\nthe bar with defensible choices, and is explicitly free to fault both\ntoo-coarse and too-fine work. It must cite specific evidence for each\nverdict — a quote or pointer, not a number. Done when every part of the bar\nhas a verdict grounded in the artifact.
\nAccount for nondeterminism. Agents flicker. A single green is not\nproof. For any case that matters or any verdict that looks borderline,\nre-run the blind run 2–3× and report the pass rate. A skill that passes\n1 of 3 is not fixed.
\nDiagnose each failure as skill-defect vs bad-case. A miss means either\nthe skill failed to drive the behavior (fixable here) or the bar was\nwrong — it asked for something the skill should not do, can't express, or\nit punished a defensible judgment call the skill was right to make (tell\nthe user; do not edit the skill to chase a wrong bar — that just encodes\nthe wrong reality). Name the defect against the write-skills failure\nmodes:\npremature completion, vague completion criterion, missing rule, no leading\nword, duplication, sediment, war story, no-op.
Revise via write-skills. Fix the named defect — and obey those\nauthoring rules while you do it: sharpen the completion criterion before\nadding bulk, prefer one leading word over more sentences, add no no-ops.\nThe failure is the spec for the edit; change only what the failure points\nat.
\nRe-eval all cases, not just the failed one. A fix can regress a case\nthat was passing. Loop until every case clears its rate bar, or until you\ncan show the skill structurally can't express a case — then report that\ninstead of forcing it.
\nA short report: per case, pass rate and the cited gap; the defect each\nfailure mapped to; the edits you made (or, if the user asked to approve\nfirst, the diff you propose); and the re-eval result. Make the before/after\nmovement legible — this is the evidence the skill actually improved.
\n