Evals are the prerequisite, not the polish. You cannot tune a prompt, a retriever, a\ncompaction strategy or a memory layer without a harness that says whether the change made\nthings better. Teams that skip this ship vibes and learn about regressions from users.
\nThis skill is the operational layer: what to measure, how to build the dataset, how to\nmake a judge trustworthy, and how to gate CI on it without teaching everyone to ignore red.
\n| The ask | \nGo to | \n
|---|---|
| \"What should I even measure?\" | \nThree levels → references/eval-taxonomy.md | \n
| \"Where do the test cases come from?\" | \nGolden set → references/golden-datasets.md | \n
| \"My judge disagrees with me / is it any good?\" | \nJudges → references/llm-judge.md | \n
| \"Verify a finding is real, not plausible\" | \nRefuters → references/adversarial-verification.md | \n
| \"Is my RAG retrieval any good?\" | \nRetrieval → references/retrieval-eval.md | \n
| \"Where do the human labels come from?\" | \nreferences/annotation-workflow.md | \n
| \"Should this block the merge?\" | \nGating → references/regression-gating.md | \n
| \"Did this change really make it worse?\" | \nIs the drop real | \n
| \"Optimise against the eval / run it overnight\" | \nHillclimbing → references/hillclimbing.md | \n
| \"Which platform should we use?\" | \nreferences/tooling-landscape.md | \n
| \"Just give me a starting file\" | \nAssets — golden set, rubric, runner, CI gate | \n
scripts/judge-calibration.py does the arithmetic and the verdict.Most teams do only the third, then wonder why quality is unpredictable.
\n| Level | \nQuestion | \nSignal | \nTypical evaluator | \n
|---|---|---|---|
| Outcome | \nIs the final artifact correct? | \nBinary or scored end state | \nDeterministic assertion, unit test, judge | \n
| Step | \nWas this tool call right? | \nPer-span: tool choice, arg shape, arg values | \nSchema/argument assertion, span-level judge | \n
| Trajectory | \nWas the path sensible? | \nSequence, loops, redundancy, cost | \nReference-trajectory match, rubric judge | \n
The failure that motivates all three: an agent reaches the right end state by an accidental\nroute - the lucky pass. Outcome-only scoring records that as a win, and the same case\nfails next week when the accident does not recur. Conversely a trajectory-only score\npunishes a legitimately novel-but-correct path. Gate on outcome; keep step and trajectory\nas the diagnostics that tell you why the gate moved.
\nFor multi-turn or stateful agents also report pass^k (all k independent runs of the same\ncase succeed) alongside pass@k (any of k succeeded). pass@k flatters a non-deterministic\nagent; pass^k is the number that predicts production. Full treatment:\nreferences/eval-taxonomy.md.
A golden set is a reviewed, versioned, deliberately frozen collection of inputs with\ntrusted expected outputs. Frozen matters: a set that grows every sprint cannot tell you\nwhether last week's number moved because the system changed or because the set did.
\nComposition - four buckets, not one:
\n| Bucket | \nSource | \nWhy | \n
|---|---|---|
| Production sample | \nReal traffic, stratified | \nKeeps the score connected to what users actually do | \n
| Failure replays | \nEvery incident that reached a human | \nRegression protection; the easiest cases to justify | \n
| Adversarial | \nInjections, contradictions, refusal-bait | \nThe class both agents and judges fail silently on | \n
| Edge cases | \nEmpty, huge, ambiguous, multilingual | \nWhere deterministic code breaks first | \n
Sizing: 20 to start, 100-300 for a working regression set, 200-500 once you have\nproduction traffic to sample. Beyond that you are usually buying latency, not signal - add\ncases when a new failure class appears, and record in the case itself why it exists.
\nscripts/goldenset-audit.py checks a set for the rot that accumulates: duplicates, a\nbucket that quietly became 90% of the set, undated cases, and drift from a frozen manifest.
python3 scripts/goldenset-audit.py evals/golden.jsonl --json | jq '.data.findings[]'\nDepth: references/golden-datasets.md.
A judge is a measurement instrument. Instruments need calibration, and this one has\ndocumented, reproducible biases:
\n| Bias | \nWhat it does | \nMitigation | \n
|---|---|---|
| Position | \nPrefers whichever candidate was shown first | \nRun both orders and average; or score absolutely, not pairwise | \n
| Verbosity | \nRates longer answers higher regardless of quality | \nSeparate correctness from style in the rubric; penalise unsupported length | \n
| Self-preference | \nRates its own model family's output higher | \nJudge with a different family than the one under test | \n
| Scale drift | \n1-5 scores cluster and shift between model versions | \nBinary pass/fail against explicit criteria; pin the judge model version | \n
Panel vs N-identical. Three calls to the same judge with the same rubric mostly buys the\nsame bias three times. A panel with distinct lenses - one asks \"is this supported by the\nsource?\", one \"does it follow the stated policy?\", one \"would this reproduce?\" - finds\nfailure modes redundancy structurally cannot. Use N-identical only to measure the judge's\nown variance, which is a different question worth asking once.
\nWhen a judge is the wrong tool: if you can express the criterion as code, do. A judge\ncosts money, adds latency, drifts across model versions, and has variance a regex does not.\nJudges earn their place on faithfulness, tone, policy compliance, and \"is this a reasonable\nanswer to an open question\" - nowhere else.
\nCalibrate before you trust. Label 50-200 cases by hand, run the judge on the same cases,\nand compute Cohen kappa (raw agreement lies when classes are imbalanced):
\npython3 scripts/judge-calibration.py evals/labels.jsonl --min-kappa 0.6\n# exit 0 = calibrated; exit 10 = below threshold, fix the rubric before shipping it\nkappa >= 0.8 production-ready - 0.6-0.8 substantial, usable with care - below 0.6 the rubric\nis the problem, not the model. Re-sample ~50 fresh cases periodically; judges drift when the\nunderlying model version moves. Depth, including bias-probe design: references/llm-judge.md.
Measure the human-human ceiling first. A judge cannot beat the agreement two people\nachieve with each other. Two annotators on 30-50 shared cases gives you that number - and\nif it is below ~0.6, the rubric is ambiguous and every label you produce against it is\nwasted. How to run the sessions, stratify the sample, and adjudicate disagreements:\nreferences/annotation-workflow.md.
For findings rather than scores - bug reports, audit results, review comments - flip the\nprompt: ask the verifier to REFUTE, not to confirm. \"Try to refute this finding; default\nto refuted if uncertain\" kills plausible-but-wrong results that an \"is this correct?\" prompt\nwaves through, because agreement is the path of least resistance for a model.
\nThen take a majority: run 3 refuters, keep the finding only if at least 2 fail to refute it.\nPrefer perspective-diverse refuters (correctness / security / does-it-actually-reproduce)\nover three identical skeptics - same reasoning as judge panels.
\nThis composes with the parallel-work skills rather than duplicating them: fleet-ops and\nparallel-ops own the fan-out mechanics; this skill owns the scoring contract the refuters\nreturn. See references/adversarial-verification.md.
RAG is the most common eval target and the most commonly mis-measured. Scoring only the\nfinal answer averages two independent failures into one uninterpretable number:
\n| \n | Right context | \nWrong context | \n
|---|---|---|
| Answer correct | \nWorking | \nLucky - the model knew it anyway; scores as a pass | \n
| Answer wrong | \nGeneration bug - chunking, prompt, model | \nRetrieval bug - embeddings, index, query rewriting | \n
Record the retrieved chunk ids next to every answer and that opaque score becomes a 2x2\nyou can assign to a team. Gate on recall@k (a precision failure degrades an answer; a\nrecall failure makes a correct one impossible) and on citation-id validity, which is free\nand catches confident answers attached to unrelated sources. Retrieval is the one place\ndeterministic scoring genuinely dominates - you have ground-truth ids, so skip the judge.
\nThe bucket almost everyone omits: questions the corpus cannot answer. Without them the\nsuite cannot detect hallucination under retrieval failure, which is what users hit most.\nMetrics, the six failure classes, and ground-truth construction: references/retrieval-eval.md.
The rule that keeps a gate alive: a blocking check must never be flaky.
\n| Check | \nGate | \n
|---|---|
| Deterministic assertions (schema, tool-call, exact match) | \nBlocking. Any failure fails CI. | \n
| Judge metrics, first few weeks | \nAdvisory. Post the delta as a PR comment. | \n
| Judge metrics, calibrated (kappa >= 0.6) and variance-measured | \nBlocking with a margin below the rolling baseline | \n
| Cost and p95 latency per case | \nBlocking on a ceiling, advisory on the trend | \n
Set the threshold below the baseline by more than the measured noise floor: if the suite\nscores 0.88 +/- 0.03 across reruns, gate at 0.80, not 0.87. You cannot know the noise floor\nfrom a single run - commit a rolling window of run results to git and read the variance off\nit. That committed history is also what distinguishes \"today is noisy\" from \"today broke\".
\nA noise floor tells you the aggregate moved unusually far. It does not tell you the same\ncases moved. Two runs over one frozen set are paired binary outcomes, and the tool for\nthose is McNemar's exact test over the discordant pairs only.
\nThis matters because a change that breaks 8 cases and fixes 7 moves the headline score by\n0.01 - invisible against any noise floor - while silently swapping which 15 things work. The\npaired view names those 15 cases; a score comparison structurally cannot. Whether 8-vs-7 is\nsignificant is a separate question - it is not - but knowing which cases flipped is what\nlets you go and look.
\npython3 scripts/eval-baseline.py evals/history.jsonl --baseline-results base.jsonl --candidate-results new.jsonl\n# exit 10 = significant regression, and it names the cases that flipped\nAttribute cost and latency per eval run from the start. An eval suite is the only place\nyou find out the accuracy win cost 4x the tokens, and retrofitting attribution after the\nharness exists is far more annoying than a tokens_in / tokens_out / ms field per case.\nFull CI shape and the noise-floor method: references/regression-gating.md.
Once the harness measures, the obvious move is to optimise against it. That works, and it\nis also the fastest way to make a good suite useless.
\nThe loop belongs to iterate - this skill owns what goes wrong. Every\nhillclimbing failure is a property of the metric, not of the loop:
Banking noise. iterate keeps a change when the metric beats the previous best -\ncorrect for line coverage, a coin flip for an eval score. At 0.88 +/- 0.03, a measured\n0.90 is not evidence. Gate the keep decision on the noise floor instead:
python3 scripts/eval-baseline.py history.jsonl --candidate iter.jsonl --accept\n# exit 0 = KEEP (a real improvement), 10 = DISCARD (noise or worse)\n--accept deliberately inverts the CI meaning of \"noise\": CI asks did this get worse,\na hillclimb asks is this improvement real. Noise fails the second question.
Overfitting the frozen set. Split train / validation / held-out before optimising,\nnever show the validation set to whatever proposes changes, and treat held-out as a\nbudget you spend at milestones - not a dashboard.
\nKeeping a champion instead of a frontier. One aggregate best hides which cases a\ncandidate won. Retaining candidates that are best on at least one case is what stops the\nloop walling itself into a local optimum.
\nAnd the eval-design consequence: a scalar score gives an optimizer nothing to reflect on.\nA judge returning {\"reason\": ..., \"verdict\": ...} can be improved against; one returning\n0.4 cannot. That field costs nothing today and is what makes automated optimization\ntractable later.
Splits, the optimizer landscape (GEPA, MIPROv2, APE/ORPO/SPO), the pre-flight checklist,\nand the extraction trigger for a future prompt-optimization-ops: references/hillclimbing.md.
Trace-level observability and eval scoring have converged into the same products - you are\npicking one system, not two. Open-source cores worth knowing: DeepEval (pytest-native),\nMLflow (tracing, eval and prompt versioning in one OSS platform), Opik, Langfuse, Arize\nPhoenix. Commercial-first: Braintrust (dataset curation for non-engineers), AgentOps,\nLangSmith, Arize.
\nHonest default: start with a JSONL file and a 40-line runner. Adopt a platform when you\nneed shared dataset curation, trace search across production traffic, or scheduled runs -\nnot before. Which-one-when: references/tooling-landscape.md.
\n\nThe landscape moves fast. Treat every version, price and feature claim in that reference\nas needing re-verification; it carries a datestamp for exactly that reason.
\n
| Script | \nUse | \n
|---|---|
scripts/judge-calibration.py | \nJudge-vs-human agreement: Cohen kappa, confusion matrix, per-class breakdown, verbosity/position bias probes. Exit 10 = below --min-kappa. | \n
scripts/goldenset-audit.py | \nGolden-set health: duplicates, bucket balance, staleness, freeze-manifest drift. Exit 10 = findings. | \n
scripts/eval-baseline.py | \nNoise floor from run history, the threshold your gate should use, and McNemar's exact test naming the cases that flipped. Exit 10 = confirmed regression or a cost/latency ceiling breach; --accept turns it into a hillclimb keep/discard gate. | \n
All three accept --help and --json, and are offline and stdlib-only.
python3 scripts/judge-calibration.py labels.jsonl --json | jq '.data.kappa'\npython3 scripts/goldenset-audit.py golden.jsonl --freeze manifest.json\npython3 scripts/eval-baseline.py history.jsonl --json | jq '.data.recommended_threshold'\nCopy-and-adapt starting points, so the first hour goes on deciding what to measure rather\nthan on scaffolding:
\n| Asset | \nWhat it is | \n
|---|---|
assets/golden-set.example.jsonl | \n10 worked cases across all four buckets, with why and criteria filled in | \n
assets/eval-runner.template.py | \nThe 40-line runner this skill tells you to start with - two ADAPT blocks, cost/latency/pass^k pre-wired | \n
assets/judge-rubric.template.md | \nOne-criterion rubric with the bias-counter instructions and the calibration checklist | \n
assets/eval-gate.template.yml | \nGitHub Actions workflow encoding the tier ladder: deterministic blocks on push, judge advisory on PR, k=3 nightly | \n
references/eval-taxonomy.md - outcome/step/trajectory, lucky pass, pass@k vs pass^k, metric selectionreferences/golden-datasets.md - building, four-bucket composition, sizing, freeze discipline, rotreferences/llm-judge.md - bias catalog and mitigations, rubric design, panels, calibration methodreferences/adversarial-verification.md - refute-not-confirm, majority thresholds, lens diversityreferences/retrieval-eval.md - RAG: recall@k, the retrieval-vs-generation split, ground truth, failure classesreferences/annotation-workflow.md - where human labels come from: the human-human ceiling, sampling, adjudication, driftreferences/regression-gating.md - blocking vs advisory, noise floor, McNemar, CI shape, cost/latency attributionreferences/hillclimbing.md - optimising against an eval without destroying it: noise, overfitting, frontiers, optimizersreferences/tooling-landscape.md - platform comparison with verification datestamps