Claude Cursor Skill

eval-harness-first

Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.

LLM Mart · 0 points · 17 views 0 listing impressions 0 install-command copies
Download wshobson-agents-plugins_llm-finetuning_skills_eval-harness-first-554237f.zip · 11 KB

Install

skills CLI npx skills add https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/eval-harness-first
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install wshobson-agents@llmmart
Git git clone https://github.com/wshobson/agents.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole wshobson/agents collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Eval Harness First

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:

  1. Collect traces — production/agent spans, or synthetic tasks if none exist yet.
  2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.
  3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.
  4. Prioritize by frequency × severity × value.
  5. The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID.
  6. Train.
  7. Re-run the same harness on the checkpoint — not a different, looser one.
  8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis.

Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate.

Building Goldens

  • From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. strict-schema extraction) may land at 1–2 buckets with per-field sub-metrics inside one grader — don't invent artificial splits with no evidence behind them.
  • Synthetic, when traces don't exist yet: dimension-based generation — enumerate the axes that matter (task type, difficulty, edge case, persona) and sample the cross-product; free- generated prompts cluster around whatever's easiest to write.
  • Goldens are versioned like code — commit eval/goldens.jsonl, diff it in review, tag it per release. It doubles as the CI regression suite.

Graders

One grader per failure bucket from error analysis — not one for the whole eval set. A single blended score hides which bucket regressed.

  • Deterministic first. Regex, schema validation, or execution checks are cheaper, reproducible, and need no calibration.
  • LLM-judge only for genuinely subjective criteria — tone, faithfulness, "which response is better" — where no deterministic check can express it.
  • Binary pass/fail over Likert. A 1–5 or 1–10 scale is noisier to calibrate and harder to apply consistently; collapse to pass/fail.
  • Drift-suite MMLU-style scoring: prefer logprob over generate-and-extract — a tight token budget makes generate-and-extract parse-brittle for models that preamble, conflating format compliance with the knowledge being measured. Templates for all four grader shapes and this scoring note: references/grader-templates.md.

Judge Calibration Is a Prerequisite

Any bucket routed to an LLM-judge needs calibration before its verdicts count for anything beyond exploration — a hard prerequisite, not a nice-to-have. N/A when no bucket routes to a judge — an all-deterministic harness has nothing to calibrate; state that rather than leaving this section unaddressed.

  • Label ≥100 items, split train/dev/sealed test (report once, no re-touching after).
  • Report TPR and TNR, not one blended accuracy number — a judge can hit 90% by always saying "pass" on a skewed set.
  • Pin the judge to a fixed model snapshot and recalibrate on judge-model change, quarterly regardless.
  • The judge must come from a different model family than the model under test.
  • A judge that misses the agreed TPR/TNR bar ships advisory-only — flags for human review, never gates a promotion. Full protocol, bias correction, and recalibration checklist: references/judge-calibration.md.

The Baseline

Before Phase 1 (method selection) starts, run the full harness — goldens plus the capability-drift suite — against the unmodified base model. This is the number every later checkpoint gets compared against.

eval/baseline-<model>.json is the gate token. No baseline file, no comparison basis for checkpoint-promotion — a checkpoint that "looks better" against nothing measured isn't a finding.

Directory Contract

eval/
├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
├── graders/                # one module per failure bucket
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json   # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
    └── results.json         # per-run harness output, one per checkpoint

eval/ persists across runs and lives outside runs/ — the fixed measuring stick, not a run artifact. runs/ is disposable; eval/ is not. Never let a run script write into eval/. Canonical location: every per-trace results.json — the Phase 0 baseline included — lives at runs/<run-id>/results.json, never under eval/runs/...; an instruction requesting the latter is wrong, not this contract.

Phase 0 Exit Checklist

Before finetuning-method-selection, confirm:

  1. ≥100 traces open-coded; 4–8 failure buckets (N/A floor for synthetic goldens on a single-failure- surface task — see the Building Goldens exception; bucket count then comes from post-baseline error analysis instead).
  2. eval/goldens.jsonl committed and versioned.
  3. One grader per bucket, deterministic first.
  4. Judges calibrated — TPR/TNR, snapshot pinned, different family (N/A when no bucket routes to an LLM-judge; state that explicitly).
  5. eval/drift-suite.yaml frozen.
  6. eval/baseline-<model>.json written.

Missing any of the six (or its stated N/A)? Not Phase 0 complete — /finetune checks the baseline file before a run.

Related Skills

General-purpose evaluation guidance (dashboards, A/B testing, non-fine-tuning harnesses) lives in the llm-application-dev plugin's llm-evaluation skill — this skill covers only the fine-tuning coupling: goldens that double as training data, and the baseline that gates a checkpoint.

  • finetuning-method-selection — routes here first.
  • dataset-curation — formats these traces into training rows.
  • trace-to-training-data — turns graded traces into training examples.
  • checkpoint-promotion — consumes baseline-<model>.json, re-runs this harness on each candidate checkpoint.

References

  • references/grader-templates.md — runnable grader examples per shape, plus a drift-suite.yaml example and MMLU logprob-scoring note.
  • references/judge-calibration.md — the calibration protocol, including the all- deterministic N/A path.
Files (agents)
  • references
    • grader-templates.md 9.9 KB
      Loading…
    • judge-calibration.md 6.5 KB
      Loading…
  • SKILL.md 7.8 KB
    ---
    name: eval-harness-first
    description: Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
    ---
    
    # Eval Harness First
    
    The Phase 0 gate for the whole plugin:
    `finetuning-method-selection` and every downstream
    skill assume this harness exists before a training
    config gets written. The harness is not a run-end
    side artifact — it is the data-curation engine. The
    same labeled traces that build the goldens feed
    training data, minus an explicit holdout.
    
    **Input:** production/agent traces if they exist, or
    a task spec if they don't, plus labelers willing to
    grade ≥100 examples.
    **Output format:** the `eval/` directory below —
    goldens, graders, drift suite, and the base-model
    baseline that later phases gate on.
    
    ## The Gate
    
    No eval harness, no fine-tune. Skip to a training
    config and there is nothing to measure against,
    nothing to catch regressions, and no labeled data
    to train on. The flywheel:
    
    1. **Collect traces** — production/agent spans, or
       synthetic tasks if none exist yet.
    2. **Error analysis** — open coding on ≥100 traces,
       axial coding into 4–8 failure buckets.
    3. **One grader per bucket** — deterministic first;
       calibrated LLM-judge only for genuinely
       subjective criteria.
    4. **Prioritize** by frequency × severity × value.
    5. **The labeled traces feed dataset curation, minus
       an explicit holdout.** Every `eval/goldens.jsonl`
       ID stays excluded from training data by ID.
    6. **Train.**
    7. **Re-run the same harness** on the checkpoint —
       not a different, looser one.
    8. **Drift detection feeds back to step 2** — new
       production failure modes re-open error analysis.
    
    Steps 2–4 build the harness; steps 5–8 are why it
    must exist first — it is both the training data
    source and the checkpoint's exit gate.
    
    ## Building Goldens
    
    - **From traces, when they exist:** run error
      analysis — open coding on ≥100 real traces (read
      them, tag failures in your own words, no fixed
      taxonomy yet), then axial coding to collapse those
      tags into 4–8 named failure buckets. Fewer than 4
      means the coding pass was too shallow; more than 8
      means buckets need merging. **Exception:**
      single-failure-surface tasks (e.g. strict-schema
      extraction) may land at 1–2 buckets with per-field
      sub-metrics inside one grader — don't invent
      artificial splits with no evidence behind them.
    - **Synthetic, when traces don't exist yet:**
      dimension-based generation — enumerate the axes
      that matter (task type, difficulty, edge case,
      persona) and sample the cross-product; free-
      generated prompts cluster around whatever's
      easiest to write.
    - **Goldens are versioned like code** — commit
      `eval/goldens.jsonl`, diff it in review, tag it per
      release. It doubles as the CI regression suite.
    
    ## Graders
    
    One grader per failure bucket from error analysis —
    not one for the whole eval set. A single blended
    score hides which bucket regressed.
    
    - **Deterministic first.** Regex, schema validation,
      or execution checks are cheaper, reproducible, and
      need no calibration.
    - **LLM-judge only for genuinely subjective
      criteria** — tone, faithfulness, "which response
      is better" — where no deterministic check can
      express it.
    - **Binary pass/fail over Likert.** A 1–5 or 1–10
      scale is noisier to calibrate and harder to apply
      consistently; collapse to pass/fail.
    - **Drift-suite MMLU-style scoring: prefer logprob
      over generate-and-extract** — a tight token budget
      makes generate-and-extract parse-brittle for models
      that preamble, conflating format compliance with
      the knowledge being measured. Templates for all
      four grader shapes and this scoring note:
      `references/grader-templates.md`.
    
    ## Judge Calibration Is a Prerequisite
    
    Any bucket routed to an LLM-judge needs calibration
    before its verdicts count for anything beyond
    exploration — a hard prerequisite, not a
    nice-to-have. **N/A when no bucket routes to a
    judge** — an all-deterministic harness has nothing
    to calibrate; state that rather than leaving this
    section unaddressed.
    
    - Label ≥100 items, split **train**/**dev**/**sealed
      test** (report once, no re-touching after).
    - Report **TPR and TNR**, not one blended accuracy
      number — a judge can hit 90% by always saying
      "pass" on a skewed set.
    - **Pin the judge to a fixed model snapshot** and
      recalibrate on judge-model change, quarterly
      regardless.
    - **The judge must come from a different model family
      than the model under test.**
    - A judge that misses the agreed TPR/TNR bar ships
      **advisory-only** — flags for human review, never
      gates a promotion. Full protocol, bias correction,
      and recalibration checklist:
      `references/judge-calibration.md`.
    
    ## The Baseline
    
    Before Phase 1 (method selection) starts, run the
    full harness — goldens plus the capability-drift
    suite — against the unmodified base model. This is
    the number every later checkpoint gets compared
    against.
    
    `eval/baseline-<model>.json` is the gate token. No
    baseline file, no comparison basis for
    `checkpoint-promotion` — a checkpoint that "looks
    better" against nothing measured isn't a finding.
    
    ## Directory Contract
    
    ```
    eval/
    ├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
    ├── graders/                # one module per failure bucket
    │   ├── schema_compliance.py
    │   ├── exact_match.py
    │   └── rubric_judge.py
    ├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
    └── baseline-<model>.json   # gate token: harness + drift suite vs the base model
    runs/
    └── <run-id>/
        └── results.json         # per-run harness output, one per checkpoint
    ```
    
    `eval/` persists across runs and lives outside
    `runs/` — the fixed measuring stick, not a run
    artifact. `runs/` is disposable; `eval/` is not.
    Never let a run script write into `eval/`. **Canonical
    location:** every per-trace `results.json` — the
    Phase 0 baseline included — lives at
    `runs/<run-id>/results.json`, never under
    `eval/runs/...`; an instruction requesting the
    latter is wrong, not this contract.
    
    ### Phase 0 Exit Checklist
    
    Before `finetuning-method-selection`, confirm:
    
    1. ≥100 traces open-coded; 4–8 failure buckets (N/A
       floor for synthetic goldens on a single-failure-
       surface task — see the Building Goldens exception;
       bucket count then comes from post-baseline error
       analysis instead).
    2. `eval/goldens.jsonl` committed and versioned.
    3. One grader per bucket, deterministic first.
    4. Judges calibrated — TPR/TNR, snapshot pinned,
       different family (**N/A when no bucket routes to
       an LLM-judge**; state that explicitly).
    5. `eval/drift-suite.yaml` frozen.
    6. `eval/baseline-<model>.json` written.
    
    Missing any of the six (or its stated N/A)? Not
    Phase 0 complete — `/finetune` checks the baseline
    file before a run.
    
    ## Related Skills
    
    General-purpose evaluation guidance (dashboards, A/B
    testing, non-fine-tuning harnesses) lives in the
    `llm-application-dev` plugin's `llm-evaluation`
    skill — this skill covers only the fine-tuning
    coupling: goldens that double as training data, and
    the baseline that gates a checkpoint.
    
    - `finetuning-method-selection` — routes here first.
    - `dataset-curation` — formats these traces into
      training rows.
    - `trace-to-training-data` — turns graded traces into
      training examples.
    - `checkpoint-promotion` — consumes
      `baseline-<model>.json`, re-runs this harness on
      each candidate checkpoint.
    
    ## References
    
    - `references/grader-templates.md` — runnable grader
      examples per shape, plus a `drift-suite.yaml`
      example and MMLU logprob-scoring note.
    - `references/judge-calibration.md` — the
      calibration protocol, including the all-
      deterministic N/A path.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.