Claude Skill

finalize

Score the best candidate on the held-out TEST split exactly once and seal the run. Use as the last evaluation step, after optimization stops. The run dir enforces the seal — a second finalize raises an error — so the headline number is produced once on data the optimizer never sa

LLM Mart · 0 points · 14 views 0 listing impressions 0 install-command copies
Download skillberry-ai-cap-evolve-skills_phases_finalize-49fcedb.zip · 9 KB

Install

skills CLI npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/finalize
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
Git git clone https://github.com/skillberry-ai/cap-evolve.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

finalize — the one honest number

Optimization hill-climbs on val: every accept decision consumed val as a tuning signal, so by the end of search val is optimistic — it has been selected against. The number you report must come from data nothing was tuned against. finalize scores the run's best candidate on the sealed test split, once, and writes final.json. That file is the run's result.

One finalize is two evals, not one

A bare test number cannot be defended — a reader cannot tell whether it beat the capability you started with. So one finalize scores test twice: the best candidate as tag FINAL, and the untouched seed candidate as FINAL_seed (harness.finalize). final.json therefore carries test, test_baseline, baseline_id, and test_delta — the held-out improvement, which is the figure report, the dashboard, and the event stream all headline. If the best candidate IS the seed (nothing was accepted), the second eval is skipped and test_delta is 0 by construction.

So budget --n-trials 3 as 3 trials × 2 candidates × |test| rollouts — twice what the flag looks like it buys on a paid benchmark.

Both evals sit inside one attempt and neither is a selection event: the delta is reported, never chosen on. That is why the seal counts attempts, not evals.

The seal (why "exactly once")

The instant test informs any choice — picking between finalists, "double- checking" a low number, re-running until it looks better — it stops being held out, because each peek is a selection event that pulls the number from an unbiased estimate toward an optimistic fit metric (references/concepts.md).

cap_evolve enforces this in three parts (rundir.py:358-407), and the split between them is the whole design:

  • reserve — every split="test" eval first checks the seal without burning it, so no phase other than finalize can reach test at all.
  • commit — the seal burns only after final.json is written, so a finalize that dies before scoring leaves it unused and is honestly retryable. A transient crash must not destroy a run's headline number.
  • attempt guard — seal-on-success alone cannot tell "crashed before scoring" from "crashed after". A real run hit the second case: a finalize killed by a timeout had already scored test, the retry scored it again, and the reported headline was that second look. begin_test_attempt refuses a retry once test rollouts exist on disk, before anything is spent.

The seal refuses that mistake by default; it is not unbypassable. CAPEVOLVE_ALLOW_TEST_RESCORE=1 is a deliberate opt-in override (rundir.py:166). Its own message promises the use "is recorded in the run" — nothing records it (issue #341), so a run that took a second look currently looks identical to an honest one. If you set it, disclose it in the write-up yourself.

Corollary: all selection happens before finalize. Choose the single best candidate on val, then finalize it. Finalists that genuinely need comparing get compared on val — never on test.

If finalize refuses

A TestSealError is three situations with three different right moves. Tell them apart from <run>/rollouts/test/ and test_used in splits.json:

State What happened Do this
no test rollouts, seal unused crashed before scoring Re-run finalize — the case seal-on-success exists for.
test rollouts exist, seal unused crashed after scoring, before commit Do not re-score. Read the rollouts under <run>/rollouts/test/ and report what that attempt already computed.
test_used: true the run is finalized Read final.json and regenerate the human artifact with report alone; cap-evolve run --resume skips finalize for you.

Never delete test rollouts or edit splits.json to get past the error — that manufactures a clean-looking number from a split that has already been seen.

Dual-mode

This phase runs two ways from the same SKILL.md: standalone as the slash command /cap-evolve:finalize (the argument-hint shows its run.py args), and orchestrator-callable — cap-evolve run / the orchestrate skill invokes the same scripts/run.py headlessly and threads the run dir between phases.

How to run

python scripts/run.py --run-dir .capevolve/run_XXXX --project .capevolve/project --n-trials 3

Multiple trials give the headline an honest stderr and a pass^k reliability figure instead of one noisy point. Under cap-evolve run the count comes from num_trials in capevolve.yaml and defaults to 1 — set it to ≥3, or the orchestrated headline ships with stderr 0: the exact single point this warns against. If the split was configured with no holdout (test == train/val) the number is a fit metric, not a held-out result; the dashboard flags it, so say so in the summary too.

Then read the result instead of just filing it: test ≈ val means the val gain generalized, test ≪ val means search overfit val — a real finding, not a reason to re-score.

References

  • references/concepts.md — why each peek biases the estimate, the train-fits / val-selects / test-estimates rationale, no-holdout runs, and how this maps to public benchmark protocol, with sources.
Files (cap-evolve)
  • references
    • concepts.md 4.4 KB
      Loading…
  • scripts
    • abstract.py 165 B
      Loading…
    • check.py 1.6 KB
      Loading…
    • run.py 1.9 KB
      Loading…
    • _bootstrap.py 3.6 KB
      Loading…
  • meta.yaml 299 B
    Loading…
  • SKILL.md 5.7 KB
    ---
    name: finalize
    description: Score the best candidate on the held-out TEST split exactly once and seal the run. Use as the last evaluation step, after optimization stops. The run dir enforces the seal — a second finalize raises an error — so the headline number is produced once on data the optimizer never saw, the way an honest benchmark result must be.
    component: phase
    argument-hint: "--run-dir DIR --project DIR"
    allowed-tools: Read, Bash
    provides: [report]
    needs: [candidate]
    sources: [tau2bench]
    ---
    
    # finalize — the one honest number
    
    Optimization hill-climbs on val: every accept decision consumed val as a tuning
    signal, so by the end of search val is *optimistic* — it has been selected
    against. The number you **report** must come from data nothing was tuned against.
    finalize scores the run's best candidate on the sealed `test` split, once, and
    writes `final.json`. That file is the run's result.
    
    ## One finalize is two evals, not one
    
    A bare test number cannot be defended — a reader cannot tell whether it beat the
    capability you started with. So one finalize scores test **twice**: the best
    candidate as tag `FINAL`, and the untouched `seed` candidate as `FINAL_seed`
    (`harness.finalize`). `final.json` therefore carries `test`, `test_baseline`,
    `baseline_id`, and `test_delta` — the held-out *improvement*, which is the figure
    `report`, the dashboard, and the event stream all headline. If the best candidate
    IS the seed (nothing was accepted), the second eval is skipped and `test_delta` is
    0 by construction.
    
    So budget `--n-trials 3` as 3 trials × **2 candidates** × |test| rollouts — twice
    what the flag looks like it buys on a paid benchmark.
    
    Both evals sit inside **one** attempt and neither is a selection event: the delta
    is *reported*, never chosen on. That is why the seal counts attempts, not evals.
    
    ## The seal (why "exactly once")
    
    The instant test informs *any* choice — picking between finalists, "double-
    checking" a low number, re-running until it looks better — it stops being held
    out, because each peek is a selection event that pulls the number from an
    unbiased estimate toward an optimistic fit metric (`references/concepts.md`).
    
    `cap_evolve` enforces this in three parts (`rundir.py:358-407`), and the split
    between them is the whole design:
    
    - **reserve** — every `split="test"` eval first *checks* the seal without burning
      it, so no phase other than finalize can reach test at all.
    - **commit** — the seal burns only after `final.json` is written, so a finalize
      that dies *before* scoring leaves it unused and is honestly retryable. A
      transient crash must not destroy a run's headline number.
    - **attempt guard** — seal-on-success alone cannot tell "crashed before scoring"
      from "crashed after". A real run hit the second case: a finalize killed by a
      timeout had already scored test, the retry scored it again, and the reported
      headline was that second look. `begin_test_attempt` refuses a retry once test
      rollouts exist on disk, before anything is spent.
    
    The seal refuses that mistake by default; it is not unbypassable.
    `CAPEVOLVE_ALLOW_TEST_RESCORE=1` is a deliberate opt-in override
    (`rundir.py:166`). Its own message promises the use "is recorded in the run" —
    nothing records it (issue #341), so a run that took a second look currently looks
    identical to an honest one. If you set it, disclose it in the write-up yourself.
    
    Corollary: **all selection happens before finalize.** Choose the single best
    candidate on val, *then* finalize it. Finalists that genuinely need comparing get
    compared on val — never on test.
    
    ## If finalize refuses
    
    A `TestSealError` is three situations with three different right moves. Tell them
    apart from `<run>/rollouts/test/` and `test_used` in `splits.json`:
    
    | State | What happened | Do this |
    |---|---|---|
    | no test rollouts, seal unused | crashed before scoring | Re-run finalize — the case seal-on-success exists for. |
    | test rollouts exist, seal unused | crashed after scoring, before commit | Do **not** re-score. Read the rollouts under `<run>/rollouts/test/` and report what that attempt already computed. |
    | `test_used: true` | the run is finalized | Read `final.json` and regenerate the human artifact with `report` alone; `cap-evolve run --resume` skips finalize for you. |
    
    Never delete test rollouts or edit `splits.json` to get past the error — that
    manufactures a clean-looking number from a split that has already been seen.
    
    ## Dual-mode
    This phase runs two ways from the **same** SKILL.md: standalone as the slash command `/cap-evolve:finalize` (the `argument-hint` shows its run.py args), and orchestrator-callable — `cap-evolve run` / the `orchestrate` skill invokes the same `scripts/run.py` headlessly and threads the run dir between phases.
    
    ## How to run
    ```
    python scripts/run.py --run-dir .capevolve/run_XXXX --project .capevolve/project --n-trials 3
    ```
    Multiple trials give the headline an honest `stderr` and a pass^k reliability
    figure instead of one noisy point. Under `cap-evolve run` the count comes from
    `num_trials` in `capevolve.yaml` and **defaults to 1** — set it to ≥3, or the
    orchestrated headline ships with `stderr` 0: the exact single point this warns
    against. If the split was configured with no holdout (test == train/val) the
    number is a *fit* metric, not a held-out result; the dashboard flags it, so say so
    in the summary too.
    
    Then read the result instead of just filing it: test ≈ val means the val gain
    generalized, test ≪ val means search overfit val — a real finding, not a reason to
    re-score.
    
    ## References
    - `references/concepts.md` — why each peek biases the estimate, the
      train-fits / val-selects / test-estimates rationale, no-holdout runs, and how
      this maps to public benchmark protocol, with sources.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.