Optimization hill-climbs on val: every accept decision consumed val as a tuning\nsignal, so by the end of search val is optimistic — it has been selected\nagainst. The number you report must come from data nothing was tuned against.\nfinalize scores the run's best candidate on the sealed test split, once, and\nwrites final.json. That file is the run's result.
A bare test number cannot be defended — a reader cannot tell whether it beat the\ncapability you started with. So one finalize scores test twice: the best\ncandidate as tag FINAL, and the untouched seed candidate as FINAL_seed\n(harness.finalize). final.json therefore carries test, test_baseline,\nbaseline_id, and test_delta — the held-out improvement, which is the figure\nreport, the dashboard, and the event stream all headline. If the best candidate\nIS the seed (nothing was accepted), the second eval is skipped and test_delta is\n0 by construction.
So budget --n-trials 3 as 3 trials × 2 candidates × |test| rollouts — twice\nwhat the flag looks like it buys on a paid benchmark.
Both evals sit inside one attempt and neither is a selection event: the delta\nis reported, never chosen on. That is why the seal counts attempts, not evals.
\nThe instant test informs any choice — picking between finalists, \"double-\nchecking\" a low number, re-running until it looks better — it stops being held\nout, because each peek is a selection event that pulls the number from an\nunbiased estimate toward an optimistic fit metric (references/concepts.md).
cap_evolve enforces this in three parts (rundir.py:358-407), and the split\nbetween them is the whole design:
split=\"test\" eval first checks the seal without burning\nit, so no phase other than finalize can reach test at all.final.json is written, so a finalize\nthat dies before scoring leaves it unused and is honestly retryable. A\ntransient crash must not destroy a run's headline number.begin_test_attempt refuses a retry once test\nrollouts exist on disk, before anything is spent.The seal refuses that mistake by default; it is not unbypassable.\nCAPEVOLVE_ALLOW_TEST_RESCORE=1 is a deliberate opt-in override\n(rundir.py:166). Its own message promises the use \"is recorded in the run\" —\nnothing records it (issue #341), so a run that took a second look currently looks\nidentical to an honest one. If you set it, disclose it in the write-up yourself.
Corollary: all selection happens before finalize. Choose the single best\ncandidate on val, then finalize it. Finalists that genuinely need comparing get\ncompared on val — never on test.
\nA TestSealError is three situations with three different right moves. Tell them\napart from <run>/rollouts/test/ and test_used in splits.json:
| State | \nWhat happened | \nDo this | \n
|---|---|---|
| no test rollouts, seal unused | \ncrashed before scoring | \nRe-run finalize — the case seal-on-success exists for. | \n
| test rollouts exist, seal unused | \ncrashed after scoring, before commit | \nDo not re-score. Read the rollouts under <run>/rollouts/test/ and report what that attempt already computed. | \n
test_used: true | \nthe run is finalized | \nRead final.json and regenerate the human artifact with report alone; cap-evolve run --resume skips finalize for you. | \n
Never delete test rollouts or edit splits.json to get past the error — that\nmanufactures a clean-looking number from a split that has already been seen.
This phase runs two ways from the same SKILL.md: standalone as the slash command /cap-evolve:finalize (the argument-hint shows its run.py args), and orchestrator-callable — cap-evolve run / the orchestrate skill invokes the same scripts/run.py headlessly and threads the run dir between phases.
python scripts/run.py --run-dir .capevolve/run_XXXX --project .capevolve/project --n-trials 3\nMultiple trials give the headline an honest stderr and a pass^k reliability\nfigure instead of one noisy point. Under cap-evolve run the count comes from\nnum_trials in capevolve.yaml and defaults to 1 — set it to ≥3, or the\norchestrated headline ships with stderr 0: the exact single point this warns\nagainst. If the split was configured with no holdout (test == train/val) the\nnumber is a fit metric, not a held-out result; the dashboard flags it, so say so\nin the summary too.
Then read the result instead of just filing it: test ≈ val means the val gain\ngeneralized, test ≪ val means search overfit val — a real finding, not a reason to\nre-score.
\nreferences/concepts.md — why each peek biases the estimate, the\ntrain-fits / val-selects / test-estimates rationale, no-holdout runs, and how\nthis maps to public benchmark protocol, with sources.