# Concepts — finalize and held-out sealing

> The whole pipeline exists to produce *one* number you can defend: how good is
> the optimized capability on data nothing was tuned against. finalize produces
> it, once, and the run dir enforces that "once". Implementation:
> `harness.finalize` + the reserve/commit `test_used` seal in `cap_evolve/rundir.py`.

## Why val is not the answer

During search, every accept/reject decision read the val split. Selecting the
candidate that scores best on val means val has been *optimized against* — its
score is biased upward by exactly the selection you performed (the more
candidates you screened, the larger the bias). This is the standard reason ML
keeps a third split: train fits, validation selects, **test estimates**. The test
number is trustworthy only because nothing — no edit, no acceptance, no
hyperparameter — was ever chosen using it.

## The seal: why "exactly once" is mechanical, not advisory

A held-out set stays held out only while it is untouched. Each time you score
test you create an opportunity to *act* on the result:

- "The number looks low — let me try the second-best candidate on test too."
- "Let me re-run with more trials until it stabilizes."
- "Let me double-check after one more edit."

Every one of these is a selection event. Picking the best of several test scores
is the same best-of-noise inflation the acceptance gate guards against — now
applied to the one split that was supposed to be clean. The result is no longer
an unbiased estimate; it has quietly become a fit metric, and nothing in the
output says so.

cap-evolve refuses to let this happen by accident, in three parts
(`rundir.py:358-407`). `reserve_test` *checks* the seal on every `split="test"`
evaluation without burning it; `commit_test` burns it only once `final.json` is
written, so a finalize that crashes *before* scoring is honestly retryable rather
than a destroyed headline number; and `begin_test_attempt` refuses a retry once
test rollouts exist on disk, since a crash after scoring means the held-out set
was already observed. Any second attempt raises `TestSealError`. The honesty is
enforced by the harness, not left to the operator's discipline — the one bypass,
`CAPEVOLVE_ALLOW_TEST_RESCORE=1`, is deliberately an explicit env override.

The seal counts *attempts*, not evaluations, because one honest finalize scores
test twice by design: the best candidate (`FINAL`) and the unmodified seed
(`FINAL_seed`), so the headline is a held-out improvement rather than a bare
number. Neither is a selection event.

## Selection happens before finalize

The corollary is a workflow rule: **all model selection must complete on val (or
train) before finalize runs.** Choose the single best candidate, then finalize it.
If two finalists genuinely need comparing, compare them on val or a freshly
carved held-out slice — never on test. finalize takes `run_dir.best_id` (already
chosen on val) precisely so the choice is made before the seal is touched.

## Report the uncertainty, and flag no-holdout runs

A single-trial point estimate on test is honest about the split but dishonest
about variance. Use ≥3 trials so `final.json` carries `stderr` and a pass^k
reliability figure alongside the mean — a high mean with low pass^k is a fragile
result the report should surface. And if the run was configured with no holdout
(test == train/val, e.g. to fit a tiny task set), the number is a *fit* metric,
not a held-out estimate; the report must label it so no reader mistakes it for
generalization.

## Mapping to benchmark protocol

Public benchmarks institutionalize this same seal: a hidden test set, a
submission scored once, no resubmission tuning. τ-bench additionally reports
pass^k so the headline reflects *reliability* across trials, not a single lucky
run. finalize is the local, enforced version of that protocol for a single
optimization run.

## Sources
- Hastie, Tibshirani, Friedman, *The Elements of Statistical Learning* — train
  fits / validation selects / test estimates; why test stays sealed:
  https://hastie.su.domains/ElemStatLearn/
- τ-bench (Yao et al., 2024) — scoring once and reporting pass^k reliability:
  https://arxiv.org/abs/2406.12045
- Koehn, "Statistical Significance Tests for MT Evaluation" (EMNLP 2004) — why a
  reported difference needs uncertainty, not a bare point:
  https://aclanthology.org/W04-3250/
- `cap_evolve/rundir.py` — the `test_used` flag and `TestSealError`.

"""The 'finalize' phase composes the project adapter + shared harness; it declares no
abstract methods of its own. check.py verifies the wiring instead of stubs."""

"""Contract: finalize scores the test split exactly once (seal-on-success) — a
second finalize on the same run dir raises TestSealError.
"""

from __future__ import annotations

import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import Rollout, Score, Task, TestSealError, harness
from cap_evolve.skillcheck import Checker, import_run, temp_run_dir


class _Adapter:
    def tasks(self, split):
        return [Task(id=t, input={}) for t in ("a", "b", "c", "d")]

    def run_target(self, task, ctx, *, seed=0):
        return Rollout(task_id=task.id, output="ok")

    def score(self, task, rollout):
        return Score(task_id=task.id, reward=1.0)

    def materialize(self, candidate_dir):
        return {}


def main() -> int:
    c = Checker("finalize")
    c.require_main(import_run())

    with tempfile.TemporaryDirectory() as d:
        rd, _ = temp_run_dir(Path(d), ids=("a", "b", "c", "d"), seed=0)
        adapter = _Adapter()
        best = Path(d) / "best"
        best.mkdir()

        payload = harness.finalize(adapter, run_dir=rd, best_dir=best)
        c.check("test" in payload and payload["test"]["reward"] == 1.0,
                f"finalize did not produce a test result: {payload}",
                note="test scored once on the sealed split")

        try:
            harness.finalize(adapter, run_dir=rd, best_dir=best)
            c.fail("second finalize succeeded — the test seal was not enforced")
        except TestSealError:
            c.note("second finalize refused (test seal enforced)")

    return c.emit()


if __name__ == "__main__":
    sys.exit(main())

"""finalize — score the best candidate on the SEALED test split, exactly once."""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import RunDir, harness
from cap_evolve.check import load_adapter


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="finalize")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", required=True)
    p.add_argument("--n-trials", type=int, default=1)
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))

    try:
        from capevolve_telemetry import load_observers_from_state
        for obs in load_observers_from_state(run_dir.load_observer_state()):
            run_dir.add_observer(obs)
    except Exception:  # noqa: BLE001
        pass

    adapter = load_adapter(Path(args.project))
    best_dir = run_dir.candidate_dir(run_dir.best_id)
    # Also score the unmodified seed (baseline) on the sealed test split, so the headline
    # is the honest optimized-vs-baseline improvement on held-out tasks. `baseline()` always
    # snapshots the seed, so a missing `candidates/seed` means a corrupted run dir — fail
    # fast rather than silently producing a misleading baseline==optimized comparison.
    seed_dir = run_dir.candidate_dir("seed")
    if not seed_dir.exists():
        raise FileNotFoundError(
            f"baseline 'seed' candidate not found at {seed_dir} — the run dir looks "
            "corrupted (baseline() should have snapshotted it). Refusing to finalize "
            "without a baseline to compare on the sealed test split."
        )
    payload = harness.finalize(adapter, run_dir=run_dir, best_dir=best_dir,
                               n_trials=args.n_trials, baseline_dir=seed_dir)

    run_dir.close_observers()

    print(json.dumps(payload, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())