# Concepts — baseline, splits, and headroom

> baseline owns the split — the one-time decision the rest of the run's honesty
> depends on. This note explains why the split is frozen once and seeded, why the
> headroom check matters, and how no-holdout runs must be labelled.
> Implementation: `harness.ensure_splits` + `harness.baseline`.

## Train / validation / test — the contract baseline seals

cap-evolve follows the standard three-way protocol:

- **train** — the data the optimizer edits *against* (proposes changes from).
- **val** — the data acceptance is decided on (the gate reads it every iteration).
- **test** — scored *once*, at finalize, to estimate generalization.

baseline writes this partition to `splits.json` exactly once. Freezing it has two
purposes:

1. **Disjointness.** If a later phase re-split, a task could migrate from test
   into train/val, leaking the held-out set and inflating the final number. One
   write, never rewritten, makes that impossible.
2. **Reproducibility.** A seeded split means two runs with the same seed partition
   identically, so results are comparable and bugs are reproducible. The seed is
   recorded in the run dir.

## The headroom verdict

baseline scores the *unmodified* seed on val before any optimization, then turns
that number into a budget decision it emits (`headroom`, `headroom_verdict` in the
printed JSON and a `headroom` event in the run dir):

- **`saturated`** (val + stderr ≥ 1.0): the ceiling is already reached. Further
  iterations chase noise for marginal gain — stop and save budget.
- **`floor`** (val ≤ 0): suspicious. Usually a broken adapter (wrong runner,
  mis-wired scorer) rather than a genuinely impossible task. Re-check the contract
  before spending budget.
- **`ok`**: real headroom — proceed.

It is deliberately non-fatal. The verdict is a fact about the run, recorded where
both a human and the orchestrator can read it; deciding to stop is theirs.

Recording the baseline also gives every algorithm a fixed bar: a candidate must
beat the baseline val (by the gate's significance margin) to count as progress.
Without a frozen baseline, "improvement" has no reference point.

## No-holdout runs are fit metrics, not held-out results

Sometimes the task set is too small to split three ways and the user pins all
three splits equal (fit the whole set). That is a legitimate choice, but it means
the "test" number was computed on data the optimizer tuned against — a **fit
metric**, not an estimate of generalization. baseline still runs; the report must
flag the test number accordingly so no reader mistakes it for held-out
performance. The distinction is the difference between "fits the data we have" and
"works on data we have not seen".

## Variance starts here

If the target is stochastic, score the baseline with `--n-trials >= 3`. A
single-trial baseline reports `stderr = 0`, which the gate then inherits for the
rest of the run — see `phases/gate` for what that does to `Δ > k·SE`.

## Sources
- Hastie, Tibshirani, Friedman, *The Elements of Statistical Learning* — the
  train/validation/test protocol and disjointness:
  https://hastie.su.domains/ElemStatLearn/
- τ-bench (Yao et al., 2024) — multi-trial scoring and reliability from the very
  first measurement: https://arxiv.org/abs/2406.12045
- Koehn, "Statistical Significance Tests for MT Evaluation" (EMNLP 2004) — why a
  baseline needs a standard error, not just a point: https://aclanthology.org/W04-3250/

"""The 'baseline' phase composes the project adapter + shared harness; it declares no
abstract methods of its own. check.py verifies the wiring instead of stubs."""

"""Contract: baseline freezes a deterministic seeded split — the same seed yields
the same train/val/test partition, and the split is written once. It also refuses
what it must refuse: a red `cap-evolve check`, and a ratio split with no val/test.
"""

from __future__ import annotations

import json
import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import RunDir, harness
from cap_evolve.check import load_adapter
from cap_evolve.skillcheck import Checker, import_run, quiet
from cap_evolve.splits import make_splits

ADAPTER = '''
import random  # noqa: F401
from cap_evolve import CapabilityAdapter
from cap_evolve.types import Task, Rollout, Score


class Adapter(CapabilityAdapter):
    def tasks(self, split):
        return [Task(id=f"t{i}", input="x", target="1") for i in range(N)]

    def run_target(self, task, ctx, *, seed=0):
        return Rollout(task_id=task.id, output="1")

    def score(self, task, rollout):
        return Score(task_id=task.id, reward=REWARD, feedback="ok")
'''


def _project(tmp: Path, name: str, *, n: int, reward: str) -> Path:
    proj = tmp / name
    (proj / "adapters").mkdir(parents=True, exist_ok=True)
    (proj / "adapters" / "adapter.py").write_text(
        ADAPTER.replace("N", str(n)).replace("REWARD", reward), encoding="utf-8")
    cap = proj / "seed_capability"
    cap.mkdir(exist_ok=True)
    (cap / "policy.txt").write_text("seed\n", encoding="utf-8")
    return proj


def main() -> int:
    c = Checker("baseline")
    run = import_run()
    c.require_main(run)

    ids = [f"t{i}" for i in range(12)]
    s1 = make_splits(list(ids), seed=7, ratios=(0.5, 0.25, 0.25))
    s2 = make_splits(list(ids), seed=7, ratios=(0.5, 0.25, 0.25))
    s3 = make_splits(list(ids), seed=8, ratios=(0.5, 0.25, 0.25))
    c.check((s1.train, s1.val, s1.test) == (s2.train, s2.val, s2.test),
            "same seed produced different splits (non-deterministic)",
            note="seeded split is deterministic")
    c.check((s1.train, s1.val, s1.test) != (s3.train, s3.val, s3.test),
            "different seeds produced identical splits (seed ignored)")
    c.check(not (set(s1.train) & set(s1.test)), "train/test overlap in a held-out split")

    with tempfile.TemporaryDirectory() as d:
        tmp = Path(d)
        # Written once: a second ensure_splits with a DIFFERENT seed and ratios must
        # return the frozen partition, not re-partition (harness.py:114-115).
        rd = RunDir.create(tmp / "once", ts="b")
        proj = _project(tmp, "green", n=12, reward="1.0")
        adapter = load_adapter(proj)
        first = harness.ensure_splits(adapter, rd, seed=1, ratios=(0.5, 0.25, 0.25))
        again = harness.ensure_splits(adapter, rd, seed=99, ratios=(0.8, 0.1, 0.1))
        c.check((first.train, first.val, first.test) == (again.train, again.val, again.test),
                "ensure_splits re-partitioned an existing run dir (split NOT written once)",
                note="split written once: a second ensure_splits returns the frozen partition")
        c.check(rd.read_splits().train == first.train,
                "splits did not round-trip through the run dir",
                note="split frozen + reloadable from the run dir")

        # A red `cap-evolve check` must stop baseline BEFORE a run dir exists (#358).
        red = _project(tmp, "red", n=12, reward="random.random()")
        base = tmp / "red_base"
        rc = run.main(["--base", str(base), "--project", str(red),
                       "--capability", str(red / "seed_capability"), "--run-ts", "x"])
        c.check(rc != 0, "baseline accepted a red cap-evolve check (non-deterministic scorer)",
                note="red adapter refused: baseline exits non-zero before freezing a split")
        c.check(not list(base.glob("run_*")), "baseline created a run dir despite a red check")

        # A ratio split with no val (n=2 -> 1/0/1) must be refused, not silently run.
        tiny = _project(tmp, "tiny", n=2, reward="1.0")
        rc = run.main(["--base", str(tmp / "tiny_base"), "--project", str(tiny),
                       "--capability", str(tiny / "seed_capability"), "--run-ts", "x"])
        c.check(rc != 0, "baseline accepted a ratio split with an empty val set",
                note="degenerate ratio split (empty val/test) refused")

        # Green path: runs, warns about the tiny val, and EMITS the headroom verdict.
        with quiet() as buf:
            rc = run.main(["--base", str(tmp / "ok_base"), "--project", str(proj),
                           "--capability", str(proj / "seed_capability"), "--run-ts", "x"])
        out = json.loads(buf.getvalue()) if rc == 0 else {}
        c.check(rc == 0, "baseline failed on a healthy 12-task project")
        c.check(out.get("headroom_verdict") == "saturated" and out.get("headroom") == 0.0,
                f"headroom not emitted for a seed that already scores 1.0 on val: {out!r}",
                note="headroom is computed and emitted (saturated at val=1.0)")
        events = (Path(out.get("run_dir", tmp)) / "events.jsonl")
        text = events.read_text(encoding="utf-8") if events.exists() else ""
        c.check('"headroom"' in text, "no headroom event logged for orchestrate to read")
        c.check("val has only" in text, "no splits_warning logged for a 3-task val split")

    return c.emit()


if __name__ == "__main__":
    sys.exit(main())

"""baseline — create the run dir, freeze the splits, score the seed on val.

This establishes the starting point every algorithm compares against. It is the
first step that touches data, so it owns split creation (seeded, written once).
Prints the run-dir path and the baseline val score as JSON.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import Budget, RunDir, harness
from cap_evolve.check import load_adapter, run_check


def _refuse_degenerate_split(splits, run_dir) -> bool:
    """Refuse a ratio split with no val or no test; warn on a tiny val (#113).

    A val-gated run with zero val tasks has nothing to decide on, and a sealed-test
    run with zero test tasks produces its headline number over nothing — both fail
    silently today because ``make_splits`` clamps sizes without a floor
    (``splits.py:113-119``). Failing here costs nothing; failing at finalize costs the
    whole run. A pinned ``--split-ids`` may be deliberately degenerate (the no-holdout
    case), so only the ratio path is guarded.
    """
    if not splits.val or not splits.test:
        msg = (f"degenerate ratio split: train={len(splits.train)} val={len(splits.val)} "
               f"test={len(splits.test)} — the val gate and the sealed test number both "
               "need at least one task. Add tasks, change --ratios, or pin --split-ids "
               "deliberately.")
        run_dir.log_event("splits_warning", msg=msg)
        print(json.dumps({"step": "baseline", "error": msg}, indent=2), file=sys.stderr)
        return True
    if len(splits.val) < 5:
        run_dir.log_event(
            "splits_warning",
            msg=(f"val has only {len(splits.val)} task(s) — the gate's Δ > k·SE bar is "
                 "optimistic at this n, and a one-task improvement cannot reliably clear "
                 "it at all (#351). Prefer >= 5 val tasks."))
    return False


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="baseline")
    p.add_argument("--base", default=".capevolve", help="dir under which run_* is created")
    p.add_argument("--project", required=True, help="dir with adapters/adapter.py")
    p.add_argument("--capability", required=True, help="seed capability dir")
    p.add_argument("--seed", type=int, default=0)
    p.add_argument("--ratios", default="0.5,0.25,0.25")
    p.add_argument("--split-ids", default=None,
                   help="JSON file {train:[],val:[],test:[]} to pin the split explicitly")
    p.add_argument("--reuse-baseline", default=None,
                   help="prior run dir: reuse its splits/baseline/seed/val-rollouts and "
                        "SKIP the baseline eval (algorithm starts at iter 1 on it)")
    p.add_argument("--n-trials", type=int, default=1)
    p.add_argument("--max-iterations", type=int, default=10)
    p.add_argument("--stall", type=int, default=0)
    p.add_argument("--max-metric-calls", type=int, default=0, help="0 = unlimited")
    p.add_argument("--max-usd", type=float, default=0.0,
                   help="0 = unlimited; total spend cap (runner + optimizer + intake)")
    p.add_argument("--max-optimizer-usd", type=float, default=0.0,
                   help="0 = off; separate cap on optimizer spend alone")
    p.add_argument("--stop-at-reward", type=float, default=0.0,
                   help="0 = off; stop the loop as soon as the best val reward reaches this")
    p.add_argument("--run-ts", default=None, help="fixed timestamp for reproducible run dirs")
    p.add_argument("--resume", action="store_true",
                   help="reopen an existing run dir instead of failing; skip the baseline "
                        "eval when it already ran (baseline.json present)")
    p.add_argument("--spec", default=None,
                   help="path to capevolve.yaml spec (for observer config)")
    args = p.parse_args(argv)

    # The hard gate, on THIS path too. `cap-evolve run` checks the adapter before it
    # calls us (cli.py:721-726), but /cap-evolve:baseline is reachable directly and the
    # needs/provides DAG validates declared order, not runtime state — so without this
    # the standalone chain would freeze a split against a knowingly-broken adapter and
    # every number in the run would be measured against it (#358). Gate before the run
    # dir exists so a red check leaves nothing behind.
    rep = run_check(Path(args.project))
    if not rep.ok:
        print(json.dumps({"step": "baseline", "error": "check failed",
                          "report": rep.to_dict()}, indent=2), file=sys.stderr)
        return 1

    Path(args.base).mkdir(parents=True, exist_ok=True)
    budget = Budget(max_iterations=args.max_iterations, stall=args.stall,
                    max_metric_calls=args.max_metric_calls, max_usd=args.max_usd,
                    max_optimizer_usd=args.max_optimizer_usd, stop_at_reward=args.stop_at_reward)
    run_dir = RunDir.create(Path(args.base), ts=args.run_ts, budget=budget, exist_ok=args.resume)

    try:
        try:
            from cap_evolve.specfile import read_yaml
            from capevolve_telemetry import load_observers, load_observers_from_state
            spec_path = Path(args.spec) if args.spec else Path(args.project, "capevolve.yaml")
            spec_text = spec_path.read_text(encoding="utf-8")
            full_spec = read_yaml(spec_text)
            obs_config = full_spec.get("observers")
            caps = full_spec.get("capabilities")
            run_name = str(full_spec.get("run_name", "")).strip()
            if not run_name:
                parts = [run_dir.root.name]
                for key in ("algorithm_skill", "optimizer_skill", "target_model"):
                    v = str(full_spec.get(key, "")).strip()
                    if v:
                        parts.append(v)
                if isinstance(caps, list):
                    parts.append("+".join(str(c) for c in caps))
                elif caps:
                    parts.append(str(caps))
                run_name = " | ".join(parts)
            run_tags = {}
            for key in ("algorithm_skill", "optimizer_skill", "optimizer_model",
                         "target_model", "max_iterations", "gate_mode", "num_trials",
                         "dataset_source"):
                v = full_spec.get(key)
                if v is not None and str(v).strip():
                    run_tags[key] = str(v)
            if caps:
                run_tags["capabilities"] = "+".join(str(c) for c in caps) if isinstance(caps, list) else str(caps)
            observers = (
                load_observers_from_state(run_dir.load_observer_state())
                if args.resume
                else load_observers(
                    obs_config,
                    run_dir_root=str(run_dir.root),
                    run_name=run_name,
                    run_tags=run_tags,
                )
            )
            for obs in observers:
                run_dir.add_observer(obs)
        except Exception:  # noqa: BLE001
            pass

        # Resume fast-path: baseline already ran → the split is frozen, the seed is scored,
        # best_id is set. Re-print the recorded baseline and skip the (expensive) eval so the
        # algorithm resumes straight from the current best. state.json is left untouched.
        if args.resume and (run_dir.root / "baseline.json").exists():
            splits = run_dir.read_splits()
            recorded = json.loads((run_dir.root / "baseline.json").read_text(encoding="utf-8"))
            print(json.dumps({
                "run_dir": str(run_dir.root),
                "splits": {"train": len(splits.train), "val": len(splits.val), "test": len(splits.test)},
                "baseline_val": recorded.get("val", {}),
                "resumed": True,
            }, indent=2))
            return 0

        adapter = load_adapter(Path(args.project))

        # --reuse-baseline: copy a prior run's frozen split + baseline + seed snapshot +
        # seed val rollouts into this fresh run dir and SKIP the (expensive) baseline eval.
        if args.reuse_baseline:
            result = harness.reuse_baseline(Path(args.reuse_baseline), run_dir=run_dir)
            splits = run_dir.read_splits()
            print(json.dumps({
                "run_dir": str(run_dir.root),
                "splits": {"train": len(splits.train), "val": len(splits.val), "test": len(splits.test)},
                "baseline_val": result.to_dict(),
                "reused_baseline_from": str(args.reuse_baseline),
            }, indent=2))
            return 0

        ratios = tuple(float(x) for x in args.ratios.split(","))
        split_ids = None
        if args.split_ids:
            # Resolve the split-ids path robustly: as given (absolute or cwd-relative),
            # else relative to the project dir. `cap-evolve run` invokes baseline with
            # cwd=workdir, so a project-relative `split_ids_file: split_ids.json` in
            # capevolve.yaml would otherwise miss — this lets users author it naturally.
            sp = Path(args.split_ids)
            if not sp.exists():
                cand = Path(args.project) / args.split_ids
                if cand.exists():
                    sp = cand
            split_ids = json.loads(sp.read_text(encoding="utf-8"))
        splits = harness.ensure_splits(adapter, run_dir, seed=args.seed, ratios=ratios,
                                       split_ids=split_ids)
        if split_ids is None and _refuse_degenerate_split(splits, run_dir):
            return 1
        # Resolve the seed capability dir robustly: as given (absolute/cwd-relative),
        # else relative to the project dir. `cap-evolve run` invokes baseline with
        # cwd=workdir, so a project-relative `capability_path: seed_capability` in
        # capevolve.yaml would otherwise miss — let users author it naturally.
        cap_path = Path(args.capability)
        if not cap_path.exists():
            cand = Path(args.project) / args.capability
            if cand.exists():
                cap_path = cand
        result = harness.baseline(adapter, cap_path, run_dir=run_dir, n_trials=args.n_trials)

        # Headroom: the budget decision this phase exists to make. Saturated => every
        # later Δ chases noise; floor => usually a broken adapter, not a hard task.
        # Emitted, not just advised, so `cap-evolve run` / orchestrate can stop on it
        # without a human reading the number. Non-fatal: recording it is the job.
        headroom = round(max(0.0, 1.0 - result.reward), 4)
        verdict = ("saturated" if result.reward + max(result.stderr, 0.0) >= 1.0
                   else "floor" if result.reward <= 0.0 else "ok")
        run_dir.log_event("headroom", headroom=headroom, verdict=verdict, val=result.reward)

        print(json.dumps({
            "run_dir": str(run_dir.root),
            "splits": {"train": len(splits.train), "val": len(splits.val), "test": len(splits.test)},
            "baseline_val": result.to_dict(),
            "headroom": headroom,
            "headroom_verdict": verdict,
        }, indent=2))
        return 0
    finally:
        close_observers = getattr(run_dir, "close_observers", None)
        if callable(close_observers):
            close_observers()


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())