# Concepts — honest, variance-aware evaluation

> A reward without its uncertainty is half a measurement. Agents are stochastic;
> the same candidate scored twice gives two numbers. This note is the statistical
> backbone of `evaluate`. The implementation is `cap_evolve/stats.py`.

## Two sources of variance, one standard error

When you score a candidate on a split you are estimating a mean across *tasks*,
where each task's score is itself a mean across *trials*. There are two
independent sources of noise:

1. **Within-task (trial) variance** — run the agent on a *fixed* task k times and
   the rewards differ (sampling temperature, tool flakiness, model
   nondeterminism). Captured per task as a trial standard error.
2. **Between-task variance** — tasks differ in difficulty, so the per-task means
   spread out. Captured as the variance of the per-task means.

Reporting only one understates uncertainty. The honest figure folds both into a
**combined standard error** of the overall mean:

```
SE_total = sqrt( between_task_var / n_tasks  +  mean(per_task_SE^2) / n_tasks )
```

This is exactly `cap_evolve.stats.combined_stderr` (`stats.py:35-53`): the
between-task term is the SE of the task means; the within-task term averages each
task's squared trial SE. It is the honest SE of the number `evaluate` *reports*.
The gate's `significant` mode compares candidate-vs-current with it
(`gate.py:184-207`) — but that mode is the fallback: the loop's default is `paired`,
which recomputes an SE from the per-task deltas (`gate.py:148-182`) and never reads
this one. See the SKILL.md section "What the gate actually consumes".

**Single trial ⇒ within-task SE is 0** and pass^k/pass@k are undefined. That is
why a stochastic agent scored at `n_trials=1` produces a falsely confident
`stderr` and should never feed a significance gate.

## Small samples: where these statistics stop meaning anything

Both terms above are sample statistics, and they degrade quietly:

- **Below 2 tasks** the between-task variance is defined as 0 (`stats.py:47-50`) and
  `stats.stderr` returns 0 (`stats.py:28-30`). A 1-task val therefore reports
  `stderr = 0`, giving a significance bar of 0; the gate logs a warning and falls
  back to strict, accepting any Δ>0 (`gate.py:40-60, 186-196`).
- **Below roughly 5 tasks** the `k·SE` bar is dominated by sample size rather than by
  the effect, so it is optimistic — a couple of lucky per-task deltas clear it. There
  is no t-based small-sample correction today; issue #113 tracks adding one (and a
  minimum-val-size guard), and its own third bullet asks that the current bar at
  minimum be *documented* as optimistic on tiny val sets. This paragraph is that
  documentation.
- **An empty val** presents as fully covered (`coverage` returns 1.0 when
  `n_tasks == 0`, `loop.py:79-84`) with `reward 0.0` — also issue #113.

### The degenerate single-task delta (issue #351, open)

Under the shipped default gate (`mode: paired`, `k_se: 1.0`), a candidate that
improves exactly one val task by `m` and changes nothing else gives paired deltas
`[m, 0, …, 0]` over `n` tasks:

```
Δ̄   = m/n
var = [ m²(1 − 1/n)² + (n−1)(m/n)² ] / (n−1) = m²/n
SE  = sqrt(var/n) = m/n = Δ̄
```

`Δ̄ == SE` exactly, for every `m` and every `n` — neither a bigger gain nor a bigger
val split rescues it, because both scale identically. Since `gate.py:176` tests a
strict `Δ̄ > k·SE`, the verdict reduces to `x > x` and is settled by floating-point
representation: rejected at n=4, 8, 50; accepted at n=20; identical printed numbers
in every case. So a single-task improvement is not reliably bankable under the
current default, and a rejection of one says nothing about the edit's quality.

## pass^k vs pass@k — opposite questions

Both summarize k i.i.d. trials on a task, but they measure different things:

- **pass^k (reliability):** probability that **all** k trials pass. Introduced by
  τ-bench, which showed strong models that succeed ~50% of the time per run drop
  far lower under pass^k (e.g. GPT-4o "pass^8 < 25% in retail") — i.e. they are
  not *dependable*. Use pass^k when the agent must work *every* time (customer
  support, automation). With `c` passes of `n` trials the unbiased estimate is the
  hypergeometric `C(c,k) / C(n,k)`.
- **pass@k (capability):** probability that **at least one** of k trials passes.
  Introduced for code generation (Codex/HumanEval), where you can sample many
  candidates and keep any that works. Its unbiased estimator is
  `1 − C(n−c, k) / C(n, k)`, designed to avoid the high variance of naively
  computing `1 − (1 − c/n)^k`.

A candidate can have high pass@k (it *can* solve the task) yet low pass^k (it
*won't reliably*). cap-evolve optimizes capabilities meant to be used repeatedly,
so pass^k is the reliability signal to watch; a wide pass^1 → pass^k drop at
report time means the gain is fragile.

## Bootstrap confidence intervals (when a closed-form SE is not enough)

The combined SE assumes roughly normal task means. For small or skewed task sets,
a **percentile bootstrap** (Koehn 2004) is more robust: resample the per-task
rewards with replacement B times, recompute the mean each time, and take the
2.5th/97.5th percentiles of those means as a 95% CI. `cap_evolve.stats.bootstrap_ci`
implements this deterministically (fixed seed → reproducible CI). Koehn's point —
made for MT metrics but general — is that without resampling you cannot tell
whether a score *difference* is real or an artifact of the particular test items.

## Where the test-split refusal is enforced

Not in prose: `scripts/run.py:25` declares `--split` with
`choices=["train", "val"]`, so `--split test` exits 2 with
`invalid choice: 'test'`. Behind that, `harness.evaluate_candidate` reserves the
seal for a test split (`harness.py:240-241`) and `splits.check_test_unused` /
`rundir.begin_test_attempt` raise `TestSealError`. Note the argparse guard is
`evaluate`'s only refusal — `harness.evaluate_candidate(split="test")` is reachable
from any other caller and would write into `rollouts/test/`, which then blocks a
*legitimate* finalize (`rundir.py:385-399`) — issue #361 tracks moving the refusal
into core. Do not widen those choices.

## Sources
- τ-bench (Yao, Shinn, Razavi, Narasimhan, 2024) — pass^k reliability; the
  per-run-vs-multi-trial gap: https://arxiv.org/abs/2406.12045
- Chen et al., "Evaluating Large Language Models Trained on Code" (2021) — pass@k
  and its variance-reduced unbiased estimator: https://arxiv.org/abs/2107.03374
- Koehn, "Statistical Significance Tests for Machine Translation Evaluation"
  (EMNLP 2004) — bootstrap resampling for score differences:
  https://aclanthology.org/W04-3250/
- `cap_evolve/stats.py` — `combined_stderr`, `pass_k`, `pass_at_k`, `bootstrap_ci`
  (the single auditable place rewards are aggregated).

"""The 'evaluate' phase composes the project adapter + shared harness; it declares no
abstract methods of its own. check.py verifies the wiring instead of stubs."""

"""Contract: evaluate aggregates split × trials HONESTLY — it scores exactly the
tasks in the requested split, runs n_trials per task, reports a mean over the VALID
trials of the SCORED tasks, and reports a non-zero SE when the tasks disagree.

The last two are the properties the skill exists for: an infra-errored trial must
leave the mean (so a crash is missing data, not a capability failure of 0.0), and
the SE must actually be measured (an SE of 0 degenerates the significance gate).
"""

from __future__ import annotations

import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import Rollout, Score, Task, harness
from cap_evolve.skillcheck import Checker, import_run, temp_run_dir

# 8 ids at a 50/25/25 split give a val of 2 — enough for a between-task SE to exist
# at all (stats.combined_stderr defines it as 0 below 2 tasks).
_IDS = ("a", "b", "c", "d", "e", "f", "g", "h")


class _Adapter:
    """Synthetic adapter: reward = 1 on even trials, 0 on odd → mean 0.5 over 2 trials."""

    def tasks(self, split):
        return [Task(id=t, input={}) for t in _IDS]

    def run_target(self, task, ctx, *, seed=0):
        return Rollout(task_id=task.id, output=str(seed))

    def score(self, task, rollout):
        # An errored rollout is still handed to score() — the harness discards the
        # number afterwards, but the scorer must not crash on it.
        if rollout.output is None:
            return Score(task_id=task.id, reward=0.0)
        return Score(task_id=task.id, reward=1.0 if int(rollout.output) % 2 == 0 else 0.0)

    def materialize(self, candidate_dir):
        return {}


class _SplitAdapter(_Adapter):
    """Tasks disagree (reward keyed off the task id), so the between-task SE is > 0."""

    def score(self, task, rollout):
        return Score(task_id=task.id, reward=1.0 if task.id in ("a", "c", "e", "g") else 0.0)


class _FlakyAdapter(_Adapter):
    """One task's runner always errors: the target never ran on it."""

    def __init__(self, dead: str):
        self.dead = dead

    def run_target(self, task, ctx, *, seed=0):
        if task.id == self.dead:
            return Rollout(task_id=task.id, error="infra: runner exploded")
        return super().run_target(task, ctx, seed=seed)


def main() -> int:
    c = Checker("evaluate")
    c.require_main(import_run())

    with tempfile.TemporaryDirectory() as d:
        rd, splits = temp_run_dir(Path(d), ids=_IDS, seed=0)
        cand = Path(d) / "cand"
        cand.mkdir()

        res = harness.evaluate_candidate(_Adapter(), cand, run_dir=rd, split="val",
                                         n_trials=2, base_seed=0, tag="chk")
        # only the val split's tasks are scored
        c.check(len(res.per_task) == len(splits.val),
                f"evaluated {len(res.per_task)} tasks, val has {len(splits.val)}",
                note=f"scored exactly the val split ({len(splits.val)} tasks)")
        # 2 trials (seed 0 -> reward 1, seed 1 -> reward 0) average to 0.5 per task
        c.check(all(abs(pt["reward"] - 0.5) < 1e-9 for pt in res.per_task),
                f"per-task mean over trials wrong: {[pt['reward'] for pt in res.per_task]}",
                note="reward is the mean over n_trials per task")
        c.check(all(pt.get("n", 0) == 2 for pt in res.per_task),
                "n_trials not recorded per task")
        c.check(res.n_scored == res.n_tasks == len(splits.val),
                f"honest denominator wrong on a healthy split: "
                f"n_scored={res.n_scored} n_tasks={res.n_tasks}",
                note="n_scored == n_tasks when nothing errored")

        # An infra-errored task is MISSING DATA, not a 0.0: it leaves the mean and
        # shrinks the denominator, so coverage exposes the decimated split.
        dead = sorted(splits.val)[0]
        flaky = harness.evaluate_candidate(_FlakyAdapter(dead), cand, run_dir=rd,
                                           split="val", n_trials=2, base_seed=0, tag="flaky")
        c.check(flaky.n_scored < flaky.n_tasks and flaky.coverage < 1.0,
                f"errored task not excluded from the denominator: "
                f"n_scored={flaky.n_scored} n_tasks={flaky.n_tasks}",
                note=f"errored trials leave the mean (coverage {flaky.coverage:.2f})")
        c.check(abs(flaky.reward - 0.5) < 1e-9,
                f"errored task dragged the mean to {flaky.reward} instead of leaving it "
                "at the scored tasks' 0.5 (a crash was averaged in as a 0.0)")
        c.check((rd.rollouts / "val" / f"{dead}__flaky__t0.json").exists(),
                "errored rollout file not kept for forensics")

        # A measured SE: tasks that disagree must produce stderr > 0, otherwise the
        # significance gate has no bar and silently degenerates to strict.
        spread = harness.evaluate_candidate(_SplitAdapter(), cand, run_dir=rd, split="val",
                                            n_trials=2, base_seed=0, tag="spread")
        c.check(spread.stderr > 0.0,
                f"stderr is {spread.stderr} on a split whose tasks disagree "
                f"({[pt['reward'] for pt in spread.per_task]}) — the gate bar would be 0",
                note=f"between-task SE is measured (stderr={spread.stderr:.4f})")

    return c.emit()


if __name__ == "__main__":
    sys.exit(main())

"""evaluate — score a candidate on a split with multi-trial honesty + pass^k.

Thin wrapper over the shared harness so any host can evaluate by parsing the JSON
on stdout. Never scores the test split (that is finalize's sealed job).
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import RunDir, harness
from cap_evolve.check import load_adapter


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="evaluate")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", required=True)
    p.add_argument("--candidate", required=True, help="candidate id or dir to evaluate")
    p.add_argument("--split", default="val", choices=["train", "val"])
    p.add_argument("--n-trials", type=int, default=1)
    p.add_argument("--ks", default=None,
                   help="comma-separated k values for pass^k/pass@k "
                        "(default: 1..n-trials, so the k you paid for is reported)")
    p.add_argument("--ids", default=None,
                   help="comma-separated task ids to restrict this eval to (default: the "
                        "whole split). For agent-optimize: evaluate on train freely, on any "
                        "subset you chose yourself — by your own trajectory-similarity "
                        "clustering, or any other method — cap_evolve.harness.evaluate_candidate "
                        "already supports it. Use a tag unique to this subset eval (a fresh "
                        "candidate dir name), never one a full-split eval also writes to. NEVER "
                        "pass this for the full-val accept gate: a subset result's coverage "
                        "looks like 1.0 to gate_check.py, exactly the case its coverage guard "
                        "cannot see through.")
    args = p.parse_args(argv)
    ids = [i.strip() for i in args.ids.split(",") if i.strip()] if args.ids else None

    # ks defaults to every k the trials can support. The harness default is (1, 2),
    # which silently drops pass^3 from a --n-trials 3 run — the exact reliability
    # figure the extra trial was bought for. A k above a task's trial count is
    # omitted by aggregate_scores, so over-wide ks is safe, never a misleading 0.0.
    ks = (tuple(int(x) for x in args.ks.split(",") if x.strip()) if args.ks
          else tuple(range(1, max(1, args.n_trials) + 1)))

    if args.n_trials <= 1:
        # Loud and auditable rather than a silently falsely-confident number: with one
        # trial every per-task stderr is 0, so the combined SE is between-task only and
        # pass^k is undefined beyond k=1. Same posture as gate._warn_se_zero.
        print("evaluate: --n-trials=1 — within-task variance is unmeasured (per-task "
              "stderr=0) and pass^k is only defined at k=1. Honest only for a "
              "deterministic target; pass --n-trials 3+ for a stochastic one.",
              file=sys.stderr)

    run_dir = RunDir.open(Path(args.run_dir))
    adapter = load_adapter(Path(args.project))
    cand = Path(args.candidate)
    cand_dir = cand if cand.exists() else run_dir.candidate_dir(args.candidate)
    result = harness.evaluate_candidate(adapter, cand_dir, run_dir=run_dir,
                                        split=args.split, n_trials=args.n_trials,
                                        ks=ks, tag=cand_dir.name, ids=ids)
    print(json.dumps(result.to_dict(), indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())