# Concepts — the acceptance gate

> The gate is the single rule that decides whether a candidate edit replaces the
> current best. Get it wrong and the optimizer "improves" on noise; the held-out
> number then disappoints and you cannot say why. Implementation:
> `cap_evolve/gate.py` (`decide`).

## Why a gate at all: search amplifies noise

Optimization screens many candidates and keeps the best. If scores are noisy,
the *maximum* over many noisy candidates is biased upward even when nothing truly
improved — the more variants you try, the larger this best-of-noise inflation.
A naive "keep it if the mean went up" rule turns that statistical artifact into a
promoted candidate, and the gain evaporates on held-out data. The gate's job is
to admit only differences large enough that noise is an implausible explanation.

## The significance test (Δ > k·SE)

Each candidate carries a val mean and a standard error. The two means are
independent estimates, so the **standard error of their difference** is:

```
SE_diff = sqrt(SE_candidate^2 + SE_current^2)
```

Accept iff `Δ = candidate − current > k · SE_diff`. This is the
textual-optimization analogue of a two-sample significance test: `k` is how many
standard errors of the difference the gap must clear.

- **k = 1** (default): ~1σ — lenient; lets through gains that are *probably* real
  but lets some noise slip. Good early, when you want momentum.
- **k = 2** (≈ 95% one-sided): stricter; few false accepts, but rejects small real
  gains. Good late, or when each accept is expensive to validate.

Koehn (2004) makes the underlying point for evaluation metrics: a difference in
scores is only meaningful if it survives a significance test (he uses bootstrap
resampling). The `significant` gate enforces the same idea online, per iteration.

**The SE must be real.** With one trial per task the within-task SE is 0, so
`SE_diff` can collapse and `k·SE` → 0, silently turning `significant` into
`strict`. Run multiple trials (see the `evaluate` reference) before trusting the
significance gate.

## No-regression: the second gate

The aggregate mean is a lossy summary. A candidate can lift the mean while
*breaking* tasks the current best solved — net positive, locally harmful. The
fix is a **dual gate**, the discipline that SWE-bench-style evaluation
formalizes: a code patch is accepted only if it makes the target tests pass
(FAIL_TO_PASS) **and** leaves the previously-passing tests passing
(PASS_TO_PASS). Translated here:

> Accept only if (significance gate passes) **and** (no task in the current
> best's passing set regresses).

`diagnose` emits `kept_good` — the currently-passing task ids — exactly so the
no-regression check has a baseline to protect. Without it, hill-climbing on the
mean can quietly trade away reliability.

## The gate runs on val — never train, never test

- **train** is what the optimizer edits against. Gating acceptance on train would
  reward memorizing the data the proposal already saw — pure overfitting.
  `decide` raises `TrainGateError` if asked to gate on train.
- **test** is sealed for `finalize` (scored once). Gating on test would consume
  the held-out set as a tuning signal and make the headline number a fit metric.
- **val** is the honest middle: a held-out-from-training set that every accept
  decision is allowed to consume. It is *expected* to be slightly optimistic by
  the end of search (you selected against it) — which is precisely why the final
  number comes from the untouched test split, not val.

## Modes, briefly

| mode          | rule                              | when                                        |
|---------------|-----------------------------------|---------------------------------------------|
| `paired`      | mean(per-task Δ) > k·SE(Δ)        | **default**; both sides scored on the same val tasks, so difficulty cancels |
| `significant` | Δ > k·√(SE_c² + SE_p²)            | unpaired fallback; the two sides are independent samples |
| `strict`      | Δ > 0                             | only near-zero-variance scorers             |
| `threshold`   | Δ > T                             | you have a domain "minimum worth it"        |

Any other value raises. The bar is `Δ > k·SE` rather than `Δ > 0` because the max
over many noisy candidates is biased upward — `Δ > 0` promotes the luckiest draw.

## Sources
- Koehn, "Statistical Significance Tests for Machine Translation Evaluation"
  (EMNLP 2004) — bootstrap significance for score differences:
  https://aclanthology.org/W04-3250/
- SWE-bench (Jimenez et al., 2024) — FAIL_TO_PASS *and* PASS_TO_PASS dual-gate:
  https://arxiv.org/abs/2310.06770
- τ-bench (Yao et al., 2024) — reliability under repeated trials motivates
  variance-aware acceptance: https://arxiv.org/abs/2406.12045
- Hastie, Tibshirani, Friedman, *Elements of Statistical Learning* — why
  selection happens on validation and the test set stays sealed:
  https://hastie.su.domains/ElemStatLearn/
- `cap_evolve/gate.py` — `decide` and the `TrainGateError` guard.

"""The 'gate' phase composes the project adapter + shared harness; it declares no
abstract methods of its own. check.py verifies the wiring instead of stubs."""

"""Contract: the gate refuses split=train, and at SE=0 the significance mode
falls back to strict (accept any Δ>0) instead of silently mis-acting.
"""

from __future__ import annotations

import sys

import _bootstrap  # noqa: F401

from cap_evolve.gate import TrainGateError, decide
from cap_evolve.skillcheck import Checker, import_run


def main() -> int:
    c = Checker("gate")
    c.require_main(import_run())

    # 1. refuses gating on train
    try:
        decide(0.5, 0.9, split="train")
        c.fail("gate accepted split=train (must raise TrainGateError)")
    except TrainGateError:
        c.note("refuses split=train")

    # 2. SE=0 → strict fallback: a positive delta accepts, a zero delta does not
    d_up = decide(0.5, 0.6, split="val", mode="significant",
                  candidate_stderr=0.0, current_stderr=0.0)
    d_flat = decide(0.5, 0.5, split="val", mode="significant",
                    candidate_stderr=0.0, current_stderr=0.0)
    c.check(d_up.accept and "STRICT fallback" in d_up.reason,
            f"SE=0 positive Δ should accept via strict fallback: {d_up.to_dict()}")
    c.check(not d_flat.accept,
            f"SE=0 zero Δ must not accept: {d_flat.to_dict()}",
            note="SE=0 collapses to strict (Δ>0), not a silent pass")

    # 3. with real variance the significance bar gates a tiny improvement
    d_noise = decide(0.5, 0.52, split="val", mode="significant",
                     candidate_stderr=0.1, current_stderr=0.1, k_se=1.0)
    c.check(not d_noise.accept,
            f"tiny Δ below the SE bar should be rejected: {d_noise.to_dict()}",
            note="improvement within noise is rejected")

    return c.emit()


if __name__ == "__main__":
    sys.exit(main())

"""gate — apply the acceptance decision (always on val) and print it.

A thin, inspectable front-end to ``cap_evolve.gate.decide``. Algorithms call
the gate internally via the harness; this skill exists so an agent or a human can
reproduce/inspect a single accept/reject decision and understand the rule.

Two ways to call it:

**Scalar mode** — pass ``--current``/``--candidate`` (plus optional stderrs). This is
the unpaired significance test. It cannot express ``--mode paired``: a paired test
needs the per-task delta vector, which two scalar means do not carry.

**Rollout mode** — pass ``--run-dir --current-tag --candidate-tag``. Both sides'
``SplitResult``s are rebuilt from the persisted val rollouts, so the aligned per-task
deltas exist and ``--mode paired`` becomes reachable. This is the same gate the
deterministic loops apply (``harness.run_step`` defaults to ``paired`` whenever the
per-task data aligns), computed by the same helpers — so a human or an agent-mode
loop inspecting a decision here gets the *real* rule, not a weaker stand-in.

Prefer rollout mode whenever the rollouts exist. ``paired`` is strictly more powerful
than ``significant`` on the same data because it removes per-task difficulty variance.
"""

from __future__ import annotations

import argparse
import json
import sys

import _bootstrap  # noqa: F401

from cap_evolve.gate import decide

_MODES = ["paired", "significant", "strict", "threshold"]


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="gate")
    p.add_argument("--current", type=float, help="current best val reward (scalar mode)")
    p.add_argument("--candidate", type=float, help="candidate val reward (scalar mode)")
    p.add_argument("--mode", default="significant", choices=_MODES)
    p.add_argument("--k-se", type=float, default=1.0)
    p.add_argument("--candidate-stderr", type=float, default=0.0)
    p.add_argument("--current-stderr", type=float, default=0.0)
    p.add_argument("--threshold", type=float, default=0.0)
    p.add_argument("--run-dir", help="run dir (rollout mode: enables --mode paired)")
    p.add_argument("--current-tag", help="candidate id/tag of the current best")
    p.add_argument("--candidate-tag", help="candidate id/tag of the challenger")
    args = p.parse_args(argv)

    kw: dict = {}
    current, candidate = args.current, args.candidate
    cur_se, cand_se = args.current_stderr, args.candidate_stderr

    tags = (args.run_dir, args.current_tag, args.candidate_tag)
    if any(tags):
        if not all(tags):
            p.error("rollout mode needs --run-dir AND --current-tag AND --candidate-tag")
        from cap_evolve import RunDir
        from cap_evolve.harness import _paired_deltas, split_result_from_rollouts
        rd = RunDir.open(args.run_dir)
        cur = split_result_from_rollouts(rd, args.current_tag, "val")
        cand = split_result_from_rollouts(rd, args.candidate_tag, "val")
        # Rollouts are the source of truth here; explicit scalars would let a stale
        # number silently disagree with the deltas computed from the same files.
        current, cur_se = cur.reward, cur.stderr
        candidate, cand_se = cand.reward, cand.stderr
        kw["coverage"] = cand.coverage
        kw["run_dir"] = rd
        deltas = _paired_deltas(cur, cand)
        if deltas:
            kw["paired_deltas"] = deltas
        elif args.mode == "paired":
            # Say so instead of letting decide() quietly downgrade to `significant`.
            print(json.dumps({
                "error": "no aligned per-task val data for a paired test",
                "fix": "check both tags have val rollouts with valid trials "
                       "(tasks unscored on either side are dropped from the pairing)",
                "current_tag": args.current_tag, "candidate_tag": args.candidate_tag,
            }, indent=2))
            return 2
    elif current is None or candidate is None:
        p.error("scalar mode needs --current and --candidate "
                "(or use rollout mode: --run-dir --current-tag --candidate-tag)")
    elif args.mode == "paired":
        p.error("--mode paired needs per-task data: pass "
                "--run-dir --current-tag --candidate-tag instead of scalar means")

    d = decide(
        current, candidate, split="val", mode=args.mode, k_se=args.k_se,
        candidate_stderr=cand_se, current_stderr=cur_se,
        threshold=args.threshold, **kw,
    )
    out = d.to_dict()
    # Only report the pair count when the paired test actually ran — printing it for
    # `significant` would imply the decision used pairing when decide() ignored it.
    if args.mode == "paired" and "paired_deltas" in kw:
        out["paired_n"] = len(kw["paired_deltas"])
    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())