# Concepts — the hard gate before budget

> implement-and-check verifies the measurement apparatus before any measurement
> is trusted. A green check is the only honest entry into the optimization loop.
> Implementation: `cap_evolve.check.run_check` + each skill's `scripts/check.py`.

## What a stub actually costs

The method list itself is in SKILL.md step 1 and `docs/ADAPTER_CONTRACT.md`. What matters
here is what each one being fake does to the number:

| method                             | failure if stubbed or fake                          |
|------------------------------------|-----------------------------------------------------|
| `tasks(split)`                     | mean over nothing; unstable or non-disjoint split   |
| `run_target(task, ctx, *, seed=0)` | no behavior to score — every rollout is empty       |
| `score(task, rollout)`             | every candidate scores the same; the gate is blind  |

If any required method is a stub, the optimization still *runs* — it just produces a number
that measures nothing. The whole point of a pre-budget gate is to make that
failure loud and early instead of silent and expensive.

## Why each check exists

- **No stubs.** A `NotImplementedError` or empty body returns nothing; downstream
  the reward is vacuous. The check refuses to call a method that was never filled.
- **`tasks` non-empty and stable.** The split is computed from the task list. If
  `tasks()` is empty, there is nothing to average; if it shuffles between calls,
  the split is not reproducible and train/val/test stop being disjoint across
  reruns.
- **`materialize()` probed.** An edit that cannot be materialized onto a capability
  copy cannot be evaluated — the loop would propose into the void. The check calls it
  against a temp copy; because `materialize` is pure, the host is never mutated. This is a
  **probe, not an assertion**: a raise is reported as a note and does NOT fail the check
  (`check.py:166-167`), because a real adapter may legitimately need its full environment.

## Scorer determinism vs target stochasticity — a crucial distinction

These are *not* the same thing, and the check only forbids one:

- **Target (agent) stochasticity is expected and legitimate.** A sampling LLM
  agent gives different rollouts each run. That variance is *measured*, not
  banned — it is exactly what multi-trial evaluation, combined standard error,
  and pass^k exist to quantify.
- **Scorer nondeterminism is a bug.** If `score(task, rollout)` returns different
  rewards for the *same* rollout, the "reward" includes noise that originates in
  the measuring instrument, not in the agent. The optimizer cannot learn against
  a ruler that changes length. The check scores a fixed rollout twice and requires
  agreement.

A scorer that calls an LLM judge can still be deterministic enough: fix the
judge's decoding (temperature 0) or average enough judge samples that the per-call
variance is negligible, and treat any residual as part of the (measured) trial
variance rather than smuggling it into a single score.

## Validation gate before budget — the discipline

Self-improving systems that skip a wiring check tend to "improve" against broken
measurements and report gains that vanish on inspection. The fix, common to
skill-/agent-optimization frameworks, is a non-negotiable validation gate: prove
the contract holds, *then* spend. implement-and-check is that gate for cap-evolve —
the analog of running a test suite green before trusting a benchmark built on top
of it. The no-gold-leak rule from intake also belongs here in spirit: a scorer can
pass the determinism check and still corrupt the run if its feedback hands the
optimizer the answer, so verify feedback stays at the level of *why it failed*,
never *what the answer was*.

## Sources
- GEPA: Reflective Prompt Evolution (Agrawal et al., 2025) — the loop is only as
  honest as the feedback/score it runs on: https://arxiv.org/abs/2507.19457
- SWE-bench (Jimenez et al., 2024) — execution-based, deterministic scoring as the
  precondition for a meaningful benchmark: https://arxiv.org/abs/2310.06770
- τ-bench (Yao et al., 2024) — separating agent stochasticity (measured via
  trials) from measurement error: https://arxiv.org/abs/2406.12045

"""The 'implement-and-check' phase composes the project adapter + shared harness; it declares no
abstract methods of its own. check.py verifies the wiring instead of stubs."""

"""Contract: implement-and-check is a real gate, not a no-op.

Asserts the three behaviours the phase actually promises:
  1. a project with NO adapter is refused;
  2. a LOADABLE adapter whose method raises the IMPLEMENT-ME marker is named in
     ``rep.stubs`` (the old fixture was an ABC with unimplemented abstractmethods, so
     instantiation raised TypeError and ``stub_methods`` was never reached);
  3. a non-deterministic ``score()`` yields a "non-deterministic" problem — the one
     assertion the whole honesty story of this phase rests on.

Known gaps in the core gate are tracked in issue #358 and documented in SKILL.md under
"What the gate does and does not guarantee"; they are deliberately NOT asserted here.
"""

from __future__ import annotations

import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve.check import run_check
from cap_evolve.skillcheck import Checker, import_run

_HEAD = (
    "from cap_evolve import CapabilityAdapter\n"
    "from cap_evolve.types import Task, Rollout, Score\n"
)

# Loadable (all three abstract methods overridden) but `score` re-raises the marker.
_MARKER_STUB = _HEAD + (
    "class Adapter(CapabilityAdapter):\n"
    "    def tasks(self, split): return [Task(id='t1', input='1+1', target='2')]\n"
    "    def run_target(self, task, ctx, *, seed=0): return Rollout(task_id=task.id, output='2')\n"
    "    def score(self, task, rollout):\n"
    "        raise NotImplementedError('IMPLEMENT ME: score(task, rollout)')\n"
)

_RNG_SCORER = _HEAD + (
    "import random\n"
    "class Adapter(CapabilityAdapter):\n"
    "    def tasks(self, split): return [Task(id='t1', input='1+1', target='2')]\n"
    "    def run_target(self, task, ctx, *, seed=0): return Rollout(task_id=task.id, output='2')\n"
    "    def score(self, task, rollout):\n"
    "        return Score(task_id=task.id, reward=random.random(), feedback='rng')\n"
)


def _project(tmp: str, source: str) -> Path:
    proj = Path(tmp) / "project"
    (proj / "adapters").mkdir(parents=True)
    (proj / "adapters" / "adapter.py").write_text(source, encoding="utf-8")
    return proj


def main() -> int:
    c = Checker("implement-and-check")
    c.require_main(import_run())

    # 1. No adapter at all must not pass the gate.
    with tempfile.TemporaryDirectory() as d:
        empty = Path(d) / "project"
        empty.mkdir()
        rep = run_check(empty)
        c.check(not rep.ok and bool(rep.problems),
                "check passed a project with no adapter (gate is a no-op)",
                note="refuses a project with no adapter")

    # 2. A loadable, marker-stubbed method must be NAMED in rep.stubs.
    with tempfile.TemporaryDirectory() as d:
        rep = run_check(_project(d, _MARKER_STUB))
        c.check(not rep.ok and "score" in rep.stubs,
                f"stubbed score() not reported in stubs (got stubs={rep.stubs}, "
                f"problems={rep.problems})",
                note="names the unimplemented method in stubs[]")

    # 3. A non-deterministic scorer must produce a "non-deterministic" problem.
    with tempfile.TemporaryDirectory() as d:
        rep = run_check(_project(d, _RNG_SCORER))
        c.check(not rep.ok and any("non-deterministic" in p for p in rep.problems),
                f"RNG scorer passed the determinism probe (problems={rep.problems})",
                note="detects a non-deterministic score()")

    # 4. SKILL.md's account of WHERE the gate is enforced must match the code. PR #374
    #    added run_check() to baseline/scripts/run.py after this SKILL.md was written,
    #    so the "the standalone chain is ungated" paragraph went stale and told the
    #    reader a closed hole was still open.
    skill = (Path(__file__).resolve().parents[1] / "SKILL.md").read_text(encoding="utf-8")
    gated = "run_check(" in (Path(__file__).resolve().parents[2] / "baseline"
                             / "scripts" / "run.py").read_text(encoding="utf-8")
    claims_ungated = ("contains no check" in skill
                      or "has no gate of its own" in skill)
    c.check(gated != claims_ungated,
            "SKILL.md and baseline/scripts/run.py disagree about whether the standalone "
            f"chain is gated (baseline calls run_check={gated}, "
            f"SKILL.md says ungated={claims_ungated})",
            note="SKILL.md's standalone-gate claim matches baseline/scripts/run.py")

    return c.emit()


if __name__ == "__main__":
    sys.exit(main())

"""Pipeline wiring self-test — run AFTER ``cap-evolve check`` is green.

``cap-evolve check`` proves the adapter contract holds. This self-test proves the
*pipeline plumbing around it* is wired: the optimizer would actually receive the
trajectories, the capability guidance, and a fully-rendered (no leftover
placeholder) instructions prompt.

A full one-iteration optimization (even with the ``mock`` optimizer) needs a
baseline, a frozen split, and a run dir — none of which exist yet at
implement-and-check time, and building them is benchmark-specific. So this test
does the focused, benchmark-AGNOSTIC equivalent: it exercises the same code paths
that build the optimizer's working dir, using the real harness renderer, and
reports precisely which artifact is missing so the intake agent can iterate.

It asserts:
  1. ``capevolve.yaml`` exists (intake must scaffold the spec);
  2. the optimizer-prompt template named by ``optimizer_instructions_file`` EXISTS and
     still carries its ``{{...}}`` placeholders (intake must NOT delete them);
  3. rendering that template through the REAL harness renderer leaves NO ``{{``
     placeholder behind (the harness fills them per iteration);
  4. the adapter EITHER defines ``trajectories()`` (returns the native traj dir) OR
     intentionally inherits the base default (cap-evolve falls back to its own
     per-rollout JSON) — both are valid; we just report which.

Assertions 2-3 are SKIPPED WITH A NOTE for an algorithm that never reads the template
(see ``_TEMPLATE_CONSUMERS``): this phase declares ``algorithms: ["*"]``, so failing a
gepa/evograph run on an artifact it will never open would be an algorithm-specific
false negative. Assertions 1 and 4 always run.

Exit 0 = wiring green; non-zero = a named artifact is missing/broken.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import specfile
from cap_evolve.adapter import CapabilityAdapter
from cap_evolve.check import load_adapter
from cap_evolve.harness import _focus_instructions
from cap_evolve.loop import SplitResult

# Algorithms whose runner actually reads ``optimizer_instructions_file``. ``cli.py`` passes
# ``--instructions-file`` only when the algorithm is hill-climb, and that flag exists in
# exactly one algorithm's argparse (``skills/algorithms/hill-climb/scripts/run.py``). Add a
# name here when another algorithm's run.py grows the flag.
_TEMPLATE_CONSUMERS = frozenset({"hill-climb"})


def _synthetic_val() -> SplitResult:
    """A SplitResult with a failing + a flaky + a solid task so EVERY dynamic block
    of the template (focus summary, failures index) is exercised when rendering."""
    return SplitResult(
        split="val",
        reward=0.5,
        stderr=0.1,
        per_task=[
            {"task_id": "t_fail", "reward": 0.0, "trial_rewards": [0.0],
             "feedback": "synthetic always-failing task"},
            {"task_id": "t_flaky", "reward": 0.5, "trial_rewards": [1.0, 0.0],
             "feedback": "synthetic flaky task"},
            {"task_id": "t_solid", "reward": 1.0, "trial_rewards": [1.0],
             "feedback": "synthetic solid task"},
        ],
    )


def selftest(project: Path) -> dict:
    project = Path(project)
    problems: list[str] = []
    notes: list[str] = []

    # 1) template scaffolded + placeholders intact
    spec_path = project / "capevolve.yaml"
    spec = specfile.read_yaml(spec_path.read_text(encoding="utf-8")) if spec_path.exists() else {}
    if not spec_path.exists():
        problems.append(f"missing spec: {spec_path} (intake must scaffold capevolve.yaml)")

    # The template checks only apply to an algorithm that actually READS the template.
    # ``cli.py`` passes ``--instructions-file`` for hill-climb only, and that flag exists
    # in exactly one algorithm's argparse — so gating a gepa/evograph/agent-optimize run on
    # an artifact it will never open is a benchmark/algorithm-specific false failure.
    algo = str(spec.get("algorithm_skill") or "hill-climb")
    if algo not in _TEMPLATE_CONSUMERS:
        notes.append(f"algorithm '{algo}' does not consume optimizer_instructions_file "
                     "— optimizer-template checks skipped")
    else:
        instr_rel = str(spec.get("optimizer_instructions_file") or "optimizer/INSTRUCTIONS.md")
        # Same resolver `cap-evolve run` uses, so a spec this gate passes cannot have a
        # path run resolves differently (#252).
        instr_path = specfile.resolve_project_path(project, instr_rel)
        if not instr_path.exists():
            problems.append(
                f"optimizer_instructions_file points at a missing file: {instr_rel} "
                f"(expected an existing template under {project}/)")
            template = ""
        else:
            template = instr_path.read_text(encoding="utf-8")
            if "{{" not in template:
                problems.append(
                    f"template {instr_rel} has NO {{{{...}}}} placeholders — intake must "
                    "KEEP them (the harness fills FOCUS_SUMMARY/FAILURES/CAP_BRIEF/"
                    "ALGO_BRIEF/BENCH_REPO per iteration); did you over-customize it?")
            else:
                notes.append(f"optimizer template OK with intact placeholders: {instr_rel}")

        # 3) render through the REAL harness renderer; no {{ may survive
        if template and "{{" in template:
            caps = [c for c in (spec.get("capabilities") or []) if c]
            rendered = _focus_instructions(
                _synthetic_val(), None, "pipeline self-test",
                capabilities=caps, algorithm=algo,
                instructions_file=instr_path,
                bench_repo=(str(spec.get("runner_repo_path")) or None),
            )
            if "{{" in rendered:
                leftovers = sorted({tok.split("}}")[0] for tok in rendered.split("{{")[1:]})
                problems.append(
                    "rendered INSTRUCTIONS.md still has leftover placeholder(s): "
                    + ", ".join("{{" + x + "}}" for x in leftovers)
                    + " — the harness did not substitute them (a placeholder typo?)")
            else:
                notes.append("rendered INSTRUCTIONS.md has no leftover {{ placeholders")

    # 4) trajectories(): defined OR intentionally inherited (both valid)
    try:
        adapter = load_adapter(project)
        defines_traj = type(adapter).trajectories is not CapabilityAdapter.trajectories
        notes.append(
            "adapter defines trajectories() (native traj dir → ./trajectories/)"
            if defines_traj else
            "adapter inherits trajectories() default → falls back to cap-evolve's "
            "per-rollout JSON (valid; note it in PROJECT.md)")
    except Exception as e:  # noqa: BLE001
        problems.append(f"could not load adapter to inspect trajectories(): {e}")

    return {"ok": not problems, "project": str(project),
            "problems": problems, "notes": notes}


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="pipeline-selftest")
    p.add_argument("--project", default=".capevolve/project")
    args = p.parse_args(argv)
    report = selftest(Path(args.project))
    print(json.dumps(report, indent=2))
    return 0 if report["ok"] else 1


if __name__ == "__main__":
    sys.exit(main())

"""implement-and-check — the HARD GATE before any optimization budget is spent.

Runs ``cap-evolve check`` on the project adapter and (optionally) each involved skill's
own ``check.py``. Aggregates the results; exits non-zero if anything is unfilled
or non-deterministic, listing exactly what to fix.
"""

from __future__ import annotations

import argparse
import json
import subprocess
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve.check import run_check


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="implement-and-check")
    p.add_argument("--project", default=".capevolve/project")
    p.add_argument("--skill-check", action="append", default=[],
                   help="path to a skill's scripts/check.py to also run (repeatable)")
    p.add_argument("--no-pipeline-selftest", action="store_true",
                   help="skip the pipeline-wiring self-test that runs after the check passes")
    args = p.parse_args(argv)

    report = {"ok": True, "project": {}, "skills": []}

    proj = run_check(Path(args.project))
    report["project"] = proj.to_dict()
    report["ok"] = report["ok"] and proj.ok

    for chk in args.skill_check:
        proc = subprocess.run([sys.executable, chk], capture_output=True, text=True)
        try:
            out = json.loads(proc.stdout or "{}")
        except json.JSONDecodeError:
            out = {"ok": proc.returncode == 0, "raw": proc.stdout[-500:], "stderr": proc.stderr[-500:]}
        out["_check"] = chk
        report["skills"].append(out)
        report["ok"] = report["ok"] and bool(out.get("ok"))

    # Only once the adapter contract is green is the pipeline-wiring self-test
    # meaningful: it proves the optimizer would get its trajectories + guidance +
    # a fully-rendered INSTRUCTIONS.md. A red check short-circuits it (nothing to
    # wire yet).
    if report["ok"] and not args.no_pipeline_selftest:
        from pipeline_selftest import selftest
        report["pipeline_selftest"] = selftest(Path(args.project))
        report["ok"] = report["ok"] and report["pipeline_selftest"]["ok"]

    print(json.dumps(report, indent=2))
    return 0 if report["ok"] else 1


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())