# Concepts — reading a run honestly

> A report's job is not to make the run look good; it is to let a human decide
> whether to ship. The honest reading is always test-vs-baseline, with the
> val-test gap as the overfitting it is. Implementation: this skill's
> `scripts/run.py` + the `cap_evolve.dashboard` builder (panels + schema documented
> in `references/dashboard.md`).

## The three numbers, and what their relationships mean

| relationship          | interpretation                                          | action            |
|-----------------------|---------------------------------------------------------|-------------------|
| test ≈ baseline       | no real gain (val gain was noise/overfit)               | don't ship; retune|
| test ≫ baseline       | genuine improvement on unseen data                      | ship              |
| val ≫ test            | overfitting — learned the val set, not the capability   | the gap is the leak|
| high mean, low pass^k | gain is fragile across trials (unreliable)              | not a dependable win|

- **baseline (val)** — where the unmodified seed started.
- **best (val)** — where search landed *on the split it optimized against*. This
  is expected to be optimistic; it is not the result.
- **test** — scored once, on data nothing was tuned against. This is the result.

## The val–test gap is overfitting, quantified

Search selects the candidate that scores best on val, so the final val score is
biased upward by exactly the amount of selection performed. The test score has no
such bias. Their difference is therefore a direct measurement of how much the run
overfit the val split. A small gap means the val gains generalized; a large gap
means the optimizer learned the validation tasks rather than the capability. Report
the gap honestly — it is one of the most informative numbers in the run, and
hiding it (by quoting val as "the result") is the most common way optimization
reports mislead.

## Reliability: pass^k, not just the mean

A high mean reward can hide an unreliable agent — one that passes often but not
*every* time. τ-bench showed strong agents whose per-run success looked fine but
whose pass^k collapsed (succeeding on all of k repeated trials is much harder than
succeeding once). At report time, a wide pass^1 → pass^k drop is a flag: the gain
exists but is fragile. For any capability that must work repeatedly, pass^k is the
number that decides shippability, not the mean.

## Always report uncertainty

A point estimate without its standard error or confidence interval invites
over-reading. "0.71" and "0.71 ± 0.08" justify different decisions; a 3-point test
gain inside a ±8-point CI is not a result. Carry the combined stderr (or a
bootstrap CI) from finalize into the report so the reader sees the noise floor, not
just the headline.

## No-holdout runs

If the run was configured with no holdout (test == train/val), say so plainly: the
test number is a *fit* metric (how well it fits data it was tuned on), not an
estimate of generalization. The dashboard and `report.md` should label it so no
reader mistakes a fit for a held-out result.

## Sources
- τ-bench (Yao et al., 2024) — pass^k reliability and the per-run-vs-multi-trial
  gap: https://arxiv.org/abs/2406.12045
- Koehn, "Statistical Significance Tests for MT Evaluation" (EMNLP 2004) —
  reporting differences with uncertainty: https://aclanthology.org/W04-3250/
- Hastie, Tibshirani, Friedman, *The Elements of Statistical Learning* — the
  validation–test gap as a measure of optimism/overfit:
  https://hastie.su.domains/ElemStatLearn/

# Reference — the dashboard & terminal report

Builder: `cap_evolve.dashboard` (engine-owned, stdlib-only), imported by the report
skill via `scripts/dashboard.py`. The flow is **reduce → render**:

```
reduce_run(run_dir)  → {"graph": {...}, "summary": {...}}   (redacted)
render_html(reduced, run_dir)  → self-contained dashboard.html
render_ansi(reduced)           → colored terminal report
```

## Contents
- [Reduced run schema](#reduced-run-schema) — graph + summary the panels read from.
- [What each panel reads](#what-each-panel-reads) — the eight HTML panels.
- [Terminal report](#terminal-report) — the `--terminal` / `--ansi` mode.
- [Secret redaction](#secret-redaction) — what is scrubbed and how.
- [Graceful degradation](#graceful-degradation) — which panels hide when data is missing.

## Reduced run schema

`reduce_run` folds the run dir's append-only `events.jsonl` (the source of truth)
plus `baseline.json`, `final.json`, the persisted per-task `rollouts/`, and the git
iteration store into one structure. It never trusts `state.json` for anything it can
recompute from events.

**graph**
```jsonc
{"root": "seed", "best_id": "cand_0007",
 "nodes": [
   {"id": "cand_0001", "parent": "seed", "children": ["cand_0002"],
    "status": "accepted|rejected|seed|failed",
    "val": 0.62, "stderr": 0.04, "best_so_far": 0.62,
    "per_task": {"t1": 1.0, "t2": 0.0}, "feedback": {"t2": "…"},
    "cost_usd": 0.012, "tokens": 1840, "seconds": 4.1,
    "optimizer_seconds": 2.3, "runner_seconds": 1.8,
    "iteration": 1, "reason": "Δ=+0.12 (paired, p<0.05)",
    "parent_val": 0.50, "epoch": 0, "merge_of": ["cand_0003","cand_0005"]}
 ]}
```
`epoch` appears only for skillopt; `merge_of` only for gepa merges (multi-parent).
`status` is `failed` when a step produced neither a val score nor rollouts (e.g. the
optimizer raised); `seed` is the baseline node.

**summary**
```jsonc
{"run_id", "baseline_val", "best_val", "best_id",
 "delta_abs", "delta_pct",                    // %Δ is null off a zero baseline → use delta_abs
 "test_reward", "test_stderr", "test_pass_k", "test_sealed",
 "counts": {"accepted","rejected","failed","seed","total"},
 "frontier",                                  // gated leaves with no accepted child
 "tasks": ["t1","t2", …],
 "wall_clock_seconds", "optimizer_seconds", "runner_seconds",
 "cost": {"optimizer_usd","runner_usd","total_usd"},
 "tokens",
 "gate_warnings": [{"reason","context","mode"}],
 "diagnoses": [{"kind","candidate","text"}],   // gate reasons + diagnose/optimizer_error
 "git_log": [{"hash","subject"}]}              // one row per iteration commit
```

## What each panel reads

1. **KPI strip** — `summary` (best/baseline/Δ, counts by status, frontier, sealed
   test, wall-clock, optimizer-vs-runner cost split, tokens).
2. **Score over iterations** — `graph.nodes[*].{iteration,val,best_so_far,status,
   parent_val}`. Running-best is an SVG step polyline; champion star + value label;
   record-holder rings on each new best; per-iteration scatter colored by status;
   hover → id / status / val / Δ-from-parent.
3. **tasks×iterations heatmap** — `graph.nodes[*].per_task` + `summary.tasks`. Rows
   = tasks sorted worst-first by mean reward, cols = iterations; green=pass,
   red=fail, amber=partial, grey=not-run. Hover a cell → that task's feedback.
4. **Diff vs parent** — `build_diffs(run_dir, graph)` diffs each candidate dir
   against its parent dir (unified, split/unified toggle, add/del/hunk coloring).
5. **Lineage** — `graph` parent→child DAG; merges drawn as multi-parent edges; the
   best-lineage spine (best_id → root) highlighted gold.
6. **Cost · tokens · latency** — per-iteration stacked bars (optimizer-seconds blue,
   runner-seconds green) + a cumulative-cost dashed line; plus a separate
   cumulative-cost-vs-best-score plot (shown only when total cost > 0).
7. **Annotations & diagnoses** — `summary.gate_warnings` + `summary.diagnoses`
   rendered as an inline stream.
8. **Candidates** — full leaderboard table (id, status badge, val, Δ-parent, iter,
   reason) + the git iteration-store log.

## Terminal report

`python scripts/run.py --run-dir DIR --terminal` (alias `--ansi`) prints, instead of
the JSON summary:
- a one-line **KPI strip** (baseline / best / Δ / sealed test / counts / frontier /
  cost / tokens / wall),
- a **cumulative-best** block chart (█ running best, ○ accept, · reject, x fail),
- a **top-N** candidate table (`--top-n`, default 8), and
- up to three **gate warnings**.

It is sized to the terminal width and is **CLAUDECODE-margin-aware**: when
`CLAUDECODE=1` it subtracts ~6 columns so lines don't wrap inside the tool-output
frame. `--no-color` (or `NO_COLOR=1`) disables ANSI codes for piping/CI.

## Secret redaction

`reduce_run` returns its result through `redact()` before it reaches the HTML or the
terminal, so a shared `dashboard.html` never leaks a credential pulled in from
config/env. The redactor walks dicts/lists/strings and:
- replaces any **value under a secret-looking key** wholesale (`*api_key*`,
  `*secret*`, `*token*` — but not the `tokens` metric — `password`, `credential`,
  `watsonx`, `authorization`, `bearer`, `private_key`, `access_key`, `session`,
  `cookie`); and
- masks **value-shaped secrets inside free text**: `sk-…`, `Bearer …`, JWTs, long
  hex/base64 blobs, and inline `KEY=value` / `KEY: value` leaks (e.g. an optimizer
  error echoing `RITS_API_KEY=…`), keeping the key name and masking only the value.

Covers `RITS_API_KEY`, `BOBSHELL_API_KEY`, `WATSONX_*`, `OPENAI_API_KEY`,
`ANTHROPIC_API_KEY`, etc.

## Graceful degradation

Optional panels hide rather than crash when their data is absent:
- no per-task rollouts → heatmap is empty/hidden, scores fall back to `baseline.json`
  per_task or the step event's `val`;
- candidate dirs not snapshotted (e.g. a synthetic log) → `diffs` is `{}`, the diff
  panel hides;
- not finalized → test KPI shows "—", the test-sealed flag reads "not finalized";
- no git store → the iteration-store log is omitted;
- zero cost → the cost-vs-score plot is omitted.

"""The 'report' phase composes the project adapter + shared harness; it declares no
abstract methods of its own. check.py verifies the wiring instead of stubs."""

"""Contract: report summarizes baseline → best val → sealed test and writes
report.md; and the Wave-4 dashboard builder reduces a synthetic event log into a
well-formed candidate graph + a self-contained, parseable dashboard.html with no
leaked secrets.
"""

from __future__ import annotations

import html.parser
import json
import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve.skillcheck import Checker, import_run, quiet, temp_run_dir


def _synthetic_events():
    """A minimal but realistic event log: splits, baseline, an accept, a gate
    warning, a reject — plus an optimizer_error echoing a secret to prove redaction."""
    return [
        {"kind": "splits", "train": 4, "val": 2, "test": 2, "seed": 0},
        {"kind": "evaluate", "split": "val", "tag": "seed", "reward": 0.25,
         "stderr": 0.0, "cost_usd": 0.0, "tokens": 0, "seconds": 0.0},
        {"kind": "baseline", "val": 0.25, "stderr": 0.0},
        {"kind": "step", "candidate": "cand_0001", "accept": True, "reason": "Δ up",
         "val": 0.75, "parent": "seed", "parent_val": 0.25,
         "optimizer_seconds": 1.0, "runner_seconds": 0.5, "cost_usd": 0.01, "tokens": 500},
        {"kind": "gate_warning", "mode": "paired", "reason": "SE collapsed to 0", "context": "se=0"},
        {"kind": "step", "candidate": "cand_0002", "accept": False, "reason": "Δ<=0",
         "val": 0.6, "parent": "cand_0001", "parent_val": 0.75,
         "optimizer_seconds": 0.9, "runner_seconds": 0.4, "cost_usd": 0.008, "tokens": 400},
        {"kind": "optimizer_error", "candidate": "cand_0002",
         "error": "auth failed RITS_API_KEY=rits-supersecret0123456789 retry"},
    ]


#: The stdout contract (CONTRIBUTING.md: "scripts/run.py must print a single JSON object").
#: `dashboard` / `dashboard_server` are added only on the paths that produce them.
_SUMMARY_KEYS = {
    "run_dir", "best_id", "finalized", "no_holdout", "baseline_val", "baseline_val_stderr",
    "best_val", "test_reward", "test_stderr", "test_baseline_reward", "test_baseline_stderr",
    "test_delta", "test_pass_k", "val_test_gap", "iterations", "target_profile",
}


def _parse_one_json(c, text: str, label: str):
    """Assert stdout is exactly one JSON object carrying the documented key set."""
    try:
        obj = json.loads(text)
    except Exception as e:  # noqa: BLE001
        c.check(False, f"[{label}] stdout is not a single JSON object ({e}): {text!r}")
        return None
    c.check(isinstance(obj, dict), f"[{label}] stdout JSON is not an object: {obj!r}",
            note="stdout is exactly one JSON object")
    if isinstance(obj, dict):
        missing = _SUMMARY_KEYS - set(obj)
        c.check(not missing, f"[{label}] summary missing documented keys: {sorted(missing)}",
                note="summary carries the documented key set")
    return obj if isinstance(obj, dict) else None


def main() -> int:
    c = Checker("report")
    run = import_run()
    c.require_main(run)

    # --- 1. report.md + stdout contract: baseline → best val → sealed test ----
    with tempfile.TemporaryDirectory() as d:
        rd, _ = temp_run_dir(Path(d))
        rd.events_path.write_text(
            "\n".join(json.dumps(e) for e in _synthetic_events()) + "\n", encoding="utf-8")
        (rd.root / "baseline.json").write_text(
            json.dumps({"val": {"reward": 0.4, "stderr": 0.05}, "best_id": "seed"}),
            encoding="utf-8")
        (rd.root / "final.json").write_text(json.dumps({
            "test": {"reward": 0.8, "stderr": 0.06, "pass_k": 0.7},
            "test_baseline": {"reward": 0.5, "stderr": 0.07},
            "baseline_id": "seed", "test_delta": 0.3, "best_id": "cand_0001"}),
            encoding="utf-8")

        with quiet() as out:
            rc = run.main(["--run-dir", str(rd.root), "--no-dashboard"])
        c.check(rc == 0, "report returned nonzero")

        # stdout IS the machine contract: `cap-evolve run` prints report's stdout as its
        # own result (cli.py `print(last)`), so exactly one parseable JSON object with the
        # documented keys is the thing every automation downstream depends on.
        summary = _parse_one_json(c, out.getvalue(), "--no-dashboard")

        md_path = rd.root / "report.md"
        c.check(md_path.exists(), "report.md was not written", note="writes report.md")
        md = md_path.read_text()
        c.check("0.4" in md and "0.8" in md,
                f"report.md missing baseline/test numbers:\n{md}",
                note="report carries baseline → sealed test")
        c.check("sealed" in md.lower(),
                "report does not state the test was scored once on the sealed split")
        # Uncertainty is the phase's own headline rule ("0.71" and "0.71 ± 0.08" justify
        # different decisions), so a bare point estimate beside a known stderr is a bug.
        c.check("0.8 ± 0.06" in md and "0.4 ± 0.05" in md,
                f"report.md renders a bare point estimate despite a known stderr:\n{md}",
                note="renders reward ± stderr when stderr is known")
        if summary is not None:
            c.check(summary.get("test_stderr") == 0.06
                    and summary.get("test_baseline_stderr") == 0.07,
                    f"summary drops the stderr finalize measured: {summary}",
                    note="JSON summary carries test/baseline stderr")
            c.check(summary.get("best_val") == 0.75
                    and summary.get("val_test_gap") == round(0.75 - 0.8, 6),
                    f"summary missing best_val / val→test gap: {summary}",
                    note="JSON summary carries best_val + val→test gap")
            c.check(summary.get("finalized") is True, f"finalized not True: {summary}")
        c.check("Best val: 0.75" in md and "Val→test gap:" in md,
                f"report.md missing best val / val→test gap:\n{md}",
                note="report.md carries best val + the val→test gap")

    # --- 1b. an unfinalized run must NOT claim a seal that never happened ----
    with tempfile.TemporaryDirectory() as d:
        rd, _ = temp_run_dir(Path(d))
        (rd.root / "baseline.json").write_text(
            json.dumps({"val": {"reward": 0.4}, "best_id": "seed"}), encoding="utf-8")
        with quiet() as out:
            rc = run.main(["--run-dir", str(rd.root), "--no-dashboard"])
        c.check(rc == 0, "report on an unfinalized run returned nonzero")
        md = (rd.root / "report.md").read_text()
        c.check("sealed" not in md.lower(),
                f"report claims the test split was sealed+scored on a run that never "
                f"finalized:\n{md}", note="no false seal claim before finalize")
        c.check("NOT FINALIZED" in md, f"unfinalized report is not labelled:\n{md}")
        s = _parse_one_json(c, out.getvalue(), "unfinalized")
        if s is not None:
            c.check(s.get("finalized") is False, f"finalized not False: {s}")

    # --- 1c. a no-holdout run's report.md must say the number is a fit metric --
    with tempfile.TemporaryDirectory() as d:
        rd, _ = temp_run_dir(Path(d))
        rd.events_path.write_text(json.dumps(
            {"kind": "splits", "train": ["t1"], "val": ["t1"], "test": ["t1"], "seed": 0}
        ) + "\n", encoding="utf-8")
        (rd.root / "final.json").write_text(
            json.dumps({"test": {"reward": 0.8}, "best_id": "seed"}), encoding="utf-8")
        with quiet():
            run.main(["--run-dir", str(rd.root), "--no-dashboard"])
        md = (rd.root / "report.md").read_text()
        c.check("No holdout" in md,
                f"no-holdout run's report.md presents a fit metric as held-out:\n{md}",
                note="labels a no-holdout run as a fit metric")

    # --- 2. reducer → well-formed graph from a synthetic event log -------
    from cap_evolve import dashboard
    with tempfile.TemporaryDirectory() as d:
        rd, _ = temp_run_dir(Path(d))
        rd.events_path.write_text(
            "\n".join(json.dumps(e) for e in _synthetic_events()) + "\n", encoding="utf-8")
        (rd.root / "baseline.json").write_text(json.dumps({
            "val": {"reward": 0.25, "per_task": [
                {"task_id": "t1", "reward": 0.0, "feedback": "wrong"},
                {"task_id": "t2", "reward": 0.5, "feedback": ""}]}, "best_id": "seed"}),
            encoding="utf-8")
        (rd.root / "final.json").write_text(
            json.dumps({"test": {"reward": 0.8}, "best_id": "cand_0001"}), encoding="utf-8")

        reduced = dashboard.reduce_run(rd)
        g, s = reduced["graph"], reduced["summary"]

        c.check(set(g.keys()) == {"nodes", "root", "best_id"},
                f"graph keys malformed: {sorted(g.keys())}", note="reducer → {nodes,root,best_id}")
        nodes = {n["id"]: n for n in g["nodes"]}
        c.check(set(nodes) == {"seed", "cand_0001", "cand_0002"},
                f"graph nodes wrong: {sorted(nodes)}")
        required = {"id", "parent", "children", "status", "val", "per_task",
                    "cost_usd", "tokens", "seconds", "optimizer_seconds",
                    "runner_seconds", "iteration", "reason", "best_so_far"}
        for n in g["nodes"]:
            missing = required - set(n)
            c.check(not missing, f"node {n['id']} missing fields: {missing}")
            c.check(n["status"] in ("seed", "accepted", "rejected", "failed"),
                    f"node {n['id']} bad status {n['status']!r}")
        c.check(nodes["seed"]["children"] == ["cand_0001"]
                and nodes["cand_0001"]["parent"] == "seed",
                "lineage edges not wired", note="parent↔child edges wired both ways")
        # Assert the buckets this synthetic log actually exercises, plus the total —
        # not dict equality. `reduce_run` seeds every status bucket it knows about
        # (`indecisive` among them), so an exact-dict assertion breaks whenever a new
        # status is added even though nothing about the reducer regressed.
        want = {"accepted": 1, "rejected": 1, "failed": 0, "seed": 1, "total": 3}
        c.check({k: s["counts"].get(k) for k in want} == want
                and sum(v for k, v in s["counts"].items() if k != "total") == s["counts"]["total"],
                f"status counts wrong: {s['counts']}")
        c.check(s["best_val"] == 0.75 and s["best_id"] == "cand_0001",
                f"best wrong: {s['best_val']} / {s['best_id']}")
        c.check(len(s["gate_warnings"]) == 1, "gate warning not surfaced")

        # --- 3. self-contained, parseable HTML, no leaked secret ---------
        out = dashboard.write_dashboard(rd)
        c.check(out.exists(), "dashboard.html not written", note="renders dashboard.html")
        text = out.read_text(encoding="utf-8")
        try:
            html.parser.HTMLParser().feed(text)
            parsed = True
        except Exception:  # noqa: BLE001
            parsed = False
        c.check(parsed, "dashboard.html is not parseable HTML")
        for marker in ('src="http', 'href="http', "<link", "cdn.", "fetch("):
            c.check(marker not in text, f"dashboard pulls external resource ({marker})",
                    note="self-contained (no CDN / network)")
        for panel in ("Summary", "Score over iterations", "Per-task pass/fail", "Lineage"):
            c.check(panel in text, f"dashboard missing panel: {panel}")
        c.check("supersecret" not in text,
                "secret leaked into dashboard.html", note="secret redaction holds")

        # --- 4. ANSI terminal report renders --------------------------
        with quiet():
            rc2 = run.main(["--run-dir", str(rd.root), "--terminal", "--no-color"])
        c.check(rc2 == 0, "report --terminal returned nonzero",
                note="ANSI terminal report mode")

        # --- 5. the dashboard-on path keeps the one-JSON-object contract ---
        # This branch is where a stray print() would creep in (it launches/records a
        # server), and it is the path `cap-evolve run` actually takes.
        with quiet() as out:
            rc3 = run.main(["--run-dir", str(rd.root)])
        c.check(rc3 == 0, "report with the dashboard on returned nonzero")
        _parse_one_json(c, out.getvalue(), "dashboard-on")

    return c.emit()


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: the dashboard builder lives in ``cap_evolve.dashboard``. 

The report skill imports this module so the reducer/renderers stay engine-owned
and unit-tested under ``core/tests`` while the skill keeps a stable import name.
"""

from __future__ import annotations

import _bootstrap  # noqa: F401

from cap_evolve.dashboard import (  # noqa: F401
    build_diffs,
    reduce_run,
    redact,
    render_ansi,
    render_html,
    write_dashboard,
)

"""report — summarize a run: baseline → best val → sealed test, and the winner.

Reads the run dir's baseline.json / final.json / events and prints a human and
machine readable summary. Writes report.md next to them, plus (by default) a
self-contained dashboard.html. ``--terminal`` / ``--ansi`` prints a colored
in-terminal report instead (CLAUDECODE-margin-aware) for in-chat progress.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import RunDir


def _pm(reward, stderr) -> str:
    """``0.71 ± 0.08`` when the stderr is known, else the bare point estimate.

    A point estimate with no noise floor invites over-reading: "0.71" and "0.71 ± 0.08"
    justify different ship decisions.
    """
    if reward is None:
        return "None"
    return f"{reward} ± {stderr}" if isinstance(stderr, (int, float)) else f"{reward}"


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="report")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--no-dashboard", action="store_true", help="skip generating dashboard.html")
    p.add_argument("--terminal", "--ansi", dest="terminal", action="store_true",
                   help="print a colored ANSI terminal report (KPI strip + cumulative-best "
                        "chart + top-N table) instead of the JSON summary")
    p.add_argument("--no-color", action="store_true", help="disable ANSI colors in --terminal mode")
    p.add_argument("--top-n", type=int, default=8, help="rows in the --terminal candidate table")
    p.add_argument("--dashboard-mode", choices=("auto", "report-only", "off"), default="off",
                   help="ensure the live dashboard server is up (auto/report-only) at the final phase")
    p.add_argument("--dashboard-port", type=int, default=7878)
    p.add_argument("--dashboard-url", default="",
                   help="URL of a dashboard server the caller ALREADY started; recorded "
                        "as-is instead of launching a second one")
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))

    # --- ANSI terminal mode: reduce → render_ansi → stdout, then return ---
    if args.terminal:
        import dashboard
        reduced = dashboard.reduce_run(run_dir)
        print(dashboard.render_ansi(reduced, color=not args.no_color, top_n=args.top_n))
        return 0

    baseline = json.loads((run_dir.root / "baseline.json").read_text()) if (run_dir.root / "baseline.json").exists() else {}
    final_path = run_dir.root / "final.json"
    final = json.loads(final_path.read_text()) if final_path.exists() else {}
    finalized = bool(final.get("test"))

    base_val_obj = baseline.get("val") or {}
    base_val = base_val_obj.get("reward")
    test = final.get("test") or {}
    test_reward = test.get("reward")
    # Baseline scored on the SAME sealed test split — the honest held-out improvement.
    test_baseline = final.get("test_baseline") or {}
    test_baseline_reward = test_baseline.get("reward")
    test_delta = final.get("test_delta")
    baseline_id = final.get("baseline_id")  # "seed" normally; == best_id if best IS the seed

    # best val, the no-holdout verdict and the consuming-LLM profile are already computed by
    # the engine's reducer (``cap_evolve.dashboard.reduce_run``). Read them; never recompute
    # them here and never ask the agent to re-derive them in prose — that is how the report
    # stopped being comparable between runs.
    # ponytail: reduces a second time when the dashboard is also written (write_dashboard
    # reduces again). Thread the reduced dict through write_dashboard if that ever profiles hot.
    best_val = best_stderr = no_holdout = target_profile = None
    try:
        import dashboard
        reduced = dashboard.reduce_run(run_dir)
        s = reduced["summary"]
        best_val = s.get("best_val")
        best_stderr = next((n.get("stderr") for n in reduced["graph"]["nodes"]
                            if n.get("id") == s.get("best_id")), None)
        no_holdout = bool((s.get("splits") or {}).get("no_holdout"))
        target_profile = s.get("target_profile")
    except Exception:  # noqa: BLE001 — a broken reducer must never break the report
        pass

    # Search picks the candidate that scores best on val, so best_val is biased upward by
    # exactly the selection performed. test has no such bias, so the difference measures how
    # much the run overfit val.
    gap = (round(best_val - test_reward, 6)
           if isinstance(best_val, (int, float)) and isinstance(test_reward, (int, float))
           else None)

    summary = {
        "run_dir": str(run_dir.root),
        "best_id": run_dir.best_id,
        "finalized": finalized,
        "no_holdout": no_holdout,
        "baseline_val": base_val,
        "baseline_val_stderr": base_val_obj.get("stderr"),
        "best_val": best_val,
        "test_reward": test_reward,
        "test_stderr": test.get("stderr"),
        "test_baseline_reward": test_baseline_reward,
        "test_baseline_stderr": test_baseline.get("stderr"),
        "test_delta": test_delta,
        "test_pass_k": test.get("pass_k"),
        "val_test_gap": gap,
        "iterations": run_dir.spent.iterations,
        "target_profile": target_profile,
    }

    # pass^k for k > n_trials is UNDEFINED, so aggregate_scores omits it (see
    # loop.aggregate_scores) — never 0.0, which would read as "0% reliable" instead
    # of "not enough trials". Render exactly the ks that are PRESENT, in numeric
    # order: a hardcoded k range would drop a measured pass^3 and invent a
    # `pass^2=N/A` that was never requested (`ks` is a caller kwarg; gepa passes a
    # non-default one). final.json does not record which ks were asked for, so the
    # present keys are the only honest source.
    pk = test.get("pass_k") or {}
    if not isinstance(pk, dict):  # legacy run dirs stored a bare scalar
        pk = {"1": pk}
    pk_str = ", ".join(f"pass^{k}={float(pk[k]):.3f}" for k in sorted(pk, key=int))

    md = [f"# cap-evolve run report — {run_dir.root.name}", ""]
    if not finalized:
        # The sealed note below is the exact claim a reader relies on. Emitting it for a run
        # that never scored test is the worst failure mode this phase has — say the opposite.
        md += ["> **NOT FINALIZED** — no held-out test number. Run the finalize phase first; "
               "everything below is val-only.", ""]
    elif no_holdout:
        md += ["> **No holdout** (train == val == test). The test number below is a *fit* "
               "metric, not an estimate of generalization.", ""]
    md += [
        f"- Best candidate: `{run_dir.best_id}`",
        f"- Baseline val: {_pm(base_val, base_val_obj.get('stderr'))}",
        f"- Best val: {_pm(best_val, best_stderr)}",
        f"- **Held-out test (optimized skills): {_pm(test_reward, test.get('stderr'))}**"
        + (f"  ({pk_str})" if pk else ""),
    ]
    # When the best candidate IS the seed (no accepted gain), baseline_id == best_id and
    # baseline == optimized — label accordingly rather than implying a separate comparison.
    best_is_seed = baseline_id is not None and baseline_id == run_dir.best_id
    if test_baseline_reward is not None and not best_is_seed:
        baseline_label = f"baseline `{baseline_id}` skills" if baseline_id else "baseline skills"
        md.append(f"- Held-out test ({baseline_label}): "
                  f"{_pm(test_baseline_reward, test_baseline.get('stderr'))}")
        md.append(
            f"- **Test improvement (optimized − baseline): {test_delta:+}**"
            if isinstance(test_delta, (int, float)) else f"- Test improvement: {test_delta}"
        )
        sealed_note = (
            "Test was scored exactly once on the sealed split, for BOTH the baseline "
            f"(`{baseline_id}`) and the optimized skills — the improvement above is on "
            "held-out tasks the optimizer never saw."
        )
    else:
        sealed_note = (
            "Test was scored exactly once on the sealed split. The best candidate is the "
            "seed (no accepted improvement), so baseline and optimized are identical here."
            if best_is_seed else
            "Test was scored exactly once on the sealed split."
        )
    if gap is not None:
        md.append(f"- Val→test gap: {gap:+} — selection optimism on val; this gap IS the overfitting")
    md.append(f"- Iterations: {run_dir.spent.iterations}")
    if target_profile and target_profile.get("model"):
        # The consuming LLM the capabilities were optimized FOR — a different LLM role from
        # the optimizer model that proposed the edits.
        md.append(f"- Optimized for: {target_profile['model']}"
                  + (f" (tier {target_profile['tier']})" if target_profile.get("tier") else ""))
    if finalized:
        md += ["", sealed_note]
    (run_dir.root / "report.md").write_text("\n".join(md) + "\n", encoding="utf-8")

    if not args.no_dashboard:
        try:
            import dashboard
            dash = dashboard.write_dashboard(run_dir)
            summary["dashboard"] = str(dash)
        except Exception as e:  # noqa: BLE001 — never let the dashboard break the report
            summary["dashboard_error"] = str(e)

    # Final phase: guarantee the live dashboard server is up and opened, so "the
    # dashboard is created automatically in the last phase" holds even when early
    # auto-start was disabled. Best-effort; never fails the report.
    #
    # NOT idempotent, which is why --dashboard-url exists: maybe_launch() deliberately
    # steps past an occupied port (a stale server there would serve the wrong run), so
    # calling it again after `cap-evolve run` already launched one spawns a SECOND
    # server on a SECOND port and leaks it. When the caller hands us the URL it got,
    # record that and launch nothing.
    if args.dashboard_url:
        summary["dashboard_server"] = args.dashboard_url
    elif args.dashboard_mode in ("auto", "report-only"):
        try:
            from cap_evolve import dashboard_launch
            base = run_dir.root.resolve().parent  # absolute: subprocess cwd may differ
            status = dashboard_launch.maybe_launch(
                base, mode=args.dashboard_mode, port=args.dashboard_port, open_browser=True
            )
            summary["dashboard_server"] = status.get("dashboard")
        except Exception as e:  # noqa: BLE001
            summary["dashboard_server_error"] = str(e)

    print(json.dumps(summary, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())