Claude Skill

report

Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for "did this optimization actually work, and by how much".

LLM Mart · 0 points · 11 views 0 listing impressions 0 install-command copies
Download skillberry-ai-cap-evolve-skills_phases_report-80721f3.zip · 18 KB

Install

skills CLI npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/report
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
Git git clone https://github.com/skillberry-ai/cap-evolve.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

report — did it work, and by how much?

The result of a run is not "we made edits" — it is a defensible answer to did this actually work, and by how much. report lays three numbers side by side: where the seed started (val), where the best candidate landed (val), and the single held-out test number that counts. It is what a human reads to decide whether to ship.

Runs standalone as /cap-evolve:report, or headlessly as the last step of cap-evolve run — same scripts/run.py either way. report is a phase SCRIPT, not a cap-evolve subcommand; invoke the script.

How to read the three numbers

The honest reading is always test vs baseline, with val as a sanity check in between. scripts/run.py produces the numbers; this is the judgment you add on top:

  • test ≈ baseline → no real gain. The val improvement was overfitting or noise the gate let through. Do not ship; tighten gate_k_se or add trials.
  • test ≫ baseline → genuine improvement on data the optimizer never saw. Ship.
  • val ≫ test → the classic overfit signature: the optimizer learned the val set, not the capability. The reported val→test gap is the overfitting, quantified.
  • passk far below pass1 → the gain is fragile across trials; the agent sometimes succeeds but not reliably. A high mean with low pass^k is not a dependable win (τ-bench's point).

Every number is rendered with its stderr when one was measured, because "0.71" and "0.71 ± 0.08" support very different decisions. A gain smaller than the noise floor is not a result — say so plainly rather than quoting the point estimate alone.

Output contract

scripts/run.py owns both artifacts and writes them deterministically from the run dir — do not hand-write or paraphrase them, or two runs stop being comparable.

report.md is exactly this skeleton (bracketed lines appear only when they apply):

# cap-evolve run report — <run_id>

[> **NOT FINALIZED** — no held-out test number. Run the finalize phase first; …]
[> **No holdout** (train == val == test). The test number below is a *fit* metric, …]

- Best candidate: `<best_id>`
- Baseline val: <r> ± <se>
- Best val: <r> ± <se>
- **Held-out test (optimized skills): <r> ± <se>**  (pass^1=…, pass^k=…)
[- Held-out test (baseline `<baseline_id>` skills): <r> ± <se>]
[- **Test improvement (optimized − baseline): <+Δ>**]
[- Val→test gap: <+Δ> — selection optimism on val; this gap IS the overfitting]
- Iterations: <n>
[- Optimized for: <consuming model> (tier <t>)]

[<sealed note — omitted entirely when the run was never finalized>]

stdout is exactly one JSON object — cap-evolve run echoes it as its own result, so it is the machine contract for everything downstream. Keys (null for whatever the run dir does not carry; an unfinalized run is finalized: false with null test numbers): run_dir, best_id, finalized bool, no_holdout bool, baseline_val, baseline_val_stderr, best_val, test_reward, test_stderr, test_baseline_reward, test_baseline_stderr, test_delta, test_pass_k (k→float), val_test_gap, iterations, target_profile ({model,tier,resolution_note}), plus dashboard / dashboard_server / *_error on the paths that produce them.

How to run

python scripts/run.py --run-dir .capevolve/run_XXXX            # JSON + report.md + dashboard.html
python scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report
python scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard

--dashboard-mode / --dashboard-port / --dashboard-url are orchestrator-supplied. Re-reporting by hand after cap-evolve run needs --dashboard-url <the URL run printed> or --no-dashboard — launching is deliberately not idempotent, so a bare re-run spawns a second server on a second port and reports that one instead.

The dashboard (dashboard.html)

One self-contained static file (inline CSS/JS/SVG, no CDN, no server, no network — opens from file://); the single shareable artifact. Eight panels reduced from the event log plus baseline/final, rollouts and the git store; every value passes a recursive secret redactor so a shared dashboard leaks no API keys; optional panels degrade silently when per-task data, diffs, or finalize are missing. --terminal renders the same reduction as an ANSI chart for in-chat progress.

References

  • references/concepts.md — why the val→test gap measures overfitting, pass^k fragility, reporting uncertainty, with sources. Load when writing the human interpretation and you want the reasoning or a citation.
  • references/dashboard.md — the reduced graph + summary schema, per-panel field sources, --terminal, redaction, degradation matrix. Load when changing or debugging the dashboard; not needed to run the phase.
Files (cap-evolve)
  • references
    • concepts.md 3.6 KB
      Loading…
    • dashboard.md 6.1 KB
      Loading…
  • scripts
    • abstract.py 163 B
      Loading…
    • check.py 12.1 KB
      Loading…
    • dashboard.py 459 B
      Loading…
    • run.py 10.5 KB
      Loading…
    • _bootstrap.py 3.6 KB
      Loading…
  • meta.yaml 723 B
    Loading…
  • SKILL.md 5.2 KB
    ---
    name: report
    description: Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for "did this optimization actually work, and by how much".
    component: phase
    argument-hint: "--run-dir DIR [--terminal] [--no-dashboard]"
    allowed-tools: Read, Write, Bash
    provides: [report]
    needs: []
    sources: [evo]
    ---
    
    # report — did it work, and by how much?
    
    The result of a run is not "we made edits" — it is a defensible answer to *did this
    actually work, and by how much.* report lays three numbers side by side: where the
    seed started (val), where the best candidate landed (val), and the single **held-out
    test** number that counts. It is what a human reads to decide whether to ship.
    
    Runs standalone as `/cap-evolve:report`, or headlessly as the last step of
    `cap-evolve run` — same `scripts/run.py` either way. report is a phase SCRIPT, not a
    `cap-evolve` subcommand; invoke the script.
    
    ## How to read the three numbers
    The honest reading is always **test vs baseline**, with val as a sanity check in
    between. `scripts/run.py` produces the numbers; this is the judgment you add on top:
    
    - **test ≈ baseline** → no real gain. The val improvement was overfitting or noise
      the gate let through. Do not ship; tighten `gate_k_se` or add trials.
    - **test ≫ baseline** → genuine improvement on data the optimizer never saw. Ship.
    - **val ≫ test** → the classic overfit signature: the optimizer learned the val set,
      not the capability. The reported val→test gap *is* the overfitting, quantified.
    - **pass^k far below pass^1** → the gain is *fragile* across trials; the agent
      sometimes succeeds but not reliably. A high mean with low pass^k is not a
      dependable win (τ-bench's point).
    
    Every number is rendered with its stderr when one was measured, because "0.71" and
    "0.71 ± 0.08" support very different decisions. A gain smaller than the noise floor
    is not a result — say so plainly rather than quoting the point estimate alone.
    
    ## Output contract
    `scripts/run.py` owns both artifacts and writes them deterministically from the run
    dir — do not hand-write or paraphrase them, or two runs stop being comparable.
    
    `report.md` is exactly this skeleton (bracketed lines appear only when they apply):
    
    ```
    # cap-evolve run report — <run_id>
    
    [> **NOT FINALIZED** — no held-out test number. Run the finalize phase first; …]
    [> **No holdout** (train == val == test). The test number below is a *fit* metric, …]
    
    - Best candidate: `<best_id>`
    - Baseline val: <r> ± <se>
    - Best val: <r> ± <se>
    - **Held-out test (optimized skills): <r> ± <se>**  (pass^1=…, pass^k=…)
    [- Held-out test (baseline `<baseline_id>` skills): <r> ± <se>]
    [- **Test improvement (optimized − baseline): <+Δ>**]
    [- Val→test gap: <+Δ> — selection optimism on val; this gap IS the overfitting]
    - Iterations: <n>
    [- Optimized for: <consuming model> (tier <t>)]
    
    [<sealed note — omitted entirely when the run was never finalized>]
    ```
    
    stdout is **exactly one** JSON object — `cap-evolve run` echoes it as its own result, so
    it is the machine contract for everything downstream. Keys (null for whatever the run
    dir does not carry; an unfinalized run is `finalized: false` with null test numbers):
    `run_dir`, `best_id`, `finalized` bool, `no_holdout` bool, `baseline_val`,
    `baseline_val_stderr`, `best_val`, `test_reward`, `test_stderr`,
    `test_baseline_reward`, `test_baseline_stderr`, `test_delta`, `test_pass_k` (k→float),
    `val_test_gap`, `iterations`, `target_profile` (`{model,tier,resolution_note}`), plus
    `dashboard` / `dashboard_server` / `*_error` on the paths that produce them.
    
    ## How to run
    ```
    python scripts/run.py --run-dir .capevolve/run_XXXX            # JSON + report.md + dashboard.html
    python scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report
    python scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard
    ```
    `--dashboard-mode` / `--dashboard-port` / `--dashboard-url` are orchestrator-supplied.
    Re-reporting by hand after `cap-evolve run` needs `--dashboard-url <the URL run printed>`
    or `--no-dashboard` — launching is deliberately not idempotent, so a bare re-run spawns
    a second server on a second port and reports that one instead.
    
    ## The dashboard (`dashboard.html`)
    One self-contained static file (inline CSS/JS/SVG, no CDN, no server, no network — opens
    from `file://`); the single shareable artifact. Eight panels reduced from the event log
    plus baseline/final, rollouts and the git store; every value passes a recursive secret
    redactor so a shared dashboard leaks no API keys; optional panels degrade silently when
    per-task data, diffs, or finalize are missing. `--terminal` renders the same reduction
    as an ANSI chart for in-chat progress.
    
    ## References
    - `references/concepts.md` — why the val→test gap measures overfitting, pass^k
      fragility, reporting uncertainty, with sources. Load when writing the human
      interpretation and you want the reasoning or a citation.
    - `references/dashboard.md` — the reduced graph + summary schema, per-panel field
      sources, `--terminal`, redaction, degradation matrix. Load when changing or
      debugging the dashboard; not needed to run the phase.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.