{"slug":"report-2","title":"report","summary":"Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for \"did this optimization actually work, and by how much\".","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-01T18:59:11.258064Z","repo":{"url":"https://github.com/skillberry-ai/cap-evolve","stars":61,"forks":16,"license":"Apache-2.0","updatedAt":"2026-09-26T21:16:54Z"},"bodyHtml":"<hr>\n<h2>name: report\ndescription: Summarize a run for a human — baseline val → best val → sealed test, the winning candidate, iterations spent, and pass^k. Use after finalize. Writes report.md and prints a compact JSON summary; the source of truth for \"did this optimization actually work, and by how much\".\ncomponent: phase\nargument-hint: \"--run-dir DIR [--terminal] [--no-dashboard]\"\nallowed-tools: Read, Write, Bash\nprovides: [report]\nneeds: []\nsources: [evo]</h2>\n<h1>report — did it work, and by how much?</h1>\n<p>The result of a run is not \"we made edits\" — it is a defensible answer to <em>did this\nactually work, and by how much.</em> report lays three numbers side by side: where the\nseed started (val), where the best candidate landed (val), and the single <strong>held-out\ntest</strong> number that counts. It is what a human reads to decide whether to ship.</p>\n<p>Runs standalone as <code>/cap-evolve:report</code>, or headlessly as the last step of\n<code>cap-evolve run</code> — same <code>scripts/run.py</code> either way. report is a phase SCRIPT, not a\n<code>cap-evolve</code> subcommand; invoke the script.</p>\n<h2>How to read the three numbers</h2>\n<p>The honest reading is always <strong>test vs baseline</strong>, with val as a sanity check in\nbetween. <code>scripts/run.py</code> produces the numbers; this is the judgment you add on top:</p>\n<ul>\n<li><strong>test ≈ baseline</strong> → no real gain. The val improvement was overfitting or noise\nthe gate let through. Do not ship; tighten <code>gate_k_se</code> or add trials.</li>\n<li><strong>test ≫ baseline</strong> → genuine improvement on data the optimizer never saw. Ship.</li>\n<li><strong>val ≫ test</strong> → the classic overfit signature: the optimizer learned the val set,\nnot the capability. The reported val→test gap <em>is</em> the overfitting, quantified.</li>\n<li><strong>pass<sup>k far below pass</sup>1</strong> → the gain is <em>fragile</em> across trials; the agent\nsometimes succeeds but not reliably. A high mean with low pass^k is not a\ndependable win (τ-bench's point).</li>\n</ul>\n<p>Every number is rendered with its stderr when one was measured, because \"0.71\" and\n\"0.71 ± 0.08\" support very different decisions. A gain smaller than the noise floor\nis not a result — say so plainly rather than quoting the point estimate alone.</p>\n<h2>Output contract</h2>\n<p><code>scripts/run.py</code> owns both artifacts and writes them deterministically from the run\ndir — do not hand-write or paraphrase them, or two runs stop being comparable.</p>\n<p><code>report.md</code> is exactly this skeleton (bracketed lines appear only when they apply):</p>\n<pre><code># cap-evolve run report — &lt;run_id&gt;\n\n[&gt; **NOT FINALIZED** — no held-out test number. Run the finalize phase first; …]\n[&gt; **No holdout** (train == val == test). The test number below is a *fit* metric, …]\n\n- Best candidate: `&lt;best_id&gt;`\n- Baseline val: &lt;r&gt; ± &lt;se&gt;\n- Best val: &lt;r&gt; ± &lt;se&gt;\n- **Held-out test (optimized skills): &lt;r&gt; ± &lt;se&gt;**  (pass^1=…, pass^k=…)\n[- Held-out test (baseline `&lt;baseline_id&gt;` skills): &lt;r&gt; ± &lt;se&gt;]\n[- **Test improvement (optimized − baseline): &lt;+Δ&gt;**]\n[- Val→test gap: &lt;+Δ&gt; — selection optimism on val; this gap IS the overfitting]\n- Iterations: &lt;n&gt;\n[- Optimized for: &lt;consuming model&gt; (tier &lt;t&gt;)]\n\n[&lt;sealed note — omitted entirely when the run was never finalized&gt;]\n</code></pre>\n<p>stdout is <strong>exactly one</strong> JSON object — <code>cap-evolve run</code> echoes it as its own result, so\nit is the machine contract for everything downstream. Keys (null for whatever the run\ndir does not carry; an unfinalized run is <code>finalized: false</code> with null test numbers):\n<code>run_dir</code>, <code>best_id</code>, <code>finalized</code> bool, <code>no_holdout</code> bool, <code>baseline_val</code>,\n<code>baseline_val_stderr</code>, <code>best_val</code>, <code>test_reward</code>, <code>test_stderr</code>,\n<code>test_baseline_reward</code>, <code>test_baseline_stderr</code>, <code>test_delta</code>, <code>test_pass_k</code> (k→float),\n<code>val_test_gap</code>, <code>iterations</code>, <code>target_profile</code> (<code>{model,tier,resolution_note}</code>), plus\n<code>dashboard</code> / <code>dashboard_server</code> / <code>*_error</code> on the paths that produce them.</p>\n<h2>How to run</h2>\n<pre><code>python scripts/run.py --run-dir .capevolve/run_XXXX            # JSON + report.md + dashboard.html\npython scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report\npython scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard\n</code></pre>\n<p><code>--dashboard-mode</code> / <code>--dashboard-port</code> / <code>--dashboard-url</code> are orchestrator-supplied.\nRe-reporting by hand after <code>cap-evolve run</code> needs <code>--dashboard-url &lt;the URL run printed&gt;</code>\nor <code>--no-dashboard</code> — launching is deliberately not idempotent, so a bare re-run spawns\na second server on a second port and reports that one instead.</p>\n<h2>The dashboard (<code>dashboard.html</code>)</h2>\n<p>One self-contained static file (inline CSS/JS/SVG, no CDN, no server, no network — opens\nfrom <code>file://</code>); the single shareable artifact. Eight panels reduced from the event log\nplus baseline/final, rollouts and the git store; every value passes a recursive secret\nredactor so a shared dashboard leaks no API keys; optional panels degrade silently when\nper-task data, diffs, or finalize are missing. <code>--terminal</code> renders the same reduction\nas an ANSI chart for in-chat progress.</p>\n<h2>References</h2>\n<ul>\n<li><code>references/concepts.md</code> — why the val→test gap measures overfitting, pass^k\nfragility, reporting uncertainty, with sources. Load when writing the human\ninterpretation and you want the reasoning or a citation.</li>\n<li><code>references/dashboard.md</code> — the reduced graph + summary schema, per-panel field\nsources, <code>--terminal</code>, redaction, degradation matrix. Load when changing or\ndebugging the dashboard; not needed to run the phase.</li>\n</ul>\n","files":[{"path":"meta.yaml","sizeBytes":723,"isText":true},{"path":"references/concepts.md","sizeBytes":3640,"isText":true},{"path":"references/dashboard.md","sizeBytes":6163,"isText":true},{"path":"scripts/abstract.py","sizeBytes":163,"isText":true},{"path":"scripts/_bootstrap.py","sizeBytes":3694,"isText":true},{"path":"scripts/check.py","sizeBytes":12367,"isText":true},{"path":"scripts/dashboard.py","sizeBytes":459,"isText":true},{"path":"scripts/run.py","sizeBytes":10741,"isText":true},{"path":"SKILL.md","sizeBytes":5351,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"notes-only","suspicious":0,"notes":1,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-01T18:59:51.895694Z","sha256":"F10B397BC353D4D050C07610AD01A307BC058BF86D521CB9AF6EFDA9D77C4B66","sizeBytes":18634},"review":null,"source":{"repositoryUrl":"https://github.com/skillberry-ai/cap-evolve","path":"skills/phases/report","license":"Apache-2.0","commit":"da4781c8c51a48f2c2bd7fb0cfc779d3940bf05b","subtreeSha":"FA7705244E0DDBBAE0F6F962103DAC88FD2F0B07A7352101C4F3285F6D728B34","lastSyncedAt":"2026-09-26T23:11:58.154287Z"},"reviewedAt":"2026-09-01T19:01:43.350528Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/report"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart"},{"target":"git","command":"git clone https://github.com/skillberry-ai/cap-evolve.git"}]}