# agent-optimize — rationale, and how honesty survives full autonomy

## Contents

- [Why a free-form agentic algorithm](#why-a-free-form-agentic-algorithm)
- [How honesty survives handing the agent the wheel](#how-honesty-survives-handing-the-agent-the-wheel)
- [Subset screening](#subset-screening-where-the-cost-actually-goes-and-why-a-screen-may-not-accept)
- [The constraint surface](#the-constraint-surface-free-text-stop_condition-parsed-and-re-read)
- [Sibling candidates by default](#why-n3-sibling-candidates-is-the-default-not-one-candidate-at-a-time)
- [Provisional candidates](#provisional-candidates-sequential-evidence-not-compounded-edits)
- [JOURNAL.md write protocol](#journalmd--the-append-only-handover-and-its-write-protocol)
- [Parallelism](#parallelism-fan-out-on-the-cheap-steps-stay-serial-where-state-moves)
- [The final measurement](#the-final-measurement-one-table-and-the-things-it-refuses-to-pretend)
- [Gate as evidence, not a verdict](#gate-as-evidence-not-a-verdict)
- [Measuring only what the edit reaches](#measuring-only-what-the-edit-reaches)
- [Caveats](#caveats)
- [Process snapshot](#process-snapshot)
- [Sources](#sources)

## Why a free-form agentic algorithm

The deterministic algorithms (hill-climb, gepa, skillopt) fix the *schedule* of the
search: which tasks each round reflects on, when the optimizer is called, how the parent
is selected. That is exactly right when rollouts are cheap and the schedule is known. It
is a poor fit when the best move is judgment: *this* failure cluster is worth a targeted
policy edit, *that* one is uncontrollable infra noise to ignore; a subset eval is enough to
kill a bad idea before paying for full val; the score goal is already met so stop now.

agent-optimize hands that judgment to the conversational agent. There is no fixed round
count and no delegated per-iteration optimizer subprocess — the agent decides what to edit,
what to evaluate, when, and when to stop, bounded by a free-text `stop_condition`.

## How honesty survives handing the agent the wheel

Full autonomy is only safe because the honesty guarantees are **not** the agent's to keep —
they live in `core/cap_evolve/{gate,rundir,splits,check}.py` and hold no matter what the
agent does:

- **Test is sealed by code.** The evaluate phase only accepts `--split train|val`; the test
  split is scored solely by the finalize phase, once, after which `RunDir.commit_test()`
  burns the seal and a second finalize raises `TestSealError`. The agent cannot peek at test
  mid-run even if it tries.
- **Acceptance is a code gate on val.** `gate.decide` applies Δ > k·SE; the agent's subset
  triage can only *kill*, never accept — `scripts/screen.py` emits `kill`/`promote` and has
  no accept path at all. The agent reaches the *same* gate the
  deterministic loops use — the **paired** test — through `scripts/gate_check.py`, which
  rebuilds both sides' `SplitResult` from the persisted rollouts
  (`harness.split_result_from_rollouts`), builds the aligned per-task delta vector
  (`harness._paired_deltas`) and calls `gate.decide(mode="paired", …)`. The `phases/gate`
  CLI takes only two scalar means, so it can express only the weaker *unpaired*
  `significant` test; it stays the human-inspection front-end.
- **No-regression is enforced, not advised.** `gate_check.py` also vetoes a mean gain that
  strictly drops any val task the current best measured and passed — the same rule, and the
  same "tasks with no valid trial are missing data" exclusion, that `hill_climb_loop`
  applies. It is what diagnose's `kept_good` list exists to protect.
- **Edits are audited.** Every candidate is snapshotted in the git-backed store and every
  round is appended to `events.jsonl`, so the search is fully reconstructable.
- **A broken framework file stops the round, it does not get hand-worked-around.** On run
  33492876620 round 3, `_gate()` swallowed a `gate_check.py` crash into `{"error": ...}` and
  the caller `.get()`'d the missing verdict keys into `null`s in the round table; the agent
  noticed, recomputed the verdicts itself from `gate_check.py`'s raw output, booked the round
  `inconclusive` by hand, and said so in `JOURNAL.md`. That specific case is now closed by
  code: `_gate()` raises `GateCheckFailed` on a non-zero exit or unparsable stdout,
  `assert_rows_were_judged` refuses to publish any row whose reward is `null` when its own
  eval succeeded, and `round.py` exits non-zero with nothing written to the table — so there
  is no broken table left standing to work around. The **general** rule survives that fix and
  covers every other framework artifact (a malformed `screen.py`/`measure.py`/`diagnose`
  output, a torn JSON file on disk): treat it the same way `GateCheckFailed` is treated,
  never the way the old `_gate()` did. Stop, report the malformed file and what it should have
  contained, and let the caller re-run it — the underlying rollouts are already on disk, so
  re-deriving the verdict is free. Do **not** recompute a substitute number by hand and keep
  going, even when the hand-computed number turns out right, because the run is no longer
  telling anyone when this happens.

So the "free" in free-form is freedom of *strategy*, not freedom to fake a result. The
headline number is still produced once, on data the search never saw.

## Subset screening: where the cost actually goes, and why a screen may not accept

The unit of cost in a run is one full-val evaluation — `val_n × num_trials` rollouts, paid
once per candidate per round. Most candidate edits are not close calls, so most of that
spend buys a conclusion a fraction of it would have reached. GEPA already exploits this on
train (`gepa._eval_minibatch` plus its `sum(child) > sum(parent)` local gate); agent-optimize
ports the same economy to val and makes it **variance-aware**, which is the part Arbor's
structure lacks: Arbor's `merge_threshold` is log-only, never blocks, and there is no
repeated-trial, standard-error or significance machinery behind it at all.

Three design choices carry the honesty:

1. **The parent side is free.** The current best already has full-val rollouts persisted, so
   `screen.py` re-reads its per-task rewards instead of re-running it. Only the candidate
   pays, and only for the subset. That is the whole saving; there is no cleverer trick.
2. **A screen may kill, never accept.** `cap_evolve.subsample.screen_decision` returns
   `kill`/`promote` only. Acceptance needs the full split, because a subset chosen from the
   parent's failing tasks is *deliberately biased toward the tasks the edit targeted* — an
   excellent triage signal and an invalid basis for a decision. A subset `SplitResult`'s
   `coverage` is also 1.0 by construction (its denominator *is* the subset), so it would sail
   past the gate's low-coverage guard if it were ever handed over.
3. **The bias runs toward promote, deliberately.** With k≈4 and one trial the delta vector is
   coarse and the SE is large. A **false kill** discards a good edit and leaves no trace — the
   run simply fails to improve and nothing says why. A **false promote** costs exactly one
   full-val eval, after which the honest gate is correct anyway. So `screen_decision` kills
   only on `Δ̄ + k·SE < 0` or a unanimous negative (SE legitimately 0), and everything else —
   flat Δ̄ included — promotes with `inconclusive: true`. Lowering `--k-se` to make the screen
   "decisive" is the one tuning knob that makes the algorithm worse.

Subset composition is `broken_ids` (tasks a previous edit is known to have broken) → most
informative remaining (`(1-reward) + stderr`: headroom plus instability) → a seeded **random
holdout** drawn from tasks the parent *passes*. The holdout is the only part of a screen that
can see a regression, and it is why the classic churn candidate (fixes 2, breaks 2, identical
mean) at least surfaces its `regressed` list at tier 1 instead of looking like a tie. It is a
partial correction, not a complete one: with small k, a regression outside the holdout screens
clean and only shows up in the full-val gate's `regressions` list (which is diagnosis — it names
which part of a bundle to drop — and blocks an accept only under `--veto-regressions`). Selection
is deterministic given
the seed, and the whole record (ids, seed, holdout fraction, deltas, decision, measured
rollout economics) is written to `$R/screens/<tag>__tier<N>.json`, so a kill is auditable
after the fact rather than a decision that happened once inside an agent's context.

Rungs are cumulative — tier 2 merges tier 1's rollouts across `<tag>__screen*` tags and pays
only for the ids it adds — and savings are reported as measured integers
(`+ (full_val − fired)` on a kill, `− fired` on a promote) so a run's ledger sums to the truth
instead of to a flattering estimate.

### The break-even, and when the ladder cannot pay for itself

Screening is an economic bet, not a free improvement, and the arithmetic is one division:
`savings.breakeven_kill_rate = fired / full_val_rollouts` — the fraction of candidates the screen
must **kill** just to recover what its own rollouts cost. Screen only when that number sits below
the kill rate you have actually observed on this project.

The floor is what makes it unreachable on a small val. `screen.py` never fires a rung below an
absolute minimum number of tasks (currently 6), because a rung decided on two or three tasks is a
coin flip dressed as evidence. So on a 12-task val the tier-1 subset is half the split and the
break-even is **0.5** — every second candidate must be provably harmful. Measured across four real
runs on that project: the screen killed **0 of 8** promoted candidates while producing one
documented false positive (a 3-task tier-1 reported `fixed: ["44"]` for a candidate that full val
showed never fixed 44 — which is why the floor is 6, not 3). A ladder that cannot pay for itself and
mis-reports is worse than no ladder: pay full val directly.

Where the screen *is* worth paying for (large val, cheap tier), read it as **direct evidence about
the tasks the edit targeted**, not as a statistical test: a tier-1 subset containing every failing
val task that comes back 0-for-N on them is a sound reason to stop spending on that candidate. Book
that as a budget decision on screen evidence (`--reject-basis budget`), never as a gate decision.

### Targeted screening: draw the collateral-damage holdout from outside the cluster

`--broken <ids>` biases a screen toward the tasks an edit is meant to fix — the right bias for
"did the targeted tasks improve", the wrong one for "did anything else break", because those
targeted ids crowd out the holdout that would otherwise catch a regression elsewhere. Two
separate calls answer the two questions cleanly: `--broken <cluster-ids> --k <n> --holdout-frac
0` for a pure read on the cluster, and a second call with no `--broken` and a normal
`--holdout-frac` for collateral damage — its holdout draws from tasks the parent already
*passes* (`select_screen_subset`'s `passing` set), which a diagnosed failure cluster is not, so
it checks the untouched surface without having to name it. One `--k`-sized draw that mixes both
either drowns the targeted signal in noise or lets the same tasks stand for both target and
holdout at once.

### Choosing your own subset

`select_screen_subset`'s broken/informative/holdout heuristic is the DEFAULT selection, not the
only one you may use. `screen.py --ids <comma-separated task ids>` bypasses it entirely: name the
val tasks you want screened — from your own read of the candidate's target failure cluster's
rollouts, grouped by trajectory similarity (same tool misused, same failure site, same violated
expectation), or by any other method — and the same paired-delta kill/promote decision runs on
exactly those ids, with the same audit trail under `$R/screens/`. It can still only kill, never
accept, for the same reason the heuristic path can't: a subset you picked because it targets the
edit is biased toward that edit, which is excellent triage and an invalid basis for acceptance.

On **train** there is no screen/gate ceremony to bypass at all: `evaluate.py --split train --ids
<your subset>` (`cap_evolve.harness.evaluate_candidate`'s `ids` parameter, exposed on the CLI)
runs exactly the ids you name and nothing else, under a tag you control. Iterate on it as many
times as you find useful — different clustering, different candidate, different cluster size —
before ever touching val; train carries none of the honesty machinery, so there is nothing to
protect there.

**Never pass `--ids` to the step 4 full-val gate eval.** That call is the one place the loop's
freedom stops: `gate_check.py`'s coverage guard (`min_coverage`) exists to catch an
under-measured candidate, and a deliberately-chosen subset's `coverage` reads 1.0 by construction
(its denominator IS the subset) — the exact blind spot the guard cannot see through. The full-val
eval stays the whole split every round, same as the heuristic-screened path always required.

### `phases/gate` is an inspection front-end, not the round's gate

`phases/gate/scripts/run.py --mode paired` reaches the *same* paired gate off the *same* persisted
rollouts, so it is a faithful way to re-derive the significance half of a decision by hand — but
only in **rollout mode** (`--run-dir` plus `--current-tag`/`--candidate-tag`). Passing scalar
`--current`/`--candidate` means cannot do a paired test at all, since two means carry no per-task
delta vector, so that combination is refused rather than quietly downgraded to an unpaired test
whose number would look the same and mean something else.

It is still not the round's gate, for two reasons that matter to the audit trail: it does not read
`regressions`, so it cannot tell you which part of a bundled edit to drop, and it does not book the
decision into the run dir. Decide with `gate_check.py`; use this to inspect.

## The constraint surface: free-text stop_condition, parsed and re-read

Per the design, agent-optimize adds **no** new budget fields and no status command: the
project's free-text `stop_condition` plus the already-tracked run-dir spend is the whole
surface. What it *does* add is a normalizer, because prose is the right input and a terrible
thing for a loop to enforce. `cap_evolve.constraints.parse_constraints` turns

> "reach val mean >= 0.75, or stop after $40 / 90 minutes; don't regress task 12"

into `target_val_score >= 0.75`, `max_usd <= 40`, `max_wallclock_seconds <= 5400`,
`protect_task == "12"` — keeping the prose verbatim alongside — and
`check_constraints` re-checks each predicate against **measured** actuals every round,
emitting one `stop | continue | narrow_scope` recommendation. The tightest ceiling wins when
one is stated twice; the score goal is checked against the **full-val** mean only.

Two properties matter more than the parsing:

- **Ambiguity is reported, never guessed.** "don't spend too much", or a bare number with no
  unit, lands in `constraints.ambiguous` with the offending span and a reason, and an
  unparseable condition is explicitly *not* treated as "no constraint". SKILL.md's Phase 0
  says to clear that list with the user before the loop runs unattended — the one moment when
  asking is free.
- **Nothing is remembered.** Spend comes from `state.json`, wallclock from the first entry in
  `events.jsonl`, the val mean from the persisted rollouts, regressions from a seed-vs-best
  per-task comparison. A running total carried in an agent's context is how a $6.00
  per-iteration cap became $6.01 in a previous real run.

`spend.py --n-siblings N` closes the last gap: it prices N full-val evaluations at this run's
own **measured** `usd / metric_calls` and answers `afford.affordable` *before* the fan-out.
Before the first rollout is paid for it honestly answers `null` rather than "yes".

The recording half matters
just as much: the evaluate phase already books `metric_calls`/`usd`/`runner_seconds`, but in
agent mode *the agent is the proposer*, so `scripts/commit.py` takes `--optimizer-usd` /
`--optimizer-tokens` / `--optimizer-seconds` alongside `iterations` and the accepted flag
(which drives the stall counter). Without those, a cost- or stall-based `stop_condition` is
unenforceable no matter how carefully it is written.

Those per-round figures **attribute** cost inside the total the host already meters for your
whole process; they do not add to it. The host books the residual (`host.optimizer_spend_to_book`),
so passing them moves spend from the run-level `unattributed` row onto the round that caused it
and never double-counts — which is exactly what it used to do: the host booked its metered total
*on top of* whatever the agent had booked, so an agent following this instruction made the run
report up to twice the optimizer spend it used, and a `max_usd` stop then fired on money nobody
spent. Pass them when you can estimate your own cost for a round; a round you cannot price is
better left unattributed than guessed, since the run total is right either way.

## Why N≥3 sibling candidates is the default, not one candidate at a time

A round pays fixed overhead regardless of how many candidates it gates: the null-control
replicates (`--control-replicates 2` by default) plus, on the deterministic path, a baseline
re-check. That overhead is paid ONCE whether the round proposes one candidate or several, so a
round that proposes exactly one candidate spends nearly all of it on a single shot at the gate —
observed directly in a live run, where two consecutive rounds each proposed one candidate and
both were rejected, at the same fixed cost as three or more addressed in parallel would have
been. `round.py` already supports evaluating N candidates as parallel *processes* (each with its
own adapter `apply()`), gating them serially — the mechanism was already there; only the default
behavior of proposing one at a time was the gap. Default to N≥3, one failure cluster each, and
drop to fewer only when `spend.py --n-siblings N` says the remaining budget cannot afford it.

## Provisional candidates: sequential evidence, not compounded edits

A candidate that clears `Δ>0` but not `Δ > k·SE` is not the same as one at `Δ<=0`: the first is
a real positive direction the gate could not yet resolve at this `n`; the second has no
direction to chase. Discarding both alike wastes the first kind of evidence — three real rounds
on one benchmark each proposed a single candidate, paid the round's full fixed overhead, and
bounced off the noise floor with nothing carried forward, even though at least one of those
candidates' `Δ` was positive and simply under-measured.

The fix is a THIRD decision, `commit.py --decision provisional`, for exactly that case
(`gate_check.py` flags it as `directionally_positive_but_inconclusive`). It is deliberately
narrow: the only thing that happens next is buying more trials on the **same, unmodified**
candidate (`scripts/grow.py`) — never a new edit stacked on top of it. This is the line that
keeps sequential evidence-gathering from becoming the failure mode this project's own
post-mortems already named — "fewer mechanisms beat more", i.e. compounding edits on
unconfirmed ground. A provisional candidate buys precision on ONE measurement; it never becomes
the base a sibling edit builds from until it has actually been accepted.

**The pooling has to be real pooling, not two verdicts averaged.** `grow.py` runs the additional
trials under a throwaway tag, then `loop.pool_split_results` concatenates each shared task's
`trial_rewards` (not its mean) before re-aggregating — so the pooled `SplitResult` is what a
single evaluation at `n = n_old + n_new` would have produced, and the paired significance test
that follows sees the honest combined variance. Averaging the two rounds' *deltas* instead would
silently discard the fact that more trials narrow the SE; averaging their *verdicts* would treat
a 60%-confidence read and a 40%-confidence read as one vote each. `grow.py` then merges the new
rollout files onto the candidate's own tag (renumbered past its existing trial indices), so every
later reader — `gate_check.py`, `commit.py`, the dashboard — sees one candidate at the pooled
`n` with no special-casing for having been grown.

**The honest risk, and its cap.** Could a provisional lineage get stuck accumulating trials on a
false positive for many rounds, burning budget chasing noise that looked promising once? Yes —
that is exactly the failure mode a gate without a stopping rule invites, and it is why growth is
capped at **`--max-growth-rounds 2`** (`grow.py`'s default): at most two extra rounds of added
trials before the candidate must be finally promoted or abandoned, never grown a third time.
Two rounds is enough to roughly double or triple `n` on a candidate that started with a small
val, which is where most of the SE reduction from added trials actually lives (diminishing
returns set in well before a fourth round would). A candidate that still has not cleared the bar
after two growth rounds is abandoned (`commit.py --decision reject --reject-basis gate`) — the
cap turns "maybe next round" into a bounded, auditable cost rather than an open-ended one.

## JOURNAL.md — the append-only handover, and its write protocol

`$R/work/$TAG/JOURNAL.md` exists in your working copy from the moment you `cp -r
"$R/candidates/$BEST" "$R/work/$TAG"` — the framework seeds it (`harness._seed_journal`) onto
the seed candidate before round 1, and re-seeds it onto whichever candidate is `$BEST` after
every `commit.py` call, so it is always present at the start of a round regardless of how many
rounds came before. It is YOUR file (append-only, never reset): the run-level copy at
`$R/JOURNAL.md` accumulates one entry per iteration, accepted and rejected alike — but that
accumulation is done BY THE FRAMEWORK (`harness._reconcile_journal`), never by you directly.
Edit `$R/work/$TAG/JOURNAL.md` only; never append to `$R/JOURNAL.md` yourself (e.g. via a bash
`cat >>`) — it is rewritten wholesale on the next `_seed_journal`/`_reconcile_journal` pass, so a
direct edit there is invisible to the parser and gets clobbered.

The write protocol:

1. Read the WHOLE file before proposing — not just the last entry — so you build on every prior
   attempt and never re-test a refuted idea.
2. Append your new entry BELOW the marker line, in `$R/work/$TAG/JOURNAL.md`
   `<!-- cap-evolve:journal-append-below — add your Iteration entry under this line; do not edit
   anything above it -->`. Never edit or delete anything above the marker.
3. Use the heading `## Iteration <candidate id> — <one-line headline of what you tried>`,
   followed by: the changes you made, the expected effect of each, which prior RESULTS you built
   on (or explicitly avoided because a prior RESULT proved it regressed), refuted hypotheses, and
   your focus for the next iteration. You cannot know your own gate result while you write it —
   `commit.py` stamps a `**RESULT (framework, objective):**` line right below your entry
   afterward (accept/reject, Δ, and the exact tasks fixed/broken vs the parent), which is the
   authoritative record of what actually worked — read it, don't guess at it.

This is a general convention for ANY continuous-session algorithm (one long-running optimizer
subprocess spanning many rounds, as opposed to the deterministic loops' fresh per-iteration
optimizer workdir): the framework re-seeds the same file at the same two points
(`harness._seed_journal` called once before the session starts, and once per `commit.py` call
afterward) so a session that never gets a fresh workdir per iteration still gets a fresh append
target every round.

## Parallelism: fan out on the cheap steps, stay serial where state moves

Arbor's discipline — dispatch independent workers into separate worktrees, evaluate each on
a dev signal, merge only what clears a held-out margin — ports cleanly, with the boundaries
cap-evolve's own state model dictates (see `docs/SUBAGENT_PATTERNS.md`):

- **Diagnosis** is read-only and costs no rollouts, so it fans out without limit.
- **Proposal** fans out across *different* parents/working copies only, never two proposers
  on one candidate dir. Each sibling needs a **unique tag**, because rollouts are written as
  `<task>__<tag>__t<k>.json` and the evaluate phase derives the tag from the candidate dir
  name: two concurrent evals sharing a tag interleave into the same filenames and corrupt
  both scores.
- **The gate is serial.** `set_best` mutates run state and paired deltas are computed against
  the *current* baseline, so admitting sibling A invalidates sibling B's deltas. B must be
  re-gated against the new best before it is committed; skipping that double-counts one gain.
- **Test is never parallel.** The seal is single-use.
- **Budget is checked before the fan-out**, for N evals rather than one: N siblings can
  exhaust a budget that had room for a single round (`spend.py --n-siblings N`).
- **Screening is where fan-out pays best.** N tier-1 screens cost roughly one full-val eval
  between them, so the expensive stage runs only for survivors.

Inside a single evaluation there are two further, composable sources of concurrency: an
adapter's own `run_batch`/`run_trials` fast path (some adapters run their whole task
grid at their own concurrency setting), and framework-level pooling via `trials.run_trials_pool`
(`screen.py --workers N`). The pool only parallelizes rollout *generation* — scoring and
persistence stay serial and in task order — so pass^k, SE and the gate see exactly the numbers
a serial run produces. It is opt-in because `adapter.run_target` is not required to be
thread-safe.

## The final measurement: one table, and the things it refuses to pretend

A val number is the signal the gate optimized against, so quoting it as *the* result quotes
the training signal. `scripts/measure.py` produces the run's one reportable table — seed vs
best on val (free, off the rollouts the gate used), on train when that adds information, and
on the sealed test split via the same `harness.finalize` the finalize phase calls — with
mean, stderr, n scored / n in split, the paired delta vector's mean + SE + n, the recomputed
gate decision (val only; `gate.decide` raises `TrainGateError` for anything else), and the
per-task fixed/broke/unchanged movement.

Three refusals are the point of it: an **empty** split reports `empty` rather than a 0.0 that
reads like a measured failure; a **no-holdout** spec (test overlapping train/val — some benchmarks ship a
default split file that makes all three the same ids) is labelled a **FIT metric, not
generalisation**, with the overlap counted; and `best_id == "seed"` emits a warning that every
delta is 0 *by construction* and must be reported as a null result with a diagnosed cause.
`--train auto` also declines to pay for a train evaluation whose ids equal val's, because the
numbers would be a copy.

### Never abandon a running FINAL eval

`measure.py` opens the sealed test split by logging `eval_start(split=test, tag=FINAL)` before
it starts scoring — the same pairing every eval in the run uses (`harness.py`'s own docstring:
"an eval_start with no evaluate after it is an evaluation that never returned"). The test seal
is single-use, so this is the one eval in the whole run where abandoning it mid-flight is not
just a wasted wait:

- If your turn or process ends while `measure.py`/`finalize.py` is still running, nothing is
  left to read `final.json` even if the eval finishes on its own a minute later — the same
  failure mode `host.py`'s Unattended briefing warns about for a round's gate, just at the
  one point in the run where it also costs the seal.
- Worse than a mid-round eval: the abandoned attempt's PARTIAL test rollouts are enough to make
  a retry refuse. `begin_test_attempt` (`rundir.py`) sees test already has rollout files on
  disk and raises `TestSealError`, because it cannot tell "crashed before scoring" from
  "crashed after scoring but before the result was read" — the second case must not be
  silently re-scored, so it isn't, even when the first case is what actually happened.
- Measured on three separate runs: `eval_start(split=test, tag=FINAL)` on disk, no matching
  `evaluate`, no `final.json` — the seal was never consumed, just spent on nothing. `host.py`
  flags this shape as an `eval_abandoned` event and reports it in `dangling_eval` when its own
  seal backstop (`_seal`) also fails for exactly this reason, so at least the gap is visible
  instead of silently read as "the run just never finalized."

So: run `measure.py` in the foreground, and do not end your turn — or let a headless session
exit — until it has printed its result. If you are running low on turns/budget, that is a
reason to run it SOONER, not to launch it and move on.

## Gate as evidence, not a verdict

The statistics come from two scripts, and it matters which one prints what: `gate_check.py` prints
`delta`, `stderr`, `resolvable_effect_size` and its own `"verdict"` for ONE candidate; `round.py` prints
the round-scoped numbers no single-candidate gate can see — `noise_floor_from_control` and
`verdict_by_reference`/`verdict_stable`, the sign-agreement check across the round's null-control
replicates. Neither decides for you. Nothing in `commit.py` or `round.py` checks that field against the `--decision` you pass: `set_best()`
is an unconditional setter, and `--reject-basis driver_judgement` exists precisely so you can log a
considered disagreement. Treat the printed numbers the way a careful researcher reads a stats printout,
not the way code reads a boolean:

- Read `resolvable_effect_size` first. It is the smallest true effect this round could have detected at
  all — a `delta` at or below it is not evidence either way, whatever the printed verdict says.
- Does `delta` clear the noise floor measured for THIS round (`round.py`'s
  `noise_floor_from_control`), not just the a priori `k_se` threshold baked into the printed verdict?
- Does the verdict survive the choice of control replicate (`round.py`'s `verdict_by_reference` /
  `verdict_stable`, always computed once there is more than one control block)? A round where they
  disagree is telling you the noise floor itself is unstable this round, not just that one candidate is
  borderline.
- Before any accept, or any decision that disagrees with the printed verdict, write one sentence in
  `commit.py`'s `--note` citing the actual numbers (delta vs resolvable effect size vs noise floor). This
  is not optional ceremony — it is the audit trail a human reviewer uses afterward to check your judgment,
  the same way a rigorous post-mortem checks the numbers behind every claim.

**Why this doesn't regress to the coin-flip-accept failure a significance bar this loose already produced
on a real benchmark**: the arithmetic that caused that failure — banking `delta > 0` as progress when the
significance bar sat below the measured noise floor — is untouched. `gate_check.py` still computes the same
rigorous statistics and prints them prominently; what changes is only that you must engage with those
numbers in your own reasoning rather than pattern-matching a boolean field. The historical failure mode was
"the bar sat below the noise floor and nobody could see it" — with `resolvable_effect_size` and
`noise_floor_from_control` printed first, that is now structurally hard to miss.

**The new risk this introduces, honestly stated**: a strong optimizer can rationalize accepting something
the numbers don't support, dressing noise-chasing in confident-sounding prose — the failure mode moves from
bad math to motivated reasoning, which is harder to catch mechanically than a wrong formula was. The only
mitigation available is the audit trail above: every override gets a human-readable justification citing
real numbers, reviewable after the fact, the same way this repo's own run post-mortems already work.

## Measuring only what the edit reaches

Two things a round prints are claims about *causality*, and both were being made at a precision the
measurement did not have. The evidence is run_finalrun6 — 7 candidates, 30 val tasks, 10 trials.

### The footprint: which tasks could the edit have moved at all?

An edit touches a handful of named surfaces. The gate, though, used to measure its delta across every val
task. The tasks the edit cannot reach do not sit at zero — they wobble, because measuring a task twice is
not the same as measuring it once — and that wobble goes straight into the SE of the mean. On
run_finalrun6, `SE(paired Δ)` ran **0.022-0.035** while the real per-edit effects were **0.011-0.05**: the
bar was as wide as the thing it was measuring. The same run's 7-way null-control replicate spread
(0.567-0.603) was statistically indistinguishable from its 7-way across-candidate spread (0.570-0.607) —
the round could not tell an edit from a re-measurement of the same code.

The fix is to measure over the tasks the edit can causally reach. `gate_check.py` diffs the candidate
against its reference, reads the identifier-like names off the changed lines, and asks which tasks'
persisted rollouts mention any of them (`cap_evolve.footprint` — deliberately a flat substring search over
the whole rollout record, so it works for any adapter's trace shape and any capability). Tasks outside that
set enter the delta vector as **0.0 rather than as their measured wobble**: an edit that cannot reach a
task has no effect on it by construction. The vector keeps its full LENGTH, so `Δ̄` stays on the same SCALE
as the val reward, and every threshold, ledger row and val-curve point derived from it stays comparable.

It changes **both** halves of the test, not just the variance: `Δ̄` moves too, because the out-of-footprint
deltas that used to be averaged in are now zeros. That is the mechanism, not a side effect — and it is why
the footprint must never UNDER-include. Zeroing a task that really regressed raises `Δ̄` *and* lowers the SE
together, both pushing toward accept, so a net-harmful candidate could be accepted on a fabricated gain.
Two rules prevent it: the enclosing definition is **always** part of the footprint (an edit inside a body
reaches everything that calls the body, not just the rare helper it happens to call), and a surface that
reaches ≥80% of tasks makes the edit **unlocalizable** — abandon and measure the full split — rather than
being dropped so the rarer surfaces beside it can define a confident, narrow, wrong footprint.

A third rule guards the other end. On a SMALL footprint the paired SE, which is estimated from the
cross-task spread of the vector, understates the real uncertainty: 4 tasks whose deltas are
{0, +0.1, 0, +0.1} have a small spread, so the gate reads SE 0.0046 and accepts — on two moves of one
flipped rollout each, the very moves `broke`/`fixed` below refuses to call real. So a restricted vector's
SE is floored by `harness.paired_se_floor`, the SE those tasks' own per-trial noise implies. Unrestricted
vectors keep the SE they always had, so no pre-existing verdict moves.

On the worked case in `core/tests/`, a real +0.1 on 3 of 30 tasks goes from unresolvable to accepted with
the bar falling roughly 4x. On run_finalrun6's real rollouts the mechanism is far more conservative: five
of seven edits reach the split too broadly to restrict at all, and neither restricted verdict changes
(cand_5 17/30, SE 0.0318 → 0.0268; cand_7 4/30, SE 0.0346 → 0.0113 floored). Its value on that run is that
it ABSTAINS rather than manufacturing confidence — an earlier revision without the three rules narrowed
cand_7 to 4 tasks at SE 0.0046 and accepted a docstring-only edit.

Read the `footprint` block in each row before the delta:

- `restricted: true` — `n_in_footprint` of `n_tasks` were in play. The SE describes those tasks.
- `restricted: false` — the surface could not be localized: no diff, a rewrite-sized diff (more distinct
  symbols than a targeted edit would name), no rollouts on disk, a footprint that covers every task
  anyway, or one of the edit's own surfaces reaching ≥80% of tasks. The measurement fell back to the whole
  split, so it carries every task's noise. **A null result here is weak evidence about the edit, not
  evidence against it.** `--no-footprint` forces this mode.
- `paired_se_floor` — the bar the restricted SE was floored at. When it equals `gate_threshold / k_se`, the
  verdict rests on per-task trial noise rather than on the cross-task spread.

The over-inclusion is deliberate: a symbol mentioned only in prose still counts as a hit. Over-including
gives back some of the noise the restriction removes; under-including would zero out a task the edit really
did move and manufacture a gain. Every unknowable case degrades to `restricted: false`, never to a crash
and never to a restriction invented from nothing.

This is also the mechanical argument for **sibling candidates over bundles**: a bundle's footprint is the
union of its parts', so a two-surface bundle is measured at close to full-split noise while each part
measured alone would have been resolvable.

**Not built, and why**: the round does not yet spend the freed budget on more trials over the smaller
in-footprint task set. That needs subset-scoped evaluation plumbed through `round.py`, and `grow.py`
already buys more `n` on an unresolved candidate — footprint-scoped growth is the natural next step, not a
prerequisite for the restriction being correct.

Free on the reference side, and already done: `gate_check.py --current` accepts a comma-separated list of
tags and **pools their trials per task**. `round.py` passes every one of the round's byte-identical
null-control replicates, so the control-relative comparison is against a pooled estimate rather than
whichever replicate happened to carry the round-scoped tag — which was a coin flip (the same candidate read
+0.0867 against one replicate and +0.0067 against the other). No new rollouts: those files are on disk.

### broke / fixed: is this task's move a claim the measurement supports?

`broke` and `fixed` used to be thresholded at `1e-9`, i.e. at nothing. At 10 trials a task that goes
`1.0 → 0.9` has had **one rollout out of ten** flip, and that was being stamped as a task the candidate
broke. It is not a hypothetical failure: on run_finalrun6, byte-identical code measured twice — `cand_2`
and its own fresh-tag re-measurement `cand_4` — got *different* `broke`/`fixed` labels, and the optimizer
reasoned from them for three rounds, including asserting a regression on a docstring-only edit that cannot
change behaviour at all.

Every such claim now has to clear **2·SE of its own per-task measurement** — the two sides' per-task SEs in
quadrature, the same "smallest resolvable effect" the gate reports for the split mean, applied per task.
One function, `harness.move_is_resolved`, is the single bar behind every place the framework makes the
claim: `LEDGER.md` + the journal RESULT stamp (`_candidate_task_impact`); the `no_regression` veto in both
the hill-climb and gepa loops, which is the strongest consequence of the set since it turns a
gate-PASSING candidate into a rejection; the round table's diagnosis list (`gate_check.regressions`);
skillopt's within-epoch improved/regressed buffer (`_categorize`); the sealed report's seed→best movement
and its improved/regressed counts (`measure.py`); and the "don't regress task X" constraint check
(`spend.py`). Seven copies of a bar is how six of them stay at `1e-9` while one gets fixed.

A sub-threshold move is reported as **`unresolved`**, which is neither "broke" nor "unchanged": the reward
moved, and the measurement cannot say the edit did it. Do not redesign an edit because a task appears
there, and do not cite one as a regression — re-measure it, or ignore it. At one trial per task every
per-task SE is 0, the bar collapses to `eps`, and the classification is exactly what it always was.

## Merging accepted candidates before you finalize

`merge_search.py` (#438) exists precisely because run_agentoptv3/run_agentoptv4 produced 3-6
narrow, single-issue candidates per round and never combined them (see its own module
docstring). SKILL.md's "Before finalizing" step makes running it, once 2+ accepted candidates
target disjoint clusters, a REQUIRED step rather than an available tool nobody reaches for under
time pressure — the same gap that let `screen.py` sit unused for a whole run before its own
compliance event existed (#420 item 4).

Practically: `--survivors` takes any tag under `$R/work/`, whether or not it individually cleared
the gate — an `accepted` graph node works exactly like a screening survivor for this purpose, since
disjointness is a property of what the edits TOUCHED, not of how they were judged. `--targets`
(or a `mechanisms.jsonl` row per tag) supplies each one's task ids; a tag with neither is skipped,
never silently merged on an empty objective. The merge itself pays `integrate.py`/`funcmerge.py`,
same as any hand-driven merge (per-task-fanout.md); a real edit collision (both branches touch the
same function differently) is refused, never force-merged.

`measure.py` runs `merge_search.check_merge_compliance` at the start of every finalize-time call:
it reads `graph.jsonl` for `accepted` nodes, their target ids (`cluster_ids` when populated, else
the same `mechanisms.jsonl` fallback `merge_search.py` itself uses), and whether any node anywhere
in the graph carries `edit_kind == "merge"`. Two or more accepted candidates with disjoint targets
and NO merge attempt anywhere in the run logs `merge_compliance_warning` to `events.jsonl` — visible
in the dashboard's activity log like any other event. It never blocks: host.py owns no algorithm
decisions (the "orchestration freedom" invariant), so this is an audit signal for the same reason
`agent_optimize_compliance` is one for the screen ladder, not a second enforcement mechanism.

## Caveats

- With `train == val` the val gate is a *fit*, not a held-out check — only the sealed test
  number generalizes. Label val a fit metric in any report.
- At `num_trials: 1` on a stochastic benchmark, single-trial val means carry real variance;
  the paired k·SE gate curbs false accepts, but consider a re-eval before sealing if the
  score goal is only just met.

## Process snapshot

`graph.jsonl` (#446) already records every candidate's parents, `cluster_ids`, screened
`subset`, `micro_tests`, and gate verdict as it's committed — a VIEW over `events.jsonl` /
`round.py` / `screen.py`, not a new source of truth (`graph.py`'s own docstring). Nothing reads
it back mid-run by default, so the reasoning behind a run is invisible until the `report` phase
renders it at the very end.

Regenerating it live closes that gap with no new mechanism: `python -m cap_evolve dashboard
--export "$R"` calls the exact same `dashboard.reduce_run` + `render_html` the `report` phase
uses, and writes the result to `$R/dashboard.html` — a single self-contained HTML file (inline
CSS/JS/SVG, no CDN) with the candidate DAG (nodes colored by accept/reject, edges for parent
links, a tooltip per node with its screened task subset and cluster ids), gate decisions, cost,
and your own accept/reject notes. Call it after every `commit.py` — the run is small enough that
re-reducing it each time is cheap, and it overwrites in place, so there is always exactly one
current snapshot on disk, not a growing pile of stale ones.

The live dashboard's "Process" tab appears automatically once `dashboard.html` exists in the run
dir (`capabilities.process_html`) — this is the artifact both a human watching the run AND you,
re-reading it on your next turn, use to see the shape of the search so far without re-deriving it
from raw events.

## Sources

- GEPA: reflective prompt evolution with a Pareto frontier (arXiv:2507.19457) — the
  deterministic sibling this loop's "read the feedback, propose one targeted edit" step echoes,
  and the prior art for the minibatch economy `subsample.py` ports to val.
- Arbor (github.com/RUC-NLPIR/Arbor) — the source of the *structure* here: independent workers
  in separate worktrees, a cheap dev-signal screen, merge only what clears a bar. Its looseness
  is deliberately **not** ported: Arbor has no repeated trials, no standard error, no
  significance test, and its `merge_threshold` is log-only and never blocks. Every idea taken
  from it is re-expressed as a variance-aware decision (`screen_decision`'s `Δ̄ + k·SE < 0`) that
  can only kill, with acceptance left to the paired val gate.
- cap-evolve honesty model: `docs/HONEST_EVAL.md`, `docs/ARCHITECTURE.md` (splits/gate/seal in core).

# Edit design — what the scorer audit found, and how guards backfire

Read this before writing an edit on a surface you have not edited before, and always when two rounds
have been rejected. SKILL.md carries the checklist and the edit-form table; this file carries the
failures that produced them, with what each one cost.

The numbers come from real runs on a multi-turn tool-use benchmark with a mid-tier agent model. The
*shape* of each finding transfers; the figures are that run's.

## Auditing the measurement before you credit a failure

Every optimizer that skipped this step optimized against its own instrumentation. Scoring is
deterministic on persisted rollouts, so each repair below re-derives at **zero** rollout cost.

**Does the feedback name the actual defect, or only the tool?** If it says "Failed action(s): a
write tool" when the real defect is a wrong *argument value*, no edit can be localized. Measured:
argument-value errors were the majority of failed gold actions. A predecessor read the wrong field
name for the action check, reported "0 actions missed" for every task, and stayed blind to the
dominant failure mode for an entire effort. Fix the adapter's feedback, then re-derive.

**Does any feedback helper fail SILENTLY?** Grep the adapter for bare `except` around signal
construction and make each one loud. Measured: a localizer called a helper method that did not
exist; the `AttributeError` was swallowed, so every failed numeric check degraded to the generic
*"1 required piece(s) of information were not clearly communicated"*. An optimiser read that as "the
checker is unsatisfiable" and spent **seven rounds** instructing the agent to state a value it was
already stating. The repaired signal distinguishes *never stated a figure* from *stated one and it
was wrong* — and re-deriving it over 125 already-persisted rollouts cost nothing.

**Distinguish "silent" from "wrong" for every value-bearing check.** They are different defects
needing opposite edits (add a REQUIRED slot vs. fix arithmetic/scope), and a message that conflates
them sends the round in the wrong direction. Report the value the AGENT stated, never the expected
one — a check's `info` field often *is* the expected value (one benchmark stores a bare `"1628"`), so
use its SHAPE and never echo it.

**Did the rollout run, or did the infrastructure fail?** A wallclock timeout, a starved endpoint or a
dropped connection is missing data, not a zero. If it lands in the mean as 0.0, a whole evaluation
can read as a catastrophic capability with no error anywhere. Check `termination_reason` and the
coverage the gate reports — a low-coverage split must be `indecisive`, never a score.

**Would a clearly-wrong candidate score worse?** If not, the metric is not discriminating and no gate
built on it can work.

## Why churn needs a bundle you can take apart

The failure mode to design against is churn, and it is measured, not hypothetical: in a real run two
of three candidates had an *identical* mean to their parent while a different set of tasks passed —
each fixed 2 tasks and broke 2. A mean-only gate calls that a tie; the paired gate rejects it because
it sees per-task movement in both directions. That is why a multi-part edit is safe to attempt at
all — and why, when you bundle, the parts must be *independent* (different files or different rules),
so that a rejected bundle can be resubmitted as its surviving part next round. Read `regressed` out
of the screen and `regressions` out of the gate to know which part to drop.

## Form, not wording

- **No nuance clauses.** Appending one qualifying clause to an otherwise-winning recipe degraded it
  from consistent to noisy. If a rule needs a caveat, restructure the rule. This applies to refusal
  text too: adding two *correct* discrimination clauses to a working refusal took a task 0.2 → 0.1,
  with all ten trials failing. The clauses were right; the longer refusal traded follow-through for
  precision. A refusal has a budget — every sentence competes with the one saying what to do next.
- **Exemption clauses do not scope.** "This limit does not apply to X" still suppresses X.
  Restructure so the rule cannot reach the exempt case in the first place.
- **Prefer an in-code guard to a prose rule when the capability owns its tools.** The only edit that
  ever carried a large accepted gain on that benchmark was tool-level (`tools.py` 593 → 832 lines,
  +0.176 val): a precondition that refuses the illegal write and returns a recovery-oriented error
  changes behaviour deterministically, where a policy sentence changes it only probabilistically.
  Prose is the right form when the agent *lacks* a decision criterion; code is the right form when it
  has one and violates it.

## Confirmation-without-execution: the agent narrates the change instead of making it

A named failure class, and the one most reliably mis-diagnosed. The trajectory shows the agent
proposing a change, the user approving it, and the final message reporting the change as done —
with specifics — while **no mutating tool call appears anywhere in the trace**. The model has
taken its own completion signal (the approval) as satisfying the task and substituted narration
for the call. It is a documented property of LLM agents, not a property of any benchmark, so
expect it on any multi-turn capability that asks before it writes. `diagnose` names it
mechanically as the `narrated_without_action` cluster; without that it hides inside a
"wrong write" cluster, because a scorer describes both the same way, and the round then ships
an argument fix for a call that never happened.

**A prose reminder will not fix it.** "Always call the tool after the user confirms" has been
tried here and rejected: the agent already knows the rule and violates it anyway, which is
exactly the case the edit-form table sends to code rather than to prose.

The fix is structural — make *confirmed by the user* and *mutation executed* the SAME action, so
no code path can reach one without the other. In practice: one call that takes the approved
change and performs it, sharing the body with whatever the confirmation path already does, and
`remove` the primitives that let the two come apart. Then check the fix FIRES on the failing
trajectory: re-run the new body on that trajectory's own arguments. Ship nothing whose only
change is a sentence telling the agent to act.

## Guard closure: a guard that forbids the harmless option can force the harmful one

**Ask what the agent does INSTEAD.** Measured: a guard refusing a change that changes nothing ("this
call would change nothing, so do not quote a price") is locally correct — an unchanged record
genuinely cannot produce a credit. On the task where the right answer was *make no change at all*, it
cost 0.288. Removing that one guard, policy byte-identical, halved the damage (−0.288 → −0.147, no
longer resolvable) while the paired task kept its +0.498.

The failure is not the guard's logic, it is the guard's *closure*. Refusing the no-op left the agent
with only real changes to choose from, and it chose one. So before adding a refusal, name the action
set it leaves behind and check that "do nothing" is still reachable — a refusal that removes the
correct answer converts a pass into a fail while looking like a safety improvement.

This is also why the first ablation was worth running even though it refuted its own hypothesis: the
paragraph suspected of causing the loss turned out mildly *helpful* (removing it cost the paired task
0.141), and without that null the round would have shipped the wrong fix and kept the real cause.

## Auto-repair can accelerate a wrong action — a rejected call is sometimes a brake

When a tool bounces a recoverable argument slip, the agent spends a turn recovering, and turns are
scarce, so repairing the slip inside the tool looks like a free win. Measured counter-example: a
transaction whose payment id used an unrecognised alias and omitted the amount was rejected by the
parent and repaired by the candidate — but that transaction was itself premature, made with a
defaulted payment method the customer had never been asked about, and the customer then asked for a
different one, forcing an undo-and-redo that left an extra stale row in the database. The rejection
had been holding back a wrong write.

So before shipping a repair, ask what the rejected call would have DONE had it succeeded. If it would
have written the right thing a turn later, repair it. If it would have written the wrong thing
immediately, the repair needs to be paired with the precondition that makes the call correct — not
shipped alone, and not abandoned either.

# Measured lessons — what a number on this loop can and cannot resolve

## Contents

- [Measurement discipline — what a number here can and cannot resolve](#measurement-discipline--what-a-number-here-can-and-cannot-resolve)
  — per-task gradient noise, re-running the null, what the gate can resolve, the ceiling, the three
  phases of a fan-out, merge granularity and what a merge silently drops, and the four things that
  keep the phase honest.
- [The binomial floor, and what an aggregate mean can resolve](#the-binomial-floor-and-what-an-aggregate-mean-can-resolve)
  — the SE formula against a measured null, why temperature 0 does not make it "sampling error", and
  why narrowing to the hard tasks makes an artifact measurement worse.
- [Load is the other half of the noise](#load-is-the-other-half-of-the-noise) — the concurrency
  tables, the oversubscription incidents, and why total in-flight requests is the knob.
- [Gate the sum, not each addend](#gate-the-sum-not-each-addend) — the six-branch merge that gated
  negative, and the cost of certifying each mechanism by rate.
- [Take the error ACROSS whole runs](#take-the-error-across-whole-runs) — the two-seed-block table
  and the accept that had to be retracted.

Every rule here was paid for by a measurement on a real run, and each states the number that bought
it. They live outside SKILL.md because the loop has to stay readable: the body carries the contract,
this file carries the evidence behind it. Read it before your first gate decision on a new benchmark,
and again whenever a result surprises you. The figures come from a multi-turn tool-use benchmark with
a mid-tier agent model; the *shape* of each finding is what transfers, and where a number is likely
benchmark-specific the rule says so.

## Measurement discipline — what a number here can and cannot resolve

Everything below is about the instrument, not the edits. It is placed inside the fan-out section
because that is where it was learned, but it applies to every round: on this benchmark the
measurement floor turned out to be larger than most of the effects being chased, and four separate
conclusions in one round had to be retracted for ignoring it.

**Measure your per-task gradient's own noise before you trust it as a gradient.** The per-task
fan-out rests on `k/n` per task being informative. Measure whether it is, by running the *same
bytes* twice and diffing per task. Here, at n=5 on identical seeds with temperature 0:

| identical bytes, run 1 vs run 2 | value |
|---|--:|
| mean per-task \|difference\| | **0.160** |
| tasks that moved at all | **19 / 30** |
| tasks that moved >= 0.40 | 3 |
| worst single-task swing | **0.60** |

Task rates of 0.20 -> 0.80, 0.40 -> 0.80 and 0.60 -> 1.00 all occurred **with no change to the
code**. That floor is larger than most per-task effects worth chasing, so at this trial count the
`k/n` gradient is mostly noise, and a per-task "improved / regressed" list is close to
uninformative. At n=10 the floor is roughly 0.11 — still large against a claimed 0.20 step.

One nuance rescues the canary discipline: **the variance is concentrated in particular tasks, not
spread uniformly**. Across five byte-identical readings of one 12-task subset, two tasks read 1.00
in *every* run while three carried nearly all the movement. So a canary set drawn from
demonstrably stable tasks is trustworthy even though per-task rates in general are not — which is
why canaries must be chosen from REPEATED measurements at the real trial count, never from a
3-trial screen, and why "canaries intact" remained a meaningful statement all round even as the
target-task rates became unreadable.

The consequence is not "give up on per-task work"; it is **stop using rates as the per-task
evidence and use MECHANISM instead**. Everything from this run that survived scrutiny was
established structurally rather than by a rate delta: a crash found in live tool returns and
confirmed by its error string going 3 -> 0; a docstring section shown to reach the model 0% of the
time by rendering the schema and counting characters; a reward component shown to be non-gating
because the tool it names was invoked in 0 of 300 rollouts while the task still scored 0.8. Every
finding that rested on a rate difference at n=5 or n=10 was later retracted or downgraded.

So the per-task loop's real output is a *diagnosis you can verify without the metric* — a wrong
argument visible in a trace, a tool that raised, text that never arrived. Use the rate only to
decide whether to keep looking, never as the proof.

**Re-run the null ITSELF, not just once — the control's own run-to-run spread is the real bar.**
A single null-edit control tells you the noise floor only if that control is itself stable, and on
this benchmark it is not. Three full-val readings at n=5, all on **identical seeds** with
temperature 0:

| arm | reading |
|---|--:|
| control, run 1 | 0.6467 |
| control, run 2 (**byte-identical, same seeds**) | 0.7267 |
| candidate | 0.7333 |

The two control runs are **+0.0800 apart, and that null "passes" a k_se=1.0 gate** (bar 0.0379).
The candidate reads +0.0867 against run 1 and **+0.0067** against run 2 — the verdict is decided by
which control reading you happened to take. Anything measured at that trial count and below ~0.08
is unresolvable, which on this run included every panel comparison and the headline candidate.

Note what this is NOT: seeds were identical across all three runs and temperature is 0, and single
model calls are perfectly deterministic (six identical completions by hash). The variance enters
through the multi-turn conversation. So a determinism check cannot substitute for it, and neither
can more trials in the same block — you have to run the whole arm again.

The cheap discipline that follows: **evaluate the control twice before you believe any candidate**,
and set the bar from the null's own spread rather than from a formula. A gate whose bar is smaller
than the null's re-run delta is not a gate.

**Know what your gate can RESOLVE, not just what it costs.** Two numbers decide whether a round
can even see its own result. Measure the per-task noise (`sd` of a task's rate across repeat runs
of identical bytes — ~0.16 at `n=10` near p=0.5), then the paired mean's SE is
`sd*sqrt(2)/sqrt(val_n)`. In one run that is **0.041**, so at `k_se = 1.0` the gate can
resolve a gain above ~0.041 of val — about **1.24 task-equivalents**. That single number tells you
three things up front: a +0.057 gain is 1.4 SE and detectable; anything worth less than ~1.2
task-equivalents cannot be distinguished no matter how confident the per-task readings look; and
closing a 0.165 gap is a **4 SE** move, which is a different kind of ask from a 1.4 SE one. Compute
it before the round, alongside the headroom, and say both out loud.

**State the ceiling before you spend.** From the baseline's per-task rates, the recoverable
loss is `Σ(1 − rate)` over failing tasks, in task-equivalents; reaching a target `T` from a
current mean `M` needs `(T − M) × val_n` of it. Say out loud what fraction that is and which
tasks hold it. In one run: 9.66 equivalents available, 6.67 needed for 0.90 — **69% of
all remaining loss**, with 6.0 of it sitting in six tasks that score exactly 0.0. That makes
the target's shape explicit (every hard task must be fixed, not most of them) and it is the
difference between a plan and a hope. If the arithmetic says the target needs ~100% of the
available loss, say so in the report before the first rollout, not after the last.

**The three phases.**

1. **Fan out one optimiser per defect.** One subagent per `DEFECT` task; one per `UNSTABLE`
   *cluster* (unstable tasks share a mechanism more often than broken ones do). Each gets its
   own `cp -r` of the current best and edits only inside it.
2. **Merge.** `merge_taskopt.py` (git 3-way, one branch per optimiser). Two rules earn their
   keep here. **Declare a rebased optimiser's parent** — `--include u67b t21 t17b:u67b` — or
   its diff re-applies everything the parent already did and collides with the parent's own
   branch. And **classify every conflict before resolving it**, because the two kinds take
   opposite treatment:

   - **Semantic conflict** — two optimisers arbitrate the *same decision* differently (rival
     guards on one write, contradictory guidance at one moment). **Drop one bundle**; a union here
     ships contradictory instructions nobody measured. The tie-break is which side has a
     measurement, not which text reads better: the one such conflict observed came from
     a bundle measuring 0.50 against its own 0.60 baseline, whose author recommended against
     merging it, so the round shipped the verified copies and excluded it.
   - **Textual collision of distinct additions** — two new functions, or two new dict keys, that
     happen to land on adjacent lines. Here the union IS what both optimisers measured, and
     dropping one throws away a verified gain over a whitespace accident. Resolve with
     `--union-on-conflict`, which names the union-resolved files so the claim stays checkable.

   Union resolution has one hard follow-up: **render the live toolset**. Keeping both sides can
   duplicate a definition or break syntax, and an import check does not catch what registration
   does. In one run the union of five branches gave 596 added lines, 14 tools registering
   and no duplicated methods — checked, not assumed. The union is still a shape nobody measured in
   isolation, so the gate decides it: union to avoid losing gains, gate to find out whether you did.
   **The conflict may be an artifact of merging whole files.** Before treating a conflict as a
   real disagreement, check the granularity. Ten independently-verified branches in one
   round produced a whole-file merge that kept **four** of them; the "conflicts" were not
   disagreements at all. Every optimiser had added one state field to the *same* `__init__` and
   one independent guard call to the *same* tool method right after the same existing check, so
   their edits landed on adjacent lines of a shared insertion point. Line-level 3-way merge
   cannot tell *two people appended different things here* from *two people rewrote the same
   thing*, and diff3 conflicts on both. Forcing them through with `--union-on-conflict` produced
   a file that **did not parse** and carried five duplicated `def`s.

   So merge per FUNCTION, not per file — `funcmerge.py`, which runs the same git 3-way merge at
   a granularity where independent additions never interact. That raised retention from 4/10 to
   the full set. It resolves in three escalating steps, each of which reports what it did:

   - **pure insertions** (`--union-pure-insertions`): if *no* branch rewrites a base line, apply
     every branch's insertions, anchored to positions in the base so branch order cannot change
     the result. Provably safe, and it is the case that covers a shared `__init__`.
   - **priority trunk + insertions**: when branches *did* rewrite one function, one becomes the
     trunk and the rest contribute only their insertions, re-anchored by the CONTENT of the base
     line they followed. Pick the trunk by **which branch changed that function most**, not by
     whose task holds the most headroom — the branch owning a full task-equivalent turned out to
     have added exactly ONE line to the contested function (its real fix was in
     another function), so ranking by headroom discarded the branch that had actually rewritten
     the return value and kept nothing. *What a function is worth is not what its author's task
     is worth.*
   - **forced trunk** (`--force-priority`): a last resort that drops the *rewrites* of losing
     branches but still applies the *insertions* of every branch that only added. Dropping a
     whole branch because someone else rewrote the function is how a merge silently loses a
     measured fix — here it would have discarded task 42's guard call to settle a disagreement
     about a money string. Every drop is reported per function and must be re-measured.

   **Audit what the merge failed to carry, and read it against the ledger's rejected entries.**
   A forced-trunk resolution does not just lose a branch's gain — it can silently RE-APPLY a
   subtraction that branch had already measured and reverted. Observed live: one optimiser had
   added a sentence to a `payment_id` argument description and had separately logged, twice,
   that removing it was harmful; the merge dropped that branch's rewrite of the function and
   re-performed exactly that subtraction. Nothing conflicted, so nothing was reported, and a
   gate would have measured the regression without ever naming its cause. Trace evidence bore
   it out: of the stored rollouts on that task which made every write on the correct record
   and still scored 0, four of seven charged a credit card when the customer had asked to pay by
   gift card — precisely the defect the deleted sentence addressed.

   **A merge that carries a function but not a CONSTANT it needs produces a crash that looks
   like a policy failure.** This is the same lost-work class one level lower, and it is the most
   expensive single defect this run produced. `_check_bags_before_cabin_change` read
   `self.CABIN_LADDER` at four sites; the merge carried the helper *and* its call site and left
   the class attribute behind. The live tool return was
   `Error: '<ToolsClass>' object has no attribute 'SOME_CONSTANT'` — the tool layer turns the
   `AttributeError` into a string, the agent reads it, abandons the write, and the reward records
   a **missing write**, indistinguishable from the agent choosing not to act. It silently
   contaminated four measurements across two candidates and two ablations, and it was found by a
   per-task optimiser reading a live trace, not by any aggregate.

   So `funcmerge.py` now **refuses to write** a result in which any constant-shaped attribute
   read off `self` is undefined. Two details make that check safe rather than merely strict.
   Instance fields are routinely declared *with annotations* (`self.x: set[str] = set()`), which
   is `ast.AnnAssign` and not `ast.Assign` — collecting only the latter reported six valid fields
   as undefined and rejected a good merge. And only UPPER_CASE names hard-fail: the class under
   merge normally has a base class, an inherited method reached through `self` is not resolvable
   from one file, and refusing those would reject valid merges. A hard check with false positives
   is worse than no check.

   `funcmerge.py` therefore reports `dropped_additions`: every non-trivial line a branch added
   that the result does not contain. It is advisory, since some drops are the deliberate outcome
   of a conflict decision, but it must be read before gating. Run cold on seven branches it
   flagged lost work from **six of them**, including a `next_step` block that was part of an
   already-verified mechanism nobody had noticed was missing.

   Two things this exposed that no rate would have. A guard **helper** can survive a merge while
   its **call site** does not, leaving dead code that costs context and buys nothing — so verify
   the call, not the definition: `grep -c '_check_foo(record)'`, never `grep -c 'def _check_foo'`.
   And a ledger `touches` field named a function (`_remaining_upcoming`) that **no branch ever
   defined**, which is why the merged artifact must be checked against the code rather than
   against the ledger's own description of itself.

3. **Gate the merge once, on full val, against `ctl_null`.** Nothing from phase 1 or 2 is
   believed until this. Per-task fan-out changes where the search spends its rollouts; it
   does not change what counts as evidence.

   **A per-task gain is verified against ONE base and is not transitive to another.** This is the
   sharpest limit on the whole per-task fan-out idea, so measure it rather than assuming it. On a
   seed-matched comparison (identical trials for both arms, so seed variance cancels entirely),
   adding an optimiser's independently-verified task-14 edits to an artifact that already carried
   three other optimisers' work measured **-0.0617** overall — and **task 14 itself fell**, from
   0.40 to 0.20, despite the very same edits having measured 0.50 -> 0.70 at n=10 on the base they
   were developed against. Two other tasks fell 0.80 -> 0.00 and 0.80 -> 0.20.

   So "verified on my task, canaries intact" is a necessary result and not a shippable one. What a
   fan-out produces is a set of *candidate mechanisms*, each with evidence that it can work
   somewhere; which subset survives together is a separate measurement, and the only reliable form
   of it is a seed-matched paired comparison of the composed artifact against the artifact without
   the addition. Budget for that: it is not free, and skipping it is how a round of nine
   confirmed wins becomes a candidate that loses.

   **Select the merge on a headroom panel before you gate it.** A full-val gate answers one bit
   for 300 rollouts, and it answers it about a *sum*. In one round the merged artifact
   scored +0.0126 and was rejected — correctly — while containing, per task, both real gains
   (task 40 `0.10 -> 1.00`, task 21 `0.20 -> 0.80`, +2.1 task-equivalents gross) and real losses
   (task 10 `0.80 -> 0.10`, task 9 `0.80 -> 0.40`, -1.6). The gate could not see either. Keeping
   only the gaining half would have measured ~0.78. So evaluate merge variants on the tasks that
   can actually move — the ones below 1.0 — pick there, and spend the full-val gate on the
   winner alone.

   Two corollaries. **Tasks already at 1.0 cannot contribute a gain**, so a panel of the
   below-1.0 tasks is both cheaper and strictly more informative per rollout than full val for
   *selection* (it is not a substitute for the gate, which is what protects the tasks at 1.0).
   And **compute the headroom before choosing a target**: sum `1 - rate` over val at the real
   trial count. That arithmetic is what says whether the goal is reachable at all — in one run
   it read 8.70 task-equivalents over 30 tasks, so 0.90 needed 5.7 of them, i.e. 65% of
   everything left, with 5.4 of it sitting in six tasks. A target nobody has costed against
   measured headroom is a wish.

   **Regression attribution is free once rollouts are on disk.** Before spending anything to
   explain a drop, diff the stored failure feedback of the two arms per task. In one run that
   showed the regressed tasks had *identical* feedback strings in both arms at different
   frequencies — the edit shifted a tendency rather than introducing a bug, which is a different
   thing to fix and would have been invisible from the means. The same pass costs nothing and
   rules out infrastructure: 5 of 2040 val rollouts (0.25%) died for infrastructure reasons and
   were scored 0.0, concentrated on one task — small enough to ignore, but *measured* small
   rather than assumed small.

**Each optimiser's loop** — target task at full `n_trials`, plus a canary of tasks measured
**1.0** at baseline, in the same call:

The inner step is one optimiser's own task at full trials plus the canary, in a single call; after
the fan-out, combine and gate the merge like any other candidate:

```bash
python "$A/taskeval.py" "$R/work/$TAG" 7,17 --project "$P" --n <num_trials> \
       --canary 0,3,12 --canary-n 3 --traces /tmp/tr_$TAG.json
python "$A/merge_taskopt.py" --root "$R/work" --base "$R/work/<parent>" \
       --out "$R/work/cand_merged" --include t7 t17 u33
python "$A/round.py" --run-dir "$R" --project "$P" --candidates cand_merged \
       --n-trials <num_trials> --k-se <gate_k_se>
```

Run each eval **detached** (`nohup ... &`, then poll for the output file). Under endpoint
contention a per-task eval can take 15-50 minutes, and one optimiser lost a 64-minute round to a
harness-level timeout killing an eval that was still healthy — detached, the same round survived.

Then read `/tmp/tr.json` — the agent's own tool calls with arguments, per failing trial — and
aim the next edit at an observed decision. For an UNSTABLE task, **diff a failing trial
against a passing one**: the divergence point is the ambiguity, and removing it beats adding
a rule (see *Match the Form to the Failure*).

**The four things that keep it honest.** Skip any one and the phase manufactures a number:

- **A per-task rate is a training number by construction.** The optimiser tuned on it. Quote
  it as a search signal, never as a result. Only the full-val gate and the sealed test are
  evidence.
- **No task-specific literals — and ENFORCE it with a script, not a promise.** No record
  id, confirmation code, item number, person name, date, user id, payment id or location pair
  from the trace may appear in a line the edit ADDS. Write a ~40-line auditor that diffs each
  candidate against the base and greps the ADDED lines for your domain's id shapes; make a
  `clean` verdict a merge precondition, independent of what the rate says. Two details decide
  whether it works: diff the **added lines only** (the pristine seed's own airport tables and
  example ids would flood a whole-file grep), and **skip any literal the base already
  contains** — a reindented pristine docstring shows up as an addition and made three clean
  candidates look guilty until that filter went in. It caught exactly one
  real case: a docstring enumerating *"New York is JFK, LGA or EWR; Chicago is ORD or MDW"* —
  its task's own cities, dressed as a general rule. The underlying idea (match a route by city,
  not airport code) was fine; the enumeration is what made it memorisation.
- **Diagnose from behaviour, never from the target — but the COORDINATOR may audit the spec.**
  The task's `target` / `evaluation_criteria` are the grader's, not the optimiser's: for an
  optimiser, the feedback string and the agent's own trace are the whole permitted input. The
  coordinator has one narrow extra permission, and it is a measurement-integrity permission, not
  an optimisation one: **read the spec to answer "is this task winnable, and is the optimiser
  chasing the right criterion?"** — then relay only what the agent itself can already observe.
  This unblocked two dead tasks in one run. On one, an optimiser had concluded the
  communicate check was unsatisfiable; the audit showed it required a single figure the agent was
  computing wrongly, so the relay was *"you speak, your arithmetic or scope is wrong"* — no value
  echoed. On the other, an optimiser had built a same-date-duplicate detector, plateaued, and
  reported that the correct write set was unreachable that way; the audit showed the criterion is
  the itinerary **the customer states in her own message**, so the relay was *"cancel what
  conflicts with the trips she stated, keep what matches"* — again nothing the agent could not
  see for itself.

  The line to hold: relay a **criterion the agent can evaluate from the conversation**, never a
  value, an id, or an expected write. If the only way to state the fix is to name the answer, the
  task is not winnable and that is the finding — say so in the report instead of leaking it. And
  keep the permission asymmetric: an optimiser that reads targets has stopped optimising the
  agent and started memorising the grader.
- **A guard must fire on a DECISION, not on a tool.** A precondition that refuses on every
  call of a write tool derails tasks it was never aimed at: one such guard dropped a canary from
  1.0 to 0.333 and pushed that eval's wall time from 299s to 1493s, because every extra refusal
  costs a turn in a turn-budgeted rollout. Key the guard to the specific contested situation and
  let it fire **once per situation** — re-keying one guard from per-user to per-contested-date
  was worth 0.0 → 0.333 on its own, because a second independent decision needs its own prompt.
  And keep the refusal directive: softening the same guard's wording to "otherwise proceed" flipped
  the failure from over-writing to under-writing and gave the whole gain back.
- **Measure the canary at the SAME `n` as the target before you use it, and never set the bar
  at 1.0.** A task that reads 1.0 off a 3-trial baseline has a CI wide enough to hold 0.4, so
  a canary chosen that way manufactures phantom collateral damage and every optimiser burns
  iterations chasing it. Measured cost of getting this wrong twice: one
  canary task read 1.0 at 3 trials and 0.67 at 10; a second read 1.0 at 3 trials and then
  0.667 / 0.333 / 0.0 / 0.333 / 0.333 across five independent 10-trial runs — so
  `canary_mean == 1.0` was unreachable for reasons no candidate caused. The bar is **no canary
  task below its own measured band**, and the band comes from the same `n` you judge at.
  The same warning applies to the target: one "0.0 DEFECT" task measured 0.444 at n=10.

**Decompose the reward before you fan out.** If the metric is composite — one benchmark scores a
database check, action checks and communicate checks and then returns a *binary* task reward —
a task that wrote the database correctly and only failed to state a required confirmation
scores 0.0, identical to one that did nothing. `taskeval.py` reports the per-component means
(`component_rates`) for exactly this reason: it turns one useless number into one number per
failure mode, and the two need different edit forms. Do this first; it is free and it
re-aims the whole round. In one run it showed all 14 failing tasks missing the database-state
component and only 4 also missing COMMUNICATE — which killed a plausible-sounding
communicate-first plan before any rollouts were spent on it.

Then tell each optimiser **which components its own task even has**: 25 of the 30 val
tasks have no communicate check at all, so on those, nothing the agent *says* can change the
score and any edit aimed at phrasing is guaranteed dead. `component_rates` lists only the
components a task actually carries, so this is free to read and it deletes whole categories of
wasted iteration.

**Check which reward components actually GATE before you read the feedback as a to-do list.**
A grader that reports several component scores does not necessarily use all of them. One
benchmark publishes `reward_basis`, and there it is `["DB", "COMMUNICATE"]` — **`ACTION` is absent**,
so action checks cannot change the score. The feedback nonetheless led with "Action-level
defects", which sends an optimiser after calls that provably do not matter: task 12's feedback
names `calculate: was never called` on every failing rollout, `calculate` was invoked in **0 of
300** rollouts, and the task still scores 0.8. Label non-gating detail as diagnostic and name
the components that do gate, or the loudest line in the feedback is the one worth least.

The same read is worth doing per task before choosing an edit form: a task with no communicate
check cannot be moved by anything the agent *says*, and a task scored only on `DB` cannot be
moved by fixing which reads it performed.

**Measure what the model actually RECEIVES before you write another word of it.** A tool
docstring is not delivered whole. One harness builds each tool's schema `description` from the docstring
**summary plus the prose before `Args:`** and drops the `Returns:` section entirely. Measured over
a 14-tool set: **5469 of 12929 docstring characters (42%) never reach the model**, and on
one tool it was 115 of 1906 delivered — **94% dropped**. Rounds of behavioural guidance
had been written into that void. One "verified" mechanism (*read these cards and pick the ONE that
matches the description*) turns out to work only because the **return VALUE** changed shape, which
the model does see at call time — not because anything documented it.

So there are exactly two places guidance can live, and a third that looks identical and does
nothing:

| surface | reaches the model | use it for |
|---|---|---|
| docstring summary + prose before `Args:` | **yes**, in the tool schema | preconditions, scope, what not to do |
| `Args:` per-parameter descriptions | **yes** | argument-level constraints |
| the returned VALUE (a `next_step` key) | **yes**, at call time | what to do next, with its constraints |
| `Returns:` docstring section | **no — silently dropped** | human readers only |

Verify it, per candidate, rather than trusting the file: render the toolset and sum the delivered
characters. Two hazards when moving text into the delivered region — a lifted line must not begin
a recognised section (`Example:`, `Returns:`), because some harnesses parse those and a stray header raises
at REGISTRATION time and kills every rollout as `INFRASTRUCTURE_ERROR`; and it must be inserted
*before* `Args:`, or it lands back in the dropped region.

**Findings go in the ledger, not in the coordinator's head.** Independent optimisers on
different tasks keep rediscovering *one* cause. In one run four of nine independently
found writes being lost to turn starvation, and two independently implemented the same tool
enrichment — which collided at merge, where only one of the two had actually been measured.
So every optimiser **lists before it diagnoses and appends when it finds**:

```bash
python "$A/mechanisms.py" list --run-dir "$R" --task "$TASK" --compact
python "$A/mechanisms.py" add --run-dir "$R" --owner "$TAG" --status proposed \
       --mechanism "<the cause, one sentence>" --evidence "<what you measured>" \
       --touches <function-the-fix-edits>
```

**Filter the ledger per optimiser, but never filter out the task-independent rows.** A real
fan-out ledger reaches a size that stops being an asset: this one hit 99 findings / 65 KB, and
pasting all of it into each of K subagents spends their context on other people's tasks.
`--task N --compact` cuts it to 24 KB while keeping every row about task N **plus every row
with no task attached** — those are the cross-cutting facts (canary bands, variance warnings,
measurement defects) that apply to everyone, and hiding them is exactly how a fan-out re-pays
for a defect someone already found.

**Retire a finding that turns out to be wrong — `--supersedes <seq>`.** Contradicting it with a
newer row is not enough: three separate `verified` rows were disproved on this run (a merge-retention
percentage computed from single readings, and two different claims about one task's ceiling), and a
reader of the listing saw both the claim and its refutation with no way to tell which won. A
superseded row drops out of `verified`/`proposed` and is reported under
`superseded_do_not_act_on`, so the history stays auditable without misleading the next optimiser.
**A disproved claim left in `verified` is worse than no ledger at all** — it is the one thing a
fan-out will confidently build on.

`--touches` is the collision key and `--status` is the point: `verified` means reuse it and
never rewrite it (**rebase onto that copy** and spend your iterations elsewhere), `proposed`
means its owner is already on it, `rejected` means a retry must be structurally *different*.
Cross-pollination is the main reason K parallel optimisers beat K sequential rounds; the
ledger is what makes it survive the coordinator forgetting to send a broadcast.

**Ablate a read+enforce pair TOGETHER, or you will throw away the half that carries it.** The
strongest single per-task result of one round was a two-part edit: a tool return printing
the concrete candidate values, plus a write-side refusal ordering the agent to re-read them.
Measured alone the read block moved 0.400 -> 0.500, inside noise, and looked worthless; the
refusal looked like the whole gain. Removing the read while keeping the refusal collapsed the task
from **0.625 to 0.200** and brought the original wrong writes straight back — a refusal that tells
the agent to re-read a value only works if that value is actually printed somewhere it can read.
The converse held too: the same content as *passive* fields with no refusal was worth nothing,
because extra fields deep in a large payload never reach the decision. Enforcement without the
read is a dead end, the read without enforcement is decoration, and ablating either half in
isolation gives you the wrong answer about both.

**A tool return that advertises a path must state that path's constraints in the same breath.**
Adding a price table showing what one option would cost pulled the agent toward that option, and it
discovered three turns later that the option could not be paid for the way the customer wanted — too
late to pivot, so it escalated. The table alone cost a task 0.28; the same table with the option's
payment rules printed beside it recovered that and carried another task to 0.90. Information that
makes a path *attractive* without making its preconditions visible is worse than no information.

**A numeric fix must not be phrased as an instruction to address the customer.** Telling the agent
to quote a figure after every write did exactly what it said — the communicate component went to
1.0 — while the database component collapsed (one task 0.50 → 0.11, and a reliable canary to 0.667),
because "report this to the customer" sends it to the customer *mid-flow*: on a cancel-then-rebook it
spent the turn announcing the refund and never booked. Fix the value, not the audience: put the
figure where the agent will use it, and never make stating it a turn-taking instruction.

**Forcing a decision to be STATED is not forcing it to be CARRIED OUT.** A guard that refused a
write until the agent named which competing item it was keeping measured 0.2 -> 0.1. It worked at
its literal job — the blind retry disappeared — but the agent then treated *having named* a keep as
having resolved the situation and never issued the second write. If the defect is a missing action,
the guard has to be satisfiable only by that action; a guard satisfiable by an assertion buys you a
better-documented failure.

**"No nuance clauses" applies to refusal text too.** Adding two *correct* discrimination clauses to
a working refusal took the same task 0.2 -> 0.1, with all ten trials failing — three cancelled
nothing and two escalated. The clauses were right and the longer refusal traded follow-through for
precision. A refusal has a budget: every sentence competes with the one that says what to do next.

**A traces file holding only FAILING trials is evidence about the passes.** If a specific action is
absent from every failing trial across dozens of rollouts, and the task sometimes passes, then that
action is what the passes are doing — a free inference from an artifact you already have, and often
the fastest route to naming the residual defect.

**A precondition can be misread as a platform limitation — say what it is.** One guard refused an
inconsistent write and the agent apologised to the customer ("the system won't let us do that"),
offered to split the request in two, and argued about it for the rest of the conversation instead of
fixing the argument. Rewriting the same refusal to say *this is not a limitation, here is the
corrected value, retry now* removed the false apology. It was score-neutral — and worth keeping
anyway, because the agent stopped telling the customer something untrue. Not every improvement
shows up in the metric, and a refusal's wording decides whether the agent treats it as a bug to
route around or an instruction to follow.

**A retryable refusal is safe when the retry path is the CORRECT action, and poison when it is a
free choice among options.** Both shapes were measured on the same task in adjacent rounds, which
is what makes the distinction trustworthy. A one-shot retryable refusal that named *the* fix
("upgrade first, then add bags") took the task 0.60 -> 0.70 and could never dead-end a legitimate
request. The same retryable shape applied to a choice — refuse the charge, list the valid payment
ids by kind, let the agent pick — measured **0.70 -> 0.10**: the agent treated the retry as a
formality, re-sent the same wrong id, and trials that had previously chosen correctly switched to
the wrong option, because the refusal's own list read as permission. So before shipping a guard
that enumerates alternatives, ask whether the retry leads to one determined action or to a menu. A
menu turns a mistake into a sanctioned choice.

**In a turn-budgeted rollout, a fix that costs a turn can cost more than the bug.** This is the
constraint that decided more edits in one run than any other, and it is easy to miss
because the edit reads as obviously correct. Telling the agent to *ask* for a missing piece of
information measured 0.5 -> 0.3 — even when scoped to exactly one call site, which is normally the
fix for that kind of regression. The mechanism: the user simulator ends the conversation a few
messages in, so the question trades a write the agent would otherwise have made for an answer it
never gets to use. Two other optimisers hit the same wall from the opposite side, where the *bug*
was a wasted confirmation round-trip. So before shipping any edit that adds an agent message,
count the turns it costs against the turns the failure costs — and prefer a form that puts the
information in a tool return, where it costs nothing.

Two corollaries measured the same way. **Place text at exactly one call site**: the same directive
inside a helper with four callers splattered across seven tool returns and into a post-success
summary where it read as self-contradictory (0.5 -> 0.3). And **do not tell the agent to stop
reasoning about eligibility and defer to the tools** — that was the single worst edit of the run
(0.5 -> 0.0, and a reliable canary 1.0 -> 0.667): the guard still catches flagrant violations, but
the agent starts attempting actions policy forbids and the conversation derails.

**Elimination evidence is only as good as the classifier feeding it — verify a classifier by
PRINTING, not by measuring.** One optimiser ran a careful elimination over 30 scored trials, ruled
out every candidate write-set it could construct, and concluded the task was unwinnable without
hardcoding. The reasoning was sound and the conclusion was false: the helper deciding which items
matched the customer's stated requirement was inverted, so every set it built excluded the right
item. The bug was one line of classification logic, visible in five seconds by printing the
helper's output next to the raw data — and instead it cost three eval rounds and a wrong verdict
about the benchmark. So whenever a round's conclusion rests on a derived label ("this one
conflicts", "this one is eligible"), dump the label beside the input it came from and read it by
eye **before** spending rollouts on any hypothesis built from it. Cheap checks first: a classifier
is a function, not an experiment.

**Find a task's own ceiling, then stop.** Not every task can reach your target, and grinding one
that cannot is the most expensive mistake in this phase. Two optimisers spent 13 rounds between
them on one task without moving it off 0.0. Its ceiling was structural: the user simulator
terminates the conversation a few messages in, and 3-4 of 10 rollouts died right after a
*mandatory* question the customer's opening message had not answered — so with a binary reward
needing both components, the achievable rate was ~0.6 whatever the edit. Say the ceiling out loud
in the report, subtract it from the headroom, and move the budget to a task that can move. A
bounded task is a finding, not a failure — and "we never reached 0.9 on task X" plus *why* is
worth more than a third optimiser.

**A regression LIST at `n=10` is noise, and the control proves it in the same round.** In the
one gate the byte-identical control reported **four** regressed tasks and the candidate
reported **four** — identical counts, disjoint sets, and one of the two artifacts provably
unchanged. That is the whole case for `--veto-regressions` being off by default: with the veto on,
a copy of the parent would have been rejected for the same reason as the candidate. Read
`regressions` as a pointer to look at, never as a verdict, and always next to the control's own list.

**One per-task reading cannot attribute a per-task change — and that trap caught this skill's
own author.** At `n = 10` the standard error on a task near 0.5 is about 0.16, so a difference
below roughly 0.3 is indistinguishable from re-measurement. Measured: one task
read **0.6 / 0.9 / 0.5** across three independent runs of *byte-identical* files, another
**0.5 / 0.4 / 0.4 / 0.4** against a single solo reading of 0.70. A "74% of the gain was retained
by the merge" figure was computed from single readings, reported, and then withdrawn when an
optimiser re-measured the same bytes twice and found the giveback was noise — the merge was
structurally clean, verified by diff.

So the between-phases check is still worth its ~70 rollouts, but read it as a **smoke test, not
an attribution**: it catches a merge that dropped an edit or broke a canary outright, which is
what it is for. To claim a per-task delta, pool runs (report `k/N` across every run of those
bytes, not the last one), and treat any single-reading per-task comparison as a hypothesis. The
per-task rate was always a training number; this is the second reason not to quote it as a result.
The full-val gate against its own control is the arbiter precisely because it averages 30 tasks
instead of trusting one.

## The binomial floor, and what an aggregate mean can resolve

**First compute the BINOMIAL floor. Most of what looks like mysterious nondeterminism is n.** Each
rollout is pass/fail, so a task's rate is a binomial proportion and an arm mean over `m` tasks at `n`
trials has

    SE(arm difference) = sqrt( sum_over_tasks 2·p(1-p)/n ) / m

Do that arithmetic BEFORE blaming the provider, the seeds, or the load. Measured on 10 tasks at n=10
with p≈0.35: predicted SE **0.0615**, observed gap between two byte-identical arms **0.0778** — a
ratio of **1.27**, i.e. plain sampling. Mean per-task movement was 0.0978 against a binomial
prediction of 0.1445, so the observed movement was *smaller* than chance requires. There was nothing
left to explain.

One precision about what this is and is not. At temperature 0 with identical seeds a fully
deterministic system would return *identical* arms, so this is not sampling error in the textbook
sense — there is no sample being drawn. What the arithmetic shows is that the observed variation is
**statistically indistinguishable in magnitude from independent per-rollout coin flips**. That
matters because no further mechanism needs to be posited to explain it, and — whatever its physical
cause — the remedy is the same one that works for binomial noise: more trials. Do not report it as
"sampling noise" without that caveat, and do not go hunting for a cause you have no evidence for.

That reframes the concurrency result rather than cancelling it: at conc 25 per-task movement was
0.250, genuinely **above** the 0.1445 floor, and dropping to conc 8 removed that excess and exposed
the floor underneath. Lowering concurrency fixes what it can; the rest is n.

The consequence is uncomfortable and worth stating plainly: **an aggregate mean over a dozen HARD
tasks at n=10 cannot resolve any realistic edit.** Reaching 2 SE on a +0.05 effect on that subset
needs ~60 trials per task.

But be careful about the obvious inference, which is wrong. Narrowing to the hard tasks makes the
measurement *worse*, not better, for judging an artifact — because a task sitting at 1.00 contributes
signal to the mean with almost no variance, so dropping it removes a free denominator. Computed from
that benchmark's own per-task rates:

| arm | rollouts/arm | SE of paired difference |
|---|--:|--:|
| 12 hard tasks, n=10 | 120 | **0.0496** |
| full val 30 tasks, n=10 | 300 | **0.0262** |
| full val 30 tasks, n=20 | 600 | 0.0185 |

Full val at n=10 is nearly twice as precise as the hard subset, and the four prior gate rounds there
were run at full val n=5 (SE 0.0371) — so their problem was never the task set. It was that they ran
at a concurrency carrying excess noise on top of that, and chased effects smaller than the sum.

So the two questions need opposite designs, and conflating them is the actual error:

| you want to know | measure | why |
|---|---|---|
| does this mechanism work | ONLY the tasks where it fires, at high n, per-task test | a mechanism firing on 2 tasks is diluted to nothing in a 30-task mean |
| does it break anything | canaries, cheap precisely because they sit at 1.00 | zero variance means a single drop is real |
| what is the artifact worth | FULL val, both arms in one batch | the 1.00 tasks are free precision for the mean |

A mechanism that fires on two tasks and lifts them 0.15 → 0.45 is resolvable at n=40 on those two
tasks (≈3 SE) and invisible in a 12-task mean at n=10 (≈0.5 SE). Same edit, same rollout budget: one
design answers the question and the other cannot.

**When an effect sits below the floor, pre-register directional predictions and use a SIGN TEST.**
The floor bounds what a *mean* can resolve; it does not bound what a *pattern of directions* can.
Write down, before its arm runs, which way each prediction should go, then count. Measured: **9 of 10
predictions positive gave p = 0.0107** on a set where no individual z reached 2 — the effect was real
and every per-arm reading was individually inconclusive. Two conditions make it a test rather than a
story: the predictions are recorded *before* the arms run (a direction chosen after the fact is not
evidence of anything), and they are directions, not magnitudes. A post-hoc count of which way things
happened to go is worthless, so if you did not write them down, you did not run the test.

**A screening band is not a baseline.** Per-task rates from a small trial count tell you where to
*look*; they do not tell you where you *are*. In one run, ten tasks whose 3-trial bands summed to
2.33 measured **4.04** at n=10 — the screen understated the artifact by 1.71 task-equivalents (0.057
of val), and one task went the other way (0.33 → 0.10). Every "0.0 DEFECT" label was suspect: three
of them measured 0.30, 0.444 and 0.60. So screen at low `n`, but re-measure at the gate's `n` before
you quote a number, compute headroom, or tell an optimiser what its starting point is — and never let
a low-`n` band be the thing a delta is computed against.

## Load is the other half of the noise

**Run the GATE at low concurrency — most of the re-measurement noise is load-induced.** This is the
one lever that makes everything else measurable, and it is cheap to verify: measure the same bytes on
the same seeds twice at your search concurrency, then twice again at a low one.

| identical bytes and seeds, 12 tasks | conc 25 | conc 8 |
|---|--:|--:|
| arm-level \|delta\| between the two runs | **0.1167** | **0.0333** |
| mean \|per-task\| movement | **0.250** | **0.100** |
| tasks that moved at all | 10 / 12 | 5 / 12 |

Five of the twelve tasks became perfectly repeatable at conc 8 having each moved 0.20-0.40 at conc
25. So the practical split is: **search fast, gate slow.** Per-task exploration can run high, because
its output is a mechanism you verify from a trace, not a rate; the accept decision must run at a
concurrency where the null actually reproduces, which costs roughly 3x wall clock for the one
evaluation that matters.

Two caveats to state whenever you quote this, both real: the low-concurrency runs were sequential, so
load and elapsed-time drift are confounded; and 12 tasks x 2 runs makes the variance comparison thin.
The direction was consistent across all three metrics, which is why it is worth acting on, but it is
not settled.

**Concurrency composes only up to the endpoint's sustainable rate, and past it the failure is
silent.** The sources multiply: K optimisers x C concurrency each is K*C in flight, and the serving
endpoint does not know about your fan-out. Measured: nine per-task optimisers at concurrency 8 put
~72 requests in flight against a proxy whose sustainable band is 24-90, and a single per-task eval
went from ~4 minutes to ~50. Nothing errored — latency just grew, so it read as "the model got
slower" rather than "I oversubscribed". An earlier incident on the same proxy is the extreme version:
concurrency 300 pushed 292 of 300 rollouts into wallclock timeouts and the evaluation reported
**0.0067** as capability. So measure the sustainable band once, divide it by the number of concurrent
optimisers, and treat a sudden wall-clock blowout as an oversubscription symptom first.

**The knob is TOTAL IN-FLIGHT REQUESTS, not the runner's per-process concurrency flag.** That flag is
per-process, so it does not bound load when several evaluations run at once — and running them at once
is the normal case. A gate launched at `--conc 8` alongside four exploring optimisers at `--conc 12`
puts about 56 requests in flight, so it is a HIGH-load measurement wearing a low-load flag, and it
will reproduce the wide null rather than the narrow one. Serialise the gate: let the fan-out finish,
or pause it, and run the gate arms alone. Both arms still belong in the same batch as each other —
pairing is what removes drift — but that batch must be the only thing running. If you cannot quiet
the machine, say so next to the verdict instead of quoting a per-process number as though it were the
load.

## Gate the sum, not each addend

**A multi-branch artifact is assembled with `integrate.py`, never by one merge.** Stated as a rule
because the author of that script skipped it on the very round it was written: six branches were
merged in one step, and the resulting artifact gated at **−0.0146** with seven replicated per-task
losses against two replicated gains — while the same round's *single*-mechanism artifact gated at
**+0.0115**. Fewer mechanisms beat more mechanisms, and a one-shot merge cannot tell you that,
because it yields one number for N simultaneous changes.

`funcmerge` merging cleanly is **not** evidence the branches compose — every branch retained cleanly
in that failed artifact, with zero conflicts and no undefined attributes. Clean merge is a syntactic
property; composition is an empirical one.

**Gate the SUM, not each addend.** Measured: one tool-level mechanism is worth roughly 0.04–0.13 on
the one or two tasks it touches, and resolving an effect that size at 2 SE needs about **n=100 trials
on that task**. Certifying seven mechanisms that way is ~1400 rollouts to establish by rate what a
deterministic replay establishes for free. So the economical order is:

1. **Prove it engages** — replay a real failing payload against the edited tool and show the guard
   fires; replay the passing payload and show it does not. Costs zero rollouts, and it is a stronger
   statement about the mechanism than any rate.
2. **Establish incidence from rollouts you already have** — how often does the condition occur, and
   is it skewed toward failures? Also free. One guard fired on 8 of 76 matching calls, 8 in failures
   and 0 in passes.
3. **Confirm the sign at modest n, with canaries** — you are checking for a regression and a
   direction, not measuring a size.
4. **Gate the accumulated artifact ONCE on full val**, where SE was 0.0262 at n=10 and several
   mechanisms can clear it together even though none clears it alone.

Expect the measured per-task effect to land well below the upper bound incidence implies — there
~40% of it — because the guard fires correctly and the agent then still fails for an unrelated
reason. That gap is not evidence the mechanism failed; check the task's other reward components
before concluding anything.

## Take the error ACROSS whole runs

**Take the error across whole runs, not across tasks within one run.** A single paired run's SE is
computed over tasks, so it cannot see run-to-run nondeterminism at all — and on that benchmark it was
the dominant term. Repeat the entire paired comparison on distinct seed blocks and use the spread of
the per-run deltas:

| seed block | candidate | control | paired Δ |
|---|--:|--:|--:|
| 0-4 | 0.7333 | 0.6467 | +0.0867 |
| 100-104 | 0.6867 | 0.6667 | +0.0200 |
| **combined** | | | **+0.0533, SE 0.0333 across runs (t ~ 1.6) — NOT demonstrated** |

```bash
# one paired run per seed block, both arms in the SAME batch, then combine ACROSS runs
python "$A/taskeval.py" "$R/work/cand" <val ids> --n 5 --base-seed 0   --json /tmp/c0.json
python "$A/taskeval.py" "$R/work/ctl"  <val ids> --n 5 --base-seed 0   --json /tmp/k0.json
python "$A/taskeval.py" "$R/work/cand" <val ids> --n 5 --base-seed 100 --json /tmp/c1.json
python "$A/taskeval.py" "$R/work/ctl"  <val ids> --n 5 --base-seed 100 --json /tmp/k1.json
python "$A/multirep.py" /tmp/c0.json:/tmp/k0.json /tmp/c1.json:/tmp/k1.json
```

`multirep.py` refuses to return a verdict from a single paired run at all, because that is exactly
where the retracted accept came from. `--base-seed` matters: raising `--n` only extends the same seed
block, so a rerun at the same seeds is a determinism check.

The first run alone reported SE 0.0548 across tasks and an "accept". Two runs show the same candidate
at +0.0867 and +0.0200, and a byte-identical control re-run moved +0.0800 by itself. The across-run
estimator needs no assumption about where the noise comes from, which matters because on that run its
source was never identified: LLM sampling, seed assignment, concurrent batching, timeouts, infra
accounting and set-iteration order were each ruled out by direct measurement, and the leading
remaining hypothesis (transient errors below the `max_errors` threshold being fed back into the
conversation) stayed unverified.

Budget for it up front: a credible verdict on a sub-0.10 effect there is **several full paired runs**,
not one. If that is unaffordable, the honest output of the round is "not resolvable at this budget" —
which is a result, and is what the earlier single-run accept should have been.

**The false-veto rate that made `--veto-regressions` opt-in.** In one gate the byte-identical control
reported **four** regressed tasks and the candidate reported **four** — identical counts, disjoint
sets, one of the two artifacts provably unchanged. The old veto fired on a byte-identical copy of the
seed **42.8%** of the time at 5 trials, and in one run it vetoed *both* candidates that had passed the
significance test. Read `regressions` as a pointer to look at, never as a verdict, and always next to
the control's own list.


**"Not resolvable" is a decision you can book, and booking it as a reject costs the run.**
`round.py` marks a candidate `verdict: inconclusive` when `verdict_stable: false` — its verdict changed
depending on which byte-identical control replicate happened to be the reference, so the round cannot tell
its edit from re-measurement. Measured on a smoke run: cand_2 at Δ +0.0433 against a threshold of 0.0492,
`verdict_by_reference: {ctl_null_i1: reject, ctl_null_i1r1: accept}`.

Book it as `commit.py --decision inconclusive`. Two things follow from booking it as a `reject` instead, and
neither is cosmetic:

- **A reject advances the stall counter, and stall ends the run.** `update_spent(accepted=False)` increments
  it, and `budget_exhausted()` stops on the cap (3 in the smoke tier). Stall means *the optimizer has run out
  of ideas* — the one thing an ambiguous measurement is no evidence of. On a benchmark whose replicate noise
  makes ambiguity common, two unresolved rounds can end a run for a reason that never happened.
  `inconclusive` charges `iterations` (the budget really was spent) and leaves stall alone.
- **A reject files the edit in `rejected.jsonl`**, which later rounds read as *tried, did not work*. An edit
  nothing could judge has not been tried in that sense, so filing it there teaches you to avoid your own
  untested idea. `inconclusive` skips that file and logs `step_indecisive` instead — which is also the event
  the dashboard reads to render the step as `indecisive` rather than red.

The `JOURNAL.md` RESULT line follows the same rule: an unresolved round is stamped `UNRESOLVED (not judged)`,
not `REJECTED … its WHOLE batch was reverted`, because the correct next move for an unresolved edit is to
**re-measure** it, not to redesign it.

**To re-measure, use a FRESH tag.** Rollouts are written `<task>__<tag>__t{k}.json` for
`k in range(n_trials)`, so re-running a tag **replaces** `t0..t9` rather than adding `t10..t19` — it swaps a
reading for another reading and buys no evidence. Either pick a new tag (`cand_2b`) or ask for the higher
`--trials` in ONE evaluation. This is not hypothetical: told to "re-run with more trials", one run
re-measured its own control under the same tag, spent 100 metric calls, replaced a 0.4967 replicate with a
0.5067 one, and *widened* the round's replicate spread. `harness` now logs a `rollout_overwrite_warning`
naming the reading that was destroyed, because once the files are gone it exists nowhere else.

**`evidence_bar` is necessary, not sufficient.** It is the noise floor to compare a delta against, but
`gate_threshold` (k·SE on the paired per-task differences) is what each `verdict` is actually computed from,
and it is usually stricter. A delta above `evidence_bar` and below `gate_threshold` is not an accept — cand_2
above cleared 0.0167 and missed 0.0492.

**A re-gate must not eat the evidence it was run to add.** `round.py` gave its *table* a `.r<k>` suffix on a
same-iteration re-run — "since a re-gate is usually being COMPARED with the first one" — but gave the control
*rollouts* the same `ctl_null_i<N>` tags as the first attempt. So the one operation the script explicitly
supports preserved the summary and deleted the measurements it summarises. Measured: the second attempt at
iteration 1 replaced `ctl_null_i1` (0.4967 → 0.5067) and `ctl_null_i1r1` (0.4800 → 0.4367), spending 200 of
the run's 900 metric calls to swap two readings for two others; the replicate spread went 0.0167 → 0.0700, so
the bar grew 4.2×, and `round_i1.json` was left quoting an `evidence_bar` derived from two numbers that no
longer existed anywhere on disk. The round's identity is (iteration, attempt) and **both** halves have to
reach the names on disk: control tags are now `ctl_null_i<N>a<k>` from the second attempt on, the table name
comes from the same attempt index rather than a second independent probe, and `prior_attempt_controls` pools
the earlier attempts' replicates into `null_delta_between_control_replicates` (`max − min` needed no change
to accept four samples instead of two). Re-gating is now accumulative: it reports the null over every
replicate the round has paid for.

**An UNCHANGED parent's noise floor is measured once, not once per round.** Across six real runs of one
multi-turn benchmark the mandatory two null-control replicates consumed roughly **40% of every rollout
spent** — nearly as much as all candidates combined — and most of that bought nothing: while `best_id` has
not moved, the control is the same bytes as the control the previous round already measured, so re-running
it re-measures a floor the run has already paid for. `round.py` therefore reuses those replicates,
reporting it as `control_reuse` in the table, and the requirement itself is untouched: an accept moves
`best_id`, the new parent has no established floor, and the next round measures two fresh replicates for
it. Reuse also needs the same `measurement` context (split, trials, concurrency — both trial count and
load move the reading) and the replicates' rollouts still on disk, since the gate re-reads them. It is
ski

# Micro-tests — TDD-style, before any rollout is spent

A candidate claims to fix a diagnosed defect via a specific MECHANISM (a guard, a call-shape
change, a different tool selected). That claim is checkable in seconds, deterministically, on the
one call or turn where the defect lives — no LLM judge, no multi-turn episode, no rollout. This
adopts Harbor/Terminal-Bench's task shape (`task.yaml` + fixture/environment + a `tests/` dir that
asserts pass/fail and writes a reward without an LLM judge), scoped down from "a whole task" to
"the one call or turn the defect lives in" (#434 section 3, #436).

## Schema

```
$R/microcases/<cluster_id>/
  case.yaml       # {id, cluster_id, source_task_ids, source_rollout, timeout_s,
                  #  expects: guard_fires|call_shape|tool_selected, assert: {metric, op, value}}
  fixture/          # extracted VERBATIM from the diagnosed rollout — never invented
  reproduce.py      # replays the fixture against the candidate directly — one tool call
                    # or one narrow unit, never a full multi-turn episode, no LLM
  assert.py         # deterministic pass/fail against case.yaml's `assert` spec, exit 0/1
```

`reproduce.py`/`assert.py` are necessarily project-specific (they know the candidate's tool
module) — `microcase.py` cannot write them for you, only scaffold the fixture and the case
metadata from a REAL diagnosed rollout, so the one project-aware step (how to instantiate the
minimal state the fixture's calls need) is the only thing left to fill in.

## Authoring a case from a diagnosed cluster

```bash
python "$A/microcase.py" gen \
    --rollout "$R/rollouts/val/<task>__<tag>__t<k>.json" \
    --cluster-id <the diagnose.py cluster's id> --expects guard_fires \
    --description "<one line: what the mechanism must do>" \
    --assert-metric <field-name-in-reproduce.py's-result> --assert-op "<=" --assert-value 1 \
    --out "$R/microcases/<cluster_id>"
```

This extracts every tool call in the rollout's trace verbatim into `fixture/calls.json` and
scaffolds `case.yaml` + a `reproduce.py` stub with a marked TODO. Finish the TODO once per
cluster (construct the minimal state the fixture's calls need, replay them against the
candidate's own tool module, write the observed field(s) `assert.py` will check) — every
candidate that later targets the same cluster reuses the finished case for free.

## Running it

```bash
python "$A/microcase.py" run-all --cases-dir "$R/microcases" --project "$P" --candidate "$R/work/$TAG"
```

Three outcomes, not two — this is the same "missing data is not a zero" discipline
`taskeval.py`'s `infra` counter already applies:

* `pass` — the mechanism fires as claimed.
* `fail` — it provably does not. `micro_test_fail: true` in the summary →
  `commit.py --decision reject --reject-basis micro_test_fail`, no screen, no full val paid.
* `error` — `reproduce.py` hit an environment gap (a missing project dependency, not a
  candidate defect) or its own contract was violated. Fix the case before trusting either a
  pass or a fail measured while it was erroring.

A candidate with no applicable micro-case skips this step; it is a filter, not a requirement, and
`run-all` never touches the run dir's budget or state — a micro-test cannot itself accept anything,
only kill cheaply before a rollout is spent.

## Worked example

`skills/algorithms/agent-optimize/scripts/microcase.py`'s own docstring and
`core/tests/test_microcase.py` carry a full worked example grounded in a real diagnosed run: a
duplicate ledger write from re-issuing the same mutating call twice on one record. The seed fails
the case in ~0.15s (2 entries instead of 1); a candidate carrying the real accepted merge-guard
mechanism passes it — before either was ever run through a task rollout.

# Per-task fan-out — economics, briefing, canaries, assembly

Read this when the baseline's per-task `k/n` shows the loss **concentrated in a few named tasks**
rather than spread thin, and you want to spend the round's rollouts on those tasks instead of on
full-val gate rounds. SKILL.md carries the three rules that decide whether the shape is safe; this
file carries the arithmetic, the briefing contract, the canary-selection rule and the commands.

The numbers below come from real runs on a multi-turn tool-use benchmark with a mid-tier agent
model. The *shape* of each finding transfers; the figures are that run's.

## Why it is cheaper

A full-val gate round costs `val_n × n_trials` rollouts and returns *one bit per candidate*: accept
or reject. A single task at `n_trials` costs `n_trials` rollouts and returns the same bit — about the
failure that actually exists. At `val_n = 30, n_trials = 10` that is 300 rollouts per learning step
versus 10: **a 30× cheaper gradient**, on the unit the defect lives in.

Measured on one long run: five classic rounds spent ~1500 rollouts to produce **10 learning steps
and 1 accept**. The same budget under this shape buys **nine optimisers × six iterations = ~54
steps**, each aimed at one measured defect, and the per-task evals run concurrently so the
wall-clock cost is one round's.

## What to ask a parallel optimiser for

**A parallel optimiser's deliverable is a MECHANISM WITH TRACE PROOF, not a rate.** K optimisers
each evaluating is K processes, so a fan-out is BY CONSTRUCTION a high-load regime — the very regime
in which a per-task rate cannot resolve the effect anyone is looking for. Briefing K subagents to
"measure whether your edit helps" therefore asks them for the one thing their situation cannot
provide, and what comes back is K rate deltas drawn from a distribution wider than the effect.
Several prior rounds did exactly this and accepted edits on it.

Ask instead for evidence that does not depend on load:

| evidence | load-sensitive? | good for |
|---|---|---|
| the guard fired on the observed call | no | proving the mechanism engages |
| the agent's next action changed after it fired | no | proving the mechanism works |
| a direct call with the exact bad payload now succeeds / still refuses | no | proving repair logic, deterministically |
| the delivered docstring text contains the keys | no | proving the description reaches the model |
| count of clarification turns before the first write | barely | proving a behavioural prose change |
| per-task pass rate | **yes, heavily** | almost nothing, at fan-out load |

Then gate the surviving mechanisms yourself, serialised, on a quiet machine. The division of labour
is: **the fan-out finds falsifiable mechanisms and proves them structurally; the driver alone turns
mechanisms into numbers.** A subagent that reports "indistinguishable from noise" while showing its
guard firing correctly has done its job completely.

## Canary selection

**Draw canaries from the WHOLE suite, not from the neighbourhood of your mechanisms.** This is the
mistake that sank an artifact whose individual mechanisms all measured positive. The canary set was
nine tasks picked near the targets; the artifact then damaged four high scorers nobody was watching
— two at 1.00, one at 0.90, one at 0.80 — and the gate failed on exactly that collateral. **A canary
set that only covers what you aimed at cannot catch what you hit by accident.**

`integrate.py --canary-auto BASELINE.json` selects them mechanically: every task at or above
`--canary-floor` (default 0.90) that is not a target, lowest-rate-first so the most fragile high
scorers are the ones kept. Run against that round's own baseline it recovers three of the four tasks
that were actually damaged.

The fourth is the honest limit: it sat at **0.80**, under the floor. Lowering the floor catches it
and admits a noisier guard — a task at 0.80 moves about ±0.13 at n=10 by chance, so it will veto good
work at random. There is no floor that is both complete and quiet. Pick it deliberately: 0.90 for a
wide sweep where false vetoes are expensive, lower when you are integrating one mechanism and can
afford to investigate every flag.

**A canary needs two separately-launched readings, not 20 trials in one.** Measured: a task read
1.00 in 20/20 rollouts and then 2/5 the next day on byte-identical code at the same seeds. It had
been promoted to canary on the strength of that 20/20 — evidence which does not support the claim,
because repeats launched inside one occasion share whatever makes the task come out the way it does.
Two separate readings caught it; more trials in the first reading never would have. The consequence
cuts both ways:

* a task that disagrees between occasions must be dropped from the canary set, **and** must not be
  used to judge a candidate either — two of twelve target tasks moved +0.45 between occasions, so a
  per-task delta on them is uninterpretable;
* the discipline itself survives — the other eight canaries read exactly 1.00 on both occasions — so
  the fix is the selection criterion, not the idea.

**And a per-task effect that clears 2 SE once can still be wrong.** Measured: a task regression of
−0.288 at n=40 (z −2.61, resolvable) read −0.80 in one full-val block and **+0.30** in the next. A
single powered reading is not the floor for a per-task claim; agreement across separately-seeded
occasions is.

Cross-occasion drift on a hosted gateway turned out NOT to be the dominant term: mean per-task
movement was 0.123 across a day at low load versus 0.100 within a day, against 0.250 at high load.
Load dominates elapsed time. Check it rather than assuming either way.

## Running the fan-out, merging it, and remembering what it found

Each optimiser evaluates its own task at full trials **plus a canary of tasks measured stable at
baseline**, in one call, and writes traces so the next edit aims at an observed decision:

```bash
python "$A/taskeval.py" "$R/work/$TAG" <its-tasks> --project "$P" --n <num_trials> \
       --canary <stable-task-ids> --canary-n 3 --conc <low> --traces /tmp/tr_$TAG.json
```

Run every eval **detached** (`nohup … &`, then poll for the output file): under endpoint contention
a per-task eval can take 15-50 minutes, and a harness-level timeout has killed an eval that was
still healthy.

Findings go in a shared ledger, never in the coordinator's head — independent optimisers on
different tasks keep rediscovering one cause, and two of them implementing the same fix collide at
merge with only one of the two actually measured:

```bash
python "$A/mechanisms.py" list --run-dir "$R" --task "$TASK" --compact
python "$A/mechanisms.py" add --run-dir "$R" --owner "$TAG" --status proposed \
       --mechanism "<the cause, one sentence>" --evidence "<what you measured>" \
       --touches <function-the-fix-edits>
python "$A/mechanisms.py" add --run-dir "$R" --owner "$TAG" --status rejected \
       --supersedes <seq> --mechanism "<what turned out to be false>" --evidence "<the test>"
```

Assemble the result **one branch at a time, measuring after each** — never in a single merge:

```bash
python "$A/integrate.py" --base "$R/work/<parent>" --branches "$R/work/t7" "$R/work/t17" \
       --out "$R/work/cand_merged" --tasks <targets> --canary-auto "$R/<baseline-per-task>.json" \
       --n <num_trials> --conc <low> --floor <measured-null-delta>
python "$A/round.py" --run-dir "$R" --project "$P" --candidates cand_merged \
       --n-trials <num_trials> --k-se <gate_k_se>
```

`funcmerge.py` is the merge engine `integrate.py` drives per step; call it directly only to inspect
a single combination, and read its `dropped_additions` — a line a branch ADDED that the merge did not
carry is how a rejected subtraction gets silently re-applied:

```bash
python "$A/funcmerge.py" --base <parent>/<file> --out /tmp/try.py \
       --inputs <branchA>/<file> <branchB>/<file> --union-pure-insertions --json /tmp/fm.json
```

`merge_taskopt.py` remains for the whole-file case (a capability whose artifact is prose, where
per-function merging does not apply):

```bash
python "$A/merge_taskopt.py" --root "$R/work" --base "$R/work/<parent>" \
       --out "$R/work/cand_merged" --include t7 t17
```

"""agent-optimize has no per-skill abstract methods beyond the project adapter.

Like hill-climb, it composes the contract methods (tasks/run_target/score/
materialize) via the shared harness and the phase skills; there is no per-iteration
optimizer subprocess (the conversational agent *is* the optimizer in agent mode).
Nothing here needs filling, so ``check.py`` verifies the agent-mode contract
(the SKILL.md loop + honesty invariants + the deterministic-invocation guard)
rather than any implementation.
"""

from __future__ import annotations

from pathlib import Path

# This algorithm carries no tunable policy: the "policy" is the free-form loop
# in SKILL.md, bounded by the project's free-text stop_condition.
DEFAULT_POLICY: dict = {}


def materialize(capability_dir: Path) -> dict:  # noqa: ARG001
    return {}

"""Behavioral contract for agent-optimize.

agent-optimize has no deterministic loop, so **SKILL.md's prose IS the
implementation** — which means the only honest contract test is to *execute* the
commands SKILL.md documents against a real throwaway run dir and fail if they don't
work. (The previous version only grepped SKILL.md for substrings, which is why a gate
invocation using an argparse choice that doesn't exist, a shell heredoc that never
expanded ``$R``, and a ``cp`` into a directory nobody creates all shipped.)

Offline and deterministic: a synthetic adapter + hand-written rollouts, zero model calls.

Asserts, end to end on a temp run dir:
  * the deterministic-invocation guard emits a JSON error naming the fix;
  * ``mkdir -p $R/work`` + copy-the-best is a real path (RunDir does NOT create ``work/``);
  * the documented **diagnose** command runs and yields ``kept_good``;
  * ``gate_check.py`` reaches the **paired** gate off real rollouts (``paired_n > 0``) and
    accepts a genuine improvement;
  * no-regression vetoes a mean gain that breaks a passing task;
  * ``commit.py`` moves ``best_id`` + ``iterations`` + the stall counter;
  * ``screen.py`` runs a real SUBSET eval, records the subset + seed on disk, reports
    MEASURED savings, kills a clearly-worse candidate, promotes a churn candidate, and
    NEVER emits ``accept``;
  * ``spend.py`` surfaces budget_exhausted + the free-text stop_condition **parsed into
    predicates with measured actuals** + a ``recommendation`` + a pre-fan-out
    affordability answer for N siblings;
  * ``measure.py`` prints the train/val/sealed-test table, labels a no-holdout split as a
    FIT metric, and burns the test seal exactly once;
  * ``taskeval.py`` per-task evals exist as documented and ``merge_taskopt.py`` combines
    disjoint per-task edits while REPORTING (never auto-resolving) overlapping ones;
  * every script SKILL.md names exists and is named in SKILL.md (no drift);
  * SKILL.md carries none of the known-broken patterns;
  * the test seal is never consumed.
"""

from __future__ import annotations

import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import harness
from cap_evolve.skillcheck import (
    Checker, SyntheticAdapter, import_run, quiet, seed_capability_dir, temp_run_dir,
    write_val_rollout,
)

HERE = Path(__file__).resolve().parent
SKILLS = HERE.parents[2]
SKILL_MD = HERE.parent / "SKILL.md"
DIAGNOSE = SKILLS / "phases" / "diagnose" / "scripts" / "run.py"


def _run(c: Checker, label: str, argv: list[str], *, expect_rc: int = 0) -> dict | None:
    """Execute a documented command; return its parsed JSON (None on failure).

    ``expect_rc`` lets a check assert a REFUSAL (a script that must fail loudly with a
    JSON error) as well as a success — a guard that returns rc 0 is not a guard.
    """
    env = dict(os.environ, CAPEVOLVE_CORE=str(SKILLS.parent / "core"))
    p = subprocess.run([sys.executable, *argv], capture_output=True, text=True, env=env)
    if p.returncode != expect_rc:
        c.fail(f"documented command returned rc={p.returncode}, expected {expect_rc} "
               f"({label}): "
               f"{(p.stderr or p.stdout).strip()[:400]}")
        return None
    try:
        return json.loads(p.stdout)
    except Exception:  # noqa: BLE001
        c.fail(f"documented command did not emit JSON ({label}): {p.stdout[:200]}")
        return None


def _guard(c: Checker) -> None:
    run = import_run()
    c.require_main(run)
    with quiet() as buf:
        rc = run.main(["--run-dir", "R", "--project", "P", "--optimizer", "x"])
    try:
        payload = json.loads(buf.getvalue())
    except Exception:  # noqa: BLE001
        payload = {}
    c.check(rc != 0 and "orchestration_mode" in str(payload.get("fix", "")),
            "run.main must refuse a deterministic invocation with a JSON fix naming "
            f"orchestration_mode (rc={rc}, payload={payload})",
            note="deterministic invocation is rejected with an actionable fix message")


def _prose(c: Checker, skill: str) -> None:
    for section in ("## Agent-mode loop", "Phase 0", "Honesty invariants",
                    "## Parallel round"):
        c.check(section in skill, f"SKILL.md missing section: {section!r}")
    for needle in ("FULL val", "stop_condition", "finalize", "budget_exhausted",
                   "no-regression", "unique per sibling",
                   # A: the subset ladder and its non-negotiable limit
                   "never accept", "holdout", "net_rollouts",
                   # B: textual constraints re-read from the run dir
                   "predicates", "ambiguous", "recommendation",
                   # C: multi-opportunity rounds and the churn they must not admit
                   "churn",
                   # D: the final full-split measurement
                   "sealed test"):
        c.check(needle in skill, f"SKILL.md missing honesty/loop marker: {needle!r}")

    # Known-broken patterns that previously shipped.
    # `--mode paired` used to be an invalid choice (argparse exit 2). The phase CLI now
    # reaches it, but ONLY in rollout mode: two scalar means carry no per-task deltas.
    # Join backslash continuations so a multi-line invocation is judged as one command.
    _joined = skill.replace("\\\n", " ")
    c.check(all("--run-dir" in ln for ln in _joined.splitlines() if "--mode paired" in ln),
            "SKILL.md pairs `--mode paired` with scalar means — that combination is "
            "refused; a paired test needs --run-dir/--current-tag/--candidate-tag",
            note="any --mode paired invocation supplies a run dir")
    c.check("<<'PY'" not in skill and '<<"PY"' not in skill,
            "SKILL.md uses a heredoc for run-dir Python (a quoted one never expands $R)",
            note="no heredoc-Python: the commit step is a real script with real flags")
    c.check('mkdir -p "$R/work"' in skill,
            "SKILL.md copies into $R/work without creating it (RunDir.create does not)",
            note="$R/work is explicitly created before it is used")
    c.check("Example only" in skill,
            "SKILL.md must mark the capability file layout as example-only")

    # Every helper the loop names must exist, and every helper must be named.
    helpers = ["gate_check.py", "commit.py", "spend.py", "screen.py", "measure.py",
               "funcmerge.py", "multirep.py",
               "round.py", "taskeval.py", "merge_taskopt.py", "mechanisms.py"]
    for h in helpers:
        c.check((HERE / h).is_file(), f"missing documented helper script: scripts/{h}")
        c.check(h in skill, f"scripts/{h} exists but SKILL.md never uses it")
    c.check("Task" in (skill.split("---")[1] if skill.count("---") >= 2 else ""),
            "frontmatter allowed-tools must include Task for the parallel round",
            note="allowed-tools declares Task (parallel fan-out is actionable)")


def _progressive_disclosure(c: Checker, skill: str) -> None:
    """The body stays inside the 500-line budget, and every reference is pointed AT.

    skill-creator: the SKILL.md body is re-read on every trigger, so it is capped at ~500 lines,
    and each bundled reference must be linked from the body with what it contains AND when to
    load it. References are ONE level deep: a reference that links to another reference cannot
    be read on its own, which is the whole point of the hierarchy.
    """
    body = skill.splitlines()
    c.check(len(body) < 500,
            f"SKILL.md body is {len(body)} lines; the recurring per-trigger budget is 500 — "
            "move depth into references/ with an explicit pointer",
            note=f"SKILL.md body is {len(body)} lines, inside the 500-line budget")

    refs = sorted(p for p in (HERE.parent / "references").glob("*.md"))
    c.check(bool(refs), "no references/ — the split is the point of the hierarchy")
    for ref in refs:
        rel = f"references/{ref.name}"
        c.check(rel in skill, f"{rel} exists but SKILL.md never links it")
        # The pointer must say WHEN to load it, not merely that it exists.
        c.check(f"({rel})" in skill and "**Load**" in skill,
                f"{rel} needs a pointer in SKILL.md's `## References` stating what it contains "
                "and when to load it")
        text = ref.read_text(encoding="utf-8")
        c.check("](references/" not in text,
                f"{rel} links to another reference — references are one level deep, because a "
                "reader may only partially read either one")
        if len(text.splitlines()) > 300:
            c.check("## Contents" in text,
                    f"{rel} is over 300 lines and needs a table of contents")
    c.note(f"{len(refs)} references, each linked with a when-to-load pointer, one level deep")

    # Retrospective narrative belongs in a reference or the run log, never in the body: it is
    # re-read on every trigger and an agent driving a round acts no differently for having it.
    for anecdote in ("in one run", "Measured here", "A real run", "four consecutive null runs"):
        c.check(anecdote not in skill,
                f"SKILL.md carries retrospective narrative ({anecdote!r}) — keep the rule and the "
                "command in the body, move the number that bought it into references/")


def _project(tmp: Path, *, n: int) -> Path:
    """A minimal project dir the SUBPROCESS scripts can load: adapter + spec.

    ``screen.py`` / ``measure.py`` take ``--project`` and go through
    ``check.load_adapter``, so the check needs a real ``adapters/adapter.py`` — not just
    the in-process ``SyntheticAdapter``. It subclasses the same synthetic adapter with
    the same ``n``, so the subprocess sees byte-identical tasks and the numbers line up
    with what this check computed in-process. Still zero model calls.
    """
    project = tmp / "project"
    (project / "adapters").mkdir(parents=True, exist_ok=True)
    (project / "adapters" / "adapter.py").write_text(
        "from cap_evolve.skillcheck import SyntheticAdapter\n\n\n"
        "class Adapter(SyntheticAdapter):\n"
        f"    def __init__(self):\n        super().__init__(n={n})\n",
        encoding="utf-8")
    (project / "capevolve.yaml").write_text(
        "num_trials: 1\ngate_mode: paired\ngate_k_se: 1.0\n"
        'stop_condition: "reach val mean >= 0.9, or stop after $5 or 30 minutes"\n',
        encoding="utf-8")
    return project


def _screen_round(c: Checker, tmp: Path) -> None:
    """The subset promotion ladder, executed: a kill, a promote, and never an accept."""
    from cap_evolve import RunDir, harness
    from cap_evolve.skillcheck import SyntheticAdapter

    # n=48 -> a 12-task val, so tier 1 (max(MIN_K, 25%) = 6) stays a STRICT subset.
    # A 20-task project gives val=5, which MIN_K=6 would round up to the whole split —
    # the screen would silently stop being a screen.
    adapter = SyntheticAdapter(n=48)
    project = _project(tmp, n=48)
    run_dir = RunDir.create(tmp / ".capevolve_screen", ts="chk")
    harness.ensure_splits(adapter, run_dir, seed=0)
    R = str(run_dir.root)
    val_ids = run_dir.read_splits().ids("val")

    # A parent that PASSES every val task, written straight to disk — no rollouts paid.
    # This is the point of the design: the screen re-reads the parent instead of re-running it.
    for tid in val_ids:
        write_val_rollout(run_dir, tid, tag="cur", reward=1.0, feedback="solved")
    run_dir.set_best("cur")
    run_dir.snapshot("cur", seed_capability_dir(tmp / "curcap", level=48))

    work = Path(R) / "work"
    work.mkdir(parents=True, exist_ok=True)
    worse = seed_capability_dir(work / "worse_src", level=0)   # solves nothing
    shutil.copytree(worse, work / "worse")
    sc = _run(c, "screen.py (kill)", [str(HERE / "screen.py"), "--run-dir", R,
                                      "--project", str(project),
                                      "--candidate", str(work / "worse"), "--tier", "1"])
    if sc:
        c.check(sc.get("decision") == "kill",
                f"screen.py did not kill a candidate that regresses every task: "
                f"{sc.get('decision')} / {sc.get('reason')}",
                note=f"subset screen kills clear harm: {sc.get('reason')}")
        c.check(0 < len(sc["subset"]["ids"]) < len(val_ids),
                f"screen.py screened {len(sc['subset']['ids'])} of {len(val_ids)} val "
                "tasks — a subset screen must be a strict subset to be cheap")
        c.check(sc["subset"].get("seed") is not None and sc["subset"].get("holdout_frac"),
                "screen.py did not record the subset seed / holdout fraction")
        c.check(sc["savings"]["avoided"] > 0 and
                sc["savings"]["net_rollouts"] == sc["savings"]["full_val_rollouts"]
                - sc["savings"]["fired"],
                f"screen.py savings are not the measured difference: {sc['savings']}",
                note=f"kill saved {sc['savings']['avoided']} of "
                     f"{sc['savings']['full_val_rollouts']} rollouts (measured)")
        rec = run_dir.root / "screens" / f"{sc['screen_tag']}.json"
        c.check(rec.is_file() and json.loads(rec.read_text())["subset"] == sc["subset"],
                f"screen.py did not persist a reproducible record at {rec}",
                note="every screen decision is recorded in $R/screens/ (auditable, seeded)")
        c.check(sc.get("decision") != "accept" and "accept" not in str(sc.get("decision")),
                "a subset screen emitted an accept — subsets may only kill or promote")

    # A candidate that changes nothing measurable must PROMOTE, not be killed on noise.
    shutil.copytree(seed_capability_dir(work / "same_src", level=48), work / "same")
    sc2 = _run(c, "screen.py (promote on a flat subset)",
               [str(HERE / "screen.py"), "--run-dir", R, "--project", str(project),
                "--candidate", str(work / "same"), "--tier", "1"])
    if sc2:
        c.check(sc2.get("decision") == "promote" and sc2.get("inconclusive") is True,
                f"screen.py must promote (not kill) a flat/inconclusive subset: {sc2}",
                note="the screen is biased against false kills: a flat Δ̄ promotes")
        c.check(sc2["savings"]["net_rollouts"] < 0,
                f"a promote must be reported as a COST, not a saving: {sc2['savings']}",
                note="promote costs are booked honestly as negative net_rollouts")

    # Rungs are cumulative: tier 2 must not re-run tier 1's tasks.
    sc3 = _run(c, "screen.py (tier 2 reuses tier 1)",
               [str(HERE / "screen.py"), "--run-dir", R, "--project", str(project),
                "--candidate", str(work / "same"), "--tier", "2"])
    if sc3:
        c.check(sc3.get("reused_from_earlier_tiers"),
                f"tier 2 re-ran tasks tier 1 already screened: {sc3.get('fired_ids')}",
                note="the promotion ladder is cumulative — each rung pays only for new ids")

    c.check(not run_dir.read_splits().test_used,
            "the subset screen consumed the sealed test split")


def _measure(c: Checker, tmp: Path) -> None:
    """The final table: a held-out verdict, a no-holdout warning, and one seal."""
    from cap_evolve import RunDir, harness
    from cap_evolve.skillcheck import SyntheticAdapter

    adapter = SyntheticAdapter(n=20)
    project = _project(tmp, n=20)
    run_dir = RunDir.create(tmp / ".capevolve_measure", ts="chk")
    harness.ensure_splits(adapter, run_dir, seed=0)
    harness.baseline(adapter, seed_capability_dir(tmp / "mseed", level=3),
                     run_dir=run_dir)
    R = str(run_dir.root)

    # A genuinely better candidate, accepted through the documented path.
    work = Path(R) / "work"
    work.mkdir(parents=True, exist_ok=True)
    shutil.copytree(run_dir.candidate_dir("seed"), work / "cand_1")
    (work / "cand_1" / "level.txt").write_text("25", encoding="utf-8")
    harness.evaluate_candidate(adapter, work / "cand_1", run_dir=run_dir, split="val",
                               n_trials=1, tag="cand_1")
    run_dir.snapshot("cand_1", work / "cand_1")
    run_dir.set_best("cand_1")

    m = _run(c, "measure.py (final table + seal)",
             [str(HERE / "measure.py"), "--run-dir", R, "--project", str(project),
              "--train", "on"])
    if m:
        rows = {r.get("split"): r for r in m.get("splits") or []}
        c.check(set(rows) == {"train", "val", "test"},
                f"measure.py did not report every split: {sorted(rows)}")
        c.check(m["holdout"]["test_is_held_out"] is True,
                f"measure.py mislabelled a disjoint split: {m['holdout']}")
        v = rows.get("val") or {}
        c.check((v.get("gate") or {}).get("accept") is True and v["paired"]["n"] > 0,
                f"measure.py did not recompute the val gate on the paired vector: {v}")
        c.check((rows["test"].get("gate") or {}).get("note", "").startswith("no gate"),
                f"measure.py must not gate on test: {rows['test'].get('gate')}")
        c.check(rows["train"].get("best", {}).get("reward") is not None,
                f"--train on did not measure the train split: {rows['train']}")
        c.check(m["val_per_task_movement"]["fixed"],
                f"measure.py reported no per-task movement: {m['val_per_task_movement']}",
                note="measure.py prints per-task fixed/broke movement, not just means")
        c.check(run_dir.read_splits().test_used,
                "measure.py did not burn the test seal (test was never scored)",
                note="measure.py seals test exactly once, via harness.finalize")
        c.check((run_dir.root / "measure.json").is_file(),
                "measure.py did not persist measure.json")
        c.check("net_rollouts" in (m.get("screen_ledger") or {}),
                f"measure.py did not sum the screen ledger: {m.get('screen_ledger')}",
                note="measure.py totals the screens' MEASURED rollout economics")

    # A second call must NOT re-score test; it reports the sealed final.json instead.
    m2 = _run(c, "measure.py (second call, seal already burned)",
              [str(HERE / "measure.py"), "--run-dir", R, "--project", str(project),
               "--train", "off"])
    if m2:
        t = next(r for r in m2["splits"] if r["split"] == "test")
        c.check("already sealed" in t.get("status", ""),
                f"a second measure.py re-scored the sealed test split: {t.get('status')}",
                note="the seal is single-use: a second measure reads final.json")

    # A no-holdout spec must be labelled a FIT metric, not generalisation.
    from cap_evolve.splits import Splits
    nh = RunDir.create(tmp / ".capevolve_nh", ts="chk")
    ids = [t.id for t in adapter.tasks("all")]
    nh.write_splits(Splits(train=list(ids), val=list(ids), test=list(ids), seed=0))
    harness.baseline(adapter, seed_capability_dir(tmp / "nhseed", level=3), run_dir=nh)
    m3 = _run(c, "measure.py (no-holdout spec)",
              [str(HERE / "measure.py"), "--run-dir", str(nh.root), "--project",
               str(project), "--train", "auto", "--skip-test"])
    if m3:
        c.check(m3["holdout"]["test_is_held_out"] is False
                and "FIT metric" in m3["holdout"]["verdict"],
                f"measure.py presented a no-holdout fit as generalisation: {m3['holdout']}",
                note="a no-holdout spec is labelled a FIT metric, with the overlap counted")
        c.check(m3.get("warning") and "null result" in m3["warning"],
                f"best==seed must be flagged as a null result: {m3.get('warning')}")
        tr = next(r for r in m3["splits"] if r["split"] == "train")
        c.check("identical to val" in tr.get("status", ""),
                f"--train auto paid for a train eval identical to val: {tr}")


def _live_round(c: Checker, tmp: Path) -> None:
    """Walk the SKILL.md round against a real run dir, executing each command."""
    from cap_evolve import Budget, RunDir, harness

    # n=20 → ~5 val tasks, and a seed that already solves a few, so the paired delta
    # vector is MIXED (some tasks flip, some don't) and the paired SE is genuinely
    # non-zero — i.e. the check proves the real significance test, not its SE=0 fallback.
    adapter = SyntheticAdapter(n=20)
    seed = seed_capability_dir(tmp, level=3)
    run_dir = RunDir.create(tmp / ".capevolve", ts="chk", budget=Budget(max_iterations=5))
    harness.ensure_splits(adapter, run_dir, seed=0)
    harness.baseline(adapter, seed, run_dir=run_dir)
    R = str(run_dir.root)

    project = _project(tmp, n=20)

    # step 0 — the affordability readout + the parsed textual constraints
    sp = _run(c, "spend.py (pre-round)", [str(HERE / "spend.py"), "--run-dir", R,
                                          "--project", str(project),
                                          "--n-siblings", "3"])
    if sp:
        c.check(sp.get("best_id") == "seed" and sp.get("stop") is False,
                f"spend.py did not report a fresh run: {sp}")
        c.check("val mean >= 0.9" in sp.get("stop_condition", ""),
                "spend.py did not echo the project's free-text stop_condition")
        c.check(sp.get("test_sealed") is True, "spend.py reported an unsealed test split")
        c.check(isinstance(sp.get("wallclock_seconds"), (int, float)),
                "spend.py did not measure wallclock from the run dir")
        cons = sp.get("constraints") or {}
        kinds = {p["kind"]: p for p in cons.get("predicates") or []}
        c.check({"target_val_score", "max_usd", "max_wallclock_seconds"} <= set(kinds),
                f"spend.py did not parse the prose into predicates: {sorted(kinds)}",
                note="spend.py parses stop_condition prose into checkable predicates")
        c.check(kinds.get("target_val_score", {}).get("actual")
                == (sp.get("best_val") or {}).get("reward")
                and kinds["target_val_score"]["satisfied"] is False,
                f"target_val_score not checked against the FULL-val mean: "
                f"{kinds.get('target_val_score')}")
        c.check(sp.get("recommendation") == "continue",
                f"spend.py should recommend continue on a fresh run: "
                f"{sp.get('recommendation')} {sp.get('recommendation_reasons')}")
        af = sp.get("afford") or {}
        c.check(af.get("rollouts_needed") == 3 * af.get("val_n", 0),
                f"spend.py --n-siblings did not price N full-val evals: {af}",
                note="spend.py answers affordability for N siblings BEFORE a fan-out")

    # step 1 — the documented diagnose command
    dg = _run(c, "diagnose phase", [str(DIAGNOSE), "--run-dir", R, "--tag", "seed"])
    if dg is not None:
        c.check("kept_good" in dg, f"diagnose emitted no kept_good: {list(dg)}",
                note="diagnose gives clusters to fix + kept_good to protect")

    # step 2 — mkdir work + copy the best (the path RunDir does not create)
    work = Path(R) / "work"
    work.mkdir(parents=True, exist_ok=True)
    tag = "cand_1"
    shutil.copytree(run_dir.candidate_dir("seed"), work / tag)
    c.check((work / tag / "level.txt").is_file(),
            "copying $R/candidates/$BEST into $R/work/<tag> did not produce a working copy",
            note="$R/work/<tag> working-copy flow is real")
    (work / tag / "level.txt").write_text("12", encoding="utf-8")  # a genuine improvement

    # step 4 — full-val eval under the candidate's own tag, then the real gate
    harness.evaluate_candidate(adapter, work / tag, run_dir=run_dir, split="val",
                              n_trials=1, tag=tag)
    g = _run(c, "gate_check.py (accept)", [str(HERE / "gate_check.py"), "--run-dir", R,
                                           "--candidate", tag, "--k-se", "1.0"])
    if g:
        c.check(g.get("paired_n", 0) > 0,
                f"gate_check did not reach the PAIRED gate (paired_n={g.get('paired_n')})",
                note=f"paired gate reachable from the CLI: {g['gate']['reason']}")
        c.check(g.get("verdict") == "accept" and not g.get("regressions"),
                f"gate_check rejected a genuine improvement: {g}")

    # step 5 — commit moves best_id, iterations and the stall counter
    cm = _run(c, "commit.py (accept)", [str(HERE / "commit.py"), "--run-dir", R,
                                       "--candidate-id", tag, "--from-dir", str(work / tag),
                                       "--decision", "accept", "--val", "1.0",
                                       "--note", "raise coverage generally",
                                       "--optimizer-usd", "0.25"])
    if cm:
        c.check(cm.get("best_id") == tag, f"commit.py did not set best: {cm}")
        c.check(cm["spent"]["iterations"] == 1 and cm["spent"]["stall"] == 0
                and cm["spent"]["optimizer_usd"] == 0.25,
                f"commit.py under-recorded spend: {cm['spent']}",
                note="commit.py records iterations + stall + the proposer's own cost")
        c.check(run_dir.candidate_dir(tag).is_dir(), "commit.py did not snapshot the candidate")

    # a reject must advance the stall counter (what budget_exhausted's stall rule reads)
    rj = _run(c, "commit.py (reject)", [str(HERE / "commit.py"), "--run-dir", R,
                                        "--candidate-id", "cand_2", "--from-dir", str(work / tag),
                                        "--decision", "reject", "--note", "no gain"])
    if rj:
        c.check(rj["best_id"] == tag and rj["spent"]["stall"] == 1,
                f"reject changed best or did not advance stall: {rj}")

    c.check(not run_dir.read_splits().test_used,
            "the agent-optimize round consumed the sealed test split",
            note="test split sealed throughout the round")


def _no_regression(c: Checker, tmp: Path) -> None:
    """Regressions are REPORTED by default and only VETO under --veto-regressions.

    Both halves are asserted against the same fixture, because the default flip is the
    fix for four consecutive null results (see gate_check.regressions): a per-task drop
    at n trials is an estimate, not evidence of harm, and the veto fired on a
    byte-identical copy of the seed 43% of the time at 5 trials.
    """
    run_dir, _ = temp_run_dir(tmp / "regr", ids=("a", "b", "c", "d"))
    # current: a passes, b/c/d fail  → mean 0.25
    for tid, r in (("a", 1.0), ("b", 0.0), ("c", 0.0), ("d", 0.0)):
        write_val_rollout(run_dir, tid, tag="cur", reward=r, feedback="fb")
    # candidate: a BREAKS, b/c pass  → mean 0.50 (a real mean gain, a real regression)
    for tid, r in (("a", 0.0), ("b", 1.0), ("c", 1.0), ("d", 0.0)):
        write_val_rollout(run_dir, tid, tag="regr", reward=r, feedback="fb")
    run_dir.set_best("cur")

    g = _run(c, "gate_check.py (regressions reported, not vetoed)",
             [str(HERE / "gate_check.py"), "--run-dir", str(run_dir.root),
              "--candidate", "regr", "--mode", "strict"])
    if g:
        c.check(g.get("regressions") == ["a"] and g.get("verdict") == "accept",
                f"default must REPORT the task-'a' regression and still accept the "
                f"gate-passing mean gain: {g}",
                note="the veto is opt-in; the paired/strict gate is the decision rule")
        c.check(g["gate"]["accept"] is True,
                f"expected the raw gate to accept the mean gain: {g['gate']}")

    v = _run(c, "gate_check.py (--veto-regressions)",
             [str(HERE / "gate_check.py"), "--run-dir", str(run_dir.root),
              "--candidate", "regr", "--mode", "strict", "--veto-regressions"])
    if v:
        c.check(v.get("regressions") == ["a"] and v.get("verdict") == "reject",
                f"--veto-regressions must still veto a mean gain that broke task 'a': {v}",
                note="the old behaviour stays reachable for anyone who wants it")


def _tag_isolation(c: Checker, tmp: Path) -> None:
    """The screen tag and the full-val tag must NEVER read each other's rollouts.

    This is the one flaw that would silently corrupt every gate decision: the reader
    globs ``*__<tag>__t*.json``, so ``<tag>__screenN`` files must be invisible to a
    read of ``<tag>`` and vice versa. Probe it adversarially — screen says 1.0, full
    val says 0.0, on the SAME candidate — and require the full-val read to return 0.0.
    """
    run_dir, _ = temp_run_dir(tmp / "iso", ids=("a", "b", "c", "d"))
    for tid in ("a", "b", "c", "d"):
        write_val_rollout(run_dir, tid, tag="cur", reward=1.0, feedback="ok")
        # the LIE: the screen tag claims a perfect candidate …
        write_val_rollout(run_dir, tid, tag="cand__screen1", reward=1.0, feedback="ok")
        # … while the candidate's real full-val rollouts are all zeros.
        write_val_rollout(run_dir, tid, tag="cand", reward=0.0, feedback="failed")
    run_dir.set_best("cur")

    full = harness.split_result_from_rollouts(run_dir, "cand", "val")
    c.check(abs(full.reward) < 1e-9 and full.n_scored == 4,
            f"full-val read of tag 'cand' leaked its screen rollouts: reward="
            f"{full.reward} n_scored={full.n_scored} (expected 0.0 over 4)",
            note="rollout isolation: a full-val read of <tag> cannot see <tag>__screenN")
    scr = harness.split_result_from_rollouts(run_dir, "cand__screen1", "val")
    c.check(abs(scr.reward - 1.0) < 1e-9,
            f"screen-tag read leaked the full-val rollouts: reward={scr.reward}",
            note="rollout isolation: a screen read of <tag>__screenN cannot see <tag>")
    g = _run(c, "gate_check.py (tag isolation)",
             [str(HERE / "gate_check.py"), "--run-dir", str(run_dir.root),
              "--candidate", "cand", "--mode", "paired", "--k-se", "1.0"])
    if g:
        c.check(g["candidate"]["reward"] == 0.0 and g["verdict"] == "reject",
                f"the gate read the screen's optimistic rollouts: {g['candidate']}",
                note="the gate scores the full-val tag only, never the screen tag")


def _tag_collision(c: Checker, tmp: Path) -> None:
    """commit.py must REFUSE to reuse a candidate id that already has a decision.

    A real run had two concurrent drivers tag a candidate ``cand_r2``: two reject
    events, ONE set of rollouts — one edit judged on another's evidence, and the
    second snapshot overwrote the first. The guard belongs in commit.py because that
    is where every caller routes.
    """
    run_dir, project = temp_run_dir(tmp / "coll", ids=("a", "b"))
    work = run_dir.root / "work" / "dup"
    work.mkdir(parents=True, exist_ok=True)
    (work / "policy.md").write_text("v1", encoding="utf-8")
    argv = [str(HERE / "commit.py"), "--run-dir", str(run_dir.root),
            "--candidate-id", "dup", "--from-dir", str(work),
            "--decision", "reject", "--note", "first"]
    first = _run(c, "commit.py (first use of a tag)", argv)
    c.check(bool(first) and first.get("decision") == "reject",
            f"the first commit of a fresh tag was refused: {first}")
    second = _run(c, "commit.py (tag reuse)", argv, expect_rc=2)
    c.check(bool(second) and "already" in json.dumps(second),
            f"commit.py accepted a duplicate candidate id: {second}",
            note="commit.py refuses a tag that already carries a decision (no "
                 "two candidates can collapse onto one set of rollouts)")
    forced = _run(c, "commit.py (--force overrides)", argv + ["--force"])
    c.check(bool(forced) and forced.get("decision") == "reject",
            f"--force did not override the collision guard: {forced}")


def _round_control(c: Checker, tmp: Path) -> None:
    """round.py builds the null control itself and reports the round's own noise floor.

    The control is not optional and not the driver's job to remember: three of the four
    null runs on a multi-turn tool-use benchmark compared a candidate against a parent mean measured in
    an *earlier* round, so re-measurement noise read as signal in both directions. This
    asserts the control is materialised from the current best and shows up in the table.
    """
    from cap_evolve import RunDir, harness
    from cap_evolve.skillcheck import SyntheticAdapter

    adapter = SyntheticAdapter(n=24)
    project = _project(tmp, n=24)
    run_dir = RunDir.create(tmp / ".capevolve_round", ts="chk")
    harness.ensure_splits(adapter, run_dir, seed=0)
    R = str(run_dir.root)

    # A parent that solves half of val, snapshotted so round.py has something to copy.
    for i, tid in enumerate(run_dir.read_splits().ids("val")):
        write_val_rollout(run_dir, tid, tag="cur", reward=float(i % 2), feedback="fb")
    run_dir.set_best("cur")
    run_dir.snapshot("cur", seed_capability_dir(tmp / "roundcap", level=12))

    work = Path(R) / "work"
    work.mkdir(parents=True, exist_ok=True)
    shutil.copytree(seed_capability_dir(work / "_src", level=24), work / "cand_x")

    r = _run(c, "round.py (null control + parallel eval + serial gate)",
             [str(HERE / "round.py"), "--run-dir", R, "--project", str(project),
              "--candidates", "cand_x", "--n-trials", "1", "--k-se", "1.0",
              # This check exercises round.py's own gate mechanics, not the screen ladder
              # (covered separately below) — cand_x never went through screen.py.
              "--skip-screen-ladder"])
    if r:
        ctl_tag = r.get("control", {}).get("tag") or ""
        c.check(ctl_tag.startswith("ctl_null_i") and (work / ctl_tag).is_dir(),
                f"round.py must materialise a ROUND-SCOPED $R/work/ctl_null_i<n> from the "
                f"current best (got {ctl_tag!r})",
                note="the control is built by the script and tagged per round, so it can "
                     "neither be skipped nor overwrite a previous round's noise floor")
        c.check(r.get("noise_floor_from_control") is not None,
                "round.py did not report the round's measured noise floor",
                note="every round reports what ZERO change measures, from its own control")
        c.check([x["tag"] for x in r.get("candidates") or []] == ["cand_x"],
                f"round.py candidate rows wrong: {r.get('candidates')}")
        c.check("commit" in (r.get("next") or ""),
                "round.py must hand the accept/reject decision back to the driver",
                note="round.py never commits: which part of a bundle to keep is a judgement")
        # The table must survive on disk, not only on stdout. A driver that forgot to redirect
        # left the round's gate verdict nowhere: on run 32814848187 the abandoned round was
        # reconstructible only because the driver happened to have redirected it somewhere
        # someone guessed. host.py's un-booked-round backstop reads these files.
        table = r.get("table_path") or ""
        c.check(bool(table) and Path(table).is_file()
                and Path(table).parent == work
                and json.loads(Path(table).read_text(encoding="utf-8")).get("candidates"),
                f"round.py did not persist its gate table under $R/work/ (got {table!r})",
                note="the round's verdict is the run's evidence; it must not depend on the "
                     "driver remembering to redirect stdout")

    # A round whose --n-trials differs from the parent's must be able to pair against the
    # control instead, or every delta silently carries a precision mismatch.
    shutil.rmtree(work / "cand_y", ignore_errors=True)
    shutil.copytree(seed_capability_dir(work / "_src2", level=24), work / "cand_y")
    r2 = _run(c, "round.py --gate-against control",
              [str(HERE / "round.py"), "--run-dir", R, "--project", str(project),
               "--candidates", "cand_y", "--n-trials", "1", "--k-se", "1.0",
               "--gate-against", "control", "--skip-screen-ladder"])
    if r2:
        ref = (r2.get("gated_against") or {})
        c.check(str(ref.get("tag", "")).startswith("ctl_null_i")
                and ref.get("mode") == "control",
                f"--gate-against control did not pair against the control: {ref}",
                note="a round can pair candidates against its OWN control, removing the "
                     "precision mismatch when its n-trials differs from the parent's")
    r3 = _run(c, "round.py --gate-against control --no-control (refused)",
              [str(HERE / "round.py"), "--run-dir", R, "--project", str(project),
               "--candidates", "cand_y", "--n-trials", "1", "--gate-against", "control",
               "--no-control", "--skip-screen-ladder"], expect_rc=2)
    c.check(bool(r3) and "control" in json.dumps(r3),
            f"gating against a control that was skipped must be refused, got {r3}",
            note="asking to gate against a control while disabling it is refused, not ignored")


def _control_replicates(c: Checker, tmp: Path) -> None:
    """A round must evaluate MORE THAN ONE control, and must report the gap between them.

    One control does not bound the noise. Measured on a multi-turn tool-use benchmark: two byte-identical
    controls, same seeds, temperature 0, read 0.6467 and 0.7267 — a paired delta of +0.0800 that
    PASSES a k_se=1.0 bar on zero change. The same candidate then read +0.0867 against one of
    those controls and +0.0067 against the other, so a single-control gate hands out a coin flip.
    Asserted here: the default is at least two replicates, and the round names the gap between
    them rather than leaving the caller to compute it.
    """
    src = (HERE / "round.py").read_text(encoding="utf-8")
    c.check("--control-replicates" in src,
            "round.py must offer control replicates",
            note="one control cannot separate a small gain from re-measurement")
    c.check('"--control-replicates", type=int, default=2' in src.replace("\n", " ")
            or 'default=2' in src.split("--control-replicates")[1][:200],
            "control replicates must DEFAULT to at least 2, not be opt-in",
            note="the failure mode is trusting a single null reading, so the safe value is the "
                 "default")
    c.check("null_delta_between_control_replicates" in src,
            "the round must report the gap between identical control replicates",
            note="identical bytes on identical seeds: that gap IS the bar")


def _multirep(c: Checker, tmp: Path) -> None:
    """A verdict must take its error ACROSS runs, because within-run SE cannot see run noise.

    Measured on a multi-turn tool-use benchmark: one paired run reported SE 0.0548 over tasks and "accept" at
    +0.0867; a second paired run of the same candidate on a different seed block gave +0.0200, and
    a byte-identical control re-run moved +0.0800 on its own. So the across-task SE understates the
    real uncertainty and the tool must refuse to call a single run a demonstration.
    """
    d = tmp / "mr"
    d.mkdir(parents=True, exist_ok=True)

    def arm(name, rates):
        (d / name).write_text(json.dumps(
            {"per_task": {t: {"rate": r} for t, r in rates.items()}}))

    # two runs, same candidate: one looks like a win, the other does not
    arm("c1.json", {"a": 1.0, "b": 0.8, "c": 0.6})
    arm("k1.json", {"a": 0.8, "b": 0.6, "c": 0.4})
    arm("c2.json", {"a": 0.8, "b": 0.6, "c": 0.6})
    arm("k2.json", {"a": 0.8, "b": 0.6, "c": 0.6})
    r = _run(c, "multirep.py (across-run error, inconsistent runs)",
             [str(HERE / "multirep.py"), f"{d/'c1.json'}:{d/'k1.json'}",
              f"{d/'c2.json'}:{d/'k2.json'}"])
    c.check(bool(r) and r.get("n_runs") == 2 and r.get("se_across_runs") is not None
            and "NOT DEMONSTRATED" in str(r.get("verdict")),
            "two runs that disagree must NOT be reported as a demonstrated gain",
            note="a single run's across-task SE cannot see run-to-run nondeterminism")
    r2 = _run(c, "multirep.py (single run refuses to conclude)",
              [str(HERE / "multirep.py"), f"{d/'c1.json'}:{d/'k1.json'}"])
    c.check(bool(r2) and r2.get("se_across_runs") is None
            and "need >= 2" in str(r2.get("verdict")),
            "one paired run must not yield a verdict at all",
            note="the retracted accept in this project came from exactly one paired run")


def _merge_taskopt(c: Checker, tmp: Path) -> None:
    """Per-task fan-out only pays if the merge is trustworthy.

    K optimisers edit the same files from the same base, so combining them is where a good
    round quietly becomes a bad one. Two properties are asserted: disjoint edits from
    different optimisers BOTH survive (a merge that silently drops one turns K measured gains
    into one), and an overlapping edit is reported as `conflicted` rather than auto-resolved
    (two optimisers guarding the same moment is a judgement call, not a textual one).
    """
    root, rel = tmp / "taskopt", "policy.md"
    base = tmp / "mergebase"
    base.mkdir(parents=True, exist_ok=True)
    # A multi-line base: with a ONE-line file every edit touches the same line, so even
    # genuinely independent optimisers would "conflict" and the check would prove nothing.
    (base / rel).write_text("\n".join(f"rule {i}" for i in range(1, 21)) + "\n")

    def copy(name: str, transform) -> None:
        d = root / name
        d.mkdir(parents=True, exist_ok=True)
        (d / rel).write_text(transform((base / rel).read_text()))

    copy("optA", lambda t: "RULE_FROM_A\n" + t)
    copy("optB", lambda t: t + "\nRULE_FROM_B\n")
    copy("optC", lambda t: "RULE_FROM_C\n" + t)   # same first line as optA -> conflict

    argv = [str(HERE / "merge_taskopt.py"), "--root", str(root), "--base", str(base),
            "--files", rel, "--subdirs", "", "--repo", str(tmp / "mergerepo")]
    r = _run(c, "merge_taskopt.py (disjoint edits)",
             argv + ["--out", str(tmp / "merged_ok"), "--include", "optA", "optB"])
    if r:
        merged = (tmp / "merged_ok" / rel).read_text()
        c.check("RULE_FROM_A" in merged and "RULE_FROM_B" in merged,
                "merge dropped an optimiser's edit: both disjoint per-task edits must survive",
                note="a merge that loses one optimiser turns K measured gains into one")
        c.check(not r.get("conflicted"),
                f"disjoint edits reported as conflicting: {r.get('conflicted')}")

    r2 = _run(c, "merge_taskopt.py (overlapping edits)",
              argv + ["--out", str(tmp / "merged_conflict"), "--include", "optA", "optC"],
              expect_rc=1)
    c.check(bool(r2) and bool(r2.get("conflicted")),
            f"an overlapping per-task edit was not reported as conflicted: {r2}",
            note="conflicts are reported, never silently resolved: a semantic conflict means "
                 "dropping a bundle, not shipping a hybrid neither optimiser measured")

    # Opt-in union resolution, for a TEXTUAL collision of two distinct additions. Dropping a
    # verified gain over a whitespace accident is the failure this prevents; the guard against
    # abusing it is that the affected files are named and the result must still be rendered.
    r3 = _run(c, "merge_taskopt.py (--union-on-conflict keeps both sides)",
              argv + ["--out", str(tmp / "merged_union"), "--include", "optA", "optC",
                      "--union-on-conflict"])
    if r3:
        got = (tmp / "merged_union" / rel).read_text()
        c.check("RULE_FROM_A" in got and "RULE_FROM_C" in got,
                "union resolution must keep BOTH colliding additions",
                note="a textual collision of distinct additions is resolved by union, since the "
                     "union is what both optimisers actually measured")
        c.check(r3.get("union_resolution_enabled") is True and r3.get("union_candidates"),
                f"union resolution must name what it resolved: {r3}",
                note="union-resolved files are named so the distinct-additions claim is checkable")
        c.check("RENDER" in (r3.get("next") or "").upper(),
                "union resolution must demand the live toolset be rendered afterwards",
                note="keeping both sides can duplicate a definition or break syntax, which an "
                     "import check does not catch")


def _mechanisms(c: Checker, tmp: Path) -> None:
    """The fan-out ledger must survive concurrent writers and must gate reimplementation.

    Its entire job is to stop K optimisers rediscovering one cause and writing K colliding
    fixes for it, so two properties matter: concurrent appends do not lose or tear rows, and
    a listing names what is already owned so a second implementation is refused by policy
    rather than discovered at merge time.
    """
    import concurrent.futures as cf

    run = tmp / "mechrun"
    run.mkdir(parents=True, exist_ok=True)
    argv = [sys.executable, str(HERE / "mechanisms.py"), "add", "--run-dir", str(run)]
    env = dict(os.environ, CAPEVOLVE_CORE=str(SKILLS.parent / "core"))

    def one(i: int) -> int:
        return subprocess.run(
            argv + ["--owner", f"t{i}", "--status", "proposed",
                    "--mechanism", f"cause {i}", "--evidence", f"trace {i}",
                    "--touches", f"tool_{i}"],
            capture_output=True, text=True, env=env).returncode

    with cf.ThreadPoolExecutor(max_workers=9) as ex:
        rcs = list(ex.map(one, range(9)))
    c.check(all(r == 0 for r in rcs), f"concurrent ledger appends failed: {rcs}")

    r = _run(c, "mechanisms.py list", [str(HERE / "mechanisms.py"), "list",
                                       "--run-dir", str(run)])
    if r:
        c.check(r.get("count") == 9,
                f"ledger lost rows under 9 concurrent writers: count={r.get('count')}",
                note="the mechanism ledger is append-atomic across parallel optimisers")
        c.check(sorted(r.get("already_owned_do_not_reimplement") or [])
                == [f"tool_{i}" for i in range(9)],
                f"ledger did not report owned surfaces: {r.get('already_owned_do_not_reimplement')}",
                note="listing names what is already owned, so a duplicate fix is refused "
                     "by policy instead of discovered at merge time")

    _run(c, "mechanisms.py add (rejected)",
         [str(HERE / "mechanisms.py"), "add", "--run-dir", str(run), "--owner", "t0",
          "--status", "rejected", "--mechanism", "m", "--evidence", "e"])
    r2 = _run(c, "mechanisms.py list (after reject)",
              [str(HERE / "mechanisms.py"), "list", "--run-dir", str(run)])
    c.check(bool(r2) and len(r2.get("rejected") or []) == 1,
            "a rejected mechanism must be listed so a retry can be made structurally different",
            note="rejected attempts are remembered, not silently retried")

    # Relevance filtering: a real fan-out ledger reached 99 findings, and pasting all of them
    # into each of K subagents spends their context on other optimisers' tasks. Filtering must
    # narrow to the task WITHOUT hiding the task-independent rows, which are the cross-cutting
    # measurement facts (canary bands, variance warnings) that apply to everyone.
    _run(c, "mechanisms.py add (task-scoped row)",
         [str(HERE / "mechanisms.py"), "add", "--run-dir", str(run), "--owner", "tA",
          "--status", "verified", "--mechanism", "task-nine-only finding",
          "--evidence", "e", "--task", "9", "--touches", "f_nine"])
    _run(c, "mechanisms.py add (other-task row)",
         [str(HERE / "mechanisms.py"), "add", "--run-dir", str(run), "--owner", "tB",
          "--status", "verified", "--mechanism", "task-fortytwo-only finding",
          "--evidence", "e", "--task", "42", "--touches", "f_ft"])
    r3 = _run(c, "mechanisms.py list --task",
              [str(HERE / "mechanisms.py"), "list", "--run-dir", str(run), "--task", "9"])
    blob = json.dumps(r3 or {})
    c.check(bool(r3) and "task-nine-only" in blob and "task-fortytwo-only" not in blob,
            "--task must keep this task's rows and drop other tasks' rows",
            note="K subagents should not each read K-1 other task histories")
    c.check(bool(r3) and "rejected" in blob and any(
        not (row.get("tasks") or [])
        for grp in ("verified", "proposed", "rejected") for row in (r3.get(grp) or [])),
            "--task must still show task-INDEPENDENT rows",
            note="cross-cutting measurement facts apply to every optimiser; hiding them is "
                 "how a fan-out repeats a defect someone already paid for")
    # A finding that turns out to be WRONG must stop appearing as verified. Three separate
    # `verified` rows were disproved on the real run, and without this a reader saw both the
    # claim and its refutation with no way to tell which won.
    r_old = _run(c, "mechanisms.py add (a claim that will later be disproved)",
                 [str(HERE / "mechanisms.py"), "add", "--run-dir", str(run), "--owner", "tX",
                  "--status", "verified", "--mechanism", "CLAIM_TO_BE_RETIRED",
                  "--evidence", "single reading", "--touches", "f_x"])
    seq = str((r_old or {}).get("added", {}).get("seq"))
    _run(c, "mechanisms.py add (superseding row)",
         [str(HERE / "mechanisms.py"), "add", "--run-dir", str(run), "--owner", "tX",
          "--status", "rejected", "--supersedes", seq,
          "--mechanism", "RETIREMENT", "--evidence", "remeasured", "--touches", "f_x"])
    r_after = _run(c, "mechanisms.py list (superseded row retired)",
                   [str(HERE / "mechanisms.py"), "list", "--run-dir", str(run)])
    blob2 = json.dumps((r_after or {}).get("verified") or [])
    sup = (r_after or {}).get("superseded_do_not_act_on") or []
    c.check(bool(r_after) and "CLAIM_TO_BE_RETIRED" not in blob2
            and any("CLAIM_TO_BE_RETIRED" in (x.get("mechanism") or "") for x in sup),
            "a superseded finding must drop out of verified and be listed as retired",
            note="a disproved claim left in `verified` is worse than no ledger")



def _funcmerge(c: Checker, tmp: Path) -> None:
    """Whole-file merge conflicts on additions that do not disagree; per-function must not.

    This is the exact shape that cost a real round 6 of 10 verified branches: every optimiser
    appends one independent line to the SAME function, so their edits land on adjacent lines
    and diff3 conflicts even though nobody disagrees. Asserted here: independent insertions
    into one shared function all survive, the result PARSES, no `def` is duplicated, and a
    genuine rewrite of the same line still conflicts rather than being silently unioned.
    """
    d = tmp / "fm"
    d.mkdir(parents=True, exist_ok=True)
    base = d / "base.py"
    base.write_text(
        "class T:\n"
        "    def __init__(self):\n"
        "        self.a = 1\n"
        "\n"
        "    def go(self, x):\n"
        "        self.check(x)\n"
        "        return x\n"
    )
    # three branches, each adding one state field AND one guard call to the same two functions
    for tag, field, guard in [("bA", "self.b = 2", "self.gb(x)"),
                              ("bB", "self.c = 3", "self.gc(x)"),
                              ("bC", "self.d = 4", "self.gd(x)")]:
        (d / f"{tag}.py").write_text(
            base.read_text()
            .replace("        self.a = 1\n", f"        self.a = 1\n        {field}\n")
            .replace("        self.check(x)\n", f"        self.check(x)\n        {guard}\n")
        )
    out = d / "out.py"
    r = _run(c, "funcmerge.py (independent insertions in one shared function)",
             [str(HERE / "funcmerge.py"), "--base", str(base), "--out", str(out),
              "--union-pure-insertions", "--inputs",
              str(d / "bA.py"), str(d / "bB.py"), str(d / "bC.py")])
    text = out.read_text() if out.exists() else ""
    c.check(bool(r) and r.get("written") and all(
        f in text for f in ("self.b = 2", "self.c = 3", "self.d = 4",
                            "self.gb(x)", "self.gc(x)", "self.gd(x)")),
            "every branch's independent insertion into a shared function must survive",
            note="whole-file 3-way merge conflicts here and keeps one; that is the bug this fixes")
    ok_parse = False
    dups: list[str] = []
    if text:
        import ast as _ast
        try:
            t = _ast.parse(text)
            names = [n.name for n in _ast.walk(t)
                     if isinstance(n, (_ast.FunctionDef, _ast.AsyncFunctionDef))]
            dups = sorted({n for n in names if names.count(n) > 1})
            ok_parse = True
        except SyntaxError:
            ok_parse = False
    c.check(ok_parse and not dups,
            "the merged file must parse and define no function twice",
            note="the union-on-conflict path this replaces produced an unparseable file "
                 "with five duplicated defs")

    # a genuine rewrite of the same base line must NOT be unioned away
    (d / "rA.py").write_text(base.read_text().replace("        return x\n",
                                                      "        return x + 1\n"))
    (d / "rB.py").write_text(base.read_text().replace("        return x\n",
                                                      "        return x * 2\n"))
    out2 = d / "out2.py"
    r2 = _run(c, "funcmerge.py (rival rewrites of one line)",
              [str(HERE / "funcmerge.py"), "--base", str(base), "--out", str(out2),
               "--union-pure-insertions", "--inputs", str(d / "rA.py"), str(d / "rB.py")],
              expect_rc=1)
    c.check(bool(r2) and not r2.get("written") and r2.get("conflicts"),
            "two branches rewriting the same line must conflict, not be auto-unioned",
            note="that is a disagreement about the right answer and belongs to a human")

    # A forced-trunk resolution can re-apply a subtraction the losing branch already measured
    # and reverted. That is invisible in the conflict report, so the merge must SAY what it
    # failed to carry.
    (d / "fA.py").write_text(base.read_text().replace(
        "    def go(self, x):\n", "    def go(self, x):\n        # KEEPME_A rationale line\n"))
    (d / "fB.py").write_text(base.read_text().replace(
        "        return x\n", "        return x  # rewritten by B\n"))
    out3 = d / "out3.py"
    r4 = _run(c, "funcmerge.py (forced trunk reports dropped additions)",
              [str(HERE / "funcmerge.py"), "--base", str(base), "--out", str(out3),
               "--union-pure-insertions", "--force-priority", "--priority", "fB", "fA",
               "--inputs", str(d / "fA.py"), str(d / "fB.py")])
    got = json.dumps((r4 or {}).get("dropped_additions") or {})
    c.check(bool(r4) and (("KEEPME_A" in got) or ("KEEPME_A" in out3.read_text())),
            "a branch addition the merge did not carry must be REPORTED, not silently lost",
            note="a dropped rewrite can re-apply a subtraction its owner measured as harmful")

    # A merged function may reference a class CONSTANT the merge did not carry. That is a
    # runtime crash, not a style problem: the tool layer turns AttributeError into an error
    # string, the agent abandons the write, and the reward records a MISSING WRITE. It
    # contaminated four measurements on a real run before a live tool return exposed it, so it
    # must be a hard failure. The second half of this check matters just as much: the fields it
    # inspects are routinely declared WITH annotations (`self.x: set[str] = set()`), and a
    # collector that only walks ast.Assign reports those as undefined and rejects a good merge.
    (d / "cA.py").write_text(
        "class T:\n    db: int\n    LADDER = (1, 2, 3)\n\n"
        "    def go(self, x):\n        return x\n\n"
        "    def need(self):\n        return self.LADDER[0]\n")
    (d / "cB.py").write_text(
        "class T:\n    db: int\n\n    def go(self, x):\n        return x + 1\n")
    r5 = _run(c, "funcmerge.py (class constant is CARRIED, not left behind)",
              [str(HERE / "funcmerge.py"), "--base", str(base), "--out", str(d / "out4.py"),
               "--union-pure-insertions", "--force-priority", "--priority", "cB", "cA",
               "--inputs", str(d / "cA.py"), str(d / "cB.py")])
    got5 = (d / "out4.py").read_text() if (d / "out4.py").exists() else ""
    import re as _re
    c.check(bool(r5) and r5.get("written")
            and _re.search(r"^    LADDER = \(1, 2, 3\)", got5, _re.M) is not None,
            "a class CONSTANT a merged function needs must be carried into the class body",
            note="merging only functions left it behind; the reference then crashed at runtime "
                 "and presented as a missing write")

    # Backstop: a constant that exists in NO input cannot be carried, and must still refuse.
    (d / "gA.py").write_text(
        "class T:\n    db: int\n\n    def go(self, x):\n        return x\n\n"
        "    def need(self):\n        return self.NOWHERE[0]\n")
    r5b = _run(c, "funcmerge.py (truly undefined constant still refuses)",
               [str(HERE / "funcmerge.py"), "--base", str(base), "--out", str(d / "out4b.py"),
                "--union-pure-insertions", "--inputs", str(d / "gA.py")],
               expect_rc=1)
    c.check(bool(r5b) and not r5b.get("written") and "NOWHERE" in str(r5b.get("error")),
            "an attribute defined in no input must refuse to write, not ship a crash",
            note="the crash is invisible in the reward, so it cannot be left to the gate")

    # A branch's inserted lines may legitimately REPEAT. De-duplicating by line text deleted the
    # second copy and truncated a multi-line statement into a syntax error whose only symptom was
    # "'{' was never closed" — a corrupted artifact, not a refusal.
    (d / "rA.py").write_text(
        "class T:\n    def __init__(self):\n        self.a = 1\n\n"
        "    def go(self, x):\n        return x\n")
    (d / "rB.py").write_text(
        "class T:\n    def __init__(self):\n        self.a = 1\n"
        "        self.p = {\n            k: 1\n            for k in self.src\n        }\n"
        "        self.q = {\n            k\n            for k in self.src\n        }\n\n"
        "    def go(self, x):\n        return x\n")
    out6 = d / "out6.py"
    r7 = _run(c, "funcmerge.py (repeated lines inside one insertion survive)",
              [str(HERE / "funcmerge.py"), "--base", str(d / "rA.py"), "--out", str(out6),
               "--union-pure-insertions", "--inputs", str(d / "rB.py")])
    got7 = out6.read_text() if out6.exists() else ""
    c.check(bool(r7) and r7.get("written") and got7.count("for k in self.src") == 2,
            "a line that legitimately repeats within one insertion must not be de-duplicated",
            note="dedupe is only for the SAME hunk from two branches at the same anchor")

    (d / "nA.py").write_text(
        "class T:\n    db: int\n\n    def __init__(self):\n"
        "        self.seen: set[str] = set()\n\n"
        "    def go(self, x):\n        return x in self.seen\n")
    r6 = _run(c, "funcmerge.py (annotated instance attribute is not 'undefined')",
              [str(HERE / "funcmerge.py"), "--base", str(d / "nA.py"),
               "--out", str(d / "out5.py"), "--union-pure-insertions",
               "--inputs", str(d / "nA.py")])
    c.check(bool(r6) and r6.get("written"),
            "an attribute declared with an annotation must not be reported undefined",
            note="ast.AnnAssign, not ast.Assign — a hard check with false positives is worse "
                 "than no check")



def _gate_concurrency(c: Checker, tmp: Path) -> None:
    """The gate must default to LOW concurrency, and must SAY SO when it is run high.

    Measured o

"""commit — persist one agent-optimize round's decision through the run dir.

Replaces the old SKILL.md shell heredoc, which was unrunnable: it used a *quoted*
heredoc (``<<'PY'``) so ``$R`` never expanded and ``RunDir.open("$R")`` opened a
literal ``$R``, and it carried bare ``<placeholder>`` tokens that aren't Python.

Does exactly what the deterministic loops do at the end of a step:
  * ``snapshot`` the working copy as a candidate (always — the audit trail should
    show rejects too),
  * ``set_best`` on accept only,
  * ``log_event`` the decision (agent-mode detail the other scripts read), and
  * ``harness.record_iteration`` — the ONE shared iteration step every algorithm
    routes through (#216/#224): charges ``iterations=1`` **and** ``accepted=`` so the
    stall counter that ``budget_exhausted()`` reads actually moves, writes the
    canonical ``step`` record every consumer enumerates, and reconciles the
    run-level ``JOURNAL.md``. Do NOT open-code those three here again.

Beyond ``accept``/``reject`` there are two more outcomes, and they are NOT the same thing:

  * ``inconclusive`` — the measurement could not resolve the edit (the verdict flips between
    byte-identical control replicates). A booked round: it charges the iteration but not the
    stall, files no ``rejected.jsonl`` record, and must be re-measured under a FRESH tag.
  * ``provisional`` — the candidate is directionally positive (Δ>0) but under the significance
    bar, and the driver wants to buy more trials on this SAME, UNMODIFIED candidate
    (``scripts/grow.py``) before calling it. NOT a booked round: it snapshots and logs the
    event, but does NOT ``set_best`` and does NOT call ``record_iteration``, so the stall
    counter, LEDGER.md and JOURNAL.md do not advance. The same candidate id later gets a real
    ``accept``/``reject``/``inconclusive`` commit once ``grow.py`` has re-gated it at the
    pooled n.

Runner-side spend (metric_calls / usd / tokens / seconds) is already recorded by the
evaluate phase; ``--optimizer-*`` is for the *proposer's* own cost, which in agent
mode is you and would otherwise never be counted.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

# Imported for its side effect ONLY: seeds sys.path so `cap_evolve` resolves when
# this script is run standalone (`python <this-file>`). Must precede the
# cap_evolve imports below; not "unused" — deleting it breaks standalone runs.
import _bootstrap  # noqa: F401  # side-effect import, see above

from cap_evolve import RunDir, harness


def _memory_skill_from_spec(run_dir: RunDir) -> str | None:
    """``memory_skill`` from the sibling project spec, or ``None``.

    Agent mode invokes this script standalone with only ``--run-dir`` — no ``--project``,
    no spec object — so the choice has to be read off disk the same zero-dependency way
    ``dashboard.py``'s ``_algorithm_from_spec`` reads ``algorithm_skill``: flat ``key: value``
    lines only, from ``<base>/project/capevolve.yaml`` next to the run dir. Best-effort — a
    missing/unreadable spec just means the default (``md-files``) applies, same as today.
    """
    spec_path = run_dir.root.parent / "project" / "capevolve.yaml"
    if not spec_path.is_file():
        return None
    try:
        for line in spec_path.read_text(encoding="utf-8").splitlines():
            if line.startswith("memory_skill:"):
                val = line.split(":", 1)[1].split("#", 1)[0].strip().strip("'\"")
                return val or None
    except OSError:
        pass
    return None


def _round_gate_numbers(run_dir: RunDir, candidate_id: str) -> dict:
    """The gate's NUMBERS for ``candidate_id``, read back from ``round.py``'s own table.

    ``dashboard.reduce_run`` builds the published ``gate_decisions[]`` (read by the dashboard,
    the TUI and CI's live snapshot) by regex-parsing the deterministic gate's reason string
    (``Δ̄ = …, SE=…, n=…, k·SE=…``). An agent writes free prose, so nothing matched and every
    numeric column came back null — on run 32971129203 the deltas and thresholds existed only
    inside sentences like "delta +0.033 within control noise 0.044". This function is why that
    is now avoidable: ``round.py`` persists the whole gate table to ``$R/work/round_i<N>.json``,
    so the numbers are on disk, structured, already.

    ``N`` is ``spent.iterations`` at gate time, and ``record_iteration`` has not charged this
    round yet — so the current count still names this round's table. A same-iteration re-gate
    gets a ``.r<k>`` suffix rather than overwriting; the highest suffix is the operative one,
    because a re-gate is run to supersede the first.

    Returns {} when there is no table (``gate_check``-only rounds never write one, and
    ``round.py``'s write is best-effort) — a missing measurement must stay missing, never
    become a fabricated 0.
    """
    work = run_dir.root / "work"
    stem = f"round_i{int(run_dir.spent.iterations)}"
    # Numeric, not lexical: a plain string sort ranks `.r10` below `.r2`, so the tenth re-gate
    # would lose to the second. Re-gates are rare but a wrong one here is silently wrong.
    def _suffix(p: Path) -> int:
        tail = p.name[len(stem):].removesuffix(".json")
        return int(tail[2:]) if tail.startswith(".r") and tail[2:].isdigit() else 0
    tables = sorted(work.glob(f"{stem}.json")) + sorted(work.glob(f"{stem}.r*.json"),
                                                        key=_suffix)
    # A grown candidate's POOLED row (``grow.py``) supersedes the round's pre-growth row by
    # construction: growth exists precisely to re-measure the same candidate at a larger n, so
    # the round table's numbers are the ones being superseded. Sorted by growth round, last wins.
    tables += sorted(work.glob(f"grow_{candidate_id}_r*.json"),
                     key=lambda p: int(p.stem.rsplit("_r", 1)[-1])
                     if p.stem.rsplit("_r", 1)[-1].isdigit() else 0)
    if not tables:
        return {}
    try:
        table = json.loads(tables[-1].read_text(encoding="utf-8"))
    except (OSError, ValueError):
        return {}
    entry = next((c for c in (table.get("candidates") or [])
                  if c.get("tag") == candidate_id), None)
    if entry is None:
        return {}
    parent = table.get("parent") or {}
    # The SE of the PAIRED per-task deltas — the number every ``gate_threshold`` in these
    # tables is ``k_se ×``, and therefore the only SE that belongs in the same row.
    # DERIVED, never read off a stderr column: neither table records the paired SE, and both
    # carry a ``stderr`` that is a mean-over-tasks SE instead. Publishing one of those is the
    # defect this is replacing — on run_finalrun6 it put an identical ``gate_stderr``
    # 0.0738 on rounds i0, i3 and i6 while the threshold it is supposed to generate moved
    # 0.0222 → 0.0244 → 0.0346, i.e. a constant beside three different bars derived from it.
    # ``resolvable_effect_size`` is ``2·SE``, written by ``gate.decide`` from the vector it
    # actually gated on, so halving it recovers that round's real SE exactly. Absent when the
    # gate reported no SE (threshold/strict modes, or too few paired samples), and a missing
    # measurement must stay missing rather than become a fabricated number.
    _res = entry.get("resolvable_effect_size")
    out = {
        "parent_val": parent.get("reward"),
        "gate_stderr": (round(float(_res) / 2.0, 6)
                        if isinstance(_res, (int, float)) else None),
        # The parent block is the round table's; a grow table has no parent, so fall back to the
        # candidate entry's own paired n — which IS what the gate used.
        "gate_n": parent.get("n_tasks", entry.get("n")),
        "gate_delta": entry.get("gate_delta"),
        "gate_threshold": entry.get("gate_threshold"),
        "gate_mode": (table.get("gated_against") or {}).get("mode"),
        "gate_table": tables[-1].name,
        # The significance multiplier the bar was built from, and the smallest true effect this
        # round could have resolved (2·SE). Without the latter a null result is unreadable: four
        # consecutive runs of nulls were read as "the edits were bad" when the measurement simply
        # could not resolve anything that small.
        "gate_k_se": entry.get("k_se"),
        "gate_resolvable_effect_size": entry.get("resolvable_effect_size"),
    }
    # The drift-free second opinion, when the round measured one. On run 32971129203 this is
    # the whole finding: cand_1 was rejected against the parent's STORED reward (Δ 0.0333 vs
    # threshold 0.0440) while the control-relative comparison ACCEPTED it (Δ 0.0556 vs 0.0341).
    # Recording only the booked verdict hides that the decision was reference-dependent.
    ctl = entry.get("control_relative") or {}
    if ctl:
        out["control_relative_verdict"] = ctl.get("verdict")
        out["control_relative_delta"] = ctl.get("gate_delta")
    bar = (table.get("evidence_bar") or {}).get("value")
    if bar is not None:
        out["evidence_bar"] = bar
    return {k: v for k, v in out.items() if v is not None}


def _record_memory(run_dir: RunDir, candidate_id: str, *, accepted: bool, reason: str,
                   val: float | None, parent_val: float | None) -> None:
    """File this round in the optimizer memory every OTHER algorithm already writes.

    ``rejected.jsonl`` / ``history.jsonl`` are what ``memory.py`` calls the readers of its two
    audit records: the dashboard's Memory panel (``GET /api/runs/{id}/memory`` and the static
    export) and any optimizer prompt that greps ``rejected.jsonl`` for approaches already
    refuted. The deterministic loops write them inline — ``harness``'s hill-climb, ``gepa``
    (including its merge paths) and ``skillopt`` all call ``.add`` themselves. Agent mode never
    did, so run 33046360451 published ``{"history": [], "rejected": []}``: a whole run of
    rejected approaches that the run itself could not enumerate.

    NOT hoisted into ``harness.record_iteration`` alongside the other three per-iteration
    records, though that is where it belongs by the argument in that docstring: gepa's
    local-gate merge reject (``_try_merge``) files a rejection WITHOUT routing through
    record_iteration, so centralising there would silently drop it, and the deterministic
    callers pass richer, algorithm-specific summaries than record_iteration's arguments can
    reconstruct. Both are fixable; neither is fixable safely in the same change as this.
    """
    from cap_evolve.memory import History, RejectedMemory

    delta = (val - parent_val if isinstance(val, (int, float))
             and isinstance(parent_val, (int, float)) else None)
    bits = [f"candidate {candidate_id}"]
    if isinstance(val, (int, float)):
        bits.append(f"(val {val:.3f}" + (f", Δ {delta:+.3f})" if delta is not None else ")"))
    summary = " ".join(bits)
    try:
        if accepted:
            History(run_dir.history_path).add(candidate_id, summary, float(val or 0.0))
        else:
            RejectedMemory(run_dir.rejected_path).add(candidate_id, summary, reason, val)
    except OSError as e:
        # Memory is an audit record, not the decision. Never lose a booked round over it.
        run_dir.log_event("optimizer_context_warning", what="memory", error=str(e)[:300])


def _prior_decision(run_dir: RunDir, candidate_id: str) -> dict | None:
    """The first decision event already recorded for ``candidate_id``, if any.

    ``inconclusive`` counts: an unresolved round is a booked round, and resolving it needs a
    FRESH tag anyway (re-running a tag REPLACES its rollouts — see ``harness``'s
    ``rollout_overwrite_warning``), so silently re-booking the old one is the same collision
    this guard exists for.

    Reads ``events.jsonl`` (the audit log, not memory) so the guard holds across
    processes — which is the only way it can catch two concurrent drivers, the exact
    failure it exists for.
    """
    try:
        with run_dir.events_path.open(encoding="utf-8") as f:
            for line in f:
                try:
                    ev = json.loads(line)
                except Exception:  # noqa: BLE001 — a torn line is not a decision
                    continue
                # NB: log_event writes the event name under "kind", not "event".
                if ev.get("kind") in ("accept", "reject", "inconclusive") and \
                        str(ev.get("candidate")) == str(candidate_id):
                    return {"kind": ev.get("kind"), "t": ev.get("t"),
                            "note": ev.get("note"), "val": ev.get("val")}
    except FileNotFoundError:
        return None
    return None


def _gate_row(run_dir: RunDir, candidate_id: str) -> dict | None:
    """This candidate's row from ``round.py``'s persisted table, if one exists.

    Readable only because ``round.py`` now writes its table to ``work/`` instead of leaving
    stdout the sole copy — before that, ``commit.py`` had no way to know what the gate had said
    and could not tell an agreeing reject from an override.

    Newest table wins: a same-iteration re-gate is written alongside the first (``.r1.json``),
    and the later measurement is the one being booked against.
    """
    work = run_dir.root / "work"
    if not work.is_dir():
        return None
    for log in sorted(work.iterdir(), key=lambda p: p.stat().st_mtime, reverse=True):
        if not log.is_file() or log.suffix != ".json":
            continue
        try:
            payload = json.loads(log.read_text(encoding="utf-8"))
        except Exception:  # noqa: BLE001 — not a round table
            continue
        if not isinstance(payload, dict):
            continue
        for row in payload.get("candidates") or []:
            if isinstance(row, dict) and str(row.get("tag")) == str(candidate_id):
                return row
    return None


def _gate_verdict(run_dir: RunDir, candidate_id: str) -> str | None:
    row = _gate_row(run_dir, candidate_id)
    return row.get("verdict") if row else None


def _has_grown(run_dir: RunDir, candidate_id: str) -> bool:
    """Has ``scripts/grow.py`` already bought this candidate at least one extra round of
    trials? True iff a ``work/grow_<candidate_id>_r*.json`` table exists.

    On run 33492876620 round 3 a candidate landed exactly on ``grow.py``'s reason for
    existing — Δ>0, below the significance bar, verdict flipping between control
    replicates — and was booked ``inconclusive`` and left there. The transcript shows the
    agent had read ``grow.py --help``, so this was not a discovery gap: the tool was
    optional, so it went unused. See ``main``'s guard below.
    """
    return any((run_dir.root / "work").glob(f"grow_{candidate_id}_r*.json"))


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="commit")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--candidate-id", required=True,
                   help="candidate id == the tag its rollouts were written under")
    p.add_argument("--from-dir", required=True, help="the working copy to snapshot")
    # ``inconclusive`` is the third outcome ``round.py`` can actually return (``verdict_stable:
    # false`` — the verdict flips depending on which byte-identical control replicate is the
    # reference, so the round cannot separate the edit from re-measurement). Without it an
    # unresolvable round had to be booked as one of two things it was not, and booking it as a
    # reject moves the STALL counter — the signal that means "the optimizer has run out of
    # ideas", which is the one thing an ambiguous measurement is no evidence of.
    #
    # ``provisional`` is a FOURTH outcome, and unlike ``inconclusive`` it does not book the
    # round at all: the candidate is directionally positive (Δ>0) but under the significance
    # bar, and the driver wants to buy more trials on this SAME, UNMODIFIED candidate
    # (``scripts/grow.py``) before making a call. The iteration is not over, so the stall
    # counter, LEDGER.md and JOURNAL.md must not advance for it.
    p.add_argument("--decision", required=True,
                   choices=["accept", "reject", "inconclusive", "provisional"],
                   help="accept=new champion; reject=the edit was judged and refuted; "
                        "inconclusive=the measurement could not resolve it (charges the "
                        "iteration, not the stall; re-measure under a FRESH tag); "
                        "provisional=Δ>0 but unresolved, buying more trials on the SAME "
                        "candidate next (books nothing; re-commit it once grown)")
    p.add_argument("--val", type=float, default=None, help="candidate's full-val mean")
    p.add_argument("--note", default="", help="one line: why this edit, in general terms")
    # The DRIVER's disposition, recorded machine-readably alongside the screen's own
    # verdict. screen.py may only say kill/promote (invariant 1), so a candidate the
    # screen PROMOTED can still never reach full val — the driver may drop it on an
    # arithmetic ceiling or a budget call. Without this field the two artifacts read as a
    # contradiction ("promote" + a prose commit note saying "not promoted to full val");
    # with it, "promote" + basis=ceiling is one coherent story. `gate` is the only basis
    # that asserts a full-val paired gate actually ran.
    p.add_argument("--reject-basis", default=None,
                   choices=["gate", "screen_kill", "ceiling", "budget", "infra",
                            "micro_test_fail", "driver_judgement"],
                   help="what evidence the reject rests on: gate=full-val paired gate ran AND "
                        "rejected; screen_kill=screen proved harm; ceiling=arithmetic proof no "
                        "accept was reachable, so full val was never paid; budget=screen "
                        "evidence plus a budget call; infra=missing data, not a judgement; "
                        "micro_test_fail=microcase.py proved the candidate's own targeted "
                        "mechanism does not fire, before any rollout was spent (#436); "
                        "driver_judgement=the gate ACCEPTED and you are overriding it (say why "
                        "in --note)")
    p.add_argument("--optimizer-usd", type=float, default=0.0)
    p.add_argument("--optimizer-tokens", type=int, default=0)
    p.add_argument("--optimizer-seconds", type=float, default=0.0)
    p.add_argument("--force", action="store_true",
                   help="commit even though this candidate id already has a decision "
                        "(audit/repair only — it overwrites the earlier snapshot); also "
                        "overrides the inconclusive-without-growth guard below")
    # graph.jsonl (#435): a MERGE candidate (integrate.py/funcmerge.py/merge_taskopt.py)
    # has 2+ parents this script has no other way to learn — those scripts produce a
    # merged artifact dir, not a commit. An ordinary edit's single parent is already
    # known (best_id at gate time, below) and needs no flag.
    p.add_argument("--parents", default=None,
                   help="comma-separated parent candidate ids, for a MERGE candidate "
                        "(2+ parents). Omit for an ordinary edit.")
    p.add_argument("--edit-kind", default=None, choices=["prompt", "code", "merge"],
                   help="graph.jsonl node kind; defaults to 'merge' when --parents has "
                        "2+ ids, else 'code'.")
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))
    src = Path(args.from_dir)
    if not src.is_dir():
        print(json.dumps({"error": f"--from-dir does not exist: {src}"}, indent=2))
        return 2

    if not args.force:
        prior = _prior_decision(run_dir, args.candidate_id)
        if prior:
            print(json.dumps({
                "error": f"candidate id {args.candidate_id!r} already has a "
                         f"{prior.get('kind')!r} decision in this run — refusing.",
                "why": "Rollouts are <task>__<tag>__t<k>.json, so two candidates sharing "
                       "a tag write into the same files: one edit gets judged on the "
                       "other's evidence and the second snapshot overwrites the first. "
                       "This happened for real (see docs/RESULTS.md, cand_r2).",
                "prior_event": prior,
                "fix": "pick a tag no sibling has used (e.g. suffix the round AND the "
                       "cluster: cand_r3_bags), or pass --force if you are deliberately "
                       "repairing this candidate's record.",
            }, indent=2))
            return 2

    accepted = args.decision == "accept"
    indecisive = args.decision == "inconclusive"
    provisional = args.decision == "provisional"

    # An inconclusive round is UNRESOLVED, not refuted — ``grow.py`` exists precisely to
    # resolve it by buying more trials on this same candidate, and issue #420 item 3 found
    # it had never run once across two real runs that hit this exact case. Making it
    # optional is why: the fix is a guard here, not a third restatement in SKILL.md prose
    # (the same argument round.py's own concurrency guard already makes). ``--force`` is
    # the deliberate override, for a candidate growth genuinely cannot help (e.g. Δ<=0).
    if indecisive and not args.force and not _has_grown(run_dir, args.candidate_id):
        print(json.dumps({
            "error": f"--decision inconclusive for {args.candidate_id!r}, but scripts/grow.py "
                     "has not bought it any extra trials yet",
            "why": "an inconclusive verdict means the measurement could not resolve the edit, "
                   "not that the edit was refuted. grow.py exists to buy more trials on this "
                   "SAME candidate and re-gate at the pooled n before it is left unresolved.",
            "fix": f"run scripts/grow.py --candidate {args.candidate_id} --growth-round 1 "
                   "--add-trials <n> first (it recommends promote/grow_again/abandon), then "
                   "commit its recommendation; or pass --force here if growth genuinely "
                   "cannot help this candidate (e.g. its delta is <= 0) and say why in --note.",
        }, indent=2))
        return 2

    if args.decision != "reject" and args.reject_basis:
        print(json.dumps({
            "error": f"--reject-basis is meaningless on an {args.decision}",
            "why": "it records what evidence a REJECT rests on. An unresolved round rests on "
                   "no evidence about the edit at all — that is what makes it unresolved.",
            "fix": "drop it, or pass --decision reject"}, indent=2))
        return 2
    # `--reject-basis gate` asserts the gate rejected this candidate. On run 32871360361 it was
    # booked for cand2, which round_i1.json recorded as `verdict: accept` at +0.19 against a
    # concurrent control — so events.jsonl, the run's audit record, said the gate had rejected the
    # best candidate of the run when in fact the driver had overridden it. Overriding is
    # legitimate (round.py leaves the decision to the driver on purpose); misattributing it is
    # not, and it is the one thing this log exists to get right.
    # An ``inconclusive`` gate verdict is the same misattribution one step further: the gate did
    # not refute the edit, it failed to resolve it. Observed live on run 33046360451 i1, where
    # cand_2's verdict was `{ctl_null_i1: reject, ctl_null_i1r1: accept}` — booking that as
    # "the gate rejected it" makes events.jsonl assert a judgement no measurement supports.
    gate_verdict = _gate_verdict(run_dir, args.candidate_id)
    overrode_gate = bool(args.decision == "reject" and gate_verdict == "accept")
    if args.reject_basis == "gate" and gate_verdict in ("accept", "inconclusive"):
        verb = ("ACCEPTED" if gate_verdict == "accept"
                else "could not resolve (verdict: inconclusive)")
        fix = ("pass --reject-basis driver_judgement and say in --note why you are overriding "
               "the gate — e.g. a task you care about regressed"
               if gate_verdict == "accept" else
               "book it as --decision inconclusive (charges the iteration, not the stall) and "
               "re-measure under a FRESH tag; or, if you are choosing to drop the edit anyway, "
               "pass --reject-basis driver_judgement and say so in --note")
        print(json.dumps({
            "error": f"--reject-basis gate, but the gate {verb} {args.candidate_id} "
                     "(see its row in work/round_*.json)",
            "gate_verdict": gate_verdict,
            "fix": fix,
        }, indent=2))
        return 2

    # The parent this candidate was gated against — ``gate_check --current`` defaults to
    # ``best_id``, so read it BEFORE ``set_best`` moves it.
    parent_id = run_dir.best_id or "seed"
    parents = ([p.strip() for p in args.parents.split(",") if p.strip()]
               if args.parents else [parent_id])
    run_dir.snapshot(args.candidate_id, src)
    if accepted:
        run_dir.set_best(args.candidate_id)
    # Carry the proposer's own spend on the EVENT as well as into state.json. update_spent
    # alone leaves the dashboard's cost ledger unable to attribute it: state.json has the
    # total, but no cost-bearing event exists to explain it, so an agent-mode run reported
    # 100% of its optimizer spend as unattributed. opt_cost_usd/opt_tokens are the field
    # names the ledger already reads from headless optimizer backends.
    # The gate's NUMBERS, IN ADDITION to the prose --note, so the dashboard can render them
    # without regex-parsing a hand-typed string — read from round.py's persisted table, and
    # simply absent when it wrote none. On the EVENT as well as the step record below, because a
    # `provisional` decision never reaches record_iteration and would otherwise carry none.
    gate = _round_gate_numbers(run_dir, args.candidate_id)
    parent_val = gate.pop("parent_val", None)
    run_dir.log_event(args.decision, candidate=args.candidate_id, val=args.val,
                      gate_verdict=gate_verdict, overrode_gate=overrode_gate,
                      note=args.note,
                      reject_basis=args.reject_basis,
                      verdict=args.decision,
                      opt_cost_usd=args.optimizer_usd or None,
                      opt_tokens=args.optimizer_tokens or None,
                      opt_seconds=args.optimizer_seconds or None,
                      **gate)
    run_dir.update_spent(optimizer_usd=args.optimizer_usd,
                         optimizer_tokens=args.optimizer_tokens,
                         optimizer_seconds=args.optimizer_seconds)
    # Did the agent write the INTENT half of its handover? ``_reconcile_journal`` (inside
    # record_iteration) folds ``<workdir>/JOURNAL.md`` into the run-level journal and silently
    # substitutes "(no handover written by the optimizer)" when there is none — which is what
    # every round of runs 32971129203 and 33046360451 recorded, because nothing asked the agent
    # for one. Read it BEFORE booking, and report the answer so a forgotten handover is
    # correctable while rounds remain rather than discovered when the run is over.
    # ``pending_handover``, not ``_journal_tail``: a working copy cloned from the last round
    # still holds THAT round's entry, and _reconcile_journal's dedup guard books the placeholder
    # rather than the same entry twice — so the plain tail reports "recorded" for exactly the
    # round whose handover went missing.
    handover = bool(harness.pending_handover(src, run_dir, args.candidate_id))
    reason = args.note or args.decision
    if indecisive:
        reason = f"indecisive (gate): {reason}"
    warnings: list[str] = []
    # `provisional` books the decision event above but stops here: the iteration is not over
    # (the SAME candidate gets a real accept/reject/inconclusive commit later, once `grow.py`
    # has re-gated it at a pooled n), so the stall counter, LEDGER.md and JOURNAL.md must not
    # advance for it — that would spend an iteration's worth of "the run learned something new"
    # bookkeeping on a decision that has not actually been made yet. It files no memory record
    # either, for the same reason `inconclusive` does not: nothing has been refuted.
    if not provisional:
        memory_skill = _memory_skill_from_spec(run_dir)
        # The shared iteration step: charges iterations/stall, writes the canonical ``step``
        # record, reconciles the run-level JOURNAL.md. The gate's numbers ride along so the
        # dashboard's ``gate_decisions[]`` does not have to regex them out of an agent's prose.
        harness.record_iteration(run_dir, src, args.candidate_id, parent_id=parent_id,
                                 accepted=accepted, reason=reason,
                                 val=args.val,
                                 parent_val=parent_val,
                                 indecisive=indecisive, memory_skill=memory_skill,
                                 parents=parents, edit_kind=args.edit_kind,
                                 opt_cost_usd=args.optimizer_usd or None,
                                 opt_tokens=args.optimizer_tokens or None,
                                 optimizer_seconds=args.optimizer_seconds or None,
                                 **gate)
        # Re-seed the framework's cross-iteration memory onto whichever candidate is now
        # $BEST, so the NEXT round's `cp -r "$R/candidates/$BEST" "$R/work/$TAG"` carries a
        # CURRENT copy forward — the round-2+ half of the fix in host.py's `_stage_context`
        # (which seeds round 1 the same way onto the seed candidate).
        # `record_iteration` above already folded THIS round's tail into the run-level journal
        # and wrote its `step` event, so the LEDGER/RUNMAP rebuilt here include this round.
        # Everything the staged CLAUDE.md pointer names, not JOURNAL.md alone: LEDGER.md,
        # RUNMAP.md and prior_iterations/<id>/diff.patch are the files the pointer (and
        # JOURNAL.md's own seed text) tell the agent to read, and re-seeding only the journal
        # is what left them absent for a whole run — see harness.seed_framework_memory.
        # Falls back to THIS candidate's own just-taken snapshot when there is no best_id yet
        # (a run with no baseline) — that dir always exists (``run_dir.snapshot`` above just
        # created it) — and is best-effort: losing the re-seed must not fail the commit.
        try:
            harness.resolve_memory(memory_skill).seed(
                run_dir.candidate_dir(run_dir.best_id or args.candidate_id), run_dir)
        except Exception as exc:  # noqa: BLE001
            run_dir.log_event("optimizer_context_warning", what="framework_memory",
                              error=str(exc)[:300])
        if indecisive:
            # The event the dashboard/TUI already read to render a step as `indecisive` rather
            # than rejected (``dashboard`` keys its status, badge and banner off this exact
            # kind), and the only thing that distinguishes an unresolved round from a refuted
            # one downstream.
            run_dir.log_event("step_indecisive", candidate=args.candidate_id, reason=reason,
                              val=args.val, gate_verdict=gate_verdict)
        else:
            # Deliberately NOT for an unresolved round: ``rejected.jsonl`` is fed back as "these
            # edits did not work", and an edit the measurement could not judge says nothing of
            # the kind — filing it there teaches the next round to avoid a change never
            # evaluated. Same reasoning as the deterministic hill-climb's own indecisive branch.
            _record_memory(run_dir, args.candidate_id, accepted=accepted,
                           reason=reason, val=args.val, parent_val=parent_val)
        if not handover:
            warnings.append(
                "no handover recorded for this round: the run-level JOURNAL.md now reads "
                "'(no handover written by the optimizer)' for "
                f"{args.candidate_id}, so the next round can see WHICH tasks moved but not what "
                "you tried or why. Before the next commit.py, write your entry to "
                "<from-dir>/JOURNAL.md as a '## Iteration <candidate> — <headline>' block "
                "(changes made, expected effect, hypotheses prior RESULT lines already refuted, "
                "focus next).")
    spent = run_dir.spent
    run_dir.record_spend_warnings()
    stop, reason = run_dir.budget_exhausted()
    print(json.dumps({"decision": args.decision, "candidate": args.candidate_id,
                      "reject_basis": args.reject_basis,
                      "gate_verdict": gate_verdict,
                      "overrode_gate": overrode_gate,
                      "handover_recorded": handover,
                      "warnings": warnings,
                      "best_id": run_dir.best_id, "spent": spent.to_dict(),
                      "stop": stop, "stop_reason": reason}, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Merge N per-task-optimised copies of one Python file by running 3-way merge PER FUNCTION.

Why this exists. `merge_taskopt.py` runs git's 3-way merge on whole files, and on a real
fan-out that reports conflicts it should not. Measured on the one multi-turn tool-use benchmark: ten
independently-verified optimiser branches, and a whole-file merge kept only four of them. The
"conflicts" were not disagreements. Every optimiser had added

  * one state field to the SAME `__init__`, and
  * one independent guard call to the SAME tool method, right after the same existing check,

so their edits landed on adjacent lines of a shared insertion point. Line-level 3-way merge
cannot tell "two people appended different things here" from "two people rewrote the same
thing", and diff3 conflicts on both. Enabling `--union-on-conflict` to force them through
produced a file that DID NOT PARSE and carried five duplicated `def`s.

The granularity is the bug. Merge each function against its own base instead of merging the
file, and independent additions inside different functions stop interacting at all; only two
branches editing the SAME function can still conflict, which is the question actually worth a
human decision. On the same ten branches this raised retention from 4/10 to the full set.

    python funcmerge.py --base BASE.py --out OUT.py --inputs A.py B.py C.py

Reports, per function: which branches changed it, whether the merge was clean, and any
remaining conflict. A conflict here is a genuine semantic overlap — two branches rewriting one
function — and is left for a decision, never auto-resolved.

Guarantees enforced before writing OUT:
  * the result parses (`ast.parse`), and
  * no `def` name is defined twice,
because the failure this replaces produced a file that violated both.
"""

import argparse
import ast
import difflib
import json
import subprocess
import tempfile
from pathlib import Path


def blocks(src: str) -> tuple[list[str], dict[str, str]]:
    """Split a module into (preamble_lines, {qualified_def_name: source_text}).

    Splitting is done on the AST, not on indentation heuristics, so a `def` inside a
    docstring or a string literal cannot create a phantom block.
    """
    tree = ast.parse(src)
    lines = src.splitlines(keepends=True)
    spans: list[tuple[int, int, str]] = []

    def walk(node, prefix=""):
        for child in ast.iter_child_nodes(node):
            if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                name = f"{prefix}{child.name}"
                start = min([child.lineno] + [d.lineno for d in child.decorator_list]) - 1
                spans.append((start, child.end_lineno, name))
            elif isinstance(child, ast.ClassDef):
                walk(child, prefix=f"{prefix}{child.name}.")
            elif prefix and isinstance(child, (ast.Assign, ast.AnnAssign)):
                # Class-level CONSTANTS are merge units too. Without this they are invisible to
                # a function-granularity merge, which is the exact root cause of the worst bug
                # this tool has produced: a merged function referencing `self.CABIN_LADDER`
                # while the constant stayed behind, crashing at runtime and presenting as a
                # missing write. Carrying them as named blocks makes the whole class of defect
                # impossible rather than merely detected.
                tgts = ([child.target] if isinstance(child, ast.AnnAssign)
                        else list(child.targets))
                names = [t.id for t in tgts if isinstance(t, ast.Name)]
                if len(names) == 1:
                    # a leading `#:` comment block belongs to the constant it documents
                    start = child.lineno - 1
                    while start > 0 and lines[start - 1].lstrip().startswith("#"):
                        start -= 1
                    spans.append((start, child.end_lineno, f"{prefix}{names[0]}"))

    walk(tree)
    spans.sort()
    out: dict[str, str] = {}
    covered = set()
    for start, end, name in spans:
        out[name] = "".join(lines[start:end])
        covered.update(range(start, end))
    pre = [ln for i, ln in enumerate(lines) if i not in covered]
    return pre, out


def pure_insertions(base: str, variant: str) -> list[tuple[int, list[str]]] | None:
    """If `variant` only ADDS lines to `base`, return those insertions; else None.

    This is the test that decides whether a same-function collision is safe to union. Every
    optimiser in a fan-out tends to append one guard call to a shared tool method and one state
    field to a shared `__init__`; those are insertions at a common anchor, and diff3 conflicts on
    them even though the branches do not disagree about anything. But a branch that REWRITES a
    base line is asserting the old line was wrong, and two such assertions cannot both be
    honoured — that one still needs a human decision. So union is offered only for the
    provably-additive case, keyed on position in the BASE so the order of branches cannot change
    the result.
    """
    bl, vl = base.splitlines(keepends=True), variant.splitlines(keepends=True)
    ins: list[tuple[int, list[str]]] = []
    for tag, i1, _i2, j1, j2 in difflib.SequenceMatcher(None, bl, vl).get_opcodes():
        if tag == "equal":
            continue
        if tag == "insert":
            ins.append((i1, vl[j1:j2]))
        else:                      # replace / delete -> a real rewrite
            return None
    return ins


def union_insertions(base: str, variants: list[tuple[str, str]]) -> str | None:
    """Apply every branch's insertions to `base`, deduplicated, anchored to base positions."""
    per = []
    for tag, text in variants:
        got = pure_insertions(base, text)
        if got is None:
            return None
        per.append((tag, got))
    bl = base.splitlines(keepends=True)
    at: dict[int, list[str]] = {}
    # Dedupe WHOLE HUNKS across branches, never individual lines. A branch's own inserted lines
    # are already correct and may legitimately repeat: two dict comprehensions in one `__init__`
    # both contain one shared iteration expression, and de-duplicating by line
    # deleted the second one, truncating the statement into a syntax error that only surfaced as
    # `'{' was never closed`. The only thing worth collapsing is two branches contributing the
    # SAME insertion at the SAME anchor, which is exactly (anchor, hunk).
    seen_hunks: set[tuple[int, tuple[str, ...]]] = set()
    for _tag, ins in per:
        for pos, lines in ins:
            key = (pos, tuple(ln.rstrip("\n") for ln in lines))
            if key in seen_hunks:
                continue
            seen_hunks.add(key)
            at.setdefault(pos, []).extend(lines)
    out: list[str] = []
    for i, ln in enumerate(bl):
        out.extend(at.pop(i, []))
        out.append(ln)
    for pos in sorted(at):
        out.extend(at[pos])
    return "".join(out)


def priority_union(base: str, variants: list[tuple[str, str]], order: list[str],
                   force: bool = False) -> str | None:
    """Resolve a same-function collision as: ONE branch's rewrite + everyone else's insertions.

    The collisions left after `union_insertions` share a shape. Several optimisers each rewrote
    the SAME tool docstring (a real rewrite, so union is not allowed) while ALSO each adding one
    independent guard call to the body (pure insertions, which union is exactly right for).
    Dropping the whole function to a single branch would throw away the other branches' guards —
    on that benchmark that meant losing two branches' fixes to keep a third's
    docstring, which is not a trade anyone would choose deliberately.

    So split the decision. The highest-priority branch (caller-supplied `order`, normally by how
    much val headroom the branch's task holds) wins the rewrite and becomes the trunk. Every
    other branch contributes only its insertions, re-anchored by the CONTENT of the base line
    they followed rather than by line number, since the trunk has shifted those numbers. An
    insertion whose anchor the trunk no longer contains is reported as dropped rather than
    guessed at.
    """
    variants = sorted(variants, key=lambda tv: _trunk_key(base, tv, order))
    trunk_tag, trunk = variants[0]
    bl = base.splitlines(keepends=True)
    tl = trunk.splitlines(keepends=True)
    dropped: list[str] = []
    for tag, text in variants[1:]:
        ins = pure_insertions(base, text)
        if ins is None:
            # A second genuine rewrite of the same function. Without --force-priority this
            # needs a human. WITH it, only this branch's REWRITE is dropped; branches that
            # merely inserted still contribute, because dropping a whole branch for someone
            # else's rewrite is how a merge silently loses a measured fix (here it would have
            # dropped task 42's guard call to resolve a disagreement about a money string).
            if not force:
                return None
            FORCED_REWRITES.append(f"{tag}")
            continue
        for pos, lines in ins:
            anchor = bl[pos - 1] if pos > 0 else None
            if any(ln.strip() and ln.strip() in "".join(tl) for ln in lines):
                continue                     # trunk already carries it
            if anchor is None:
                tl = lines + tl
                continue
            try:
                at = next(i for i in range(len(tl) - 1, -1, -1) if tl[i] == anchor)
            except StopIteration:
                dropped.append(f"{tag}:{lines[0].strip()[:60]}")
                continue
            tl = tl[: at + 1] + lines + tl[at + 1 :]
    out = "".join(tl)
    if dropped:
        out += ""      # reported by caller via PRIORITY_DROPPED
        PRIORITY_DROPPED.extend(dropped)
    return out


def _trunk_key(base: str, tv: tuple[str, str], order: list[str]) -> tuple[int, int]:
    """Sort key choosing which branch becomes the trunk of a contested function.

    Trunk = the branch that CHANGED THIS FUNCTION MOST, measured in lines differing from base;
    the caller's `order` is only a tiebreak. Ordering by the branch's task headroom instead is
    a trap that cost a whole resolution on that benchmark: the branch with the most headroom
    (task 7, a full task-equivalent) turned out to have added exactly ONE line to the contested
    the contested function — its real fix was elsewhere — so making it the trunk
    discarded the branch that had actually rewritten the return value, and kept nothing. What a
    function is worth is not what its author's task is worth.
    """
    tag, text = tv
    bl = base.splitlines()
    changed = sum(1 for ln in difflib.unified_diff(bl, text.splitlines(), lineterm="", n=0)
                  if ln.startswith(("+", "-")) and not ln.startswith(("+++", "---")))
    rank = {t: i for i, t in enumerate(order)}
    return (-changed, rank.get(tag, len(order)))


PRIORITY_DROPPED: list[str] = []
FORCED: list[dict] = []
FORCED_REWRITES: list[str] = []


def merge3(base: str, a: str, b: str) -> tuple[str, bool]:
    """git merge-file on three strings. Returns (text, clean)."""
    with tempfile.TemporaryDirectory() as td:
        d = Path(td)
        (d / "base").write_text(base)
        (d / "a").write_text(a)
        (d / "b").write_text(b)
        r = subprocess.run(["git", "merge-file", "-p", "--diff3",
                            str(d / "a"), str(d / "base"), str(d / "b")],
                           capture_output=True, text=True)
        return r.stdout, r.returncode == 0


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", required=True)
    ap.add_argument("--out", required=True)
    ap.add_argument("--inputs", nargs="+", required=True)
    ap.add_argument("--json", dest="json_out", default="")
    ap.add_argument("--priority", nargs="*", default=[],
                    help="branch tags, most important first. When one function was REWRITTEN by "
                         "several branches, the first-listed becomes the trunk and the others "
                         "contribute only their insertions. Order by val headroom, not by name.")
    ap.add_argument("--force-priority", action="store_true",
                    help="for a function TWO branches rewrote, ship the highest-priority "
                         "branch's version whole and REPORT the branches dropped. This is a "
                         "real loss of measured work, so it is reported per function and must "
                         "be re-measured, never assumed harmless.")
    ap.add_argument("--union-pure-insertions", action="store_true",
                    help="resolve a same-function collision by applying every branch's "
                         "insertions when NO branch rewrites a base line (see "
                         "pure_insertions). Anything that rewrites base still conflicts.")
    args = ap.parse_args()

    base_src = Path(args.base).read_text()
    base_pre, base_fns = blocks(base_src)

    variants: dict[str, list[tuple[str, str]]] = {}   # fn -> [(branch, text)]
    pres: list[tuple[str, list[str]]] = []
    order: list[str] = list(base_fns)
    for p in args.inputs:
        tag = Path(p).parent.parent.name if Path(p).parent.name == "tools" else Path(p).stem
        pre, fns = blocks(Path(p).read_text())
        pres.append((tag, pre))
        for name, text in fns.items():
            if name not in base_fns:
                variants.setdefault(name, []).append((tag, text))
                if name not in order:
                    order.append(name)
            elif text != base_fns[name]:
                variants.setdefault(name, []).append((tag, text))

    report: dict[str, dict] = {}
    merged_fns: dict[str, str] = dict(base_fns)
    conflicts: list[str] = []

    for name in order:
        vs = variants.get(name, [])
        if not vs:
            continue
        if name not in base_fns:
            # a NEW function. Two branches adding the same name with different bodies is a
            # real collision; identical bodies (a shared helper) is not.
            uniq = {t for _, t in vs}
            merged_fns[name] = vs[0][1]
            report[name] = {"kind": "added", "branches": [t for t, _ in vs],
                            "identical": len(uniq) == 1}
            if len(uniq) > 1:
                conflicts.append(name)
                report[name]["conflict"] = "same new name, different bodies"
            continue
        cur, clean_all = base_fns[name], True
        for tag, text in vs:
            cur, clean = merge3(base_fns[name], cur, text)
            if not clean:
                clean_all = False
        how = "diff3"
        if not clean_all and args.union_pure_insertions:
            u = union_insertions(base_fns[name], vs)
            if u is not None:
                cur, clean_all, how = u, True, "union-pure-insertions"
            elif args.priority:
                pu = priority_union(base_fns[name], vs, args.priority)
                if pu is not None:
                    cur, clean_all, how = pu, True, "priority-trunk+insertions"
                elif args.force_priority:
                    FORCED_REWRITES.clear()
                    pf = priority_union(base_fns[name], vs, args.priority, force=True)
                    if pf is not None:
                        trunk = min(vs, key=lambda tv: _trunk_key(base_fns[name], tv,
                                                                  args.priority))[0]
                        cur, clean_all, how = pf, True, f"forced-trunk:{trunk}"
                        FORCED.append({"function": name, "trunk": trunk,
                                       "rewrites_dropped": list(FORCED_REWRITES),
                                       "insertions_kept_from": [t for t, _ in vs
                                                                if t != trunk
                                                                and t not in FORCED_REWRITES]})
        merged_fns[name] = cur
        report[name] = {"kind": "modified", "branches": [t for t, _ in vs],
                        "clean": clean_all, "resolved_by": how}
        if not clean_all:
            conflicts.append(name)

    # preamble: base plus any import lines a branch added, in first-seen order
    pre_out = list(base_pre)
    have = set(base_pre)
    for _, pre in pres:
        for ln in pre:
            if ln.startswith(("import ", "from ")) and ln not in have:
                pre_out.insert(len([x for x in pre_out if x.startswith(("import ", "from "))]), ln)
                have.add(ln)

    # Reassemble in the base file's own layout so the diff stays readable: walk the base
    # source and swap each function block for its merged text, appending genuinely new
    # functions after the last function of the class they came from.
    src_lines = base_src.splitlines(keepends=True)
    _, spans = blocks(base_src), None
    tree = ast.parse(base_src)
    placed: list[tuple[int, int, str]] = []

    def walk(node, prefix=""):
        for child in ast.iter_child_nodes(node):
            if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                s = min([child.lineno] + [d.lineno for d in child.decorator_list]) - 1
                placed.append((s, child.end_lineno, f"{prefix}{child.name}"))
            elif isinstance(child, ast.ClassDef):
                walk(child, prefix=f"{prefix}{child.name}.")

    walk(tree)
    placed.sort()
    new_names = [n for n in merged_fns if n not in base_fns]
    # A new CONSTANT must land inside the class body, before the first method, or it silently
    # becomes a module-level name and `self.NAME` still fails.
    new_consts = [n for n in new_names if n.rsplit(".", 1)[-1].isupper()]
    new_names = [n for n in new_names if n not in new_consts]
    out_parts: list[str] = []
    cursor = 0
    first_method = placed[0][0] if placed else 0
    for i, (s, e, name) in enumerate(placed):
        if i == 0 and new_consts:
            out_parts.append("".join(src_lines[cursor:s]))
            for cn in new_consts:
                out_parts.append(merged_fns[cn].rstrip("\n") + "\n\n")
            cursor = s
        out_parts.append("".join(src_lines[cursor:s]))
        out_parts.append(merged_fns[name])
        cursor = e
        if i + 1 < len(placed):
            continue
        for n in new_names:                       # append new helpers after the last method
            out_parts.append("\n" + merged_fns[n])
    out_parts.append("".join(src_lines[cursor:]))
    text = "".join(out_parts)

    # add any imports the branches needed
    if pre_out != base_pre:
        added = [ln for ln in pre_out if ln not in base_pre]
        head, sep, rest = text.partition("\n\n")
        text = head + "\n" + "".join(added) + sep + rest

    # POST-MERGE AUDIT: which lines that a branch ADDED did the merge fail to carry?
    #
    # This exists because a forced-trunk resolution can silently re-apply a change the losing
    # branch had already MEASURED AND REVERTED. Observed live: one branch had added a sentence
    # to a `payment_id` Args description and separately recorded, twice, that REMOVING that
    # sentence was harmful. The merge dropped that branch rewrite of the function, which
    # re-performed the exact subtraction its owner had rejected - invisible at whole-file
    # level, and absent from the conflict report, because from the merge point of view nothing
    # conflicted. A gate would then measure the regression without ever naming its cause.
    #
    # The check is cheap and purely structural: any non-trivial line a branch added that is
    # absent from the result is reported. It is advisory, not fatal - some drops are the
    # deliberate outcome of a conflict decision - but it must be READ against the ledger
    # rejected entries before the merged artifact is gated.
    base_lines = set(base_src.splitlines())
    dropped_additions: dict[str, list[str]] = {}
    _fences = ('"""', "'''")
    for p_in in args.inputs:
        tag = (Path(p_in).parent.parent.name if Path(p_in).parent.name == "tools"
               else Path(p_in).stem)
        lost = []
        for ln in Path(p_in).read_text().splitlines():
            t = ln.strip()
            if len(t) < 12 or ln in base_lines or t in _fences:
                continue
            if t not in text:
                lost.append(t[:120])
        if lost:
            dropped_additions[tag] = lost[:12]

    ok, err = True, ""
    try:
        t = ast.parse(text)
        names = [n.name for n in ast.walk(t)
                 if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))]
        dups = sorted({n for n in names if names.count(n) > 1})
        if dups:
            ok, err = False, f"duplicate defs: {dups}"
        # Merging FUNCTIONS can drop a class-level CONSTANT that a merged function needs. This
        # is not a style problem, it is a crash: the helper survives, its call site survives,
        # and `self.CABIN_LADDER` raises AttributeError at runtime. The tool layer catches it
        # and hands the agent an error string, so the agent abandons the write and the failure
        # presents as a MISSING WRITE — indistinguishable from a policy failure in the reward,
        # and it silently contaminated four separate measurements before a live tool return
        # exposed it. So resolve it statically: every attribute the result reads off `self`
        # must be defined in the result.
        # Instance attributes are set both plainly (`self.x = 1`) and WITH ANNOTATIONS
        # (`self.x: set[str] = set()`). The latter is ast.AnnAssign, not ast.Assign; collecting
        # only Assign made this check report six valid fields as undefined and hard-fail a good
        # merge. A hard-fail check with false positives is worse than no check at all.
        assigned = set()
        for n in ast.walk(t):
            if isinstance(n, ast.Assign):
                for tgt in n.targets:
                    for x in ast.walk(tgt):
                        if isinstance(x, ast.Attribute):
                            assigned.add(x.attr)
            elif isinstance(n, ast.AnnAssign) and isinstance(n.target, ast.Attribute):
                assigned.add(n.target.attr)
            elif isinstance(n, (ast.AugAssign,)) and isinstance(n.target, ast.Attribute):
                assigned.add(n.target.attr)
        class_attrs, methods = set(), set()
        for n in ast.walk(t):
            if isinstance(n, ast.ClassDef):
                for stmt in n.body:
                    if isinstance(stmt, (ast.FunctionDef, ast.AsyncFunctionDef)):
                        methods.add(stmt.name)
                    for tgt in getattr(stmt, "targets", []) or []:
                        if isinstance(tgt, ast.Name):
                            class_attrs.add(tgt.id)
                    if isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name):
                        class_attrs.add(stmt.target.id)
        read = {n.attr for n in ast.walk(t)
                if isinstance(n, ast.Attribute) and isinstance(n.value, ast.Name)
                and n.value.id == "self" and isinstance(n.ctx, ast.Load)}
        undefined = sorted(read - assigned - class_attrs - methods - set(dir(object)))
        # Only CONSTANT-shaped names hard-fail. The class under merge normally has a base class
        # (so `self.x` may resolve on a base class), and an inherited method called through `self` is not
        # resolvable from this file — hard-failing on those would reject valid merges, which is
        # worse than not checking. Upper-case class constants are the case actually observed
        # crashing (`self.CABIN_LADDER`), and they are not inherited in practice. Everything
        # else is reported for a human to read.
        missing_const = [n for n in undefined if n.isupper()]
        maybe_inherited = [n for n in undefined if not n.isupper()]
        if ok and missing_const:
            ok, err = False, ("undefined on self (a merged function needs a definition the "
                              f"merge did not carry): {missing_const}")
    except SyntaxError as exc:
        ok, err = False, f"SyntaxError: {exc}"

    if ok and not conflicts:
        Path(args.out).write_text(text)
    elif not ok:
        # Write the rejected text next to the target so a syntax failure is INSPECTABLE.
        # A refusal that leaves nothing behind forces the caller to reconstruct the assembly
        # by hand to find out what broke, which is how a tool bug gets worked around instead
        # of fixed.
        Path(str(args.out) + ".rejected").write_text(text)

    result = {
        "base": args.base,
        "inputs": args.inputs,
        "written": bool(ok and not conflicts),
        "out": args.out,
        "parses": ok,
        "error": err,
        "conflicts": conflicts,
        "functions_touched": {k: v for k, v in report.items()},
        "priority_dropped_insertions": PRIORITY_DROPPED,
        "forced_single_branch": FORCED,
        "dropped_additions": dropped_additions,
        "self_attrs_not_defined_here": locals().get("maybe_inherited") or [],
        "dropped_additions_warning": (
            "lines a branch ADDED that the merge did not carry. Check each against the "
            "ledger REJECTED entries: a dropped rewrite can re-apply a subtraction its "
            "owner already measured as harmful." if dropped_additions else ""),
        "next": ("render the LIVE toolset (validate_capability.py) before spending rollouts"
                 if ok and not conflicts else
                 "resolve the listed conflicts by hand — two branches rewrote one function"),
    }
    print(json.dumps(result, indent=2))
    if args.json_out:
        Path(args.json_out).write_text(json.dumps(result, indent=2))
    return 0 if result["written"] else 1


if __name__ == "__main__":
    raise SystemExit(main())

"""gate_check — the honest accept/reject decision for agent-optimize, from real rollouts.

Why this exists instead of ``phases/gate/scripts/run.py``: that CLI takes only two
scalar means, so it can reach the *unpaired* ``significant`` test and nothing else.
The deterministic loops all default to the **paired** gate (mean per-task Δ vs the SE
of those deltas), which needs the aligned per-task vector — data the scalar CLI has no
way to accept. So the agent had no reachable path to the same gate the rest of
cap-evolve uses.

This script closes that: it reconstructs both sides' ``SplitResult`` from the persisted
val rollouts (``harness.split_result_from_rollouts``), builds the paired delta vector
with the SAME helper the loops use (``harness._paired_deltas``), and calls the SAME
``gate.decide``. It also REPORTS **regressions** — val tasks the parent measured and passed
that dropped — as diagnosis for the next round. They do not veto an accept unless you pass
``--veto-regressions``; see ``regressions()`` for the measured reason that default flipped.

Tags are candidate dir names: the evaluate phase writes rollouts as
``<task>__<tag>__t<k>.json`` with ``tag = candidate_dir.name``.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

# Imported for its side effect ONLY: seeds sys.path so `cap_evolve` resolves when
# this script is run standalone (`python <this-file>`). Must precede the
# cap_evolve imports below; not "unused" — deleting it breaks standalone runs.
import _bootstrap  # noqa: F401  # side-effect import, see above

from cap_evolve import RunDir, footprint, harness
from cap_evolve.gate import decide
from cap_evolve.loop import has_valid_trials

EPS = 1e-9


def regressions(current, candidate) -> list[str]:
    """Val tasks the current best measured-and-PASSED that got worse. REPORTED, not a veto.

    As of one long run this list is DIAGNOSIS ONLY — it no longer blocks an accept
    unless you pass ``--veto-regressions``. The veto was measured to be the dominant cause
    of four consecutive null results on a multi-turn tool-use benchmark:

      * it fires on a byte-identical copy of the seed 42.8% of the time at 5 trials
        (12.9% at 10) — see the table below, which is why no trial count rescues it at the
        val sizes this benchmark allows;
      * in run_agentoptv4 it vetoed BOTH candidates that passed the significance test
        (``cA_partial`` Delta-bar +0.0167 > bar 0.0134, vetoed on task 8; ``cB_becabin``
        +0.0167, vetoed on 8/32/40). Those were the run's only two positive signals.

    A per-task reward at n trials is an estimate with its own error bar, so "this one task
    dropped" is not evidence of harm at the sizes involved; the PAIRED test on the mean
    already accounts for per-task movement in both directions and is the statistically
    correct decision rule. Churn (fix 2 / break 2 at an identical mean) is correctly a
    non-accept under the paired test — it just fails for the right reason (no significant
    gain) instead of being vetoed after passing.

    The list stays in the output because it is the most actionable thing the next round
    reads: it names which part of a bundled edit to drop.

    Mirrors ``harness._movement`` exactly -- the parent must have scored a full 1.0
    (``par >= 1.0 - EPS``), which is what SKILL.md means by "measured-and-passed".
    Tasks with no valid trial on either side are missing data, not evidence, so an
    infra outage can't veto a genuinely better candidate.

    This USED to veto on any strict drop from any parent level, which silently made
    agent-optimize's gate stricter than every other algorithm's -- and uniquely
    broken at num_trials > 1. At 1 trial rewards are 0/1 so the two rules coincide.
    Above that, a per-task reward is a fraction and the parent's is frozen from one
    draw, so a task whose true rate is 0.45 but which drew 4/5 vetoes almost any
    re-measurement of the SAME capability. Measured on the v4 val rates,
    P(veto fires on a byte-identical seed copy):

        trials   any-drop (old)   parent-passed (this rule, == harness)
             1            0.889                                  0.889
             5            0.983                                  0.428
            10            0.990                                  0.129

    The old rule got WORSE as trials rose, so no trial count could fix it; the
    harness rule converges, which is the behaviour a variance-aware gate must have.

    And it converges FASTER than "any strict drop below 1.0", because the drop must clear
    ``2·SE`` of its own per-task measurement — the same bar ``harness._candidate_task_impact``
    applies, kept in sync by ``test_regression_gate``. Without it the list reported a task as
    regressed for a single flipped rollout out of ten: on run_finalrun6 the same "task 27"
    was reported against structurally unrelated candidates, one of them a docstring-only edit
    that cannot change behaviour, and the optimizer spent three rounds re-deriving that it was
    noise. At one trial every SE is 0, the bar collapses to ``EPS``, and the rule is
    unchanged.
    """
    cur = {pt["task_id"]: pt for pt in (current.per_task or []) if has_valid_trials(pt)}
    cand = {pt["task_id"]: pt for pt in (candidate.per_task or []) if has_valid_trials(pt)}

    def _dropped(t) -> bool:
        pr = cur[t].get("reward", 0.0) or 0.0
        cd = cand[t].get("reward", 0.0) or 0.0
        return pr >= 1.0 - EPS and cd < pr and harness.move_is_resolved(
            pr, cd, cur[t].get("stderr") or 0.0, cand[t].get("stderr") or 0.0)

    return sorted(t for t in cur if t in cand and _dropped(t))


# The gate modes THIS script implements, and the single source of truth for them.
# `round.py` forwards its own --mode here verbatim, so it imports this list rather than
# repeating it: on run 33492876620 round 3 the two disagreed (round.py had no `choices=` at
# all), `--mode val` sailed through round.py, was rejected here, and emptied the entire
# round table while `eval_rc` stayed 0. Two copies of a list is how that happens.
GATE_MODES = ["paired", "significant", "strict", "threshold"]


def _frozen_coverage(run_dir, per_task, split: str = "val") -> float:
    """Real coverage against the FROZEN split, not just the tasks a rollout exists for.

    ``SplitResult.coverage`` is ``n_scored / n_tasks`` where ``n_tasks`` counts only
    tasks that have a rollout file under this tag — reconstructed purely from disk
    (``harness.split_result_from_rollouts``). A candidate evaluated on a SUBSET of val
    (deliberately via ``--ids``, or by an eval that died partway through) therefore
    reads back ``coverage == 1.0``: every task it DID measure, it measured. That is
    exactly the blind spot ``gate.decide``'s low-coverage guard exists to catch, and it
    cannot see through it unless coverage is computed against the split cap-evolve
    actually froze, not against whatever happened to land on disk.
    """
    frozen = {str(i) for i in (run_dir.read_splits().ids(split) or [])}
    if not frozen:
        return 1.0
    scored = {str(pt.get("task_id")) for pt in (per_task or []) if has_valid_trials(pt)}
    return len(scored & frozen) / len(frozen)


def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(prog="gate_check")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--candidate", required=True, help="candidate tag (== its dir name)")
    p.add_argument("--current", default=None,
                   help="tag to compare against; default = the run's current best_id. Accepts a "
                        "COMMA-SEPARATED list, whose trials are POOLED per task into one "
                        "reference — the way a round's byte-identical null-control replicates "
                        "become a lower-variance estimate of the same parent for free, since "
                        "their rollouts are already on disk (see round.py's control_replicates).")
    p.add_argument("--no-footprint", action="store_true",
                   help="measure the delta across EVERY val task, including the ones the edit "
                        "cannot causally reach. Footprint restriction is on by default and is a "
                        "no-op whenever the edit's surface cannot be determined; see "
                        "cap_evolve.footprint for why the unrestricted vector buries real "
                        "effects in the noise of tasks the edit never touched.")
    p.add_argument("--mode", default="paired", choices=GATE_MODES)
    p.add_argument("--k-se", type=float, default=1.0)
    p.add_argument("--threshold", type=float, default=0.0)
    p.add_argument("--veto-regressions", action="store_true",
                   help="ALSO reject a gate-passing candidate that drops any val task the parent "
                        "measured-and-passed. OFF by default — see regressions() for why.")
    p.add_argument("--allow-regression", action="store_true",
                   help="deprecated no-op: regressions no longer veto unless --veto-regressions")
    return p


def main(argv=None) -> int:
    args = build_parser().parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))
    cur_tags = [t.strip() for t in (args.current or "").split(",") if t.strip()] \
        or ([run_dir.best_id] if run_dir.best_id else [])
    if not cur_tags:
        print(json.dumps({"error": "no --current tag and no best_id in the run dir "
                                   "(has baseline run?)"}, indent=2))
        return 2
    cur_tag = ",".join(cur_tags)

    cur = harness.split_result_from_rollouts(run_dir, cur_tags, "val")
    cand = harness.split_result_from_rollouts(run_dir, args.candidate, "val")
    if not cand.per_task:
        print(json.dumps({"error": f"no val rollouts for tag {args.candidate!r} — run the "
                                   "evaluate phase on FULL val first"}, indent=2))
        return 2

    # Which val tasks the edit can causally reach. The candidate snapshot is diffed against
    # the FIRST reference tag's snapshot: a pooled reference is several byte-identical copies
    # of one parent, so any of them gives the same diff. None when it cannot be determined,
    # which leaves the full-vector behaviour exactly as it was.
    fp = None
    if not args.no_footprint:
        fp = footprint.footprint(
            run_dir, parent_dir=run_dir.candidate_dir(cur_tags[0]),
            cand_dir=run_dir.candidate_dir(args.candidate),
            tags=[*cur_tags, args.candidate], split="val",
            all_task_ids=[pt.get("task_id") for pt in (cand.per_task or [])])

    # Real coverage against the frozen split (see `_frozen_coverage`), min of both sides —
    # a candidate OR a reference measured on a subset is equally invalid to gate on.
    cand_frozen_cov = _frozen_coverage(run_dir, cand.per_task, "val")
    cur_frozen_cov = _frozen_coverage(run_dir, cur.per_task, "val")
    frozen_coverage = min(cand_frozen_cov, cur_frozen_cov)
    frozen_ids = {str(i) for i in (run_dir.read_splits().ids("val") or [])}
    cand_ids = {str(pt.get("task_id")) for pt in (cand.per_task or []) if has_valid_trials(pt)}
    missing_from_frozen_val = sorted(frozen_ids - cand_ids)

    deltas = harness._paired_deltas(cur, cand, footprint=fp)
    # Only when restricted: on a small footprint the zero-padded vector's cross-task spread
    # understates the real uncertainty, so floor it with per-task trial noise. Unrestricted
    # vectors keep the SE they always had.
    se_floor = (harness.paired_se_floor(run_dir, args.candidate, cur_tags[0], fp, len(deltas))
                if fp is not None and deltas else 0.0)
    d = decide(cur.reward, cand.reward, split="val", mode=args.mode, k_se=args.k_se,
               candidate_stderr=cand.stderr, current_stderr=cur.stderr,
               threshold=args.threshold, paired_deltas=deltas,
               paired_se_floor=se_floor, coverage=frozen_coverage, run_dir=run_dir)

    regs = regressions(cur, cand)
    accept = bool(d.accept) and not (regs and args.veto_regressions)
    verdict = "indecisive" if d.indecisive else ("accept" if accept else "reject")
    # A reject with delta > 0 is not the same as a reject with delta <= 0: the first is
    # a positive direction the gate could not yet resolve at this n, and growing n on
    # this SAME candidate (never a new edit) may resolve it — see references/algorithm.md,
    # "Provisional candidates". Surfaced here so the driver notices it without having to
    # compute delta > 0 itself.
    # Off `d.accept`, not the regression-vetoed `accept`: a candidate the GATE accepted and
    # `--veto-regressions` then rejected has nothing left for more trials to resolve — the
    # veto is a per-task harm call, not a measurement-power problem.
    directionally_positive_but_inconclusive = (
        not d.indecisive and not d.accept and d.delta > 0)
    next_cmd = f"scripts/commit.py --decision {'accept' if accept else 'reject'}"
    if directionally_positive_but_inconclusive:
        next_cmd += " (or --decision provisional, then scripts/grow.py, to buy more n on this candidate)"
    print(json.dumps({
        "current": {"tag": cur_tag, "reward": cur.reward, "stderr": cur.stderr,
                    "pooled_tags": cur_tags if len(cur_tags) > 1 else None},
        "candidate": {"tag": args.candidate, "reward": cand.reward,
                      "stderr": cand.stderr, "coverage": cand.coverage,
                      "coverage_of_frozen_val": round(frozen_coverage, 4),
                      "missing_from_frozen_val": missing_from_frozen_val},
        "gate": d.to_dict(),
        "paired_n": len(deltas or []),
        # What the delta was measured over. `restricted: false` means the edit's surface could
        # not be determined, so every val task is in the vector and the SE carries the noise of
        # tasks the edit cannot reach — read the verdict knowing that.
        "footprint": ({"restricted": True, "n_in_footprint": len(fp),
                       "n_tasks": len(cand.per_task or []), "tasks": sorted(map(str, fp)),
                       "paired_se_floor": round(se_floor, 6),
                       "reading": "tasks OUTSIDE this set entered the delta vector as 0.0 (an "
                                  "edit that cannot reach a task has no effect on it by "
                                  "construction), so the SE reflects only the tasks in play — "
                                  "floored by `paired_se_floor`, the SE those tasks' own "
                                  "per-trial noise implies, so a handful of one-rollout flips "
                                  "cannot read as a significant mean"}
                      if fp is not None else
                      {"restricted": False,
                       "reading": ("disabled by --no-footprint" if args.no_footprint else
                                   "the edit's surface could not be localized (no diff, a "
                                   "rewrite-sized diff, no rollouts, or it reaches every "
                                   "task) — full-vector measurement, as before")}),
        "regressions": regs,
        "verdict": verdict,
        "directionally_positive_but_inconclusive": directionally_positive_but_inconclusive,
        "next": next_cmd,
    }, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""grow — buy more trials on a PROVISIONAL candidate, then re-gate at the pooled n.

A candidate is ``provisional`` (``commit.py --decision provisional``) when it is
directionally positive (Δ>0) but did not clear the significance gate at the n it was
measured at. That is sequential evidence, not a null result: the honest next step is
more trials on the SAME, UNMODIFIED candidate — never a new edit on top of unconfirmed
ground (see references/algorithm.md, "Provisional candidates").

This script:
  1. runs ``--add-trials`` NEW trials on the candidate's unchanged working copy, under a
     throwaway tag (so the new rollout files cannot collide with the candidate's own),
  2. pools the new trials with the candidate's existing val rollouts via
     ``loop.pool_split_results`` (concatenates per-task trial vectors, not two means),
  3. re-runs the SAME paired gate the candidate was first measured against, at the
     pooled n,
  4. merges the new rollout files onto the candidate's own tag on disk (renumbered past
     its existing trial indices) so a later ``gate_check.py --candidate <tag>`` or
     ``commit.py`` sees the full pooled history with no special-casing, and
  5. recommends ``promote`` / ``grow_again`` / ``abandon`` — capped at
     ``--max-growth-rounds`` (default 2): a provisional lineage that still has not
     resolved after 2 extra growth rounds must be abandoned, not grown again, so an
     unlucky early positive can only consume a bounded amount of extra budget.

Never edits the candidate directory — it is re-evaluated exactly as it is.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

# Imported for its side effect ONLY: seeds sys.path so `cap_evolve` resolves when
# this script is run standalone (`python <this-file>`). Must precede the
# cap_evolve imports below; not "unused" — deleting it breaks standalone runs.
import _bootstrap  # noqa: F401  # side-effect import, see above

from cap_evolve import RunDir, harness
from cap_evolve.check import load_adapter
from cap_evolve.gate import decide
from cap_evolve.loop import pool_split_results

DEFAULT_MAX_GROWTH_ROUNDS = 2


def _existing_trial_count(run_dir: RunDir, tag: str, split: str) -> int:
    """How many trial files the candidate's OWN tag already has (any task; every task
    gets the same count, errored or not — see harness.evaluate_candidate)."""
    vdir = run_dir.rollouts / split
    if not vdir.exists():
        return 0
    # Track the HIGHEST trial index per task, then convert to a count once. Comparing a
    # running count against the next index instead (`max(count, idx) + 1`) over-counts
    # whenever glob order is not ascending, which leaves gaps in the merged indices.
    highest: dict = {}
    for f in vdir.glob(f"*__{tag}__t*.json"):
        tid = f.name.split(f"__{tag}__t")[0]
        idx = int(f.name.rsplit("__t", 1)[1].removesuffix(".json"))
        highest[tid] = max(highest.get(tid, -1), idx)
    return max(highest.values()) + 1 if highest else 0


def _merge_grow_trials(run_dir: RunDir, candidate: str, grow_tag: str, split: str,
                       offset: int) -> None:
    """Rename the throwaway tag's rollout files onto the candidate's own tag, at trial
    indices starting from ``offset`` — so the candidate's tag alone now carries the full
    pooled history and every downstream reader (gate_check.py, dashboard, LEDGER.md)
    needs no special-casing for a grown candidate."""
    vdir = run_dir.rollouts / split
    for f in sorted(vdir.glob(f"*__{grow_tag}__t*.json")):
        tid = f.name.split(f"__{grow_tag}__t")[0]
        idx = int(f.name.rsplit("__t", 1)[1].removesuffix(".json"))
        f.rename(vdir / f"{tid}__{candidate}__t{offset + idx}.json")


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="grow")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", required=True)
    p.add_argument("--candidate", required=True,
                   help="the provisional candidate's tag == dir name")
    p.add_argument("--current", default=None,
                   help="tag to gate against; default = the run's current best_id")
    p.add_argument("--add-trials", type=int, required=True,
                   help="how many NEW trials to run on this candidate before re-gating")
    p.add_argument("--growth-round", type=int, required=True,
                   help="which growth attempt this is for this candidate (1, 2, ...)")
    p.add_argument("--max-growth-rounds", type=int, default=DEFAULT_MAX_GROWTH_ROUNDS)
    p.add_argument("--k-se", type=float, default=1.0)
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))
    adapter = load_adapter(Path(args.project))
    cur_tag = args.current or run_dir.best_id
    if not cur_tag:
        print(json.dumps({"error": "no --current tag and no best_id in the run dir"}, indent=2))
        return 2

    cand_dir = run_dir.candidate_dir(args.candidate)
    if not cand_dir.is_dir():
        print(json.dumps({"error": f"no snapshot for candidate {args.candidate!r} — "
                                   "commit.py --decision provisional first"}, indent=2))
        return 2

    existing = harness.split_result_from_rollouts(run_dir, args.candidate, "val")
    if not existing.per_task:
        print(json.dumps({"error": f"no existing val rollouts for {args.candidate!r} — "
                                   "this is not a candidate that was ever gated"}, indent=2))
        return 2

    # New trials go under a THROWAWAY tag first (never the candidate's own), so they
    # cannot collide with — or silently overwrite — the trials already on disk.
    grow_tag = f"{args.candidate}__grow{args.growth_round}"
    try:
        base_seed = int(run_dir.read_splits().seed)
    except Exception:  # noqa: BLE001
        base_seed = 0
    # Offset the new batch's seeds well clear of any prior growth round's, so re-running
    # a real (non-deterministic) target draws genuinely new trials rather than replaying
    # ones already on disk. ponytail: a fixed 1000-per-round stride, not exact bookkeeping
    # of how many seeds a prior round actually consumed — plenty of headroom at the trial
    # counts this gate is meant for.
    new_result = harness.evaluate_candidate(
        adapter, cand_dir, run_dir=run_dir, split="val", n_trials=args.add_trials,
        tag=grow_tag, base_seed=base_seed + 1000 * args.growth_round)

    pooled = pool_split_results(existing, new_result)

    cur = harness.split_result_from_rollouts(run_dir, cur_tag, "val")
    deltas = harness._paired_deltas(cur, pooled)
    d = decide(cur.reward, pooled.reward, split="val", mode="paired", k_se=args.k_se,
               candidate_stderr=pooled.stderr, current_stderr=cur.stderr,
               paired_deltas=deltas, coverage=pooled.coverage, run_dir=run_dir)
    verdict = "indecisive" if d.indecisive else ("accept" if d.accept else "reject")

    # Merge the new trials onto the candidate's OWN tag now that the pooled numbers are
    # computed, so a re-run of gate_check.py --candidate <candidate> (or another grow.py
    # call for round N+1) sees the same pooled n with no special-casing.
    offset = _existing_trial_count(run_dir, args.candidate, "val")
    _merge_grow_trials(run_dir, args.candidate, grow_tag, "val", offset)

    if verdict == "accept":
        recommendation = "promote"
    elif verdict != "indecisive" and d.delta > 0 and args.growth_round < args.max_growth_rounds:
        recommendation = "grow_again"
    else:
        recommendation = "abandon"

    run_dir.log_event("provisional_grow", candidate=args.candidate, growth_round=args.growth_round,
                      add_trials=args.add_trials, pooled_n=len(deltas or []),
                      pooled_val=pooled.reward, verdict=verdict, recommendation=recommendation)

    # Persist the POOLED gate row in the same `work/<table>.json` shape `round.py` writes,
    # because `commit.py` reads such a table to recover the verdict (`_gate_row`, newest mtime)
    # and the structured gate numbers it attaches to the decision event
    # (`_round_gate_numbers`, which prefers a `grow_<cand>_r<k>.json` over the round's own row
    # for exactly this candidate). Without this the final commit on a grown candidate books the
    # round's PRE-growth numbers — and a `promote` would be logged as `gate_verdict: reject`,
    # reading as a driver override of a gate that in fact accepted at the pooled n.
    work = run_dir.root / "work"
    work.mkdir(parents=True, exist_ok=True)
    (work / f"grow_{args.candidate}_r{args.growth_round}.json").write_text(
        json.dumps({"grown": args.candidate, "growth_round": args.growth_round,
                    "candidates": [{
                        "tag": args.candidate,
                        "reward": pooled.reward,
                        "gate_delta": d.delta,
                        "gate_threshold": d.threshold,
                        "stderr": pooled.stderr,
                        "n": len(deltas or []),
                        "k_se": args.k_se,
                        "resolvable_effect_size": d.resolvable_effect_size,
                        "verdict": verdict,
                    }]}, indent=2), encoding="utf-8")

    print(json.dumps({
        "candidate": args.candidate,
        "growth_round": args.growth_round,
        "max_growth_rounds": args.max_growth_rounds,
        "current": {"tag": cur_tag, "reward": cur.reward, "stderr": cur.stderr},
        "pooled": {"reward": pooled.reward, "stderr": pooled.stderr, "n_tasks": pooled.n_tasks},
        "gate": d.to_dict(),
        "paired_n": len(deltas or []),
        "verdict": verdict,
        "recommendation": recommendation,
        "next": ("scripts/commit.py --decision accept" if recommendation == "promote" else
                 f"scripts/grow.py --growth-round {args.growth_round + 1}" if recommendation == "grow_again" else
                 "scripts/commit.py --decision reject --reject-basis gate"),
    }, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""host.py — drive this skill's loop from a NON-INTERACTIVE caller (CI, cron, a script).

agent-optimize's loop is prose in ``SKILL.md``, executed by the conversational agent that
ran intake. ``cap-evolve run`` (agent mode) does check + baseline, prints a handoff, and
returns: no algorithm subprocess, no auto-finalize. That is exactly right with a human in
the loop and leaves the algorithm *unavailable* anywhere without one — a CI job gets the
handoff and then nothing happens.

This script is the missing host, and deliberately owns as little as possible. Everything it
borrows, and from where:

  * the **loop** stays in ``SKILL.md`` + this dir's helpers. The briefing points at them; it
    does not restate the algorithm, because a second copy of the loop would drift from the
    first and there would be no way to tell which one ran.
  * the **CLI invocation** goes through ``optimizers/run-optimizer``, which already resolves a
    registry row, substitutes ``{model}``, maps ``--budget``/``--usd-budget`` to that CLI's own
    flags, captures cost from its JSON output, and hard-fails when the CLI is absent — and
    whose ``load_registry`` also answers "is this a known agent?".
  * the **read-context** is ``harness.OptimizerContext``: ``inject()`` stages the capability
    skills, the diagnose method, the sources and the trajectories exactly as every
    deterministic algorithm gets them, and ``capability_brief()`` / ``reader_brief()`` /
    ``empty_seed_brief()`` supply the measured prompt blocks. This script previously
    hand-rolled thinner equivalents, and the consequence was measurable: with
    ``_CAP_EDIT_SPACE``'s "the highest-leverage edit is a new code-bearing tool" and the
    target-reader block both absent, the hosted optimizer only ever edited prose.
  * the **spec resolution** for ``optimizer_instructions_file`` is
    ``specfile.resolve_instructions_file``, shared with ``cli.py``. Two copies of a path
    resolution rule is how #252 happened.
  * the **seal** is ``measure.py``, the same script the skill documents.

What it does NOT borrow: ``OptimizerContext.instructions()``. That renders the per-iteration
contract — "fix many root causes in this ONE candidate and STOP; the harness re-scores you" —
which is false here, where the agent owns the search, the evaluation and the gate. The blocks
are composed instead, and an arm's own instructions template is included with its scope stated.

What is genuinely this script's own:

**A raised Bash-tool ceiling.** Every loop command is a shell call, and a full-val eval on
a real benchmark runs for hours. Claude Code caps one Bash call at ``BASH_MAX_TIMEOUT_MS``
(default 600000 = 10 min) and the effective ceiling is ``max(default, max)``, so both are
raised. Left alone, every eval is killed mid-flight and an entirely healthy run reads as a
broken runner.

**A guaranteed seal.** An agent that exhausts its turns or dies mid-loop leaves no
``final.json``. CI then cannot distinguish "stopped early" from "a step crashed", and there
is no honest number at all. So if the agent did not seal, the host does — through the same
``measure.py`` the skill documents, labelled ``seal: host`` so nobody mistakes it for the
agent's own judgement that it was finished.

The seal is idempotent: an already-sealed run reports ``seal: agent`` rather than raising
``TestSealError`` out of the host and failing a run that is actually complete.

**A foreground contract, and a backstop for when it is broken.** The turn budget is not the
only way an unattended loop stops short. On run 32814848187 the agent used 78 of 600 turns and
stopped on ``subtype: success`` / ``stop_reason: end_turn``: it had backgrounded round 2's gate
and ended its turn to await a notification, which ends the process here — there is no
conversation to resume it. So the briefing states the invariant (the turn that launches work is
the turn that collects it; delegate the work, never the waiting — subagents and parallel evals
stay encouraged), and ``unbooked_rounds`` reports candidates a round gated to a verdict that no
``commit.py`` booked, since ``spent.iterations`` cannot tell that from a round never attempted.
It reports rather than books: booking an accept after ``measure.py`` sealed against the old
``best_id`` would convert a visible gap into a wrong headline number.

That check runs AFTER the seal on purpose — the abandoned round's evals outlived the agent by
14 minutes, so a pre-seal check would have found an empty ``work/``.

**A heartbeat, for when THIS process dies rather than the CLI it launches.** Everything above
covers the hosted CLI stopping short while host.py itself stays alive to notice. Nothing here
covers host.py's own OS process dying — the machine sleeping, the terminal it ran in closing —
which is a distinct failure `watchdog.py` (this dir) exists to catch from outside: it reads
``host/heartbeat.json``, written every ``HEARTBEAT_INTERVAL_SECONDS`` while a CLI invocation is
in flight, and relaunches host.py against the same run dir when the heartbeat goes stale and no
process still holds its pid. Re-running host.py is already safe to do by hand (``commit.py``
refuses to double-book a decided candidate, ``_seal`` is idempotent) — the watchdog only
automates noticing and doing that, using ``host/launch_args.json`` (written on every launch) to
reconstruct the original command line. It is a process supervisor, not a resume of a hung
turn: see the briefing's Unattended section for why a turn that ends with work outstanding can
never be resumed from inside the conversation.
"""

from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
import threading
import time
from pathlib import Path

import _bootstrap  # noqa: F401

HERE = Path(__file__).resolve().parent
SKILL_DIR = HERE.parent
SKILLS = SKILL_DIR.parents[1]
RUN_OPTIMIZER = SKILLS / "optimizers" / "run-optimizer" / "scripts" / "run.py"
REGISTRY = SKILLS / "optimizers" / "registry.yaml"

#: `terminal_reason` values that name a TRANSIENT infra failure (network/DNS/rate-limit),
#: not a genuine stop. Deliberately narrow — see #430's non-goal: `error_max_turns` and any
#: other real stop condition must never be retried, only this small allow-list.
_RETRYABLE_TERMINAL_REASONS = {"api_error"}

#: #431: ``permission_denials`` entries whose ``tool_input`` shape says the driver tried to
#: DETACH a wait rather than stay in the foreground — a persistent ``Monitor``, or a
#: ``tail -f`` / ``nohup`` / backgrounding-shaped Bash command. ``registry.yaml``'s
#: ``--disallowedTools Monitor`` now makes the Monitor case structurally impossible for
#: claude-code specifically, but ``permission_denials`` is read from the CLI's own payload
#: regardless of agent or registry row, so a near-miss on any other tool/CLI is still flagged
#: instead of only visible in a raw transcript nobody was going to read.
_DETACH_COMMAND_PATTERNS = ("tail -f", "nohup ", "disown", "setsid ")


def _looks_like_a_detach_attempt(denial) -> bool:
    if not isinstance(denial, dict):
        return False
    if str(denial.get("tool_name") or "") == "Monitor":
        return True
    tool_input = denial.get("tool_input")
    if isinstance(tool_input, dict):
        if tool_input.get("persistent") is True:
            return True
        command = str(tool_input.get("command") or "")
        if any(pat in command for pat in _DETACH_COMMAND_PATTERNS):
            return True
    return False


def _backgrounding_near_misses(permission_denials) -> list[dict]:
    """``permission_denials`` entries shaped like a detached-wait attempt, not a plain denial."""
    if not isinstance(permission_denials, list):
        return []
    return [d for d in permission_denials if _looks_like_a_detach_attempt(d)]


# 4h. Long enough for a full-val eval on the slowest benchmark in this repo
# (spreadsheetbench full: one Docker container per task x trials), while still bounding a
# genuinely hung command instead of waiting forever.
BASH_TIMEOUT_MS = 4 * 60 * 60 * 1000
#: How often, while a CLI invocation is in flight, host.py touches ``host/heartbeat.json``.
#: ``watchdog.py`` treats anything older than a few multiples of this as evidence the
#: process died rather than just being between writes.
HEARTBEAT_INTERVAL_SECONDS = 60


def _write_heartbeat(run_dir: Path, *, pid: int) -> None:
    """Best-effort liveness marker for ``watchdog.py``. Never raises: a missed write is not
    worth failing an otherwise-healthy run over, and the watchdog already tolerates a
    heartbeat a few intervals stale.
    """
    path = run_dir / "host" / "heartbeat.json"
    try:
        path.parent.mkdir(parents=True, exist_ok=True)
        tmp = path.with_suffix(".tmp")
        tmp.write_text(json.dumps({"pid": pid, "ts": time.time()}), encoding="utf-8")
        tmp.replace(path)
    except OSError:
        pass


def _heartbeat_loop(run_dir: Path, pid: int, stop: threading.Event) -> None:
    while not stop.wait(HEARTBEAT_INTERVAL_SECONDS):
        _write_heartbeat(run_dir, pid=pid)


def _save_launch_args(run_dir: Path, argv: list[str]) -> None:
    """Persist this invocation's CLI args so ``watchdog.py`` can reconstruct the exact
    command line on a relaunch, without a human remembering ``--agent``/``--model``/etc.
    """
    try:
        path = run_dir / "host" / "launch_args.json"
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps({"argv": argv, "python": sys.executable}, indent=2),
                        encoding="utf-8")
    except OSError:
        pass
#: Measurement concurrency handed to the agent when the spec names none. Matches ``round.py``'s
#: own default and its refusal bound; see the concurrency note in the briefing.
GATE_CONCURRENCY = 8


def _spec(project: Path, spec_path: Path | None) -> dict:
    from cap_evolve.specfile import read_yaml

    path = spec_path or (project / "capevolve.yaml")
    if not path.exists():
        return {}
    return read_yaml(path.read_text(encoding="utf-8")) or {}


def _known_agents() -> list[str]:
    """Registry rows, read by run-optimizer's own loader rather than a second parser."""
    try:
        sys.path.insert(0, str(RUN_OPTIMIZER.parent))
        import run as _run_optimizer  # run-optimizer/scripts/run.py

        return sorted((_run_optimizer.load_registry() or {}).keys())
    except Exception:  # noqa: BLE001 — a missing registry is reported by the caller below
        return []


def _editable_files(run_dir: Path, project: Path, spec: dict) -> list[str]:
    """The capability's real files, relative — the surface the agent may actually edit.

    ``capabilities`` names which capability skills' ``validate()`` runs; it does NOT restrict
    what may be written. Handing the agent only those names is what leaves most of the
    surface untouched: in one measured run a spec declaring both a prompt capability and a
    tool-code capability produced 2 of 2 candidates that edited only the prompt file, while
    the tool code sat writable and unopened in the same candidate dir.

    Read from the materialized seed candidate (what the agent copies and edits) rather than
    from the project's capability_path, so this is the same file set it will really see.
    """
    root = run_dir / "candidates" / "seed"
    if not root.is_dir():
        cap = str(spec.get("capability_path") or "seed_capability")
        root = (project / cap).resolve()
    if not root.is_dir():
        return []
    skip_dirs = {"__pycache__", ".git"}
    files = [
        p.relative_to(root).as_posix()
        for p in sorted(root.rglob("*"))
        if p.is_file()
        and not any(part in skip_dirs for part in p.parts)
        and p.suffix not in {".pyc", ".pyo"}
    ]
    return files


#: Suffixes that make a capability file CODE rather than prose. Drives whether the briefing
#: offers code-vs-prose advice at all — see _surface_section.
#:
#: KNOWN-GOOD, NOT EXHAUSTIVE. A missing suffix means the code-guard advice silently does not
#: fire, which is the same silent-miss this host was fixed for on ``.py``/``.js`` — just
#: relocated to whichever language nobody listed. It started at 14 entries and therefore
#: treated C, C++, C#, PHP, Swift, Kotlin and Objective-C surfaces as prose. So: ADD to this
#: set freely when a workload brings a new language; absence here is a gap, never a decision
#: that the language is prose.
#:
#: A suffix list is deliberately kept as the test rather than the capability's declared kind.
#: Kind is only a proxy — a ``tools`` capability can be schema-only and a ``system-prompt`` one
#: can ship a helper script — whereas the question the briefing actually asks is "does the
#: surface I am handing you contain code you could put a guard in".
_CODE_SUFFIXES = {
    # scripting / dynamic
    ".py", ".js", ".mjs", ".cjs", ".ts", ".tsx", ".jsx", ".rb", ".pl", ".lua", ".php",
    ".r", ".jl", ".dart", ".groovy", ".tcl",
    # shell
    ".sh", ".bash", ".zsh", ".fish", ".ps1",
    # compiled / systems
    ".c", ".h", ".cc", ".cpp", ".cxx", ".hpp", ".hh", ".cs", ".go", ".rs", ".java", ".kt",
    ".kts", ".swift", ".m", ".mm", ".scala", ".zig", ".nim", ".d", ".f90", ".vb",
    # functional
    ".ex", ".exs", ".erl", ".hs", ".ml", ".mli", ".clj", ".cljs", ".fs", ".fsx", ".rkt",
    ".scm", ".lisp", ".el",
    # query / template languages that carry real logic
    ".sql", ".vim",
}
#: Above this, the listing is grouped instead of enumerated. A skill-package capability can
#: be dozens of files, and a wall of paths crowds out the rest of the briefing.
_MAX_LISTED = 20


def _surface_section(files: list[str]) -> str:
    """Render the editable-file list, and say what a prompt-only edit leaves undone.

    Scaled to what is actually there, in two ways that matter for genericity:

    * **Code advice only when there is code.** A capability of two prose files (a system
      prompt plus a task template) told to "prefer an in-code guard" goes looking for code it
      does not own — the failure that once had a prompt-only optimizer editing ``adapter.py``.
    * **Grouped, never silently truncated, when large.** A skill-package capability can run to
      dozens of files. Bounding the list is fine; bounding it without saying so is not, since
      the reader then takes it as the complete surface.
    """
    if not files:
        return ""
    if len(files) == 1:
        return (f"## Your editable surface — one file\n\n- `{files[0]}`\n\n"
                "That file is the whole capability. There is no second surface, so do not go "
                "looking for one outside it.\n")

    has_code = any(Path(f).suffix in _CODE_SUFFIXES for f in files)

    if len(files) <= _MAX_LISTED:
        listing = "\n".join(f"- `{f}`" for f in files)
        head = f"## Your editable surface — ALL {len(files)} of these files\n\n{listing}\n"
    else:
        groups: dict[str, list[str]] = {}
        for f in files:
            parts = f.split("/")
            groups.setdefault(parts[0] if len(parts) > 1 else ".", []).append(f)
        rows = []
        shown = 0
        for g, gf in sorted(groups.items()):
            examples = ", ".join(f"`{x}`" for x in gf[:3])
            more = f", +{len(gf) - 3} more" if len(gf) > 3 else ""
            shown += min(len(gf), 3)
            label = f"`{g}/`" if g != "." else "top level"
            rows.append(f"- {label} — {len(gf)} file(s): {examples}{more}")
        head = (
            f"## Your editable surface — {len(files)} files\n\n" + "\n".join(rows) + "\n\n"
            f"Grouped because there are {len(files)}; {len(files) - shown} are not listed "
            f"individually above. **Enumerate the full set yourself** (`find` the candidate "
            f"dir) before deciding a file is out of scope — everything under it is editable, "
            f"not only the files named here.\n")

    body = (
        "\nEvery one of them is in your candidate copy and every one is fair game — prose, "
        "code, data, nested files alike. A round that changes **only the obvious prompt "
        "file** leaves the rest of the agent's instruction and behaviour surface exactly as "
        "it was, and that is the most common way a run produces nothing: the fix that was "
        "needed lived in a file nobody opened.\n\n"
        "So before you write an edit, decide *which file* is the right place for it — the "
        "allowed edit space per capability, and which form has the most leverage on each, is "
        "in the capability brief below rather than restated here. Name the file you chose in "
        "the `commit.py --note` for the round, so the run records which surface each decision "
        "was made on.\n\n"
             "### Precondition on round 3 and later\n\n"
             "**If your last two rounds were both rejected, the next round may not reuse the "
             "surface *and* form those two used.** Read the rejected candidate's trace first "
             "and ask whether the agent ever exercised your rule at all: never exercised means "
             "the FORM was wrong, so a third variation of the same wording will be rejected "
             "too. Change the form, or change the surface")
    if has_code:
        body += (" — and where the failing behaviour is one the agent has a criterion for and "
                 "violates anyway (it *should* call a tool and does not, it *should* validate "
                 "and does not), the form that works is a guard in the code, not a third "
                 "restatement in prose")
    body += (".\n\nTwo rejections are not a reason to stop — they are the signal to escalate. "
             "Spend every round the budget allows.\n")
    return head + body


def _arm_section(text: str) -> str:
    """Include the arm's own instructions — with their scope stated, not silently merged.

    That file was authored for the DETERMINISTIC per-iteration optimizer, so its process half
    actively contradicts this loop: it says to stop after editing and not to evaluate, because
    there the harness re-scores the candidate. Here the agent owns the evaluation and the gate,
    and an agent obeying that line would never gate anything.

    What is uniquely valuable in it is the benchmark's own constraints — which files are
    editable, which tokens are load-bearing, what silently zeroes a score. So the precedence is
    stated explicitly instead of leaving the agent to guess which half to follow.
    """
    if not text.strip():
        return ""
    return (
        "## Benchmark-specific instructions for THIS capability\n\n"
        "Authored for this project. Read them for the **benchmark's own facts and "
        "constraints** — which files are editable, which tokens are load-bearing, what "
        "silently zeroes a score. Those are measured on this benchmark, are repeated nowhere "
        "else in this briefing, and are binding.\n\n"
        "**Scope, because they were written for the other loop:** they address a per-iteration "
        "optimizer that proposes one edit and stops while the harness scores it. You own the "
        "whole search, so anything in them about stopping after an edit, not evaluating, or "
        "iteration budget does NOT apply — the loop in SKILL.md and this briefing wins there. "
        "On benchmark facts, they win.\n\n"
        "<arm_instructions>\n" + text.strip() + "\n</arm_instructions>\n")


def _shared_blocks(ctx, run_dir: Path, context: dict) -> str:
    """The prompt blocks the DETERMINISTIC path has always had, from the same source.

    Not re-authored here. Every deterministic algorithm gets these through
    ``OptimizerContext``; agent mode used to hand-roll thinner equivalents, and the measured
    consequence was an optimizer that only ever edited prose — because the block naming tool
    code as the highest-leverage surface (``harness._CAP_EDIT_SPACE``) and the block saying a
    weak reader needs code enforcement over terse prose (the target-reader profile) were both
    absent. Reusing them is the fix; writing better prose here would not have been.
    """
    parts = []
    brief = ctx.capability_brief()
    if brief:
        parts.append(brief)
        if context.get("staged"):
            parts.append("Each `./guidance/<cap>/SKILL.md` above is staged in your working "
                         "directory and also under the native skills dir. **Read the one for "
                         "the surface you are about to edit** — an edit made without it is a "
                         "guess, and the surface you have no guidance for is the one you will "
                         "avoid by default.")
    reader = ctx.reader_brief()
    if reader:
        parts.append(reader)
    empty = ctx.empty_seed_brief(run_dir / "candidates" / "seed")
    if empty:
        parts.append(empty)
    return "\n\n".join(p.strip() for p in parts if p and p.strip())


def _briefing(*, run_dir: Path, project: Path, spec: dict, skills: Path,
              rounds: int, workdir: Path, context: dict, arm: str = "",
              ctx=None) -> str:
    """The driver briefing: the handoff facts, then a pointer to the loop itself.

    Deliberately NOT a restatement of the algorithm. SKILL.md is the implementation and the
    agent reads it; what it cannot know are the paths, the spec values, the shared prompt
    blocks, and the fact that nobody is available to answer a question.
    """
    stop = str(spec.get("stop_condition") or "").strip()
    n_trials = spec.get("num_trials", 1)
    k_se = spec.get("gate_k_se", 1.0)
    gate_mode = spec.get("gate_mode", "paired")
    caps = spec.get("capabilities") or []
    cap_path = spec.get("capability_path") or "seed_capability"
    surface = _surface_section(_editable_files(run_dir, project, spec))
    guidance = _shared_blocks(ctx, run_dir, context) if ctx is not None else ""
    arm_block = _arm_section(arm)
    skill_md = SKILL_DIR / "SKILL.md"
    helpers = HERE
    # Quoted from the constant that actually sets BASH_*_TIMEOUT_MS below, so the briefing
    # cannot promise a ceiling the env does not grant.
    hours = round(BASH_TIMEOUT_MS / 3_600_000, 1)
    hours = int(hours) if hours == int(hours) else hours
    # The interpreter that launched us — see _agent_env for why this is the authority.
    interpreter = sys.executable
    gate_conc = int(spec.get("measure_concurrency") or GATE_CONCURRENCY)

    if not stop:
        stop = (f"Spend at most {rounds} rounds, gate every candidate on FULL val, and "
                "finish by sealing test exactly once with measure.py.")

    return f"""# Drive the agent-optimize loop on an existing run — unattended

You are the optimizer for a cap-evolve run that is ALREADY set up and baselined.
`cap-evolve run` finished check + baseline and handed the loop over. Your job is to run the
`agent-optimize` loop against it and finish with one honest sealed measurement.

## Read this first

`{skill_md}`

That file IS the algorithm — its "Agent-mode loop" section is what you execute, step by
step, including Phase 0. Its helper scripts are in `{helpers}`. Do not re-derive the loop
from this briefing; this briefing only gives you the facts SKILL.md cannot know.

## The handoff

```bash
R="{run_dir}"      # the run dir: splits, baseline, candidates, rollouts, events
P="{project}"      # the project: capevolve.yaml, adapters/, {cap_path}
S="{skills}"       # the skills dir (CAPEVOLVE_SKILLS_DIR is already set to it)
A="$S/algorithms/agent-optimize/scripts"
PY="{interpreter}"      # the ONLY interpreter that can import this benchmark's adapter deps
mkdir -p "$R/work"
```

Paths are absolute; use them as given rather than relative paths, because your working
directory is not necessarily either of theirs.

`$PY` is already first on your `PATH`, so plain `python` resolves to it and SKILL.md's commands
work as written. Use `"$PY"` explicitly anywhere you build a command yourself. Do **not**
substitute another interpreter, `uv run`, or a fresh venv: the adapter's packages are
installed into this one only, and an eval run under any other dies with `ModuleNotFoundError`
on the adapter's imports — which scores the candidate `null`, not zero, and wastes the round.

## The spec values your gate needs

| key | value |
| --- | --- |
| `num_trials` | {n_trials} |
| `gate_k_se` | {k_se} |
| `gate_mode` | {gate_mode} |
| `capabilities` (which capability rules validate your edits) | {caps} |
| `capability_path` | {cap_path} |
| `--concurrency` for every gate | {gate_conc} |

Pass these explicitly — `--n-trials {n_trials}` on every evaluate, `--k-se {k_se}` on every
gate — rather than relying on a default that may not match this spec.

**The concurrency is a measurement parameter, not a speed dial.** Measured on this benchmark,
byte-identical code at identical seeds moves ~0.03 at concurrency 8 and ~0.08 above 25 — so a
gate run hot cannot resolve the effect you are looking for, and `round.py` now refuses a value
that coarse rather than warning about it. Buy wall clock with fewer candidates per round, never
with concurrency.

{surface}
{guidance}
{arm_block}

## Default to 3+ candidates per round

**Default to proposing 3 sibling candidates per round via `round.py`'s parallel mode** —
address different failure clusters in parallel, not one at a time — unless `spend.py
--n-siblings N` says your remaining budget can't afford it. A round's fixed overhead
(baseline + null-control replicates) is paid regardless of how many candidates it gates, so
one candidate per round wastes most of it on a single shot at the gate.

## The primitives every round must go through

SKILL.md says why; this is the checklist, because nobody is watching and a round that
skipped one leaves artifacts that cannot be audited afterwards:

| helper | per round | what it is for |
| --- | --- | --- |
| `$A/spend.py` | before | affordability + your stop condition as checkable predicates |
| `$A/gate_check.py` | after the full-val eval | the paired significance gate — the accept decision |
| `$A/commit.py` | always, whatever the outcome | books the decision: snapshot, best_id, iteration, event |
| `$A/measure.py` | once, at the end | seals test exactly once and prints the honest table |

`commit.py` is the one most easily skipped on a reject, and skipping it is what makes a run
report zero iterations having done real work. `screen.py` and `round.py` are optional
accelerators; the four above are not.

`--decision` has THREE values, and the third is not a formality. `accept` = new champion.
`reject` = the edit was judged and refuted. `inconclusive` = the measurement could not resolve it
— which is exactly what `round.py` reports as `verdict: inconclusive` (`verdict_stable: false`,
the verdict flipping depending on which byte-identical control replicate was the reference). Book
that as `inconclusive`, not as a reject:

- a reject increments the STALL counter, and stall is the signal that means *the optimizer has run
  out of ideas* — the one thing an ambiguous measurement is no evidence of. Two ambiguous rounds
  booked as rejects can end your run early for a reason that never happened.
- a reject files the edit in `rejected.jsonl`, which later rounds read as *this was tried and it
  did not work*. An edit nothing could judge has not been tried in that sense; filing it there
  teaches you to avoid your own untested idea.
- `inconclusive` still charges the iteration (the budget really was spent) and still snapshots the
  candidate, so nothing is hidden. To resolve it, re-measure under a **fresh tag** — re-running the
  same tag REPLACES its rollouts rather than adding to them. The control side needs no care: a
  re-gate of the same iteration measures its own `ctl_null_i<N>a<k>` replicates and pools the
  earlier attempt's, so `null_delta_between_control_replicates` covers every replicate the round
  has paid for and the earlier attempt's table stays on disk beside the new one.

`--reject-basis gate` asserts the gate ran AND rejected; `commit.py` refuses it when the gate
accepted or returned inconclusive, because that field is the run's record of what the evidence was.

## Your stop condition

{stop}

`spend.py` parses that text into checkable predicates; run it before each round and act on
its single `recommendation` (`stop` | `narrow_scope` | `continue`), as SKILL.md describes.

## `LEDGER.md`, `RUNMAP.md`, `prior_iterations/`

Your always-on instructions mention these. They are real here too — `seed_framework_memory`
builds them for this loop the same as for the deterministic one, so read them; do not assume
they're a deterministic-loop-only artifact. `rejected.jsonl` and `history.jsonl` sit next to
them and are worth reading as well — they hold every prior candidate's real outcome, not just
what made it into `JOURNAL.md`'s prose.

`JOURNAL.md` is different, and it has TWO halves — one of them is yours to write.

- The FRAMEWORK half: `commit.py` stamps an objective `RESULT` line under each entry (outcome,
  Δ, and the exact task ids that round broke and fixed). You get that for free.
- YOUR half, the handover: **before each `commit.py`, append your entry for the round to
  `<your working dir>/JOURNAL.md`** — the same `--from-dir` you are about to commit — as a
  block starting `## Iteration <candidate id> — <one-line headline>`, covering: the changes you
  made (file + cluster each targets), the effect you expected and why it was safe, which prior
  RESULT lines you built on, hypotheses a prior RESULT has already REFUTED (never re-test one),
  and your focus next round. Read the accumulated `$R/JOURNAL.md` before writing, so you build
  on every prior round rather than the last one.

Skipping your half is silent and cheap in the moment and expensive by round 3: the run-level
journal records "(no handover written by the optimizer)", and your later rounds can then see
WHICH tasks each edit broke but not WHAT WAS TRIED — so refuted ideas get re-tested with the
budget that should have gone to new ones. Measured: three-round runs where every entry read that
way. `commit.py` returns `handover_recorded` and warns when it books an empty one; if you see
that warning, write the entry before the next round rather than at the end of the run.

## Unattended — this is the one real difference from an interactive run

**Nobody is available to answer a question. Do not ask any; do not wait for input.** Where
SKILL.md's Phase 0 says to ask the user about a blocking ambiguity (including
`constraints.ambiguous` from `spend.py`), instead: pick the most conservative reading, state
the assumption in one line in your final summary, and proceed. A round spent on a
conservative assumption is worth far more than a run that stalls waiting for a reply.

Three consequences worth being explicit about:

1. **Never leave the run unsealed.** Finish with `measure.py` (which seals test exactly
   once) and the report phase, as SKILL.md's "Stop & seal" section shows. A run with no
   finalize has no result. If you are running out of budget, stop optimizing and seal —
   sealing what you have beats one more candidate.

   `measure.py` is the *last* long-running eval you launch — it opens the sealed test split
   (`eval_start(split=test, tag=FINAL)`), and this seal is single-use. Rule 3 below applies
   here MOST of all: stay in the foreground until it exits. Ending your turn while it is
   still running does not just lose the number — the abandoned attempt's partial rollouts
   then make even a RETRY refuse (`begin_test_attempt` sees test already has rollouts on it),
   so the seal is wasted, not merely delayed. Measured on three separate runs: an
   `eval_start(split=test, tag=FINAL)` with no matching `evaluate` and no `final.json` ever
   written.
2. **A null result is a valid outcome, honestly reported.** If nothing beat the baseline
   through the gate, say so and seal anyway. Do not lower the gate, gate on a screen
   subset, or present a screen `promote` as an accept to manufacture a gain.
3. **Drive the loop from the foreground, and never end a turn with work outstanding.**
   There is no conversation to come back to. When you end a turn with no tool call pending,
   this process exits and everything it started is orphaned — so anything that would report
   back *later* never reports at all: a job left running in the background, a watcher on a
   file, a completion notice, a wake-up you scheduled. There is nobody to wake.

   This is not a rule against doing several things at once. Fan out as widely as the work
   deserves — subagents, parallel diagnosers, a whole round's candidates evaluated
   concurrently (that is exactly what `round.py` is for). The one invariant is that **the
   turn that launched the work is still the turn that collects it**: stay blocked until the
   result is in your hands, read it, and act on it before that turn ends. Delegate the work,
   never the waiting.

   Waiting is safe: one Bash call may run for {hours} hours, a ceiling raised for precisely
   this reason, so a long eval does not need backgrounding to survive. If something really
   would outlast that, make it smaller — fewer trials, fewer candidates per round — rather
   than detaching it.

   Measured on run 32814848187: the driver backgrounded round 2's full-val gate and ended
   its turn to await a notification. The process exited; the gate finished 14 minutes later
   and wrote a real verdict that nobody was left to read. Two of three rounds went unspent,
   and the orphaned evals were still hitting the runner while the seal was being measured.

## When you are done

Finish your final message with the run's honest table: seed vs best on val, on train if it
adds information, and on the sealed test split — plus the accepted candidate id, the number
of rounds, and any assumption you had to make on your own.
"""


def _decided_candidates(run_dir: Path) -> set[str]:
    """Candidate tags that already carry an accept/reject decision in ``events.jsonl``.

    Same source and same event names ``commit.py`` writes and re-reads for its own
    double-booking guard — the audit log rather than in-process memory, because the driver
    that booked the decision is a different process that has already exited.
    """
    decided: set[str] = set()
    try:
        with (run_dir / "events.jsonl").open(encoding="utf-8") as f:
            for line in f:
                try:
                    ev = json.loads(line)
                except Exception:  # noqa: BLE001 — a torn line is not a decision
                    continue
                # ``inconclusive`` is a booking too. Leaving it out would report a round booked
                # with the honest decision as gated-but-never-booked, i.e. diagnose the run as
                # defective precisely for not misfiling an unresolvable verdict as a reject.
                if ev.get("kind") in ("accept", "reject", "inconclusive") and \
                        ev.get("candidate"):
                    decided.add(str(ev["candidate"]))
    except OSError:
        return decided
    return decided


def _unbooked_rounds(run_dir: Path) -> list[dict]:
    """Candidates a round GATED but nobody booked with ``commit.py``.

    ``spent.iterations`` counts ``commit.py`` calls, so a round whose full-val gate ran to a
    verdict and was then abandoned is indistinguishable from a round never attempted — the
    operator is left diffing candidate dirs by hand to find out which. Measured on run
    32814848187: ``r2_comm_search`` was gated to ``reject`` at val 0.44 against parent 0.58,
    the table was on disk, and the run reported 1 of 3 rounds with no hint the second existed.

    Recognised by SHAPE, not by filename: ``round.py`` prints its table to stdout and the
    driver chooses where to redirect it, so matching ``round*.log`` would only ever catch the
    one name that happened to be used. Any file under ``work/`` that parses as a round table
    counts.

    Deliberately reports rather than books. Which decision a verdict deserves is the driver's
    judgement — ``round.py``'s own docstring says so — and a host that booked accepts on its
    behalf would move ``best_id`` after ``measure.py`` had already sealed against the old one,
    turning a visible gap into a wrong headline number.
    """
    work = run_dir / "work"
    if not work.is_dir():
        return []
    decided = _decided_candidates(run_dir)
    seen: set[str] = set()
    found: list[dict] = []
    for log in sorted(work.iterdir()):
        if not log.is_file() or log.suffix not in (".log", ".json", ".txt"):
            continue
        try:
            payload = json.loads(log.read_text(encoding="utf-8"))
        except Exception:  # noqa: BLE001 — not a round table; nothing to say about it
            continue
        if not isinstance(payload, dict) or not isinstance(payload.get("candidates"), list):
            continue
        for cand in payload["candidates"]:
            if not isinstance(cand, dict):
                continue
            tag = str(cand.get("tag") or "")
            # A row with no verdict never reached a decision anyone could have skipped
            # (a crashed eval, or a table written before the gate ran).
            if not tag or tag in decided or tag in seen or not cand.get("verdict"):
                continue
            seen.add(tag)
            found.append({"candidate": tag, "verdict": cand.get("verdict"),
                          "reward": cand.get("reward"),
                          "parent": (payload.get("parent") or {}).get("tag"),
                          "log": log.name})
    return found


def _dangling_eval(run_dir: Path) -> dict | None:
    """The last ``eval_start`` with no matching ``evaluate`` for the same (split, tag).

    ``harness.py``'s own docstring names the invariant: "an eval_start with no evaluate
    after it is an evaluation that never returned." Measured on three separate runs: the
    driver issued ``eval_start(split=test, tag=FINAL)`` (the sealed test eval `measure.py`/
    `finalize.py` opens with — see ``harness.finalize``) near the end of its turns and the
    process exited before the matching ``evaluate`` was ever logged — no ``final.json``, no
    error, just a Bash call that outlived the turn that launched it. Reported here so the
    host's run summary (and the `incomplete` diagnosis) can say which eval was abandoned
    instead of the operator diffing rollout files by hand.

    Not benchmark-specific: split/tag are read straight off the events, whatever adapter or
    algorithm wrote them.
    """
    starts: dict[tuple, dict] = {}
    try:
        with (run_dir / "events.jsonl").open(encoding="utf-8") as f:
            for line in f:
                try:
                    ev = json.loads(line)
                except Exception:  # noqa: BLE001 — a torn line carries no eval state
                    continue
                if not isinstance(ev, dict):
                    continue
                kind = ev.get("kind")
                key = (ev.get("split"), ev.get("tag"))
                if kind == "eval_start":
                    starts[key] = ev
                elif kind == "evaluate":
                    starts.pop(key, None)
    except OSError:
        return None
    if not starts:
        return None
    # Last one issued, by event time — an earlier dangling start that a later retry of the
    # SAME (split, tag) resolved is not itself abandoned; only the most recent open one is.
    last = max(starts.values(), key=lambda e: e.get("t") or 0)
    return last


def _log_eval_abandoned(run_dir: Path, dangling_eval: dict) -> None:
    """Flag a dangling ``eval_start`` in ``events.jsonl`` so the run's audit log names it
    instead of leaving it discoverable only by hand-diffing eval_start/evaluate pairs."""
    try:
        from cap_evolve import RunDir as _RunDir

        _RunDir.open(run_dir).log_event(
            "eval_abandoned", split=dangling_eval.get("split"),
            tag=dangling_eval.get("tag"), started_at=dangling_eval.get("t"))
    except Exception:  # noqa: BLE001 — flagging the gap must not crash a run report
        pass


def _seal(run_dir: Path, project: Path, spec: dict, *, timeout: float | None) -> dict:
    """Ensure the run has a sealed test number AND the full seed-vs-best bookend
    (train + val + test, both candidates) that ``harness.finalize`` now always writes
    into ``final.json``. Idempotent.

    Returns ``{"sealed": bool, "seal": "agent"|"host"|"failed", ...}``. ``agent`` means
    final.json was already there when the host looked — the normal, desired outcome.

    The bookend guarantee lives in ``harness.finalize`` itself (called from ``measure.py``
    below, and from every other path that ever produces ``final.json``), not here — this
    function only guarantees that SOMETHING calls it. Doing it there rather than here
    means the same guarantee, and the same "reuse rollouts already on disk instead of
    re-measuring" dedup, apply to a run the agent sealed itself, not only to this fallback.
    """
    final = run_dir / "final.json"
    if final.exists():
        return {"sealed": True, "seal": "agent"}

    n_trials = int(spec.get("num_trials", 1) or 1)
    cmd = [sys.executable, str(HERE / "measure.py"), "--run-dir", str(run_dir),
           "--project", str(project), "--n-trials", str(n_trials), "--train", "auto"]
    p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout,
                       env=_child_env())
    if final.exists():
        return {"sealed": True, "seal": "host", "measure_rc": p.returncode}
    return {"sealed": False, "seal": "failed", "measure_rc": p.returncode,
            "measure_error": (p.stderr or p.stdout)[-1200:]}


def _stage_context(*, run_dir: Path, project: Path, workdir: Path, spec: dict,
                   agent: str) -> tuple:
    """Stage the SAME optimizer read-context every deterministic algorithm receives.

    ``harness.OptimizerContext`` exists so "an algorithm cannot silently run on a thinner
    prompt than its siblings": it places the declared capability skills as
    ``./guidance/<cap>/`` *and* where the agent natively discovers skills, plus the diagnose
    method, any ``capability_sources``, and the agent's own features reference.

    Agent mode never called it. ``test_optimizer_context_parity.py`` names that gap in its own
    docstring — this algorithm "declares none of the context flags and drives its own loop", so
    it "can still run blind while this file stays green". It did: measured across two runs, 4
    of 4 candidates edited only the prompt file, because the agent had guidance for prose and
    none at all for the tool code sitting beside it. Naming the files in the briefing did not
    move it, since the file list was never the missing piece.

    Reported rather than swallowed: a run that silently skipped staging is indistinguishable
    from one that staged fine, and simply optimizes less surface.
    """
    caps = tuple(c for c in (spec.get("capabilities") or []) if c)
    from cap_evolve import harness

    # from_spec, not the field-by-field constructor: it also resolves the target profile, so
    # reader_brief() is populated exactly the way the deterministic path's from_args does it.
    ctx = harness.OptimizerContext.from_spec(spec, project_dir=project, optimizer_name=agent)
    try:
        from cap_evolve import RunDir
        from cap_evolve.check import load_adapter

        rd = RunDir.open(run_dir)
        adapter = load_adapter(project)
        # split="val" + the current best's tag: the parent step the agent builds on, which is
        # the same choice the deterministic loop makes.
        ctx.inject(adapter, rd, workdir, split="val", tag=rd.best_id or "seed")

        # Seed the run's continuous JOURNAL.md into the CURRENT BEST candidate's snapshot
        # (baseline's "seed" at round 1) so the agent's first `cp -r "$R/candidates/$BEST"
        # "$R/work/$TAG"` (SKILL.md step 2) carries a marker-terminated JOURNAL.md forward.
        # Without this, agent-optimize's continuous session never gets the per-iteration
        # workdir seed the deterministic loops get from `_augment_instructions`, so
        # JOURNAL.md never exists and every round's handover reads as empty — confirmed live.
        # `commit.py` re-seeds the same way (onto whichever candidate becomes $BEST) after
        # every round, so round 2+ inherits a clean append target automatically. Own
        # try/except: a journal-seed failure must not mark the whole context (guidance,
        # trajectories) unstaged — it is a nicety on top of staging, not staging itself.
        # `seed_framework_memory`, not `_seed_journal` alone: LEDGER.md, RUNMAP.md and
        # prior_iterations/ are named by the staged CLAUDE.md pointer AND by JOURNAL.md's own
        # seed text, and seeding only the journal is what left them absent for a whole run.
        try:
            harness.resolve_memory(ctx.memory_skill).seed(rd.candidate_dir(rd.best_id or "seed"), rd)
        except Exception as exc:  # noqa: BLE001
            rd.log_event("optimizer_context_warning", what="framework_memory",
                         error=str(exc)[:300])

        guidance = (sorted(g.name for g in (workdir / "guidance").iterdir())
                    if (workdir / "guidance").is_dir() else [])
        # A DECLARED capability that got no guidance dir. `harness._stage_context` skips a
        # capability with no matching skill package (`if not src.is_dir(): continue`) and still
        # reports staged, so "some capabilities missing" was indistinguishable from "everything
        # staged" — while the all-missing case has always been loud. Silently optimizing a
        # surface with no guidance is the exact defect this host was fixed for; leaving half of
        # it quiet just moves the blind spot. The staged list was already reported; what was
        # missing is the comparison against what the spec asked for.
        staged = {"staged": True, "capabilities": list(caps), "guidance": guidance,
                  "guidance_missing": [c for c in caps if c not in guidance]}
        try:
            rd.log_event("host_context", capabilities=list(caps), agent=agent, staged=True)
        except Exception:  # noqa: BLE001 — the event is a nicety, the staging is the point
            pass
        return ctx, staged
    except Exception as exc:  # noqa: BLE001
        return ctx, {"staged": False, "capabilities": list(caps),
                     "error": f"{type(exc).__name__}: {exc}"[:400]}


def optimizer_spend_to_book(metered: dict, before: dict, now: dict) -> dict:
    """The share of THIS agent process's metered spend that is not already in the run's state.

    The host meters the whole agent process; the agent books its own proposal cost per round
    through ``commit.py --optimizer-usd/--optimizer-tokens/--optimizer-seconds``, which the skill
    asks it to do. Those are the SAME money — a round's proposal happens inside this process — so
    booking the metered total on top of them reports up to twice the optimizer spend actually
    used, and a run's `max_usd`/cost-based stop condition then fires against a number no one
    spent. Book the residual instead.

    ``before``/``now`` bracket THIS invocation: a ``--resume`` run carries an earlier host's
    optimizer spend in the same counter, and that is not this agent's attribution to net out.
    Each role is netted independently, and never below zero — an agent that over-attributes
    (guessing its own cost high) must not subtract from another role or from the run total.
    """
    out = {}
    for key in ("usd", "tokens", "seconds"):
        booked_by_agent = max(0, (now.get(key) or 0) - (before.get(key) or 0))
        out[key] = max(0, (metered.get(key) or 0) - booked_by_agent)
    return {"usd": float(out["usd"]), "tokens": int(out["tokens"]),
            "seconds": float(out["seconds"])}


def _child_env() -> dict:
    env = dict(os.environ)
    env.setdefault("CAPEVOLVE_SKILLS_DIR", str(SKILLS))
    return env


def _agent_env(model: str | None) -> dict:
    """The environment the hosted agent runs in — reported so a test can assert it."""
    env = {
        "CAPEVOLVE_SKILLS_DIR": str(SKILLS),
        # Both, because the effective ceiling is max(default, max): raising only one leaves
        # the other as the real limit.
        "BASH_DEFAULT_TIMEOUT_MS": str(BASH_TIMEOUT_MS),
        "BASH_MAX_TIMEOUT_MS": str(BASH_TIMEOUT_MS),
        # THE interpreter, first. An arm's adapter deps are installed into exactly one venv, and
        # `cap-evolve run` uses it — which is why on run 32861747778 the baseline scored 0.44
        # while every candidate eval died `ModuleNotFoundError` on the adapter's own imports:
        # CI's PATH never contains that venv's bin, and SKILL.md tells the agent to run
        # `python "$A/round.py"`. Bare `python` could therefore never resolve to the one that
        # works, and the run before it had survived on luck.
        #
        # Fixed here rather than by rewriting SKILL.md's commands: the interpreter that launched
        # this host IS the correct one (run_suite.sh invokes `"$PY" host.py`), so putting its bin
        # dir first makes every existing `python ...` line correct by construction. Prose the
        # agent must remember is the form that already failed.
        "PATH": os.pathsep.join([str(Path(sys.executable).parent),
                                 os.environ.get("PATH", "")]).rstrip(os.pathsep),
    }
    if model:
        env["CAPEVOLVE_OPTIMIZER_MODEL"] = model
    return env


def main(argv=None) -> int:
    p = argparse.ArgumentParser(
        prog="host.py",
        description="Drive the agent-optimize loop headlessly against a baselined run dir.")
    p.add_argument("--run-dir", required=True, help="run dir from the agent-mode handoff")
    p.add_argument("--project", required=True, help="project dir (capevolve.yaml, adapters/)")
    p.add_argument("--spec", default=None,
                   help="spec file; defaults to <project>/capevolve.yaml")
    p.add_argument("--agent", default="claude-code",
                   help="host agent: a row in optimizers/registry.yaml (default claude-code)")
    p.add_argument("--model", default=None, help="model for the host agent")
    p.add_argument("--budget", type=int, default=None,
                   help="whole-loop turn cap, mapped to the CLI's own budget flag")
    p.add_argument("--usd-budget", type=float, default=None,
                   help="whole-loop $ cap, mapped to the CLI's native flag where it has one")
    p.add_argument("--timeout", type=float, default=None,
                   help="wall-clock seconds for the hosted agent (default: none)")
    p.add_argument("--prompt-only", action="store_true",
                   help="render the briefing and exit without invoking the agent")
    p.add_argument("--seal-only", action="store_true",
                   help="skip the agent; only ensure the run has a sealed test number")
    p.add_argument("--run-optimizer", default=None,
                   help="path to the run-optimizer script (test seam; defaults to the "
                        "sibling optimizers/run-optimizer)")
    p.add_argument("--max-retries", type=int, default=2,
                   help="bounded retries of the optimizer CLI on a transient crash "
                        "(terminal_reason in %s), so a single DNS/network blip does not "
                        "forfeit the rest of the run's budget (default: 2)"
                        % sorted(_RETRYABLE_TERMINAL_REASONS))
    args = p.parse_args(argv)

    run_dir = Path(args.run_dir).resolve()
    project = Path(args.project).resolve()
    if not run_dir.is_dir():
        print(json.dumps({"error": f"run dir not found: {run_dir}",
                          "fix": "pass the run_dir from the agent-mode handoff printed by "
                                 "`cap-evolve run`"}, indent=2))
        return 2
    if not project.is_dir():
        print(json.dumps({"error": f"project dir not found: {project}"}, indent=2))
        return 2

    spec = _spec(project, Path(args.spec).resolve() if args.spec else None)
    rounds = int(spec.get("max_iterations", 0) or 0) or 10

    if args.seal_only:
        out = _seal(run_dir, project, spec, timeout=args.timeout)
        # This is the rescue path an operator reaches for on a run that died mid-loop, so it
        # is the path most likely to be sitting on an abandoned round. Reporting it only on
        # the full host path would hide it from exactly the reader who came looking.
        out["unbooked_rounds"] = _unbooked_rounds(run_dir)
        out["dangling_eval"] = _dangling_eval(run_dir) if not out["sealed"] else None
        if out["dangling_eval"] is not None:
            _log_eval_abandoned(run_dir, out["dangling_eval"])
        print(json.dumps({"run_dir": str(run_dir), "seal_only": True, **out}, indent=2))
        return 0 if out["sealed"] else 1

    # Refuse an unknown host agent BEFORE anything is spent. The registry is where a host
    # agent is actually added, so name it in the fix.
    known = _known_agents()
    if known and args.agent not in known:
        print(json.dumps({
            "error": f"unknown host agent: {args.agent!r}",
            "known": known,
            "fix": f"pass --agent with one of the rows in {REGISTRY}, or add a row there "
                   "(one row per shell-invokable agent CLI — no new code needed)",
        }, indent=2))
        return 2

    # So `watchdog.py` can relaunch this exact invocation without a human reconstructing
    # the flags. Not saved for --prompt-only (a render-and-exit probe, never actually
    # relaunchable) or --seal-only (handled above, before this line is even reached).
    if not args.prompt_only:
        _save_launch_args(run_dir, list(argv if argv is not None else sys.argv[1:]))

    # The agent needs write access to BOTH the run dir and the project; their common parent
    # is the natural workdir, and it is where the staged guidance + native skills land.
    workdir = _common_parent(run_dir, project)
    ctx, context = _stage_context(run_dir=run_dir, project=project, workdir=workdir,
                                  spec=spec, agent=args.agent)

    prompt_dir = run_dir / "host"
    prompt_dir.mkdir(parents=True, exist_ok=True)
    prompt_path = prompt_dir / "driver_prompt.md"
    # ONE resolution rule, shared with cli.py's deterministic path (specfile). Two copies of
    # it is how #252 happened: a relative key resolved against different cwds, and a miss
    # silently downgraded the optimizer to the generic template.
    from cap_evolve.specfile import resolve_instructions_file

    arm_p, arm_exists, arm_warning = resolve_instructions_file(spec, project)
    arm_path = str(arm_p) if arm_exists else ""
    arm_text = ""
    if arm_exists:
        try:
            # Rendered through the shared renderer, not stripped by hand: the arm's template
            # arrives with its blocks filled from the same functions the deterministic path
            # uses, instead of as literal {{SLOT}} braces.
            arm_text = ctx.render_template(arm_p.read_text(encoding="utf-8"),
                                           parent_dir=run_dir / "candidates" / "seed")
        except OSError as exc:
            arm_warning = f"could not read {arm_p}: {exc}"
    briefing = _briefing(run_dir=run_dir, project=project, spec=spec, skills=SKILLS,
                         rounds=rounds, workdir=workdir, context=context, arm=arm_text,
                         ctx=ctx)
    prompt_path.write_text(briefing, encoding="utf-8")
    # Also as <workdir>/INSTRUCTIONS.md. Staging writes an always-on CLAUDE.md pointer whose
    # first instruction is "read ./INSTRUCTIONS.md FIRST" — written for the deterministic
    # per-iteration optimizer, which has one. Agent mode passes its briefing as the prompt, so
    # without this the agent's always-on context opens by pointing at a file that is not there.
    # Same bytes, so the two can never disagree; the run-dir copy stays the audit record.
    if context.get("staged"):
        try:
            (workdir / "INSTRUCTIONS.md").write_text(briefing, encoding="utf-8")
        except OSError as exc:
            context.setdefault("warnings", []).append(f"INSTRUCTIONS.md: {exc}")

    agent_env = _agent_env(args.model)
    if args.prompt_only:
        print(json.dumps({"run_dir": str(run_dir), "agent": args.agent,
                          "model": args.model, "prompt_only": True,
                          "prompt_path": str(prompt_path), "returncode": None,
                          "agent_env": agent_env, "budget": args.budget,
                          "usd_budget": args.usd_budget,
                          "workdir": str(workdir), "context": context,
                          "instructions_file": arm_path,
                          "instructions_warning": arm_warning}, indent=2))
        return 0
    if not context["staged"]:
        print(f"::warning::optimizer context not staged for the hosted agent "
              f"({context.get('error')}) — it will optimize with no capability guidance",
              file=sys.stderr)
    elif context.get("guidance_missing"):
        # Warned, not merely recorded: a field nobody greps is not a report, and this is the
        # partial case of the failure the branch above already shouts about.
        missing = ", ".join(context["guidance_missing"])
        print(f"::warning::agent-optimize: no guidance staged for declared capability/ies "
              f"[{missing}] — no skill package of that name exists under the capabilities "
              f"root, so the agent will edit that surface with no allowed-edit-space brief. "
              f"Check the spelling in capevolve.yaml `capabilities`, or add the skill.",
              file=sys.stderr)

    # Delegate the invocation. --json switches on run-optimizer's cost capture, which is how
    # the host's own spend reaches the run dir at all: the evaluate phase records the
    # runner's cost, and nothing records the proposer's.
    runner = Path(args.run_optimizer).resolve() if args.run_optimizer else RUN_OPTIMIZER
    cmd = [sys.executable, str(runner), "--name", args.agent, "--json",
           # Same workdir the guidance was staged into, so the agent's cwd is where its
           # native skills dir and ./guidance/ live.
           "--workdir", str(workdir),
           "--prompt", str(prompt_path),
           # The loop's own record. Four hours of run 32814848187 were unaccounted for and
           # unaccountable: the only trace kept was an 800-char stdout tail, so what the agent
           # was blocked on could be narrowed to "something that hit the Bash ceiling" and no
           # further. This lands beside driver_prompt.md, so the run dir holds both what the
           # host asked for and what the agent actually did.
           "--transcript", str(prompt_path.parent / "transcript.jsonl")]
    if args.model:
        cmd += ["--model", args.model]
    if args.budget:
        cmd += ["--budget", str(int(args.budget))]
    if args.usd_budget:
        cmd += ["--usd-budget", str(float(args.usd_budget))]

    # Retry loop: one attempt, plus up to --max-retries more when the CLI's OWN payload
    # says it died on a transient infra error (#430). Each attempt re-invokes the SAME cmd
    # against the SAME run_dir — commit.py refuses to double-book a candidate that already
    # has an accept/reject event, so a retry can never re-decide a round the first attempt
    # already committed; it can only

"""integrate — fold N verified branches into ONE artifact by SEQUENTIAL, VERIFIED accumulation.

Why this exists. The obvious way to combine per-task optimiser branches is to merge them all at
once and evaluate the result. That was measured on the one multi-turn tool-use benchmark and it does not
work: every branch had been independently verified to help its own task, `funcmerge.py` retained
all of them cleanly, and the merged artifact then measured **-0.0617** against seed-matched arms
— with the very task whose fix was merged falling from 0.40 to 0.20. Verified per-task gains DO
NOT COMPOSE. A one-shot merge cannot tell you which branch broke the combination, because it
produces a single number for N simultaneous changes.

So merge one branch at a time and MEASURE AFTER EACH. The step that regresses is the step you
drop, and you learn which one it was. This costs N evaluations instead of 1, which is the price
of attribution; it is paid on a task subset (branch targets + canaries), not on full val, so N
steps of this cost less than the single full-val gate round that would otherwise be wasted.

Two disciplines are enforced here because both were violated by hand in earlier rounds:

  1. **Canaries are part of the objective, not a side check.** A branch that lifts its own task
     and quietly drops a task that used to pass at 1.00 is a net loss. Canaries must be drawn
     from tasks that are provably stable — on this benchmark two tasks read 1.00 in five
     separate readings while others swung 0.60 between byte-identical runs, so an unstable task
     used as a canary vetoes good work at random.
  2. **A step delta below the noise floor is not evidence.** Such a step is recorded as
     `kept_provisionally`, never as a gain. Pass --floor with the round's measured null delta
     (see round.py's null_delta_between_control_replicates). Steps inside the floor are kept
     only because they are cheap to carry, and the JSON says so explicitly.

    python integrate.py --base BEST --branches B1 B2 B3 --tasks 7,23,42 \
        --canary 0,1,46 --n 10 --conc 8 --floor 0.0333 --out FINAL

Measurement runs at --conc 8 by default, NOT the concurrency used for exploration: on this
benchmark the per-task movement of byte-identical code fell from 0.250 to 0.100 when
concurrency dropped from 25 to 8. Integration decisions are gate decisions, so they run slow.
"""

from __future__ import annotations

import argparse
import json
import shutil
import subprocess
import sys
from pathlib import Path

HERE = Path(__file__).resolve().parent


def _measure(pyexe: str, taskeval: Path, cand: Path, tasks: str, canary: str,
             n: int, conc: int, base_seed: int, project: str) -> dict:
    """Run taskeval on one candidate and return {task: rate} plus the objective.

    ``base_seed`` is accepted for the caller's own record-keeping (see the ``measurement``
    block in the final report) but is NOT forwarded to taskeval.py, which has no such flag —
    it always evaluates at its own internal seed 0. Forwarding it anyway used to make this
    call fail argparse outright with "unrecognized arguments", right after failing it a
    second way for the missing ``--project`` this function now supplies: this script had
    never actually been run end-to-end (see #434/#438), and both defects were latent.
    """
    out = cand.parent / f".{cand.name}_eval.json"
    cmd = [pyexe, str(taskeval), str(cand), tasks, "--project", project,
           "--n", str(n), "--conc", str(conc), "--json", str(out)]
    if canary:
        cmd += ["--canary", canary, "--canary-n", str(max(3, n // 2))]
    p = subprocess.run(cmd, capture_output=True, text=True)
    if not out.exists():
        return {"error": (p.stderr or p.stdout)[-900:]}
    d = json.loads(out.read_text())
    rates = {}
    for key in ("per_task", "tasks", "results"):
        v = d.get(key)
        if isinstance(v, dict):
            for t, row in v.items():
                rates[str(t)] = row.get("rate") if isinstance(row, dict) else row
            break
    return {"rates": rates, "raw": d}


def _objective(rates: dict, tasks: list[str], canary: list[str]) -> float | None:
    """Mean over targets AND canaries. Canaries are IN the objective on purpose: a branch that
    lifts its target while dropping a stable task is not an improvement, and scoring targets
    alone is exactly how such a branch gets accepted."""
    keys = [t for t in tasks + canary if rates.get(t) is not None]
    if not keys:
        return None
    return sum(float(rates[k]) for k in keys) / len(keys)


def main(argv=None) -> int:
    ap = argparse.ArgumentParser(prog="integrate")
    ap.add_argument("--base", required=True, help="starting artifact dir (the current best)")
    ap.add_argument("--project", required=True,
                    help="cap-evolve project dir (holds adapters/), forwarded to taskeval.py")
    ap.add_argument("--branches", nargs="+", required=True,
                    help="branch artifact dirs, applied in the order given; put the "
                         "best-evidenced branch first so a later regression is attributable")
    ap.add_argument("--out", required=True)
    ap.add_argument("--tasks", required=True, help="comma-separated target task ids")
    ap.add_argument("--canary", default="", help="comma-separated STABLE task ids to protect")
    ap.add_argument("--canary-auto", default="",
                    help="path to a baseline per-task JSON. Canaries are then chosen from the WHOLE "
                         "suite - every task at or above --canary-floor that is NOT a target - "
                         "instead of by hand. This exists because a hand-picked canary set drawn "
                         "from tasks near the mechanisms let four high-scoring tasks (1.00, 1.00, "
                         "0.80, 0.90) be damaged unguarded: they were not targets, so nobody "
                         "thought to watch them, and the artifact's gate failed on exactly that "
                         "collateral. A canary set that only covers what you aimed at cannot catch "
                         "what you hit by accident.")
    ap.add_argument("--canary-floor", type=float, default=0.9,
                    help="minimum baseline rate for an auto-selected canary (default 0.9)")
    ap.add_argument("--canary-max", type=int, default=12,
                    help="cap on auto-selected canaries, lowest-rate-first so the most fragile "
                         "high scorers are the ones kept")
    ap.add_argument("--n", type=int, default=10)
    ap.add_argument("--conc", type=int, default=8,
                    help="measurement concurrency. Default 8, deliberately low: byte-identical "
                         "code moved 0.250 per task at conc 25 and 0.100 at conc 8.")
    ap.add_argument("--base-seed", type=int, default=0)
    ap.add_argument("--floor", type=float, default=0.0,
                    help="measured null delta. Step deltas at or below this are recorded as "
                         "kept_provisionally, never as gains.")
    ap.add_argument("--file", default="tools/tools.py",
                    help="the Python file merged PER FUNCTION. Default suits the `tools` capability; "
                         "point it at whatever file the capability under optimization owns.")
    ap.add_argument("--prose", default="policy/policy.md",
                    help="comma-separated NON-Python files carried wholesale when a branch changed "
                         "them and this integration has not. Default suits `system-prompt` / "
                         "`tools`; a skill-package capability would pass SKILL.md instead. These "
                         "cannot be merged per function, so a second branch touching an "
                         "already-modified prose file is reported as contended and NOT applied - "
                         "silently concatenating two prose edits is how a policy grows "
                         "contradictory rules that no measurement can attribute.")
    ap.add_argument("--python", default=sys.executable)
    ap.add_argument("--taskeval", default="")
    ap.add_argument("--json", dest="json_out", default="")
    args = ap.parse_args(argv)

    taskeval = Path(args.taskeval) if args.taskeval else (
        Path(args.base).resolve().parents[2] / "taskeval.py")
    if not taskeval.exists():
        print(json.dumps({"error": f"taskeval.py not found at {taskeval}; pass --taskeval"}))
        return 2

    tasks = [t.strip() for t in args.tasks.split(",") if t.strip()]
    canary = [t.strip() for t in args.canary.split(",") if t.strip()]
    if args.canary_auto:
        base = json.loads(Path(args.canary_auto).read_text())
        per = base.get("per_task", base)
        pool = []
        for tid, row in per.items():
            if str(tid) in tasks:
                continue
            rate = row.get("rate") if isinstance(row, dict) else row
            if rate is not None and float(rate) >= args.canary_floor:
                pool.append((float(rate), str(tid)))
        pool.sort()                      # lowest rate first: the most fragile high scorers
        auto = [t for _, t in pool[: args.canary_max]]
        canary = sorted(set(canary) | set(auto), key=lambda x: (len(x), x))
        print(json.dumps({"canary_auto": {"selected": canary, "from_pool_of": len(pool),
                                          "floor": args.canary_floor,
                                          "note": "chosen from the WHOLE suite, excluding targets, "
                                                  "lowest-rate-first"}}, indent=2))
    base = Path(args.base).resolve()
    out = Path(args.out).resolve()
    if out.exists():
        shutil.rmtree(out)
    shutil.copytree(base, out)
    for junk in out.rglob("__pycache__"):
        shutil.rmtree(junk, ignore_errors=True)

    ev = _measure(args.python, taskeval, out, args.tasks, args.canary,
                  args.n, args.conc, args.base_seed, args.project)
    if "error" in ev:
        print(json.dumps({"error": "baseline measurement failed", "detail": ev["error"]}, indent=2))
        return 2
    cur = _objective(ev["rates"], tasks, canary)
    steps = [{"step": "base", "branch": base.name, "objective": cur, "rates": ev["rates"]}]

    for bdir in args.branches:
        b = Path(bdir).resolve()
        trial = out.parent / f".{out.name}__try_{b.name}"
        if trial.exists():
            shutil.rmtree(trial)
        shutil.copytree(out, trial)
        merged = subprocess.run(
            [args.python, str(HERE / "funcmerge.py"),
             "--base", str(base / args.file), "--out", str(trial / args.file),
             "--inputs", str(out / args.file), str(b / args.file),
             "--union-pure-insertions"],
            capture_output=True, text=True)
        if merged.returncode != 0:
            steps.append({"step": b.name, "decision": "reject",
                          "reason": "merge failed", "detail": merged.stderr[-500:]})
            shutil.rmtree(trial, ignore_errors=True)
            continue
        # Non-.py siblings (policy prose) are carried only when this integration has not
        # already changed them, so two branches cannot silently overwrite each other's prose.
        for extra in [e.strip() for e in args.prose.split(",") if e.strip()]:
            bp, op, basep = b / extra, out / extra, base / extra
            if bp.exists() and basep.exists() and bp.read_text() != basep.read_text():
                if op.read_text() == basep.read_text():
                    (trial / extra).write_text(bp.read_text())
                else:
                    steps.append({"step": b.name, "note": f"{extra} contended — branch prose "
                                  "NOT applied; base prose already modified by an earlier step"})

        ev = _measure(args.python, taskeval, trial, args.tasks, args.canary,
                      args.n, args.conc, args.base_seed, args.project)
        if "error" in ev:
            steps.append({"step": b.name, "decision": "reject", "reason": "eval failed",
                          "detail": ev["error"]})
            shutil.rmtree(trial, ignore_errors=True)
            continue
        new = _objective(ev["rates"], tasks, canary)
        delta = None if (new is None or cur is None) else round(new - cur, 4)
        dropped = [t for t in canary
                   if ev["rates"].get(t) is not None
                   and steps[0]["rates"].get(t) is not None
                   and float(ev["rates"][t]) < float(steps[0]["rates"][t])]
        keep = delta is not None and delta >= 0 and not dropped
        rec = {"step": b.name, "objective": new, "delta": delta,
               "canaries_dropped": dropped, "rates": ev["rates"],
               "decision": "accept" if keep else "reject"}
        if keep and delta is not None and delta <= args.floor:
            rec["decision"] = "kept_provisionally"
            rec["reading"] = (f"delta {delta} is at or below the measured noise floor "
                              f"{args.floor} — carried, but this is NOT evidence of a gain")
        steps.append(rec)
        if keep:
            shutil.rmtree(out)
            shutil.move(str(trial), str(out))
            cur = new
        else:
            shutil.rmtree(trial, ignore_errors=True)

    accepted = [s["step"] for s in steps[1:] if s.get("decision") == "accept"]
    prov = [s["step"] for s in steps[1:] if s.get("decision") == "kept_provisionally"]
    result = {
        "out": str(out),
        "measurement": {"n": args.n, "conc": args.conc, "base_seed": args.base_seed,
                        "floor": args.floor,
                        "note": "conc is deliberately low; integration decisions are gate "
                                "decisions and gate decisions run slow"},
        "objective_basis": f"mean over {len(tasks)} targets + {len(canary)} canaries",
        "base_objective": steps[0]["objective"],
        "final_objective": cur,
        "accepted": accepted,
        "kept_provisionally": prov,
        "rejected": [s["step"] for s in steps[1:]
                     if s.get("decision") == "reject"],
        "steps": steps,
        "honesty": ("Each step was measured on the SAME seeds at the SAME concurrency, so step "
                    "deltas are comparable to each other. They are NOT a full-val result: this "
                    "objective is a task subset chosen because those tasks were failing, so it "
                    "is upward-biased by selection. Gate the final artifact on full val."),
    }
    print(json.dumps(result, indent=2))
    if args.json_out:
        Path(args.json_out).write_text(json.dumps(result, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Link contract for this skill: every relative link resolves, and no reference links to another.

Two failures this catches, both silent in review:
  * a body pointer to `references/<x>.md` that was renamed or never created — the agent follows
    the pointer, finds nothing, and improvises the depth the reference was supposed to carry;
  * a reference that links to another reference. References are ONE level deep because the agent
    may read only part of either one, so a ref->ref hop can leave it acting on half a rule.

Run: python scripts/linkcheck.py   (prints JSON, exits non-zero on any problem)
"""

from __future__ import annotations

import json
import re
import sys
from pathlib import Path

SKILL = Path(__file__).resolve().parent.parent
LINK = re.compile(r"\[[^\]]*\]\(([^)]+)\)")


def main() -> int:
    problems: list[str] = []
    checked = 0
    files = [SKILL / "SKILL.md", *sorted((SKILL / "references").glob("*.md"))]
    for f in files:
        rel_f = f.relative_to(SKILL)
        for target in LINK.findall(f.read_text(encoding="utf-8")):
            if target.startswith(("http://", "https://", "#", "mailto:")):
                continue
            checked += 1
            path, _, anchor = target.partition("#")
            resolved = (f.parent / path).resolve()
            if not resolved.is_file():
                problems.append(f"{rel_f}: broken relative link -> {target}")
                continue
            if anchor:
                slugs = {
                    re.sub(r"[^a-z0-9\- ]", "", line.lstrip("#").strip().lower()).replace(" ", "-")
                    for line in resolved.read_text(encoding="utf-8").splitlines()
                    if line.startswith("#")
                }
                if anchor not in slugs:
                    problems.append(f"{rel_f}: link {target} names a heading that does not exist")
            if f.parent.name == "references" and resolved.parent.name == "references":
                problems.append(
                    f"{rel_f}: links to another reference ({path}) — references are one level deep")

    print(json.dumps({"files": len(files), "relative_links": checked,
                      "ok": not problems, "problems": problems}, indent=2))
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

"""measure — the run's FINAL, honest seed-vs-best table across every real split.

A val number is the thing the gate optimized against, so quoting it as the result is
quoting the training signal. This script produces the one table that is allowed to be
called "the improvement": ``seed`` vs ``best`` on **val** (from the rollouts the gate
actually used), on **train** when the spec defines a train split worth reporting, and
on the **sealed test** split — scored exactly once, through the same ``harness.finalize``
the finalize phase calls, and refused on a second attempt.

For every split it prints mean, stderr, n (tasks scored / tasks in split), the paired
per-task delta vector's mean + SE + n, and the gate decision recomputed on that vector.

What it refuses to pretend:

  * **No-holdout specs.** If ``test`` overlaps ``train``/``val`` (as some benchmarks ship by
    default, where all three are the same ids), the test column is a FIT
    metric, not generalisation, and the payload says so in ``holdout`` — with the
    overlap counted, not hand-waved.
  * **Empty splits.** A split with no ids gets ``"status": "empty"`` and no numbers,
    rather than a 0.0 that reads like a measured failure.
  * **best == seed.** Nothing cleared the gate; the deltas are 0 by construction and
    ``no_accepted_change`` says so, instead of presenting a 0.000 improvement as a
    measurement of anything.

Train is measured only when it adds information: ``--train auto`` (the default) skips
it when the train ids equal the val ids (the numbers would be a copy) and skips it when
there are no train ids. ``--train on`` forces it; ``--train off`` never pays for it.
Train and val evals here are ordinary un-sealed evaluations; only test is sealed.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import RunDir, harness
from cap_evolve.check import load_adapter
from cap_evolve.gate import decide
from cap_evolve.loop import SplitResult
from cap_evolve.specfile import spec_for_run

import merge_search


def _num(sr: SplitResult | None) -> dict:
    if sr is None:
        return {"status": "not measured"}
    return {"reward": round(sr.reward, 6), "stderr": round(sr.stderr, 6),
            "n_scored": sr.n_scored, "n_tasks": sr.n_tasks,
            "coverage": round(sr.coverage, 4), "pass_k": sr.pass_k}


def _compare(seed: SplitResult | None, best: SplitResult | None, *, split: str,
             k_se: float, mode: str) -> dict:
    """Paired delta + the gate decision on one split. Gate is only meaningful on val."""
    row = {"split": split, "seed": _num(seed), "best": _num(best)}
    if seed is None or best is None:
        return row
    deltas = harness._paired_deltas(seed, best)
    row["mean_delta_unpaired"] = round(best.reward - seed.reward, 6)
    if not deltas:
        row["paired"] = {"status": "no aligned per-task data (tasks unscored on one "
                                  "side are dropped — missing data, not a 0.0)"}
        return row
    n = len(deltas)
    mean_d = sum(deltas) / n
    se = 0.0
    if n >= 2:
        var = sum((d - mean_d) ** 2 for d in deltas) / (n - 1)
        se = (var / n) ** 0.5
    # `improved`/`regressed` count the RESOLVED movers (`_per_task_movement`, which applies
    # `harness.move_is_resolved`), not every task whose reward differs by more than 1e-9.
    # Counting the latter put two bars in one output: this block would report a task as
    # improved while `val_per_task_movement` right beside it called the same task unresolved.
    mv = _per_task_movement(seed, best)
    row["paired"] = {"n": n, "mean_delta": round(mean_d, 6), "se": round(se, 6),
                     "improved": len(mv["fixed"]), "regressed": len(mv["broke"]),
                     "unresolved": len(mv["unresolved"]),
                     "counts_reading": "improved/regressed count only tasks whose move cleared "
                                       "2*SE of its own per-task measurement; smaller movers "
                                       "are in `unresolved` and are not evidence either way"}
    if split == "val":
        d = decide(seed.reward, best.reward, split="val", mode=mode, k_se=k_se,
                   candidate_stderr=best.stderr, current_stderr=seed.stderr,
                   paired_deltas=deltas, coverage=best.coverage)
        row["gate"] = d.to_dict()
    else:
        row["gate"] = {"note": f"no gate on {split}: acceptance is val-only "
                               "(gate.decide refuses any other split)"}
    return row


def _per_task_movement(seed: SplitResult, best: SplitResult) -> dict:
    """Which tasks the whole run measurably fixed or broke, seed -> best.

    `fixed`/`broke` require the move to clear 2*SE of its own per-task measurement
    (``harness.move_is_resolved`` — the ONE bar every broke/fixed claim in the framework
    uses). A smaller move goes to `unresolved`: it is not `unchanged` (the reward did move)
    and it is not evidence the run changed that task's behaviour. This is the sealed report a
    human reads, so a task listed here as fixed or broken had better be a claim the
    measurement supports — at 10 trials the old ``1e-9`` test promoted a single flipped
    rollout to a headline behavioural change.
    """
    from cap_evolve.loop import has_valid_trials
    s = {pt["task_id"]: pt for pt in (seed.per_task or []) if has_valid_trials(pt)}
    b = {pt["task_id"]: pt for pt in (best.per_task or []) if has_valid_trials(pt)}
    shared = sorted(set(s) & set(b))

    def _r(d, t):
        return d[t].get("reward", 0.0) or 0.0

    def _resolved(t):
        return harness.move_is_resolved(_r(s, t), _r(b, t),
                                        s[t].get("stderr") or 0.0, b[t].get("stderr") or 0.0)

    return {
        "fixed": [t for t in shared if _r(b, t) > _r(s, t) and _resolved(t)],
        "broke": [t for t in shared if _r(b, t) < _r(s, t) and _resolved(t)],
        "unresolved": [t for t in shared
                       if abs(_r(b, t) - _r(s, t)) > 1e-9 and not _resolved(t)],
        "unchanged": [t for t in shared if abs(_r(b, t) - _r(s, t)) <= 1e-9],
        "unpaired": sorted((set(s) | set(b)) - set(shared)),
    }


def _screen_ledger(run_dir: RunDir) -> dict:
    """Sum every recorded subset screen. MEASURED integers, no estimates.

    ``net_rollouts`` is what the ladder actually bought: ``+ (full_val − fired)`` for each
    kill, ``− fired`` for each promote. A negative total is an honest report that screening
    cost more than it saved on this run — which is what happens when nothing gets killed.
    """
    d = run_dir.root / "screens"
    rows = []
    for f in sorted(d.glob("*.json")):
        try:
            rows.append(json.loads(f.read_text(encoding="utf-8")))
        except Exception:  # noqa: BLE001
            continue
    kills = [r for r in rows if r.get("decision") == "kill"]
    return {
        "screens": len(rows),
        "kills": len(kills),
        "promotes": len(rows) - len(kills),
        "rollouts_fired_by_screens": sum(int(r["savings"]["fired"]) for r in rows),
        "rollouts_avoided_by_kills": sum(int(r["savings"]["avoided"]) for r in rows),
        "net_rollouts": sum(int(r["savings"]["net_rollouts"]) for r in rows),
        "screen_usd": round(sum(float(r["savings"].get("screen_cost_usd") or 0.0)
                                for r in rows), 6),
        "note": ("net_rollouts > 0 means the ladder paid for itself; <= 0 means every "
                 "candidate was promoted, so screening was pure overhead this run"),
    }


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="measure")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", required=True)
    p.add_argument("--n-trials", type=int, default=0,
                   help="trials per eval; default = the spec's num_trials")
    p.add_argument("--k-se", type=float, default=None,
                   help="gate bar; default = the spec's gate_k_se")
    p.add_argument("--gate-mode", default=None,
                   help="gate mode; default = the spec's gate_mode (else paired)")
    p.add_argument("--train", default="auto", choices=["auto", "on", "off"])
    p.add_argument("--skip-test", action="store_true",
                   help="report train/val only; do NOT touch the seal (audit use)")
    p.add_argument("--workers", type=int, default=None)
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))
    project = Path(args.project)

    # Compliance signal (SKILL.md: merge disjoint-cluster accepted candidates before any
    # end-of-run measurement) — see merge_search.check_merge_compliance's own docstring.
    # Never blocks; just makes an ignored requirement visible in events.jsonl/dashboard.
    merge_warning = merge_search.check_merge_compliance(run_dir)
    if merge_warning:
        run_dir.log_event("merge_compliance_warning", **merge_warning)

    spec = spec_for_run(run_dir, project)
    n_trials = args.n_trials or int(spec.get("num_trials") or 1)
    k_se = args.k_se if args.k_se is not None else float(spec.get("gate_k_se") or 1.0)
    mode = args.gate_mode or str(spec.get("gate_mode") or "paired")

    splits = run_dir.read_splits()
    best_id = run_dir.best_id or "seed"
    tr, va, te = set(splits.train), set(splits.val), set(splits.test)
    holdout = {
        "n_train": len(tr), "n_val": len(va), "n_test": len(te),
        "test_overlaps_train": len(te & tr), "test_overlaps_val": len(te & va),
        "val_overlaps_train": len(va & tr),
    }
    held_out = bool(te) and not (te & tr) and not (te & va)
    holdout["test_is_held_out"] = held_out
    holdout["verdict"] = (
        "TEST IS HELD OUT: the test column measures generalisation."
        if held_out else
        ("NO TEST SPLIT: there is no held-out number in this run at all."
         if not te else
         "NOT HELD OUT: test overlaps train/val, so the test column is a FIT metric, "
         "not generalisation. Do not report it as a held-out result.")
    )

    rows: list = []

    # ---- val: free, straight off the rollouts the gate used ------------------
    if not va:
        rows.append({"split": "val", "status": "empty — no val ids in the frozen split"})
        seed_val = best_val = None
    else:
        seed_val = harness.split_result_from_rollouts(run_dir, "seed", "val")
        best_val = harness.split_result_from_rollouts(run_dir, best_id, "val")
        rows.append(_compare(seed_val, best_val, split="val", k_se=k_se, mode=mode))

    # ---- train: only when it adds information -------------------------------
    do_train = args.train == "on" or (args.train == "auto" and tr and tr != va)
    if not tr:
        rows.append({"split": "train", "status": "empty — no train ids in the frozen split"})
    elif not do_train:
        rows.append({"split": "train",
                     "status": ("skipped: train ids are identical to val, so the numbers "
                                "would be a copy of the val row (pass --train on to "
                                "measure anyway)" if tr == va else
                                "skipped by --train off")})
    else:
        adapter = load_adapter(project)

        def _train(cid: str, tag: str):
            """Train result for candidate ``cid``, REUSING its rollouts when complete.

            A candidate dir is an immutable snapshot, so rollouts already persisted under
            its own tag are measurements of exactly this capability — re-running them buys
            nothing and costs a full train split. (The agent-mode loop pays one
            ``evaluate --split train`` for the seed to get a diagnosis signal, so this
            reuse is the common case, not a corner case.)
            """
            have = harness.split_result_from_rollouts(run_dir, cid, "train")
            if have.per_task and have.n_scored >= len(tr):
                have.reused_rollouts = True  # type: ignore[attr-defined]
                return have, True
            return harness.evaluate_candidate(adapter, run_dir.candidate_dir(cid),
                                              run_dir=run_dir, split="train",
                                              n_trials=n_trials, tag=tag,
                                              workers=args.workers), False

        # Tags match the ones `evaluate_candidate`/`harness.finalize` use elsewhere
        # ("seed" and the candidate's own id) — not a "MEASURE_*" alias — so this eval
        # and `finalize`'s own train+val bookend (below, via measure.py's test section)
        # see the SAME rollouts on disk and never pay for the same measurement twice.
        s, s_reused = _train("seed", "seed")
        if best_id == "seed":
            b, b_reused = s, s_reused
        else:
            b, b_reused = _train(best_id, best_id)
        row = _compare(s, b, split="train", k_se=k_se, mode=mode)
        row["rollouts_reused"] = {"seed": s_reused, "best": b_reused}
        rows.append(row)

    # ---- test: the sealed split, exactly once -------------------------------
    final_path = run_dir.root / "final.json"
    if not te:
        rows.append({"split": "test", "status": "empty — no test ids; this run has no "
                                                "held-out number"})
    elif args.skip_test:
        rows.append({"split": "test", "status": "skipped by --skip-test (seal untouched)"})
    elif splits.test_used and final_path.is_file():
        fin = json.loads(final_path.read_text(encoding="utf-8"))
        rows.append({"split": "test", "status": "already sealed — reporting final.json",
                     "seed": _num(SplitResult.from_dict(fin["test_baseline"])),
                     "best": _num(SplitResult.from_dict(fin["test"])),
                     "test_delta": fin.get("test_delta"),
                     "paired": (_compare(SplitResult.from_dict(fin["test_baseline"]),
                                         SplitResult.from_dict(fin["test"]),
                                         split="test", k_se=k_se, mode=mode)
                                .get("paired"))})
    else:
        adapter = load_adapter(project)
        fin = harness.finalize(adapter, run_dir=run_dir,
                               best_dir=run_dir.candidate_dir(best_id),
                               n_trials=n_trials,
                               baseline_dir=run_dir.candidate_dir("seed"))
        row = _compare(SplitResult.from_dict(fin["test_baseline"]),
                       SplitResult.from_dict(fin["test"]),
                       split="test", k_se=k_se, mode=mode)
        row["test_delta"] = fin.get("test_delta")
        row["status"] = "sealed now (once)"
        rows.append(row)

    payload = {
        "run_dir": str(run_dir.root),
        "seed_id": "seed",
        "best_id": best_id,
        "no_accepted_change": best_id == "seed",
        "n_trials": n_trials,
        "gate": {"mode": mode, "k_se": k_se},
        "holdout": holdout,
        "splits": rows,
        "val_per_task_movement": (_per_task_movement(seed_val, best_val)
                                  if seed_val and best_val else None),
        "spent": run_dir.spent.to_dict(),
        "screen_ledger": _screen_ledger(run_dir),
    }
    if best_id == "seed":
        payload["warning"] = ("best_id == 'seed': nothing cleared the val gate, so every "
                              "delta below is 0 by construction. Report this as a null "
                              "result with a diagnosed cause, not as a 0.000 improvement.")
    (run_dir.root / "measure.json").write_text(json.dumps(payload, indent=2),
                                               encoding="utf-8")
    print(json.dumps(payload, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Shared mechanism ledger for a per-task fan-out: what was found, who owns it, what it did.

K parallel optimisers on K different tasks keep rediscovering ONE cause. Measured on one
multi-turn tool-use benchmark: four of nine independently found the same root cause, and two
independently implemented the same fix — which then collided at merge, and
only one of the two had actually been measured. Both problems are the same problem: findings
lived in the coordinator's head instead of on disk.

So put them on disk. Every optimiser LISTS before it diagnoses (someone may already have your
bug, and if they own the fix you must not write a second one) and APPENDS when it finds a
mechanism (so the next optimiser does not pay for it again). This is what makes a fan-out
compound instead of merely parallel, and it replaces a broadcast the coordinator has to
remember to send.

    python mechanisms.py list --run-dir R [--file tools/tools.py]
    python mechanisms.py add  --run-dir R --owner t17 --status proposed \\
        --mechanism "<the cause, one sentence>" \\
        --evidence "3/5 failing trials commit to the first id probed" \\
        --touches <function-the-fix-edits>
    python mechanisms.py add  --run-dir R --owner t17 --status rejected \\
        --mechanism "invite the agent to ask which listed trip they meant" \\
        --evidence "user simulator invents a non-existent trip; 2/4 failures that round"

``status`` is the whole point of the ledger, so it is required:
  * ``proposed``  — diagnosed, edit written, not yet measured. Do not build on it.
  * ``verified``  — measured to move its target with the canary intact. Reuse, never rewrite.
  * ``rejected``  — measured at or below control, or actively harmful. Do not retry this form;
    a later attempt at the same failure must be structurally DIFFERENT.
"""

import argparse
import json
import os
import time
from pathlib import Path



def ledger(run_dir: Path) -> Path:
    return Path(run_dir) / "mechanisms.jsonl"


def add(args) -> int:
    row = {
        "seq": int(time.time() * 1000),
        "owner": args.owner,
        "status": args.status,
        "mechanism": args.mechanism,
        "evidence": args.evidence,
        "touches": sorted(set(args.touches or [])),
        "tasks": sorted(set(args.task or [])),
        "supersedes": sorted(set(args.supersedes or [])),
    }
    path = ledger(args.run_dir)
    path.parent.mkdir(parents=True, exist_ok=True)
    line = json.dumps(row, sort_keys=True) + "\n"
    # O_APPEND makes a single small write atomic across the concurrent optimisers, so no
    # lock file is needed and a crashed optimiser cannot leave a half-written row.
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
    try:
        os.write(fd, line.encode("utf-8"))
    finally:
        os.close(fd)
    print(json.dumps({"added": row, "ledger": str(path)}, indent=2))
    return 0


def read(path: Path) -> list[dict]:
    if not path.exists():
        return []
    rows = []
    for ln in path.read_text(encoding="utf-8").splitlines():
        ln = ln.strip()
        if not ln:
            continue
        try:
            rows.append(json.loads(ln))
        except Exception:  # noqa: BLE001
            continue          # a torn row is skipped, never fatal
    return rows


def lst(args) -> int:
    rows = read(ledger(args.run_dir))
    if args.file:
        rows = [r for r in rows if any(args.file in t for t in r.get("touches") or [])]
    if args.task:
        # A fan-out ledger grows past what is useful to paste at an optimiser: this one
        # reached 99 findings, and handing all of them to each of K subagents spends their
        # context on other people's tasks. Relevance = rows about THIS task, plus every row
        # with no task attached, because those are the cross-cutting facts (measurement
        # defects, canary bands, variance warnings) that apply to everyone and are exactly
        # what a filtered view must not hide.
        rows = [r for r in rows
                if args.task in (r.get("tasks") or []) or not (r.get("tasks") or [])]
    if args.compact:
        rows = [{k: v for k, v in r.items() if k != "evidence"} for r in rows]
    superseded = {sq for r in rows for sq in (r.get("supersedes") or [])}
    by = {"verified": [], "proposed": [], "rejected": []}
    dead = []
    for r in rows:
        if str(r.get("seq")) in superseded:
            dead.append(r)
            continue
        by.setdefault(r.get("status", "proposed"), []).append(r)
    owned = sorted({t for r in by["verified"] + by["proposed"]
                    for t in (r.get("touches") or [])})
    print(json.dumps({
        "count": len(rows),
        "verified": by["verified"],
        "proposed": by["proposed"],
        "rejected": by["rejected"],
        "already_owned_do_not_reimplement": owned,
        "superseded_do_not_act_on": [
            {"seq": r.get("seq"), "owner": r.get("owner"), "was": r.get("status"),
             "mechanism": (r.get("mechanism") or "")[:160]} for r in dead],
        "reminder": ("if your bug is listed as verified or proposed, its owner writes the fix — "
                     "rebase onto their copy and spend your iterations elsewhere; if it is "
                     "listed as rejected, a retry must be structurally different"),
    }, indent=2))
    return 0


def main(argv=None) -> int:
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd", required=True)

    a = sub.add_parser("add")
    a.add_argument("--run-dir", required=True)
    a.add_argument("--owner", required=True, help="the optimiser tag that found it")
    a.add_argument("--status", required=True, choices=["proposed", "verified", "rejected"])
    a.add_argument("--mechanism", required=True, help="the CAUSE, in one sentence")
    a.add_argument("--evidence", required=True, help="what you measured or saw in the trace")
    a.add_argument("--touches", action="append", default=[],
                   help="function/file the fix edits (repeatable) — this is the collision key")
    a.add_argument("--task", action="append", default=[])
    a.add_argument("--supersedes", action="append", default=[],
                   help="seq id(s) this row replaces. A finding that turns out to be wrong cannot "
                        "just be contradicted by a newer row: on this run three separate "
                        "`verified` rows were later disproved, and a reader of the listing saw "
                        "both the claim and its refutation with no way to tell which won. A "
                        "superseded row is dropped from `verified`/`proposed` and reported "
                        "separately, so the ledger's own history stays auditable without "
                        "misleading the next optimiser. Repeatable.")
    a.set_defaults(fn=add)

    l = sub.add_parser("list")
    l.add_argument("--run-dir", required=True)
    l.add_argument("--file", default="", help="only rows whose touches mention this")
    l.add_argument("--task", default="", help="only rows about THIS task, plus every "
                                              "task-independent row (those apply to everyone)")
    l.add_argument("--compact", action="store_true",
                   help="drop the evidence field: mechanism + status + touches only")
    l.set_defaults(fn=lst)

    args = ap.parse_args(argv)
    args.run_dir = Path(args.run_dir)
    return args.fn(args)


if __name__ == "__main__":
    raise SystemExit(main())

"""merge_search — pairwise merge-as-graph-search over a round's disjoint-cluster survivors.

Why this exists. run_agentoptv3/run_agentoptv4 produced 3-6 narrow, single-issue candidates
per round, each individually gated on full val and rejected, and NEVER combined — no
integrate.py/funcmerge.py/merge_taskopt.py call appears in either run (#434, #438). The
merge machinery already exists and already works (see integrate.py/funcmerge.py's own
docstrings for the measured cases it was built to fix); the missing piece is simply DECIDING
which survivors are safe to try merging and DOING it, instead of leaving that to a driver
under time pressure who defaults to the cheapest step (another single candidate).

A round's "survivors" are candidates that got PAST screening (screen.py promote, or a
grow.py-provisional that ran out of growth rounds) without individually clearing the full-val
accept gate — see round.py/screen.py/grow.py. Two survivors are a MERGE CANDIDATE when the
functions/constants they changed, relative to the same base, are DISJOINT: `funcmerge.py`'s
own docstring is the reason overlapping edits are refused here rather than attempted — a
same-function collision is a genuine semantic disagreement funcmerge.py already declines to
auto-resolve, and offering it to a human via a merge conflict is not "graph search", it is
"ask a human what the graph search could not decide". Disjoint edits are exactly the
provably-safe case per-task-fanout.md already describes.

This script does NOT invent a new merge engine. Per pair it shells out to `integrate.py`
(one branch at a time, measured after each — see that file for why a one-shot N-way merge
does not compose) and forwards `--canary-auto` so the merge's objective is measured against
canaries drawn from the WHOLE suite, never just the neighbourhood of the two branches'
targets (per-task-fanout.md's "a canary set that only covers what you aimed at cannot catch
what you hit by accident" — restated here because a merge is exactly the moment two
neighbourhoods combine and the blast radius is not either one's alone).

It also does NOT gate anything. A successfully-built merge candidate is written to
`$R/work/merge_<a>_<b>` — an ordinary candidate directory, indistinguishable from any other
tag on disk — so `round.py --candidates merge_<a>_<b>,...` gates it through the EXACT SAME
cascade (null control, paired significance, no-regression veto) as a hand-authored edit. No
special-casing was added to round.py, deliberately: a merge is a candidate, not a new kind
of thing the gate has to know about.

Per-survivor target task ids come from `--targets tag:1,2,3` (repeatable) when given, else
from `mechanisms.jsonl`'s `tasks` field for rows owned by that tag (the ledger's existing
"who changed what, aimed at which tasks" record — see mechanisms.py). A survivor with
neither is skipped with a reason, never silently merged on an empty/whole-val objective.

Graph-DAG note (#435): this script currently reads survivor tags + a mechanisms ledger
directly, because no `graph.jsonl` exists yet on `main` to consume. Once #435 lands, the
survivor list and each one's `subset.rationale`/target ids should come from the DAG's
frontier nodes instead of `--survivors`/`--targets`/mechanisms.jsonl — the disjointness
check and the integrate.py call below do not change.

    python merge_search.py --run-dir R --project P --base BEST \\
        --survivors t7,t17,u33 --targets t7:1,2 --targets t17:5,9 \\
        --canary-auto R/baseline.json --n 10 --conc 8 --floor 0.0333
"""

from __future__ import annotations

import argparse
import contextlib
import io
import itertools
import json
import subprocess
import sys
from pathlib import Path

import funcmerge
import mechanisms

HERE = Path(__file__).resolve().parent


def _canary_auto_file(path: str) -> str:
    """Normalize a baseline JSON into the per-task DICT shape `integrate.py --canary-auto`
    reads (``{tid: {"rate": ...}}``, taskeval.py's own shape).

    The run's own ``$R/baseline.json`` (harness.baseline's output) is the file every run
    already has, but its ``per_task`` is a LIST of Score dicts (``harness.SplitResult``),
    one level down under ``"val"``. Reshaping it here — rather than teaching integrate.py a
    second per_task shape — keeps the merge engine itself unchanged.
    """
    raw = json.loads(Path(path).read_text(encoding="utf-8"))
    per = raw.get("per_task")
    if per is None and isinstance(raw.get("val"), dict):
        per = raw["val"].get("per_task")
    if isinstance(per, list):
        shaped = {str(row["task_id"]): {"rate": row.get("reward")}
                 for row in per if isinstance(row, dict) and row.get("task_id") is not None}
    elif isinstance(per, dict):
        return path  # already the shape integrate.py expects
    else:
        return path  # nothing recognizable — let integrate.py report the same error
    import tempfile
    fd, tmp = tempfile.mkstemp(suffix=".json", prefix="merge_search_canary_")
    Path(tmp).write_text(json.dumps({"per_task": shaped}), encoding="utf-8")
    return tmp


def changed_functions(base_src: str, variant_src: str) -> set[str]:
    """Names of functions/constants `variant_src` changed or added relative to `base_src`.

    Reuses funcmerge.blocks — the SAME per-function split integrate.py's own merge step
    runs on — so "disjoint" here means exactly what funcmerge.py would find non-conflicting,
    not a second, possibly-inconsistent notion of overlap.
    """
    _, base_fns = funcmerge.blocks(base_src)
    _, var_fns = funcmerge.blocks(variant_src)
    return {name for name, text in var_fns.items()
            if name not in base_fns or text != base_fns[name]}


def _mechanisms_targets(run_dir: Path, tag: str) -> list[str]:
    """Union of `tasks` from mechanisms.jsonl rows owned by `tag` (see mechanisms.py)."""
    path = Path(run_dir) / "mechanisms.jsonl"
    if not path.exists():
        return []
    ids: set[str] = set()
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line:
            continue
        try:
            row = json.loads(line)
        except Exception:  # noqa: BLE001
            continue
        if row.get("owner") == tag:
            ids.update(str(t) for t in (row.get("tasks") or []))
    return sorted(ids)


def find_disjoint_pairs(base_src: str, survivor_srcs: dict[str, str]) -> dict:
    """Every survivor pair, split into disjoint (mergeable) vs overlapping (skipped).

    Returns {"changed": {tag: sorted[fn...]}, "disjoint_pairs": [[a, b], ...],
    "overlapping_pairs": [{"pair": [a, b], "shared": [fn...]}]}.
    """
    changed = {tag: changed_functions(base_src, src) for tag, src in survivor_srcs.items()}
    disjoint, overlapping = [], []
    for a, b in itertools.combinations(sorted(survivor_srcs), 2):
        shared = changed[a] & changed[b]
        if shared:
            overlapping.append({"pair": [a, b], "shared": sorted(shared)})
        else:
            disjoint.append([a, b])
    return {"changed": {t: sorted(v) for t, v in changed.items()},
            "disjoint_pairs": disjoint, "overlapping_pairs": overlapping}


def _integrate(run_dir: Path, project: Path, work: Path, base_dir: Path, a: str, b: str,
               tasks: list[str], canary: list[str], canary_auto: str, canary_floor: float,
               n: int, conc: int, base_seed: int, floor: float, file_: str, prose: str,
               out_tag: str) -> dict:
    json_out = work / f".{out_tag}_integrate.json"
    cmd = [sys.executable, str(HERE / "integrate.py"),
           "--base", str(base_dir), "--project", str(project),
           "--branches", str(work / a), str(work / b),
           "--out", str(work / out_tag), "--tasks", ",".join(tasks),
           "--n", str(n), "--conc", str(conc), "--base-seed", str(base_seed),
           "--floor", str(floor), "--file", file_, "--prose", prose,
           "--taskeval", str(HERE / "taskeval.py"), "--json", str(json_out)]
    if canary_auto:
        cmd += ["--canary-auto", canary_auto, "--canary-floor", str(canary_floor)]
    elif canary:
        cmd += ["--canary", ",".join(canary)]
    p = subprocess.run(cmd, capture_output=True, text=True)
    # NOT json.loads(p.stdout): with --canary-auto, integrate.py prints the canary-selection
    # note as its OWN json.dumps call before the final result — two concatenated JSON
    # documents on one stream, which `json.loads` cannot parse as one object. `--json`
    # writes only the final result, so read that back instead.
    if json_out.exists():
        try:
            return json.loads(json_out.read_text(encoding="utf-8"))
        except Exception:  # noqa: BLE001
            pass
    return {"error": (p.stderr or p.stdout)[-1200:], "rc": p.returncode}


def check_merge_compliance(run_dir) -> dict | None:
    """Audit signal, not an enforcement: did this run merge disjoint-cluster accepted
    candidates before finalizing?

    SKILL.md requires the optimizer to run this script on its accepted candidates before
    any end-of-run measurement, whenever 2+ of them target disjoint task clusters — the
    exact shape this script exists to combine (module docstring above). Nothing in the
    framework can force the agent to actually do that (host.py owns no algorithm
    decisions), so this is the same kind of code-level compliance signal
    ``round.py``'s ``agent_optimize_compliance`` event already logs for the screen
    ladder: it never blocks, it only makes the omission visible in ``events.jsonl`` and
    the dashboard's activity log.

    Reads ``graph.jsonl`` for ``status == "accepted"`` nodes and each one's target task
    ids — its own ``cluster_ids`` if a caller has populated that field, else the same
    ``mechanisms.jsonl`` fallback this script's own ``main()`` uses to pick merge targets
    (`_mechanisms_targets`). Two or more accepted candidates whose target sets are
    pairwise DISJOINT, with no ``edit_kind == "merge"`` node anywhere in the graph, means
    the run's "best" is whichever single cluster's fix happened to score highest, never a
    combination of them.

    Returns ``None`` when there is nothing to flag (fewer than 2 accepted candidates with
    known targets, none of their target sets are disjoint, or a merge was already
    attempted this run), else a dict of fields for ``RunDir.log_event``.
    """
    import _bootstrap  # noqa: F401
    from cap_evolve import graph as graph_mod

    nodes = graph_mod.read_nodes(run_dir)
    accepted = [n for n in nodes if n.get("status") == "accepted"]

    targets: dict[str, list[str]] = {}
    for n in accepted:
        tag = n.get("id")
        if not tag:
            continue
        ids = n.get("cluster_ids") or _mechanisms_targets(run_dir.root, tag)
        if ids:
            targets[tag] = sorted(set(ids))
    if len(targets) < 2:
        return None

    disjoint_pairs = [[a, b] for a, b in itertools.combinations(sorted(targets), 2)
                      if not (set(targets[a]) & set(targets[b]))]
    if not disjoint_pairs:
        return None

    if any(n.get("edit_kind") == "merge" for n in nodes):
        return None  # a merge was attempted this run — nothing to flag

    return {
        "reason": "merge_skipped_with_multiple_clusters",
        "accepted_candidates": sorted(targets),
        "targets": targets,
        "disjoint_pairs": disjoint_pairs,
    }


def build_parser() -> argparse.ArgumentParser:
    ap = argparse.ArgumentParser(prog="merge_search")
    ap.add_argument("--run-dir", required=True)
    ap.add_argument("--project", required=True)
    ap.add_argument("--base", required=True, help="tag of the round's current parent")
    ap.add_argument("--survivors", required=True,
                    help="comma-separated tags under $R/work/ that passed screening but did "
                         "not individually clear the full-val accept gate")
    ap.add_argument("--targets", action="append", default=[],
                    help="tag:comma,separated,task,ids — the tasks this survivor targeted. "
                         "Repeatable. Falls back to mechanisms.jsonl's `tasks` field for rows "
                         "owned by that tag when omitted.")
    ap.add_argument("--file", default="tools/tools.py",
                    help="the Python file merged per function — see integrate.py --file")
    ap.add_argument("--prose", default="policy/policy.md")
    ap.add_argument("--canary-auto", default="",
                    help="path to a baseline per-task JSON. Canaries are drawn from the "
                         "WHOLE suite (per-task-fanout.md), not from the merged branches' "
                         "own neighbourhood — pass the run's baseline.json.")
    ap.add_argument("--canary", default="", help="explicit canary ids, if not using --canary-auto")
    ap.add_argument("--canary-floor", type=float, default=0.9)
    ap.add_argument("--n", type=int, default=10)
    ap.add_argument("--conc", type=int, default=8)
    ap.add_argument("--base-seed", type=int, default=0)
    ap.add_argument("--floor", type=float, default=0.0,
                    help="measured null delta — see integrate.py --floor")
    ap.add_argument("--json", dest="json_out", default="")
    return ap


def main(argv=None) -> int:
    args = build_parser().parse_args(argv)

    run_dir = Path(args.run_dir)
    project = Path(args.project)
    work = run_dir / "work"
    survivors = [t.strip() for t in args.survivors.split(",") if t.strip()]
    canary_auto = _canary_auto_file(args.canary_auto) if args.canary_auto else ""

    explicit_targets: dict[str, list[str]] = {}
    for item in args.targets:
        tag, _, ids = item.partition(":")
        explicit_targets[tag.strip()] = [i.strip() for i in ids.split(",") if i.strip()]

    base_dir = work / args.base
    if not base_dir.is_dir():
        # The round's parent usually lives in candidates/, not work/ — only a branch
        # actively being merged gets a work/ copy. Fall back to the run's own record of
        # where that tag's snapshot is, rather than requiring the caller to pre-stage it.
        import _bootstrap  # noqa: F401
        from cap_evolve import RunDir
        base_dir = RunDir.open(run_dir).candidate_dir(args.base)
    base_file = base_dir / args.file
    if not base_file.exists():
        print(json.dumps({"error": f"base file not found: {base_file}"}, indent=2))
        return 2
    base_src = base_file.read_text(encoding="utf-8")

    survivor_srcs, missing = {}, []
    for tag in survivors:
        f = work / tag / args.file
        if not f.exists():
            missing.append(tag)
            continue
        survivor_srcs[tag] = f.read_text(encoding="utf-8")
    if missing:
        print(json.dumps({"error": f"survivor file(s) missing under {work}: {missing}"},
                         indent=2))
        return 2

    disjointness = find_disjoint_pairs(base_src, survivor_srcs)

    targets = {}
    skipped_no_targets = []
    for tag in survivors:
        t = explicit_targets.get(tag) or _mechanisms_targets(run_dir, tag)
        if t:
            targets[tag] = t
        else:
            skipped_no_targets.append(tag)

    merges = []
    ready_for_gate = []
    for a, b in disjointness["disjoint_pairs"]:
        if a in skipped_no_targets or b in skipped_no_targets:
            merges.append({"pair": [a, b], "attempted": False,
                          "reason": "no target task ids for at least one branch "
                                    "(pass --targets or record them in mechanisms.jsonl)"})
            continue
        union_tasks = sorted(set(targets[a]) | set(targets[b]))
        out_tag = f"merge_{a}_{b}"
        result = _integrate(run_dir, project, work, base_dir, a, b, union_tasks,
                            [i.strip() for i in args.canary.split(",") if i.strip()],
                            canary_auto, args.canary_floor, args.n, args.conc,
                            args.base_seed, args.floor, args.file, args.prose, out_tag)
        built = bool(result.get("out")) and (work / out_tag).is_dir() and "error" not in result
        merges.append({"pair": [a, b], "attempted": True, "tag": out_tag, "built": built,
                      "targets": union_tasks, "result": result})
        if built:
            ready_for_gate.append(out_tag)
            mechanisms.ledger(run_dir).parent.mkdir(parents=True, exist_ok=True)
            mechanisms_args = argparse.Namespace(
                run_dir=run_dir, owner=out_tag, status="proposed",
                mechanism=f"pairwise merge of disjoint-cluster survivors {a} + {b}",
                evidence=f"integrate.py accepted: {sorted(result.get('accepted', []))}; "
                         f"final_objective={result.get('final_objective')}",
                touches=sorted(disjointness["changed"].get(a, []))
                        + sorted(disjointness["changed"].get(b, [])),
                task=union_tasks, supersedes=[])
            # mechanisms.add() is written as a CLI subcommand and prints its own JSON on
            # success — fine standalone, but this script has ONE JSON document on stdout
            # (see main()'s final print) and mixing a second one in breaks any caller
            # parsing stdout as JSON. Suppress its print; the ledger write itself is
            # unaffected.
            with contextlib.redirect_stdout(io.StringIO()):
                mechanisms.add(mechanisms_args)

    out = {
        "base": args.base,
        "survivors": survivors,
        "changed_functions": disjointness["changed"],
        "disjoint_pairs": disjointness["disjoint_pairs"],
        "overlapping_pairs_skipped": disjointness["overlapping_pairs"],
        "skipped_no_targets": skipped_no_targets,
        "merges": merges,
        "ready_for_gate": ready_for_gate,
        "next": (f"python round.py --run-dir {run_dir} --project {project} "
                 f"--candidates {','.join(ready_for_gate)} --n-trials {args.n} "
                 "— gates each merge through the SAME cascade as any candidate"
                 if ready_for_gate else
                 "no merge candidate was built — see merges[].reason / merges[].result.error"),
    }
    text = json.dumps(out, indent=2)
    print(text)
    if args.json_out:
        Path(args.json_out).write_text(text, encoding="utf-8")
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Merge N per-task-optimised capability copies into one candidate, via git's 3-way merge.

The per-task phase gives K optimisers a 30x cheaper feedback loop each, but they all edit the
SAME two files from the SAME base, so their results have to be combined before anything can be
gated. Hand-merging K policy rewrites is how a good round quietly becomes a bad one.

So don't hand-merge. `git merge-file` already implements 3-way merge correctly, including
conflict detection, and it has been debugged by more people than this repo has commits. Base
becomes a commit, each optimiser's copy becomes a branch, and merging happens one branch at a
time so a conflict is attributable to a specific pair rather than to "the merge".

A conflict is a SIGNAL, not an accident: two optimisers guarding the same moment in the same tool
means their diagnoses overlap, and the round should ship one of them, not a stitched-together
hybrid neither one measured. Conflicts are reported and left for a decision; they are never
auto-resolved.

    python merge_taskopt.py --root <dir-of-optimiser-copies> \\
        --base <parent-artifact> --out <merged-candidate> --include t7 t17 u33 ...
"""

import argparse
import json
import shutil
import subprocess
import sys
from pathlib import Path

FILES_DEFAULT = "policy/policy.md,tools/tools.py"


def git(repo: Path, *args: str, check: bool = True) -> subprocess.CompletedProcess:
    return subprocess.run(
        ["git", *args], cwd=repo, capture_output=True, text=True, check=check,
        env={"GIT_AUTHOR_NAME": "capevolve", "GIT_AUTHOR_EMAIL": "capevolve@local",
             "GIT_COMMITTER_NAME": "capevolve", "GIT_COMMITTER_EMAIL": "capevolve@local",
             "PATH": "/usr/bin:/bin:/usr/local/bin", "HOME": "/tmp"},
    )


def stage(repo: Path, src: Path, files: list[str]) -> None:
    """Copy the merge-relevant files from src into the repo working tree."""
    for rel in files:
        s, d = src / rel, repo / rel
        if s.exists():
            d.parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(s, d)


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", required=True, help="the parent artifact every optimiser started from")
    ap.add_argument("--out", required=True, help="destination candidate dir")
    ap.add_argument("--include", nargs="+", required=True,
                    help="subdir names under --root to merge. Use NAME:PARENT when an optimiser "
                         "was REBASED onto another one's copy (e.g. t17b:u67b) — its branch is "
                         "then cut from PARENT so only its own deltas are applied. Without this "
                         "a rebased copy's diff re-applies everything its parent already did and "
                         "conflicts with the parent's own branch.")
    ap.add_argument("--root", required=True, help="dir holding the per-task optimiser copies")
    ap.add_argument("--files", default=FILES_DEFAULT,
                    help="comma-separated capability-relative files to 3-way merge")
    ap.add_argument("--subdirs", default="policy,tools,reference",
                    help="comma-separated capability subdirs to copy into --out")
    ap.add_argument("--union-on-conflict", action="store_true",
                    help="resolve a conflict by keeping BOTH sides (a git union merge driver). "
                         "Legitimate ONLY for textual collisions of DISTINCT additions — two new "
                         "functions or two new dict keys that happen to land on adjacent lines. "
                         "Never for a semantic conflict, where two optimisers arbitrate the SAME "
                         "decision differently: union there ships contradictory guidance nobody "
                         "measured. Union-resolved files are named in the output so the claim can "
                         "be checked, and the result MUST be validated (it can break syntax) and "
                         "gated as a whole before it is believed.")
    ap.add_argument("--repo", default="/tmp/capevolve_merge", help="scratch git repo")
    args = ap.parse_args()

    base, out = Path(args.base).resolve(), Path(args.out).resolve()
    root = Path(args.root).resolve()
    files = [f.strip() for f in args.files.split(",") if f.strip()]
    subdirs = [s.strip() for s in args.subdirs.split(",") if s.strip()]
    repo = Path(args.repo)
    shutil.rmtree(repo, ignore_errors=True)
    repo.mkdir(parents=True)

    git(repo, "init", "-q", "-b", "base")
    if args.union_on_conflict:
        (repo / ".gitattributes").write_text(
            "\n".join(f"{f} merge=union" for f in files) + "\n")
        git(repo, "config", "merge.union.name", "keep both sides")
        git(repo, "config", "merge.union.driver", "git merge-file --union -L base -L ours "
                                                 "-L theirs %A %O %B")
    stage(repo, base, files)
    git(repo, "add", "-A")
    git(repo, "commit", "-qm", "base")

    spec = []
    for item in args.include:
        name, _, parent = item.partition(":")
        spec.append((name, parent or "base"))
    order = {n: i for i, (n, _) in enumerate(spec)}
    for name, parent in spec:
        if parent != "base" and order.get(parent, 1 << 30) > order[name]:
            print(json.dumps({"error": f"{name} is rebased onto {parent}, so {parent} must be "
                                       f"listed BEFORE it in --include"}, indent=2))
            return 2

    for name, parent in spec:
        src = root / name
        if not any((src / f).exists() for f in files):
            print(f"skip {name}: none of {files} present", file=sys.stderr)
            continue
        git(repo, "checkout", "-q", "-b", name, parent)
        stage(repo, src, files)
        git(repo, "add", "-A")
        r = git(repo, "commit", "-qm", name, check=False)
        if "nothing to commit" in (r.stdout + r.stderr):
            print(f"note {name}: identical to base (no edit)", file=sys.stderr)

    git(repo, "checkout", "-q", "base")
    git(repo, "checkout", "-q", "-b", "merged")

    merged, conflicts, empty, unioned = [], {}, [], []
    for name, _parent in spec:
        if not any((root / name / f).exists() for f in files):
            continue
        r = git(repo, "merge", "--no-edit", name, check=False)
        if r.returncode == 0:
            if args.union_on_conflict:
                unioned.append(name)
            if "Already up to date" in r.stdout:
                empty.append(name)
            else:
                merged.append(name)
            continue
        status = git(repo, "diff", "--name-only", "--diff-filter=U").stdout.split()
        conflicts[name] = status
        git(repo, "merge", "--abort", check=False)

    out.mkdir(parents=True, exist_ok=True)
    for sub in subdirs:
        shutil.rmtree(out / sub, ignore_errors=True)
        if (base / sub).exists():
            shutil.copytree(base / sub, out / sub, ignore=shutil.ignore_patterns("__pycache__"))
    for rel in files:
        if (repo / rel).exists():
            shutil.copy2(repo / rel, out / rel)
    shutil.rmtree(out / "tools" / "__pycache__", ignore_errors=True)

    diff = git(repo, "diff", "--stat", "base", "merged").stdout.strip()
    print(json.dumps({
        "out": str(out),
        "bases": {n: p for n, p in spec},
        "merged_cleanly": merged,
        "union_resolution_enabled": bool(args.union_on_conflict),
        "union_candidates": unioned if args.union_on_conflict else [],
        "no_edit": empty,
        "conflicted": conflicts,
        "diffstat_vs_base": diff.splitlines(),
        "next": ("syntax-check, RENDER THE LIVE TOOLSET (union resolution can break syntax or "
                 "duplicate a definition), then gate the merged dir on full val"),
    }, indent=2))
    return 1 if conflicts else 0


if __name__ == "__main__":
    raise SystemExit(main())

"""microcase — TDD-style micro-tests: reproduce ONE diagnosed failure deterministically,
in seconds, before any real rollout is spent on a candidate.

Adopts Harbor/Terminal-Bench's task shape (``task.yaml`` + fixture/environment + a
``tests/`` dir that asserts pass/fail and writes a reward, no LLM judge required for the
assertion itself), scoped down from "a whole task" to "the one call or turn where a
diagnosed defect lives" — see #434 section 3 and #436.

    $R/microcases/<cluster_id>/
      case.yaml       # {id, cluster_id, source_task_ids, source_rollout, timeout_s,
                       #  expects: guard_fires|call_shape|tool_selected, assert: {...}}
      fixture/          # extracted VERBATIM from the diagnosed rollout — never invented
      reproduce.py      # replays the fixture against the candidate directly: one tool
                        # call / one narrow unit, no full multi-turn episode, no LLM
      assert.py         # deterministic pass/fail against case.yaml's `assert` spec

``reproduce.py``/``assert.py`` are project-specific code (they know the candidate's tool
module) — this script cannot write them for you, only scaffold them from a real diagnosed
rollout so the author fills in the one project-aware step. What IS mechanical: extracting
the tool-call fixture verbatim, and running the case once it exists.

Two subcommands:

    python microcase.py gen  --rollout <path/to/task__tag__tK.json> \\
        --cluster-id netguard_duplicate_payment --expects guard_fires \\
        --description "..." --assert-metric payment_history_len --assert-op "<=" \\
        --assert-value 1 --out $R/microcases/netguard_duplicate_payment

    python microcase.py run  --case $R/microcases/<id> \\
        --candidate $R/work/<tag> --project P [--python PYEXE]

    python microcase.py run-all --cases-dir $R/microcases \\
        --candidate $R/work/<tag> --project P

``run``/``run-all`` never touch the run dir's budget or state — a micro-test is a
pre-rollout filter, not a rollout, and cannot itself accept a candidate (only the full-val
gate can). A ``fail`` here is the new cheap reject basis (``micro_test_fail``, see
``commit.py --reject-basis``): the mechanism the candidate claims to add provably does not
fire, discovered without spending a single evaluation.
"""

from __future__ import annotations

import argparse
import json
import shutil
import subprocess
import sys
import tempfile
import time
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve.specfile import read_yaml

VALID_EXPECTS = ("guard_fires", "call_shape", "tool_selected")
VALID_OPS = ("==", "!=", "<", "<=", ">", ">=")

CASE_TEMPLATE = """\
id: {id}
cluster_id: {cluster_id}
source_task_ids: [{source_task_ids}]
source_rollout: {source_rollout}
timeout_s: {timeout_s}
expects: {expects}
description: {description}
assert:
  metric: {assert_metric}
  op: "{assert_op}"
  value: {assert_value}
"""

REPRODUCE_STUB = '''\
#!/usr/bin/env python3
"""reproduce.py for microcase {case_id!r} — replays the fixture extracted from
{source_rollout} against the CANDIDATE's own code, directly (no LLM, no multi-turn
episode). Generated by microcase.py gen; TODO fill in the marked section — this part
is necessarily project-specific (it knows the candidate's tool module and how to
construct the minimal state the fixture's calls need).

Contract: write a JSON object to --out. On success, include whatever fields
assert.py's `assert.metric` in case.yaml needs. On an environment problem (a missing
project dependency, not a candidate defect), write {{"status": "error", "reason": ...}}
and exit 2 — that is infra, not a fail, and the runner reports it as such.
"""
import argparse
import json
import sys
from pathlib import Path


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--candidate", required=True)
    ap.add_argument("--project", required=True)
    ap.add_argument("--fixture", required=True)
    ap.add_argument("--out", required=True)
    args = ap.parse_args()

    calls = json.loads((Path(args.fixture) / "calls.json").read_text())["calls"]

    # TODO: import the candidate's tool module (Path(args.candidate) / ...), construct
    # the minimal state the fixture's calls need, replay `calls` against it in order,
    # and write the observed post-call state to args.out. Example:
    #
    #   sys.path.insert(0, str(Path(args.candidate)))
    #   from tools.tools import YourToolkitClass
    #   ...
    Path(args.out).write_text(json.dumps(
        {{"status": "error", "reason": "reproduce.py not yet implemented for this case"}}))
    return 2


if __name__ == "__main__":
    raise SystemExit(main())
'''

ASSERT_PY = '''\
#!/usr/bin/env python3
"""assert.py — deterministic pass/fail for a microcase, no LLM judge.

Reads reproduce.py's result JSON and case.yaml's `assert` spec (metric/op/value),
evaluates the comparison, and writes {status, metric, observed, expected} to --out.
Exit 0 = pass, 1 = fail, 2 = reproduce.py reported an environment error (not a fail).
"""
import argparse
import json
import operator
from pathlib import Path

OPS = {"==": operator.eq, "!=": operator.ne, "<": operator.lt,
       "<=": operator.le, ">": operator.gt, ">=": operator.ge}


def _read_case_yaml(text: str) -> dict:
    """Tiny reader for THIS script's own case.yaml shape (flat keys + one nested
    `assert:` block) — no PyYAML dependency needed for a file this script itself
    wrote via its rigid template.
    """
    try:
        import yaml  # type: ignore
        return yaml.safe_load(text) or {}
    except Exception:
        pass
    out: dict = {}
    section = None
    for raw in text.splitlines():
        if not raw.strip():
            continue
        if not raw.startswith(" "):
            key, _, val = raw.partition(":")
            key, val = key.strip(), val.strip()
            if val == "":
                section = {}
                out[key] = section
            else:
                section = None
                out[key] = json.loads(val) if val[:1] in ('[', '{', '"') or val[:1].isdigit() or val in (
                    "true", "false") else val
        elif section is not None:
            key, _, val = raw.strip().partition(":")
            key, val = key.strip(), val.strip().strip('"')
            try:
                section[key] = json.loads(val)
            except json.JSONDecodeError:
                section[key] = val
    return out


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--case", required=True, help="the case directory (holds case.yaml)")
    ap.add_argument("--result", required=True, help="reproduce.py's output JSON")
    ap.add_argument("--out", required=True)
    args = ap.parse_args()

    case = _read_case_yaml(Path(args.case, "case.yaml").read_text())
    result = json.loads(Path(args.result).read_text())

    if result.get("status") == "error":
        out = {"status": "error", "reason": result.get("reason", "unknown")}
        Path(args.out).write_text(json.dumps(out, indent=2))
        print(json.dumps(out, indent=2))
        return 2

    spec = case["assert"]
    metric, op, expected = spec["metric"], spec["op"], spec["value"]
    if metric not in result:
        out = {"status": "error",
               "reason": f"reproduce.py's result has no {metric!r} field"}
        Path(args.out).write_text(json.dumps(out, indent=2))
        print(json.dumps(out, indent=2))
        return 2

    observed = result[metric]
    passed = OPS[op](observed, expected)
    out = {"status": "pass" if passed else "fail",
           "metric": metric, "op": op, "observed": observed, "expected": expected}
    Path(args.out).write_text(json.dumps(out, indent=2))
    print(json.dumps(out, indent=2))
    return 0 if passed else 1


if __name__ == "__main__":
    raise SystemExit(main())
'''


def _load_rollout(path: Path) -> dict:
    return json.loads(path.read_text(encoding="utf-8"))


def _extract_calls(rollout_record: dict) -> list[dict]:
    """Every tool call in the rollout's trace, verbatim, in order.

    Same lookup ``cluster.py``'s ``_call_names`` uses (``Rollout.tool_calls`` or an
    OpenAI-style ``tool_calls`` per message in ``Rollout.trace``) — never invented,
    never summarized.
    """
    roll = rollout_record.get("rollout", rollout_record)
    calls: list[dict] = []
    for c in roll.get("tool_calls") or []:
        calls.append({"name": c.get("name"), "arguments": c.get("arguments")})
    for msg in roll.get("trace") or []:
        if not isinstance(msg, dict):
            continue
        for c in msg.get("tool_calls") or []:
            calls.append({"name": c.get("name"), "arguments": c.get("arguments")})
    return calls


def cmd_gen(args) -> int:
    rollout_path = Path(args.rollout).resolve()
    record = _load_rollout(rollout_path)
    calls = _extract_calls(record)
    if not calls:
        print(json.dumps({
            "error": f"no tool calls found in {rollout_path}",
            "why": "a micro-case fixture is extracted from a real tool call; a rollout "
                   "with none has nothing to reproduce at this grain.",
        }, indent=2))
        return 2

    out_dir = Path(args.out).resolve()
    fixture_dir = out_dir / "fixture"
    fixture_dir.mkdir(parents=True, exist_ok=True)

    fixture_dir.joinpath("calls.json").write_text(
        json.dumps({"calls": calls}, indent=2))
    score = record.get("score")
    if score is not None:
        fixture_dir.joinpath("score.json").write_text(json.dumps(score, indent=2))

    case_id = args.cluster_id
    case_yaml = CASE_TEMPLATE.format(
        id=case_id,
        cluster_id=args.cluster_id,
        source_task_ids=", ".join(args.source_task_ids.split(",")),
        source_rollout=rollout_path.name,
        timeout_s=args.timeout_s,
        expects=args.expects,
        description=json.dumps(args.description),
        assert_metric=args.assert_metric,
        assert_op=args.assert_op,
        assert_value=json.dumps(args.assert_value),
    )
    out_dir.joinpath("case.yaml").write_text(case_yaml)

    reproduce_path = out_dir / "reproduce.py"
    if not reproduce_path.exists():
        reproduce_path.write_text(REPRODUCE_STUB.format(
            case_id=case_id, source_rollout=rollout_path.name))
    assert_path = out_dir / "assert.py"
    if not assert_path.exists():
        assert_path.write_text(ASSERT_PY)

    result = {
        "case_dir": str(out_dir),
        "calls_extracted": len(calls),
        "wrote": [str(out_dir / "case.yaml"), str(fixture_dir / "calls.json"),
                  str(reproduce_path), str(assert_path)],
        "next_step": f"fill in the TODO in {reproduce_path} — the rest runs as-is",
    }
    print(json.dumps(result, indent=2))
    return 0


def _run_one(case_dir: Path, candidate: Path, project: Path, python: str) -> dict:
    case = read_yaml((case_dir / "case.yaml").read_text(encoding="utf-8"))
    timeout_s = float(case.get("timeout_s", 10))
    t0 = time.time()
    # Scratch files, NOT written beside case.yaml: a case directory is often a
    # committed fixture (this run's own worked example included), and running it
    # must not dirty that tree with per-run output.
    scratch = Path(tempfile.mkdtemp(prefix="microcase_"))
    result_path = scratch / "result.json"
    verdict_path = scratch / "verdict.json"

    try:
        return _run_one_in(case_dir, candidate, project, python, case, timeout_s, t0,
                           result_path, verdict_path)
    finally:
        shutil.rmtree(scratch, ignore_errors=True)


def _run_one_in(case_dir, candidate, project, python, case, timeout_s, t0,
                result_path, verdict_path) -> dict:
    try:
        proc = subprocess.run(
            [python, str(case_dir / "reproduce.py"),
             "--candidate", str(candidate), "--project", str(project),
             "--fixture", str(case_dir / "fixture"), "--out", str(result_path)],
            capture_output=True, text=True, timeout=timeout_s,
        )
    except subprocess.TimeoutExpired:
        return {"case_id": case.get("id", case_dir.name), "status": "error",
                "reason": f"reproduce.py exceeded timeout_s={timeout_s}",
                "wall_seconds": round(time.time() - t0, 3)}

    if proc.returncode not in (0, 2) or not result_path.exists():
        return {"case_id": case.get("id", case_dir.name), "status": "error",
                "reason": f"reproduce.py exit {proc.returncode}: "
                          f"{(proc.stderr or proc.stdout)[-2000:]}",
                "wall_seconds": round(time.time() - t0, 3)}

    if proc.returncode == 2:
        verdict = json.loads(result_path.read_text())
        verdict.setdefault("status", "error")
    else:
        assert_proc = subprocess.run(
            [python, str(case_dir / "assert.py"), "--case", str(case_dir),
             "--result", str(result_path), "--out", str(verdict_path)],
            capture_output=True, text=True, timeout=timeout_s,
        )
        if not verdict_path.exists():
            return {"case_id": case.get("id", case_dir.name), "status": "error",
                    "reason": f"assert.py exit {assert_proc.returncode}: "
                              f"{(assert_proc.stderr or assert_proc.stdout)[-2000:]}",
                    "wall_seconds": round(time.time() - t0, 3)}
        verdict = json.loads(verdict_path.read_text())

    verdict["case_id"] = case.get("id", case_dir.name)
    verdict["wall_seconds"] = round(time.time() - t0, 3)
    return verdict


def cmd_run(args) -> int:
    verdict = _run_one(Path(args.case).resolve(), Path(args.candidate).resolve(),
                        Path(args.project).resolve(), args.python)
    print(json.dumps(verdict, indent=2))
    return {"pass": 0, "fail": 1}.get(verdict.get("status"), 2)


def cmd_run_all(args) -> int:
    cases_dir = Path(args.cases_dir).resolve()
    cand = Path(args.candidate).resolve()
    proj = Path(args.project).resolve()
    verdicts = []
    for case_dir in sorted(p for p in cases_dir.iterdir()
                           if p.is_dir() and (p / "case.yaml").exists()):
        verdicts.append(_run_one(case_dir, cand, proj, args.python))
    failed = [v for v in verdicts if v.get("status") == "fail"]
    errored = [v for v in verdicts if v.get("status") == "error"]
    out = {
        "candidate": str(cand),
        "cases": len(verdicts),
        "failed": [v["case_id"] for v in failed],
        "errored": [v["case_id"] for v in errored],
        "verdicts": verdicts,
        "micro_test_fail": bool(failed),
        "recommendation": (
            "reject --reject-basis micro_test_fail: the mechanism this candidate "
            "targets provably does not fire — no rollout needed" if failed else
            "proceed to screen/full-val" if not errored else
            "cases errored (infra, not a candidate defect) — fix the case or its "
            "environment before trusting a pass here"
        ),
    }
    print(json.dumps(out, indent=2))
    return 1 if failed else 0


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="microcase")
    sub = p.add_subparsers(dest="cmd", required=True)

    g = sub.add_parser("gen", help="scaffold a case from a real diagnosed rollout")
    g.add_argument("--rollout", required=True,
                   help="path to the rollout record (<task>__<tag>__t<k>.json)")
    g.add_argument("--cluster-id", required=True,
                   help="filename-safe id — also the case's own id, by convention")
    g.add_argument("--source-task-ids", required=True, help="comma-separated task ids")
    g.add_argument("--expects", required=True, choices=VALID_EXPECTS)
    g.add_argument("--description", required=True)
    g.add_argument("--assert-metric", required=True,
                   help="key in reproduce.py's result JSON that assert.py compares")
    g.add_argument("--assert-op", required=True, choices=VALID_OPS)
    g.add_argument("--assert-value", required=True,
                   help="JSON-decoded if it parses (so 1, true, \"x\" all work); else kept as a string")
    g.add_argument("--timeout-s", type=float, default=10.0)
    g.add_argument("--out", required=True, help="the case directory to write")
    g.set_defaults(func=cmd_gen)

    r = sub.add_parser("run", help="run ONE case against a candidate")
    r.add_argument("--case", required=True)
    r.add_argument("--candidate", required=True)
    r.add_argument("--project", required=True)
    r.add_argument("--python", default=sys.executable)
    r.set_defaults(func=cmd_run)

    ra = sub.add_parser("run-all", help="run every case under a directory against a candidate")
    ra.add_argument("--cases-dir", required=True)
    ra.add_argument("--candidate", required=True)
    ra.add_argument("--project", required=True)
    ra.add_argument("--python", default=sys.executable)
    ra.set_defaults(func=cmd_run_all)

    args = p.parse_args(argv)
    if args.cmd == "gen":
        try:
            args.assert_value = json.loads(args.assert_value)
        except json.JSONDecodeError:
            pass
        args.timeout_s = int(args.timeout_s) if float(args.timeout_s).is_integer() else args.timeout_s
    return args.func(args)


if __name__ == "__main__":
    raise SystemExit(main())

"""Combine several INDEPENDENT paired runs into one verdict, with the SE taken ACROSS runs.

One paired run cannot settle anything here. Measured on this benchmark: a byte-identical control,
re-run on the SAME seeds at temperature 0, moved 0.6467 -> 0.7267 — a paired delta of +0.0800 that
"passes" a k_se=1.0 gate. So the within-run SE (across tasks) understates the real uncertainty,
because it cannot see run-to-run nondeterminism at all.

The fix is to repeat the whole paired comparison on distinct seed blocks and take the spread of
the per-run deltas as the error. That estimator sees both sources of variance, needs no assumption
about where the noise comes from, and is the only thing that would have caught the retracted accept
before it was reported.

    python multirep.py cand1.json:ctl1.json cand2.json:ctl2.json ...
"""
import json
import math
import statistics as st
import sys
from pathlib import Path

def rates(p):
    d = json.loads(Path(p).read_text(encoding="utf-8"))
    return {t: r["rate"] for t, r in (d.get("per_task") or {}).items() if "rate" in r}


def main() -> int:
    pairs = [a.split(":") for a in sys.argv[1:]]
    if not pairs:
        print("usage: multirep.py cand.json:ctl.json ...", file=sys.stderr)
        return 2
    rows, deltas = [], []
    for cand, ctl in pairs:
        c, k = rates(cand), rates(ctl)
        common = sorted(set(c) & set(k))
        if not common:
            continue
        d = sum(c[t] - k[t] for t in common) / len(common)
        rows.append({"run": Path(cand).stem, "cand": round(sum(c[t] for t in common) / len(common), 4),
                     "ctl": round(sum(k[t] for t in common) / len(common), 4),
                     "paired_delta": round(d, 4), "tasks": len(common)})
        deltas.append(d)
    n = len(deltas)
    mean = sum(deltas) / n
    sd = st.stdev(deltas) if n > 1 else float("nan")
    se = sd / math.sqrt(n) if n > 1 else float("nan")
    out = {
        "runs": rows,
        "n_runs": n,
        "mean_paired_delta": round(mean, 4),
        "sd_across_runs": None if n < 2 else round(sd, 4),
        "se_across_runs": None if n < 2 else round(se, 4),
        "t_like": None if n < 2 or se == 0 else round(mean / se, 2),
        "verdict": ("need >= 2 independent runs" if n < 2 else
                    "DEMONSTRATED (delta > 2 SE across runs)" if mean > 2 * se else
                    "NOT DEMONSTRATED at this sample size"),
        "note": ("SE here is across whole runs, so it includes run-to-run nondeterminism that a "
                 "single run's across-task SE cannot see. A byte-identical control re-run moved "
                 "0.0800 on this benchmark, which is why this estimator exists."),
    }
    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

"""round — evaluate a whole round's candidates PLUS a null control, then gate them.

Why this exists. Three things went wrong the same way in every prior agent-optimize run,
and all three are round-level bookkeeping the driver was doing by hand:

  1. **No null control.** A candidate's val mean was compared against a parent mean measured
     in an earlier round, so ordinary re-measurement noise looked like a signal in both
     directions. Three runs discovered this reactively, after the fact. Here the byte-identical
     copy ``ctl_null`` is a first-class member of every round, so the round reports its OWN
     noise floor and a candidate inside that band is visibly not evidence.
  2. **Serial evals wasted the wall clock.** A multi-turn agent rollout is tail-dominated (measured: 6 to
     40 minutes at ``max_steps=100``), so one candidate's full-val eval costs about as long as
     its slowest single rollout. Evaluating candidates one after another multiplies that tail by
     the number of candidates for no statistical benefit. Each eval is an independent process
     with its own adapter ``apply()``, so they parallelise safely — the reason this is a script
     and not prose is that ``apply()`` mutates a process-global registry, which is exactly the
     kind of footgun a driver should not have to remember.
  3. **The gate must stay serial.** ``set_best`` mutates run state, so gating is done after all
     evals land, one candidate at a time, re-reading ``best_id`` each time.

This script does NOT commit. It prints the table; the driver reads it, decides, and calls
``commit.py`` — because choosing which part of a bundled edit to keep is a judgement that
belongs to the driver, and ``regressions`` is the input to it.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import shutil
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

import _bootstrap  # noqa: F401  # side-effect import: seeds sys.path for cap_evolve

# Sibling script, imported for GATE_MODES: --mode is forwarded to it verbatim, so its list of
# accepted values is the only correct source for ours. Importable on the same terms as
# _bootstrap above — this directory is already on sys.path or that import would have failed.
import gate_check

from cap_evolve import RunDir, harness

#: Gate measurement concurrency. The default is deliberately low; the ceiling is where the
#: measured degradation is established (~0.08 at the arm level above 25, ~0.03 at 8), so above
#: it a verdict cannot resolve the effect the round is looking for and the round is refused.
DEFAULT_CONCURRENCY = 8
MAX_RESOLVING_CONCURRENCY = 25

HERE = Path(__file__).resolve().parent
SKILLS = Path(os.environ.get("CAPEVOLVE_SKILLS_DIR", HERE.parents[2]))


def _all_tables(run_dir) -> list[tuple[int, int, Path]]:
    """Every round table on disk as (iteration, attempt, path), oldest first.

    The name is matched strictly rather than by glob: ``round_i1*.json`` also matches
    ``round_i10.json``, so a glob would count iteration 10's tables as re-gates of iteration 1
    and shift every name in this round — quietly, and worse the further a run gets.
    """
    work = run_dir.root / "work"
    if not work.is_dir():
        return []
    found = []
    for path in work.iterdir():
        if not path.is_file():
            continue
        m = re.fullmatch(r"round_i(\d+)(?:\.r(\d+))?\.json", path.name)
        if m:
            found.append((int(m.group(1)), int(m.group(2) or 0), path))
    return sorted(found)


def _round_tables(run_dir) -> list[tuple[int, Path]]:
    """This iteration's tables as (attempt, path), lowest attempt first."""
    it = int(run_dir.spent.iterations)
    return [(att, path) for i, att, path in _all_tables(run_dir) if i == it]


def round_attempt(run_dir) -> int:
    """How many times this iteration has already been gated: 0 for the first attempt.

    A round's identity is (iteration, attempt), and BOTH halves have to reach the names on
    disk. The table already had the attempt half — a same-iteration re-run is written
    ``round_i1.r1.json`` rather than overwriting ``round_i1.json`` — but the control ROLLOUTS
    did not, so the one operation this script explicitly supports re-measured the previous
    attempt's control under the same tag and deleted the numbers the first table cites.
    Counting the tables here is what keeps the two halves in agreement.
    """
    tables = _round_tables(run_dir)
    # max+1 rather than len, so a hand-deleted table cannot hand out a name that is still taken.
    return max((att for att, _ in tables), default=-1) + 1


def table_stem(run_dir) -> str:
    """Filename stem for this attempt's table: ``round_i1``, then ``round_i1.r1``, …"""
    it = int(run_dir.spent.iterations)
    a = round_attempt(run_dir)
    return f"round_i{it}" if a == 0 else f"round_i{it}.r{a}"


def control_tag(run_dir) -> str:
    """Round-scoped control tag, e.g. ``ctl_null_i2`` — and ``ctl_null_i2a1`` on a re-gate.

    Rollout files are ``<task>__<tag>__t<k>.json``, so a fixed ``ctl_null`` tag makes each
    round's control OVERWRITE the previous round's on disk — destroying the one measurement
    that proves what zero change looked like at that point in the run. The noise floor is
    evidence, and it is per-round (it moves with the parent and with the provider's mood),
    so it gets its own tag per iteration.

    The same argument applies WITHIN an iteration, which is what the ``a<k>`` suffix is for. A
    re-gate is normally run to buy MORE evidence about the same round, and without the suffix it
    bought none: on run 33046360451 the second attempt at iteration 1 re-measured
    ``ctl_null_i1`` (0.4967 -> 0.5067) and ``ctl_null_i1r1`` (0.4800 -> 0.4367) under those exact
    tags, spending 200 metric calls to swap two readings for two others. The round's replicate
    spread went 0.0167 -> 0.0700 and ``round_i1.json`` was left quoting an ``evidence_bar``
    computed from two numbers that no longer existed. With the suffix the attempts accumulate,
    ``prior_attempt_controls`` pools them, and the re-gate gets the four samples it paid for.
    """
    it = int(run_dir.spent.iterations)
    a = round_attempt(run_dir)
    return f"ctl_null_i{it}" if a == 0 else f"ctl_null_i{it}a{a}"


def prior_attempt_controls(run_dir) -> list[dict]:
    """Control replicates measured by EARLIER attempts at this same iteration.

    Read from those attempts' tables rather than from rollouts, because the table is the record
    that survives: a pre-fix run has tables whose rollouts were already overwritten, and the
    table is then the only place the destroyed reading still exists.

    They are byte-identical copies of the same parent measured in the same round, so they are
    samples of the same null and belong in it. Pooling them is the entire point of re-gating.
    """
    out: list[dict] = []
    for att, path in _round_tables(run_dir):
        try:
            table = json.loads(path.read_text(encoding="utf-8"))
        except (OSError, ValueError):
            continue
        if not isinstance(table, dict):
            continue
        for row in table.get("control_replicates") or []:
            if isinstance(row, dict) and row.get("reward") is not None:
                out.append({**row, "from_attempt": att})
    return out


def measurement_context(split: str, n_trials: int, concurrency: int | None) -> dict:
    """What a control replicate's reward is only comparable WITHIN.

    Recorded in the table so a later round can tell whether an existing replicate was measured
    under the same conditions. Trial count and load both move the reading — the concurrency
    numbers this script refuses above are exactly that effect — so a replicate measured at a
    different n or a different load is not a sample of this round's null.
    """
    return {"split": split, "n_trials": int(n_trials), "concurrency": concurrency}


def prior_round_settings(run_dir) -> dict | None:
    """The most recent EARLIER iteration's ``--concurrency``/``--max-parallel``, or ``None``.

    Issue #420 item 9: round 3 of a real run got ``--max-parallel`` 4 (the default) after
    rounds 1-2 both explicitly passed 2, doubling how many candidate evals ran concurrently
    against the same target — a load change that makes the round's own noise floor
    incomparable with the rounds before it, and nobody noticed because nothing was recorded
    to notice it from.
    """
    it = int(run_dir.spent.iterations)
    for i, _att, path in reversed(_all_tables(run_dir)):
        if i >= it:
            continue
        try:
            table = json.loads(path.read_text(encoding="utf-8"))
        except (OSError, ValueError):
            continue
        return {"concurrency": table.get("measurement_concurrency"),
                "max_parallel": table.get("measurement_max_parallel")}
    return None


def parallel_drift_warning(prior: dict | None, concurrency: int | None,
                           max_parallel: int) -> str | None:
    """Did this round's load settings drift from the previous round's, unannounced?

    ``None`` on a prior round that never recorded ``max_parallel`` (a table written before
    this field existed) — a missing measurement is not evidence of drift.
    """
    if not prior:
        return None
    if prior["concurrency"] in (None, concurrency) and prior["max_parallel"] in (None, max_parallel):
        return None
    return (f"this round used --concurrency {concurrency} --max-parallel {max_parallel}, but "
            f"the previous round used --concurrency {prior['concurrency']} --max-parallel "
            f"{prior['max_parallel']}. Gate settings that drift between rounds make the "
            "rounds' noise floors incomparable — confirm this was a deliberate change.")


def _rollouts_present(run_dir, tag: str, split: str, n_trials: int) -> bool:
    """Are ``tag``'s persisted rollouts still on disk at the trial depth this round needs?

    Rollouts are ``<task>__<tag>__t<k>.json`` for k in range(n_trials), so the presence of the
    LAST index is what says the measurement went as deep as this round is asking for.
    """
    d = run_dir.rollouts / split
    return d.is_dir() and any(d.glob(f"*__{tag}__t{n_trials - 1}.json"))


def reusable_controls(run_dir, best: str, measurement: dict, want: int) -> dict | None:
    """Control replicates of THIS SAME parent, already measured in an EARLIER iteration.

    A null control is a byte-identical copy of the parent, re-measured to establish the round's
    noise floor. When ``best_id`` has not moved, the parent is the same bytes as it was, so its
    noise floor has already been paid for and re-measuring it buys nothing: across six real runs
    the controls consumed about 40% of every rollout spent, nearly as much as all candidates
    combined. Reuse only removes the redundant re-measurement — the requirement itself is
    untouched. A NEW parent (any accept) has no established floor and must be measured fresh,
    which is what the ``parent.tag`` check below enforces.

    Reuse requires ALL of:

    * an earlier iteration's table whose ``parent.tag`` is the current ``best`` — i.e. the same
      parent bytes were the parent then, so nothing has been accepted since;
    * the same ``measurement`` context (split, trials, concurrency): a reading taken at a
      different n or load is not a sample of this round's null;
    * at least ``want`` replicates with a reward, all of whose rollouts are still on disk —
      the gate re-reads them, so a pruned run dir falls back to measuring.

    The most recent qualifying iteration wins. Returns None when nothing qualifies, which is the
    signal to measure.
    """
    it = int(run_dir.spent.iterations)
    for i, _att, path in reversed(_all_tables(run_dir)):
        if i >= it:
            continue                      # this iteration's own attempts: prior_attempt_controls
        try:
            table = json.loads(path.read_text(encoding="utf-8"))
        except (OSError, ValueError):
            continue
        if not isinstance(table, dict):
            continue
        if (table.get("parent") or {}).get("tag") != best:
            continue
        if table.get("measurement") != measurement:
            continue
        rows = [r for r in (table.get("control_replicates") or [])
                if isinstance(r, dict) and r.get("reward") is not None and r.get("tag")]
        if len(rows) < want:
            continue
        if not all(_rollouts_present(run_dir, r["tag"], measurement["split"],
                                     measurement["n_trials"]) for r in rows):
            continue
        return {"from_iteration": i, "from_table": str(path),
                "tags": [str(r["tag"]) for r in rows]}
    return None


def _evaluate(run_dir: Path, project: Path, tag: str, split: str, n_trials: int,
              concurrency: int | None) -> dict:
    """Run the evaluate phase for one tag in its own process."""
    cmd = [sys.executable, str(SKILLS / "phases" / "evaluate" / "scripts" / "run.py"),
           "--run-dir", str(run_dir), "--project", str(project),
           "--candidate", str(Path(run_dir) / "work" / tag),
           "--split", split, "--n-trials", str(n_trials)]
    env = dict(os.environ)
    if concurrency:
        # Canonical, benchmark-neutral name. A runner whose knob predates this convention gets it
        # through CAPEVOLVE_CONCURRENCY_ENV (comma-separated extra names), so nothing here is
        # specific to one benchmark's environment variable.
        env["CAPEVOLVE_MAX_CONCURRENCY"] = str(concurrency)
        for name in [n.strip() for n in
                     os.environ.get("CAPEVOLVE_CONCURRENCY_ENV", "").split(",") if n.strip()]:
            env[name] = str(concurrency)
    p = subprocess.run(cmd, capture_output=True, text=True, env=env)
    try:
        return {"tag": tag, "rc": p.returncode, **json.loads(p.stdout)}
    except Exception:  # noqa: BLE001
        return {"tag": tag, "rc": p.returncode, "error": (p.stderr or p.stdout)[-800:]}


class GateCheckFailed(RuntimeError):
    """The gate did not run. That is NOT the same as a gate that decided against a candidate.

    On run 33492876620 round 3 this distinction did not exist. ``_gate`` returned
    ``{"error": ...}`` — a dict with every verdict key MISSING — and the caller ``.get()``s the
    keys it wants, so the table was written with ``reward``, ``gate_delta``, ``gate_threshold``
    and ``verdict`` all ``null`` for all three candidates AND the control, ``eval_rc: 0``,
    100/100 rollouts scored on disk. A row that reads "this candidate did not move" for a
    candidate nothing judged is the most expensive kind of wrong this script can be.
    """

    def __init__(self, tag: str, rc: int, detail: str):
        self.tag, self.rc, self.detail = tag, rc, detail
        super().__init__(
            f"gate_check.py failed for tag {tag!r} (rc={rc}): {detail}\n"
            "The round is NOT booked. Nothing was written to the round table, because a failed "
            "gate is not a verdict — fix the cause and re-run this round; the rollouts are "
            "already on disk, so re-gating costs nothing.")


def _gate(run_dir: Path, tag: str, k_se: float, mode: str, veto: bool,
          current: str | None = None) -> dict:
    cmd = [sys.executable, str(HERE / "gate_check.py"), "--run-dir", str(run_dir),
           "--candidate", tag, "--k-se", str(k_se), "--mode", mode]
    if current:
        cmd += ["--current", current]
    if veto:
        cmd.append("--veto-regressions")
    p = subprocess.run(cmd, capture_output=True, text=True)
    # A non-zero rc is a failure whether or not its stdout parses. gate_check.py's own two
    # `return 2` paths (no --current and no best_id; no rollouts for the tag) print WELL-FORMED
    # JSON, so `json.loads` succeeds on them and the old code handed the error dict straight
    # back as if it were a result. Checking rc first is what closes that second path — fixing
    # the --mode flag alone would have left it wide open.
    if p.returncode != 0:
        raise GateCheckFailed(tag, p.returncode, ((p.stderr or p.stdout) or "").strip()[-800:])
    try:
        return json.loads(p.stdout)
    except Exception as exc:  # noqa: BLE001
        raise GateCheckFailed(
            tag, p.returncode,
            f"exited 0 but stdout is not JSON: {(p.stdout or '').strip()[-800:]}") from exc


def gate_unless_eval_failed(ev: dict, run_dir: Path, tag: str, k_se: float, mode: str,
                            veto: bool, current: str | None = None) -> dict:
    """Gate a tag, unless its own EVALUATION failed — then there was never anything to gate.

    The distinction the round needs and did not have. A candidate whose eval died has no
    rollouts, so ``gate_check.py`` exits 2 for a legitimate reason and the row belongs in the
    table carrying its ``eval_rc``/``eval_error``. A candidate whose eval SUCCEEDED and whose
    gate still failed is a framework bug, and the round stops.
    """
    try:
        return _gate(run_dir, tag, k_se, mode, veto, current)
    except GateCheckFailed:
        if ev.get("rc") or ev.get("error"):
            return {}
        raise


def assert_rows_were_judged(rows: list[dict]) -> None:
    """Refuse to publish a round table row that no gate actually decided.

    The backstop for the invariant itself, independent of any particular cause: a row whose
    ``eval_rc`` is 0 was measured — its rollouts exist and were scored — so a missing reward can
    only mean the GATE failed. A row whose evaluation genuinely failed keeps its ``None`` and its
    ``eval_rc``/``eval_error``, because a real infrastructure failure must stay a REPORT rather
    than becoming a crash: that is the one case where "no verdict" is the honest answer.
    """
    unjudged = [r["tag"] for r in rows
                if r.get("reward") is None and not r.get("eval_rc") and not r.get("eval_error")]
    if unjudged:
        raise GateCheckFailed(
            ", ".join(unjudged), 0,
            "the evaluation succeeded (eval_rc 0, rollouts scored) but no gate verdict came "
            "back, so these rows would publish as 'no movement' for candidates nothing judged")


def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(prog="round")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", required=True)
    p.add_argument("--candidates", required=True,
                   help="comma-separated tags that already exist under $R/work/")
    p.add_argument("--n-trials", type=int, required=True)
    p.add_argument("--k-se", type=float, default=1.0)
    # choices= imported from gate_check.py, never repeated here: this value is forwarded to it
    # verbatim, so a value only one side accepts empties the whole round table (run 33492876620
    # round 3, `--mode val` — the caller meant `--split val`, which is the default anyway).
    p.add_argument("--mode", default="paired", choices=gate_check.GATE_MODES)
    p.add_argument("--split", default="val")
    p.add_argument("--concurrency", type=int, default=DEFAULT_CONCURRENCY,
                   help="rollout concurrency per eval process (total = this x n_tags). Default "
                        "8, deliberately LOW, because the noise this script exists to expose is "
                        "largely load-induced and therefore fixable. Measured on byte-identical "
                        "code at identical seeds: mean per-task movement 0.250 at conc 25 vs "
                        "0.100 at conc 8, tasks moving 10/12 vs 5/12, arm-level |delta| 0.1167 "
                        "vs 0.0333. A gate at conc 25 cannot resolve any effect smaller than "
                        "0.08, which is larger than most real edits. Explore fast, gate slow.")
    p.add_argument("--max-parallel", type=int, default=4,
                   help="how many candidate evals run at once")
    p.add_argument("--gate-against", choices=["parent", "control"], default="parent",
                   help="'control' pairs each candidate against THIS round's null control "
                        "instead of the stored parent rollouts. Use it whenever this round's "
                        "--n-trials differs from the trial count the parent was measured at: "
                        "the control is a byte-identical copy of the parent measured in this "
                        "same round at this same n, so pairing against it removes a precision "
                        "mismatch the parent comparison would silently carry into the delta.")
    p.add_argument("--veto-regressions", action="store_true")
    p.add_argument("--control-replicates", type=int, default=2,
                   help="how many byte-identical copies of the parent to evaluate. Default 2, "
                        "because ONE control cannot bound run-to-run noise: two identical "
                        "controls on identical seeds differed by 0.0800 paired on this "
                        "benchmark, enough to pass the gate on their own. The gap between the "
                        "replicates is the round's real bar.")
    p.add_argument("--allow-high-concurrency", action="store_true",
                   help="run the gate above MAX_RESOLVING_CONCURRENCY anyway. An explicit, "
                        "recorded choice: the verdicts cannot resolve a small effect")
    p.add_argument("--no-reuse-control", action="store_true",
                   help="measure fresh control replicates even when this parent's own replicates "
                        "already exist from an earlier round. Reuse is on by default and is "
                        "only ever applied when best_id has NOT changed since those replicates "
                        "were measured, so the same bytes are not re-measured every round; a new "
                        "parent always gets fresh ones.")
    p.add_argument("--no-control", action="store_true",
                   help="skip the null control (NOT recommended — you lose the noise floor)")
    p.add_argument("--skip-screen-ladder", action="store_true",
                   help="run full-val on a candidate with no screen.py record for it (NOT "
                        "recommended — see the compliance check above). An explicit, recorded "
                        "choice, the same idiom as --allow-high-concurrency.")
    return p


def main(argv=None) -> int:
    try:
        return _main(argv)
    except GateCheckFailed as exc:
        # A traceback would be loud enough, but the driver reads this output to decide what to do
        # next, so say it in the words the skill uses: the round is not booked, re-gating is free.
        print(f"ERROR: {exc}", file=sys.stderr)
        return 2


def _main(argv=None) -> int:
    args = build_parser().parse_args(argv)

    # A gate too coarse to resolve its own verdict is refused, not warned about. Measured on
    # run 32861747778: the driver gated at --concurrency 100 after SKILL.md had told it "do not
    # raise it to buy wall clock", and this script's own table then carried "a verdict from this
    # round can therefore not resolve an effect smaller than roughly 0.08" while the run
    # continued and booked decisions anyway. The skill's own edit-form rule applies to the
    # skill: where the agent has the criterion and violates it regardless, the form that works
    # is a guard in the code, not a third restatement in prose. Refusal (not a silent clamp) is
    # already this script's idiom for an incoherent request — see --gate-against control
    # --no-control below.
    if args.concurrency and args.concurrency > MAX_RESOLVING_CONCURRENCY \
            and not args.allow_high_concurrency:
        print(json.dumps({
            "error": f"--concurrency {args.concurrency} exceeds {MAX_RESOLVING_CONCURRENCY}: "
                     "byte-identical code at identical seeds moves ~0.08 at this load versus "
                     "~0.03 at 8, so no verdict from the round could resolve an effect smaller "
                     "than the noise the concurrency itself adds",
            "fix": f"re-run with --concurrency {DEFAULT_CONCURRENCY} (buy wall clock with "
                   "fewer candidates per round, not with load), or pass "
                   "--allow-high-concurrency to record the trade deliberately",
        }, indent=2))
        return 2

    run_dir = RunDir.open(Path(args.run_dir))
    project = Path(args.project)
    work = Path(args.run_dir) / "work"
    work.mkdir(parents=True, exist_ok=True)

    best = run_dir.best_id
    if not best:
        print(json.dumps({"error": "no best_id in the run dir — run baseline first"}, indent=2))
        return 2

    tags = [t.strip() for t in args.candidates.split(",") if t.strip()]
    missing = [t for t in tags if not (work / t).is_dir()]
    if missing:
        print(json.dumps({"error": f"tags not found under {work}: {missing}"}, indent=2))
        return 2

    # Compliance instrumentation (issue #401): log, per candidate, whether screen.py was
    # invoked for it BEFORE this full-val eval — a distinct, auditable event rather than
    # something only inferable (or not) from SKILL.md prose. `screen.py` writes
    # `<run_dir>/screens/<tag>__screenN.json`; its absence means this candidate skipped
    # straight to full-val, which the dashboard can now show as its own event kind.
    screens_dir = run_dir.root / "screens"
    unscreened = []
    for t in tags:
        screened = screens_dir.is_dir() and any(screens_dir.glob(f"{t}__screen*.json"))
        run_dir.log_event("agent_optimize_compliance", tag=t,
                          screened_before_fullval=screened,
                          iteration=int(run_dir.spent.iterations))
        if not screened:
            unscreened.append(t)

    # Made a hard refusal, not a logged fact: issue #420 item 4 found that EVERY candidate
    # in a whole run skipped screen.py, even the ones (`cand_scope`, harmful;
    # `cand_e2_verifytype`, flat) it exists to kill for a quarter of the price — because
    # using it was optional, so a real, working, cheap tool never ran once. The compliance
    # event above already proved this cannot be caught after the fact by inference; making
    # it a warning would just be a third restatement of the same prose SKILL.md already
    # carried. `--skip-screen-ladder` is the deliberate, recorded override (e.g. a
    # candidate whose val is small enough that screening buys nothing).
    if unscreened and not args.skip_screen_ladder:
        print(json.dumps({
            "error": f"candidate(s) {unscreened} went straight to a full-val eval without a "
                     "screen.py record under $R/screens/",
            "why": "screen.py triages a candidate on a cheap val SUBSET before this full-val "
                   "eval is paid — see its own docstring. Skipping it because it is optional "
                   "is the exact failure issue #420 item 4 found: every candidate paid full "
                   "val, including ones a quarter-price screen would have killed.",
            "fix": "run screen.py --tier 1 (then --tier 2 if it promotes) on each of these "
                   "tags first, or pass --skip-screen-ladder to record the deliberate choice "
                   "to skip it.",
        }, indent=2))
        return 2

    # The null control is built here, not by the driver, so it cannot silently be skipped
    # or accidentally differ from the parent.
    # Derive the round's identity ONCE: the table name and the control tags are two halves of
    # it, and independently derived halves can disagree about which attempt this is.
    ATTEMPT = round_attempt(run_dir)
    STEM = table_stem(run_dir)
    PRIOR_CTL = prior_attempt_controls(run_dir)
    CTL = control_tag(run_dir)
    MEASUREMENT = measurement_context(args.split, args.n_trials, args.concurrency)
    ctl_tags: list[str] = []
    REUSED = None
    # Not under --gate-against control: that mode's whole premise is a control measured
    # CONCURRENTLY with the candidates, so that the drift between the two cancels. A reused
    # replicate is by definition not concurrent, and pairing against it would put the drift
    # back into every delta while the output still claimed it had been removed. Reuse is a
    # rollout saving, never a change to what a comparison means.
    # Nor on a re-gate: a second attempt at the same iteration is run precisely to BUY more
    # replicate evidence, so handing it the readings it already has would answer the question
    # with the numbers that failed to answer it.
    if not args.no_control and not args.no_reuse_control \
            and args.gate_against != "control" and ATTEMPT == 0:
        # Same parent bytes as the round that already measured this parent's noise floor, so
        # there is nothing new to learn from measuring it again — see reusable_controls.
        REUSED = reusable_controls(run_dir, best, MEASUREMENT,
                                   max(1, args.control_replicates))
    if REUSED:
        # No copytree, no eval: the rollouts these tags name are already on disk and the gate
        # reads them from there, so everything downstream is unchanged.
        ctl_tags = REUSED["tags"]
        CTL = ctl_tags[0]
    elif not args.no_control:
        # MORE THAN ONE control replicate, because one control does not bound the noise. Measured
        # here: a byte-identical control, re-run on the SAME seeds at temperature 0, moved
        # 0.6467 -> 0.7267 — a paired delta of +0.0800 that PASSES a k_se=1.0 bar on identical
        # code. A candidate measured against a single control reading therefore inherits a
        # coin-flip: the same candidate read +0.0867 against one control run and +0.0067 against
        # the other. Two replicates give the round its own null delta, which is the only bar
        # worth comparing a candidate to.
        for i in range(max(1, args.control_replicates)):
            tag = CTL if i == 0 else f"{CTL}r{i}"
            if (work / tag).exists():
                shutil.rmtree(work / tag)
            shutil.copytree(run_dir.candidate_dir(best), work / tag)
            ctl_tags.append(tag)
        tags = ctl_tags + tags

    with ThreadPoolExecutor(max_workers=max(1, args.max_parallel)) as pool:
        evals = list(pool.map(
            lambda t: _evaluate(Path(args.run_dir), project, t, args.split,
                                args.n_trials, args.concurrency), tags))
    if REUSED:
        # Reused replicates are gated exactly like measured ones (gate_check reads persisted
        # rollouts), so they join the row set here without an eval behind them.
        evals = [{"tag": t, "rc": 0, "reused": True} for t in ctl_tags] + evals

    # Gate serially against the CURRENT best; the driver commits, so best_id is stable here.
    gate_ref = best
    if args.gate_against == "control":
        if args.no_control:
            print(json.dumps({"error": "--gate-against control needs the control: drop "
                                       "--no-control"}, indent=2))
            return 2
        gate_ref = CTL
    # TWO distinct objects, kept distinct. `gate_res` is what deltas and thresholds are measured
    # against (the concurrent control under --gate-against control); `parent_res` is the
    # candidate this round is climbing from. Under --gate-against parent they coincide.
    #
    # Conflating them reported the CONTROL's reward under the PARENT's tag: on run 32871360361
    # the table said `parent: {tag: 'seed', reward: 0.34}` while baseline.json said the seed
    # scored 0.38, which no reader could reconcile. Worse, the gap between the two IS this
    # round's temporal drift — measured at 0.24/0.44/0.38 on identical seed bytes across three
    # runs, i.e. several times the gate bar — so collapsing them erased the one number that says
    # whether any delta in the table means anything.
    gate_res = harness.split_result_from_rollouts(run_dir, gate_ref, args.split)
    parent_res = (gate_res if gate_ref == best
                  else harness.split_result_from_rollouts(run_dir, best, args.split))
    parent = gate_res  # deltas/thresholds are always against the gate reference
    rows = []
    for ev in evals:
        tag = ev["tag"]
        if tag == CTL:
            g = gate_unless_eval_failed(ev, Path(args.run_dir), tag, args.k_se, args.mode,
                                        args.veto_regressions)
        else:
            g = gate_unless_eval_failed(ev, Path(args.run_dir), tag, args.k_se, args.mode,
                                        args.veto_regressions,
                                        current=gate_ref if args.gate_against == "control"
                                        else None)
        rows.append({
            "tag": tag,
            "reward": (g.get("candidate") or {}).get("reward"),
            "delta_vs_gate_ref": (None if (g.get("candidate") or {}).get("reward") is None
                                  else round((g["candidate"]["reward"] or 0.0)
                                             - gate_res.reward, 4)),
            "gate_delta": (g.get("gate") or {}).get("delta"),
            "gate_threshold": (g.get("gate") or {}).get("threshold"),
            # Structured numeric fields from gate_check.py's own JSON, kept alongside
            # gate_delta/gate_threshold above so commit.py can attach them to the events it
            # writes, rather than only a hand-typed prose note (dashboard.py's gate_decisions
            # previously had to regex-parse these back out of that note — see commit.py).
            "stderr": (g.get("candidate") or {}).get("stderr"),
            "n": g.get("paired_n"),
            "k_se": args.k_se,
            "resolvable_effect_size": (g.get("gate") or {}).get("resolvable_effect_size"),
            # Which val tasks the delta was actually measured over. A row whose footprint is
            # `restricted: false` was measured across the whole split, so its SE carries the
            # noise of every task the edit cannot reach — the defect that made SE(paired Δ)
            # 0.022-0.035 on run_finalrun6 while real per-edit effects were 0.011-0.05.
            "footprint": g.get("footprint"),
            "verdict": g.get("verdict"),
            "regressions": g.get("regressions"),
            "eval_rc": ev.get("rc"),
            "eval_error": ev.get("error"),
            # True only for a control replicate this round read back instead of measuring.
            "reused": bool(ev.get("reused")),
        })

    # Nothing derived from these rows — the drift-free re-gate, the evidence bar, the noise
    # floor, the written table — is meaningful if a row was never judged. Check before any of it.
    assert_rows_were_judged(rows)

    # A parent-gated round has ALREADY measured the drift-free comparison — it just was not
    # reporting it. On run 32871360361 round 4 the table showed cand4 at +0.15 against the seed's
    # stored 0.38 with a bar of 0.11 (drift), i.e. marginal; the same round's two concurrent
    # controls both read exactly 0.27, so the drift-free answer from the identical rollouts is
    # +0.26 against a bar of 0.00. The 0.11 belongs to WHEN the seed was measured, not to cand4,
    # so parent-mode gating understated the effect and inflated the bar at the same time.
    #
    # Reported rather than made the default: changing the default gate mode on one benchmark's
    # drift would be a guess about every other workload, while an extra comparison is strictly
    # more information and simply agrees with the primary one where there is no drift. Costs no
    # rollouts — the controls are already evaluated and gate_check reads stored data.
    if args.gate_against != "control" and ctl_tags:
        for r in rows:
            if r["tag"] in ctl_tags or r.get("reward") is None:
                continue
            # POOLED over every control replicate this round has, not just the one carrying
            # the round-scoped tag. They are byte-identical copies of the same parent measured
            # in the same round, so they are draws from one distribution and pooling their
            # trials per task is the lower-variance estimate of the same quantity — for free,
            # since these rollouts are already on disk. Measuring against a single replicate is
            # what made this comparison a coin flip: the same candidate read +0.0867 against
            # one replicate and +0.0067 against the other.
            g = _gate(Path(args.run_dir), r["tag"], args.k_se, args.mode,
                      args.veto_regressions, current=",".join(ctl_tags))
            r["control_relative"] = {
                "reference": ctl_tags if len(ctl_tags) > 1 else CTL,
                "gate_delta": (g.get("gate") or {}).get("delta"),
                "gate_threshold": (g.get("gate") or {}).get("threshold"),
                "verdict": g.get("verdict"),
                "reading": ("the same comparison with the DRIFT removed: this candidate against "
                            f"{len(ctl_tags)} byte-identical control(s) measured in this round — "
                            "their trials POOLED per task, so the reference carries less of its "
                            "own measurement error than any single replicate — rather than "
                            "against a reward measured earlier. Where the two disagree, the "
                            "difference is drift, not the edit."
                            if not REUSED else
                            f"this candidate against a byte-identical control of the SAME parent "
                            f"measured in iteration {REUSED['from_iteration']} and reused here. "
                            "It removes the parent's own measurement error but NOT the drift "
                            "since that iteration, so it is not the drift-free comparison a "
                            "concurrent control gives — re-run with --no-reuse-control (or "
                            "--gate-against control, which never reuses) to buy that."),
            }

    # Would the verdict have survived a different control replicate? On run 32871360361 round 3
    # two byte-identical replicates read 0.32 and 0.20 two minutes apart, and the reference was
    # simply whichever carried the round-scoped tag (0.20) — so cand3 scored +0.17 and accepted
    # where against the other replicate it is +0.05 and rejects. The table said nothing about the
    # verdict resting on that choice. Re-gating costs no rollouts, so there is no reason not to
    # check; a verdict that flips is not evidence, whatever the picked replicate showed.
    #
    # MANDATORY two-seed-block sign agreement: this used to run only under
    # --gate-against control, so a parent-gated round (the default) never checked whether its
    # accept survived the choice of control replicate — measured on a real run to have called a
    # null result positive exactly that way, unchecked because the round gated against the stored
    # parent. With --control-replicates 2 the default, this check now always runs whenever there
    # is more than one control block, in EITHER gate mode.
    if len(ctl_tags) > 1:
        for r in rows:
            if r["tag"] in ctl_tags or r.get("reward") is None:
                continue
            by_ref = {}
            for ref in ctl_tags:
                g = _gate(Path(args.run_dir), r["tag"], args.k_se, args.mode,
                          args.veto_regressions, current=ref)
                by_ref[ref] = g.get("verdict")
            r["verdict_by_reference"] = by_ref
            verdicts = {v for v in by_ref.values() if v is not None}
            r["verdict_stable"] = (len(verdicts) <= 1)
            if not r["verdict_stable"]:
                r["verdict"] = "inconclusive"

    ctl = next((r for r in rows if r["tag"] == CTL), None)
    # The floor must be the control's delta against the STORED parent, never against whatever
    # this round gated on. With --gate-against control the control IS the reference, so
    # delta_vs_parent is 0.0 by construction — reporting that as the noise floor would claim
    # zero re-measurement noise, the single most dangerous number this script can print.
    floor = None
    if ctl is not None:
        if args.gate_against == "control":
            floor = abs(ctl["gate_delta"]) if ctl.get("gate_delta") is not None else None
        elif ctl["delta_vs_gate_ref"] is not None:
            floor = abs(ctl["delta_vs_gate_ref"])
    # The gap BETWEEN identical control replicates is the round's empirical bar. It is a
    # stronger statement than any single control's delta, because both replicates are the same
    # bytes on the same seeds: whatever separates them is pure re-measurement. Two such
    # replicates differed by 0.0800 paired on this benchmark — enough to pass a k_se=1.0 gate on
    # zero change — so a candidate that does not clear this number has shown nothing.
    #
    # Earlier attempts at THIS iteration are pooled in. They are the same parent bytes on the
    # same seeds in the same round, so they are samples of the same null, and a re-gate is run
    # precisely to buy more of them: reporting only this attempt's two would discard half the
    # evidence the round has already paid for. `max - min` needs no change to accept them.
    ctl_rows = [r for r in rows if r["tag"] in ctl_tags and r.get("reward") is not None]
    pooled_rows = [{**r, "from_attempt": ATTEMPT} for r in ctl_rows] + PRIOR_CTL
    null_delta = None
    if len(pooled_rows) > 1:
        rewards = [r["reward"] for r in pooled_rows]
        null_delta = round(max(rewards) - min(rewards), 4)
    conc_warning = None
    if args.concurrency and args.concurrency > 12:
        conc_warning = (
            f"GATE RAN AT CONCURRENCY {args.concurrency}. Measured on this benchmark, "
            "byte-identical code at identical seeds moves ~0.08 at the arm level above conc 25 "
            "and ~0.03 at conc 8. A verdict from this round can therefore not resolve an effect "
            "smaller than roughly 0.08. Re-run the gate at --concurrency 8 before believing an "
            "accept.")

    prior_settings = prior_round_settings(run_dir)
    parallel_warning = parallel_drift_warning(prior_settings, args.concurrency, args.max_parallel)
    out = {
        # Which gate of this iteration this is. On the live run nothing in the output
        # distinguished "second opinion on iteration 1" from "iteration 1", so an operator
        # watching the stream saw two identical control evaluations and no statement that the
        # second had replaced the first.
        "attempt": ATTEMPT,
        "attempt_reading": (
            f"RE-GATE: attempt {ATTEMPT} at iteration {int(run_dir.spent.iterations)}. Its "
            f"{len(PRIOR_CTL)} earlier control replicate(s) are pooled into `null_delta_...` "
            "below, so the bar here rests on every replicate this round has paid for. This "
            "attempt's candidate rollouts are written under fresh tags; the earlier attempt's "
            "table is still on disk beside this one."
            if ATTEMPT else "first gate of this iteration"),
        "parent": {"tag": best, "reward": parent_res.reward, "stderr": parent_res.stderr,
                   "n_tasks": len(parent_res.per_task or [])},
        # What the deltas and thresholds in `candidates` are actually measured against.
        "gate_reference": {"tag": gate_ref, "mode": args.gate_against,
                           "reward": gate_res.reward, "stderr": gate_res.stderr},
        # The round's OWN drift: identical-or-parent bytes measured now versus what the parent
        # measured when it was scored. Non-null only when they are different measurements.
        "parent_vs_gate_ref_drift": (None if gate_ref == best else
                                     round((gate_res.reward or 0.0)
                                           - (parent_res.reward or 0.0), 4)),
        "drift_reading": (
            "the parent's stored reward and a byte-identical control measured in THIS round "
            "differ by this much. It is re-measurement drift, not progress, and any candidate "
            "delta of comparable size is not evidence — whatever its verdict says."
            if gate_ref != best else
            "gated against the parent's stored reward, so this round cannot see how far that "
            "reward has drifted since it was measured; --gate-against control measures it."),
        # What these rewards are comparable within, and therefore what a LATER round has to
        # match before it may reuse this round's control replicates (see reusable_controls).
        "measurement": MEASUREMENT,
        # Did this round pay for its own control replicates, or read back the ones this SAME
        # parent already has? Reuse happens only while best_id has not moved: the parent is the
        # same bytes, so its noise floor is already established. Any accept invalidates it and
        # the next round measures fresh — the two-replicate requirement is unchanged either way.
        "control_reuse": ({"reused": True, **REUSED,
                           "rollouts_saved": len(REUSED["tags"]),
                           "reading": "the control replicates below were measured in iteration "
                                      f"{REUSED['from_iteration']}, when this same parent was "
                                      "already the parent. No new control rollouts were spent. "
                                      "The replicate GAP is still this parent's own null; what "
                                      "it no longer contains is drift since that iteration, so "
                                      "prefer the parent-mode reading of `evidence_bar` here."}
                          if REUSED else
                          {"reused": False,
                           "reason": ("--no-control" if args.no_control else
                                      "--no-reuse-control" if args.no_reuse_control else
                                      "--gate-against control needs a CONCURRENT control"
                                      if args.gate_against == "control" else
                                      "a re-gate is run to BUY replicate evidence, so it always "
                                      "measures" if ATTEMPT else
                                      "no earlier round measured THIS parent's replicates under "
                                      "this same measurement context — a new parent has no "
                                      "established noise floor, so it must be measured")}),
        "measurement_concurrency": args.concurrency,
        "concurrency_warning": conc_warning,
        "measurement_max_parallel": args.max_parallel,
        "parallel_warning": parallel_warning,
        "null_delta_between_control_replicates": null_delta,
        "null_delta_replicates": len(pooled_rows),
        "null_delta_reading": (
            "identical bytes on identical seeds, so this is pure re-measurement noise. Any "
            "candidate delta at or below it is NOT evidence, whatever its verdict says."
            + (f" Pooled over {len(pooled_rows)} replicates of this iteration, "
               f"{len(PRIOR_CTL)} of them from earlier attempts." if PRIOR_CTL else "")
            if null_delta is not None else
            "only one control replicate — run with --control-replicates 2 to measure the bar "
            "instead of assuming a formula gives it"),
        "gated_against": {"tag": gate_ref, "mode": args.gate_against},
        "noise_floor_from_control": floor,
        "noise_floor_basis": ("control vs the STORED parent rollouts (differing trial counts are "
                              "part of this floor, which is the point)" if args.gate_against ==
                              "control" else "control vs the parent it was copied from"),
        # ONE bar, matched to how this round actually gated. Reporting several numbers and
        # leaving the driver to choose is not neutral: on run 32871360361 round 2 the table
        # showed cand2 beating its CONCURRENT control by +0.19 (three times the k_se threshold,
        # nineteen times the 0.01 gap between the control's own replicates) alongside a
        # `noise_floor_from_control` of 0.14 — which is the control-vs-STORED-parent gap, i.e.
        # temporal drift. The reading told the driver to treat any delta at or below the floor as
        # no evidence, so it compared a control-relative delta against a drift-derived floor,
        # resolved the contradiction conservatively, and booked a REJECT on the best candidate of
        # the run.
        #
        # Which bar is right depends entirely on what the delta was measured against:
        #   * control mode — the delta is against a control measured in THIS round, so drift is
        #     already cancelled and the bar is the gap between identical replicates.
        #   * parent mode  — the delta is against a reward measured in an earlier round, so drift
        #     is inside it and the bar has to include the control's drift as well.
        "evidence_bar": {
            "value": (null_delta if args.gate_against == "control"
                      else (None if (null_delta is None and floor is None)
                            else max(null_delta or 0.0, floor or 0.0))),
            "basis": ("gap between byte-identical control replicates measured in THIS round — "
                      "drift is cancelled by gating against a concurrent control"
                      if args.gate_against == "control" else
                      "the larger of the replicate gap and the control's drift against the "
                      "stored parent, because this round's deltas ARE against that stored "
                      "reward and carry its drift"),
        },
        "reading": (
            "A candidate marked `verdict_stable: false` has an UNSTABLE verdict and is "
            "INCONCLUSIVE, never accepted: its "
            "verdict changed depending on which byte-identical control replicate happened to be "
            "the reference, so the round cannot tell its edit from re-measurement. Run "
            "`scripts/grow.py --candidate <tag> --growth-round 1 --add-trials <n>` on it BEFORE "
            "booking anything — commit.py refuses `--decision inconclusive` until grow.py has "
            "bought this candidate at least one extra round of trials (issue #420 item 3: this "
            "exact case was read-and-skipped, never run, across two prior runs). grow.py pools "
            "the new trials onto the SAME candidate and re-gates at the pooled n; commit its "
            "recommendation (promote/grow_again/abandon) once it has one. Only if growth "
            "genuinely cannot help (e.g. delta <= 0) book `commit.py --decision inconclusive "
            "--force` and say why in --note — that still charges the iteration but NOT the "
            "stall counter, because a measurement that could not resolve is no evidence you "
            "have run out of ideas, and it keeps the edit out of `rejected.jsonl` so a later "
            "round is not taught to avoid a change nothing ever judged. "
            "rollouts are written `<task>__<tag>__t{k}.json` for k in range(n_trials), so "
            "re-running the SAME tag REPLACES t0..t9 rather than adding t10..t19 — it swaps a "
            "reading for another reading and buys no extra evidence; grow.py's own throwaway "
            "tag avoids that. The control side is handled for you — a re-gate of the "
            "same iteration gets its own `ctl_null_i<N>a<k>` replicates and POOLS the earlier "
            "attempt's into `null_delta_between_control_replicates`, so re-running this script "
            "adds control evidence instead of replacing it. Do not re-use a candidate tag to "
            "get that; there is no pooling for candidates. "
            "Judge every candidate's delta against `evidence_bar` rather than any other noise "
            "number here — but clearing it is NECESSARY, not sufficient: `gate_threshold` "
            "(k·SE on the paired per-task differences) is what each `verdict` is actually "
            "computed from and is usually the stricter of the two, so a delta above "
            "`evidence_bar` and below `gate_threshold` is not an accept. "
            "`noise_floor_from_control` is the gap between a byte-identical control "
            "measured now and the parent's STORED reward: that is re-measurement DRIFT, and it "
            "bounds how far the ABSOLUTE rewards in this table can be trusted — it is not a bar "
            "a candidate gated against a concurrent control has to clear, because that "
            "comparison never contained the drift. Do not re-derive a delta against the stored "
            "parent and reject on it; that puts the drift back in."
            if args.gate_against == "control" else
            "ctl_null is a byte-identical copy of the parent, so its delta is what ZERO change "
            "measures today. This round gated against the parent's STORED reward, so that drift "
            "is inside every candidate delta here: treat any candidate at or below "
            "`evidence_bar` as no evidence, even if its verdict is accept. Gating against the "
            "control instead removes the drift from the comparison."
            if floor is not None or null_delta is not None else
            "no null control in this round — you cannot separate a small gain from re-measurement."
        ),
        "candidates": sorted((r for r in rows if r["tag"] not in ctl_tags),
                             key=lambda r: (r["reward"] is None, -(r["reward"] or 0.0))),
        "control": ctl,
        # `control_replicates` stays THIS attempt's own measurements — that is what a later
        # attempt reads back to pool, and storing the pooled set here would make attempt 2
        # count attempt 0's replicates twice. The pooled view is reported separately.
        "control_replicates": ctl_rows,
        "pooled_control_replicates": pooled_rows if PRIOR_CTL else None,
        "next": ("read regressions, then commit.py --decision accept|reject|inconclusive per "
                 "candidate — `inconclusive` for any row whose `verdict` is inconclusive, so the "
                 "round is not recorded as refuting an edit it could not judge"),
    }
    # Persist the table as well as printing it. Until now the ONLY copy lived on stdout, so
    # whether a round's verdict survived depended on the driver remembering to redirect —
    # and on run 32814848187 the round that was abandoned was only reconstructible because
    # the driver happened to have redirected it to a name someone guessed. A round's gate
    # result is the run's evidence; it should not be optional.
    #
    # Per-iteration name for the same reason `control_tag` is per-iteration: a fixed name
    # would let each round destroy the previous round's table. A same-iteration re-run gets a
    # suffix rather than overwriting, since a re-gate is usually being COMPARED with the
    # first one.
    #
    # The name comes from the SAME attempt index the control tags were built from, rather than
    # from a second, independent probe of the directory. When the two derivations disagreed the
    # table survived and the rollouts it cites did not, which is the whole defect this attempt
    # index exists to close.
    try:
        work.mkdir(parents=True, exist_ok=True)
        table = work / f"{STEM}.json"
        n = ATTEMPT
        while table.exists():  # belt-and-braces: never overwrite a sibling attempt's table
            n += 1
            table = work / f"round_i{int(run_dir.spent.iterations)}.r{n}.json"
        table.write_text(json.dumps(out, indent=2), encoding="utf-8")
        out["table_path"] = str(table)
    except OSError as exc:  # noqa: BLE001 — the printed table is still the primary output
        out["table_write_error"] = str(exc)

    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""agent-optimize — the fully-agentic, free-form optimization algorithm.

This algorithm has NO deterministic subprocess loop. The conversational agent
drives the whole search itself, following the "Agent-mode loop" in ``SKILL.md``:
understand the benchmark/inputs, run the baseline, then freely propose edits,
triage on task subsets, gate on FULL val, and stop when the free-text
``stop_condition`` is met — sealing once via the finalize phase.

Because agent mode short-circuits in ``cap-evolve run`` right after baseline (it
prints a handoff and returns *before* any algorithm subprocess is invoked), this
``run.py`` is never called on the happy path. It exists only as a loud guard: if
someone selects ``algorithm_skill: agent-optimize`` WITHOUT ``orchestration_mode:
agent`` (i.e. tries to run it deterministically), fail with a clear message rather
than silently no-op into a misleading seed-vs-seed finalize.
"""

from __future__ import annotations

import argparse
import json
import sys

import _bootstrap  # noqa: F401

ALGO = "agent-optimize"


def main(argv=None) -> int:
    # Accept the standard algorithm CLI seam so a deterministic invocation reaches
    # our guard message instead of an argparse crash.
    p = argparse.ArgumentParser(prog=ALGO)
    p.add_argument("--run-dir")
    p.add_argument("--project")
    p.add_argument("--optimizer")
    p.add_argument("--max-iterations", type=int, default=0)
    p.add_argument("--n-trials", type=int, default=1)
    p.add_argument("--gate-mode", default="auto")
    p.add_argument("--k-se", type=float, default=1.0)
    p.add_argument("--store", default="git")
    # Tolerate any other flags the deterministic seam passes through.
    p.parse_known_args(argv)

    print(json.dumps({
        "algorithm": ALGO,
        "error": "agent-optimize is agent-driven and has no deterministic loop.",
        "fix": "set `orchestration_mode: agent` in capevolve.yaml. In agent mode "
               "`cap-evolve run` does check+baseline then hands the loop to the "
               "conversational agent, which follows this skill's 'Agent-mode loop'. "
               "For a deterministic run choose hill-climb | gepa | skillopt instead.",
    }, indent=2))
    return 2


if __name__ == "__main__":
    sys.exit(main())

"""screen — cheap SUBSET triage on val. Kills bad candidates; can never accept one.

The economics this exists for: a full-val evaluation costs ``val_n × num_trials``
rollouts and is paid once per candidate per round. Most edits are not close calls, and
paying full val to discover that is the biggest waste in a run. So: screen the
candidate on a small, *deterministically chosen*, *informative* subset of val first,
kill it there if it is clearly harmful, and only promote survivors to the full-val
paired gate.

**The parent side of the comparison is free.** The current best already has full-val
rollouts on disk, so the screen re-reads its per-task rewards instead of re-running it.
Only the candidate pays, and only for the subset — that is where the saving comes from.

A promotion ladder, one call per rung (``--tier``):

    tier 1  ~25% of val, 1 trial   → kill obvious harm for a quarter of the price
    tier 2  ~50% of val, 1 trial   → a second look before paying full val
    (then)  FULL val × num_trials  → the evaluate phase + gate_check.py: the ONLY accept

Tier 2 does **not** re-run tier 1's tasks: the candidate's screen rollouts are merged
across every ``<tag>__screen*`` tag, so each rung only pays for the ids it adds.

This script prints ``"decision": "kill" | "promote"``. It never prints ``accept`` and
carries no code path that could: acceptance is ``gate_check.py`` on FULL val (Δ̄ > k·SE
plus the no-regression veto), by construction and by honesty invariant 1.

Every screen is written to ``<run_dir>/screens/<tag>__tier<N>.json`` — subset ids, the
seed, the deltas, the decision, and the MEASURED rollout economics — so any kill is
reproducible and auditable after the fact.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import RunDir, harness
from cap_evolve.check import load_adapter
from cap_evolve.subsample import (
    full_val_ceiling, paired_deltas_on, screen_decision, screen_savings,
    select_screen_subset,
)

#: Rung → fraction of val screened. Tier 3 is "almost full val" for the rare case
#: where full val is very large; the real gate is still a separate full-val eval.
TIER_FRAC = {1: 0.25, 2: 0.5, 3: 0.75}

#: Absolute floor on subset width, independent of the fraction. Was 3, and 3 is
#: MEASURED to be too narrow: on a 12-task val, tier 1 = round(0.25·12) = 3, and the
#: run in docs/RESULTS.md produced a screen that reported ``fixed: ["44"]`` on a 3-task
#: subset when full val showed task 44 was never fixed — a false positive on a third of
#: the evidence. 6 is the smallest width where the paired SE over {-1,0,+1} deltas is
#: not dominated by a single task. It only binds on small val splits; a 100-task val
#: still screens at the 25% fraction.
MIN_K = 6


def _screen_tags(run_dir: RunDir, tag: str) -> list[str]:
    """Every ``<tag>__screenN`` tag that already has val rollouts on disk."""
    seen = set()
    for f in (run_dir.rollouts / "val").glob(f"*__{tag}__screen*__t*.json"):
        parts = f.name.split("__")
        # <task>__<tag…>__screenN__t<k>.json — the screen tag is everything before __t<k>
        seen.add("__".join(parts[1:-1]))
    return sorted(seen)


def _merged_per_task(run_dir: RunDir, tags: list[str]) -> list:
    """Union of per-task val records across tags (later tags win on a collision)."""
    out: dict = {}
    for tg in tags:
        for pt in harness.split_result_from_rollouts(run_dir, tg, "val").per_task or []:
            out[str(pt.get("task_id"))] = pt
    return list(out.values())


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="screen")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", required=True)
    p.add_argument("--candidate", required=True, help="working-copy dir to screen")
    p.add_argument("--tag", default=None,
                   help="candidate tag; default = the candidate dir name")
    p.add_argument("--current", default=None,
                   help="parent tag to compare against; default = the run's best_id")
    p.add_argument("--tier", type=int, default=1, choices=sorted(TIER_FRAC),
                   help="promotion rung: 1 (~25%% of val) | 2 (~50%%) | 3 (~75%%)")
    p.add_argument("--k", type=int, default=0,
                   help="explicit subset size; overrides --tier's fraction")
    p.add_argument("--seed", type=int, default=None,
                   help="subset seed; default = the frozen splits seed + tier "
                        "(so each rung draws a different holdout, reproducibly)")
    p.add_argument("--holdout-frac", type=float, default=0.34,
                   help="fraction of the subset drawn at random from tasks the parent "
                        "PASSES, so the screen can see a regression (default 0.34)")
    p.add_argument("--k-se", type=float, default=1.0,
                   help="kill only when Δ̄ + k·SE < 0 on the subset (default 1.0)")
    p.add_argument("--broken", default="",
                   help="comma-separated task ids a previous edit broke — screened first")
    p.add_argument("--ids", default="",
                   help="comma-separated val task ids to screen on, chosen by YOUR OWN method "
                        "(trajectory-similarity clustering, reading rollouts, anything) — "
                        "bypasses select_screen_subset's fixed broken/informative/holdout "
                        "heuristic entirely. --tier/--k/--broken/--holdout-frac are ignored "
                        "when this is set. The kill/promote decision and audit trail are "
                        "unchanged — this only changes WHICH tasks are screened, never "
                        "whether a screen can accept (it still can't).")
    p.add_argument("--n-trials", type=int, default=1,
                   help="trials per screened task (1 is the point; >1 is not a gate)")
    p.add_argument("--workers", type=int, default=None,
                   help="concurrent rollouts (adapter must be thread-safe)")
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))
    cand_dir = Path(args.candidate)
    if not cand_dir.is_dir():
        cand_dir = run_dir.candidate_dir(args.candidate)
    if not cand_dir.is_dir():
        print(json.dumps({"error": f"candidate dir not found: {args.candidate}"}, indent=2))
        return 2
    tag = args.tag or cand_dir.name
    cur_tag = args.current or run_dir.best_id
    if not cur_tag:
        print(json.dumps({"error": "no --current tag and no best_id (has baseline run?)"},
                         indent=2))
        return 2

    val_ids = run_dir.read_splits().ids("val")
    parent = harness.split_result_from_rollouts(run_dir, cur_tag, "val")
    if not parent.per_task:
        print(json.dumps({
            "error": f"no val rollouts for parent tag {cur_tag!r} — the screen reads the "
                     "parent's existing full-val rollouts (that is what makes it cheap)",
            "fix": "run the baseline / a full-val evaluate for the current best first",
        }, indent=2))
        return 2

    custom_ids = [i.strip() for i in (args.ids or "").split(",") if i.strip()]
    if custom_ids:
        val_id_set = {str(i) for i in val_ids}
        chosen = sorted({i for i in custom_ids if i in val_id_set})
        sub = {"ids": chosen, "broken": [], "holdout": [], "informative": chosen,
               "k": len(chosen), "requested_k": len(custom_ids), "seed": None,
               "holdout_frac": None, "pool_n": len(parent.per_task),
               "rationale": f"optimizer-chosen subset ({len(chosen)} of {len(custom_ids)} "
                            "requested ids fell inside the frozen val split), bypassing "
                            "select_screen_subset's heuristic"}
    else:
        frac = TIER_FRAC[args.tier]
        k = args.k or max(MIN_K, int(round(frac * len(val_ids))))
        seed = args.seed if args.seed is not None else int(run_dir.read_splits().seed) + args.tier
        broken = [b for b in (args.broken or "").split(",") if b.strip()]
        sub = select_screen_subset(parent.per_task, k=k, seed=seed,
                                   holdout_frac=args.holdout_frac,
                                   broken_ids=[b.strip() for b in broken])

    # Rungs are cumulative: never re-run a task an earlier rung already screened.
    prior_tags = _screen_tags(run_dir, tag)
    already = {str(pt.get("task_id")) for pt in _merged_per_task(run_dir, prior_tags)}
    new_ids = [i for i in sub["ids"] if i not in already]

    screen_tag = f"{tag}__screen{args.tier}"
    fired = 0
    screen_cost_usd = 0.0
    if new_ids:
        res = harness.evaluate_candidate(
            load_adapter(Path(args.project)), cand_dir, run_dir=run_dir,
            split="val", n_trials=max(1, args.n_trials), tag=screen_tag,
            workers=args.workers, ids=new_ids, ks=(1,))
        fired = len(new_ids) * max(1, args.n_trials)
        screen_cost_usd = res.cost_usd

    cand_per_task = _merged_per_task(run_dir, sorted({*prior_tags, screen_tag}))
    pair = paired_deltas_on(parent.per_task, cand_per_task, sub["ids"])
    decision = screen_decision(pair["deltas"], k_se=args.k_se,
                               regressed=pair["regressed"])

    # ARITHMETIC kill. When the screened ids already cover every val task the parent
    # fails, the unscreened remainder is all tasks the parent passes, so it can only
    # stay level or regress — and the candidate's best conceivable full-val mean is
    # computable. If that ceiling cannot beat the parent, no full-val eval can ever
    # accept, and paying for one buys strictly nothing. This still cannot accept
    # anything: the only conclusion it can reach is "reject".
    ceiling = full_val_ceiling(parent.per_task, cand_per_task, sub["ids"],
                               [str(i) for i in val_ids])
    # STRICTLY negative only. A best-case Δ̄ of exactly 0.0 also cannot accept (the bar
    # is >= 0), but that is the degenerate "parent already perfect on the screened set"
    # case, and escalating it would override the deliberate promote-on-a-flat-subset
    # bias for no gain. Keep the bias; kill only when the ceiling is provably BELOW the
    # parent.
    if (ceiling.get("best_case_mean_delta") is not None
            and ceiling["best_case_mean_delta"] < -1e-9
            and decision["decision"] != "kill"):
        decision = {**decision, "decision": "kill", "provable": True,
                    "inconclusive": False,
                    "reason": "PROVABLE kill (not a statistical one): "
                              + ceiling["reason"]}

    savings = screen_savings(fired=fired, val_n=len(val_ids),
                             n_trials=max(1, args.n_trials),
                             decision=decision["decision"])

    payload = {
        "tag": tag, "screen_tag": screen_tag, "tier": args.tier,
        "current": cur_tag,
        "subset": sub,
        "reused_from_earlier_tiers": sorted(already & set(sub["ids"])),
        "fired_ids": new_ids,
        "paired": pair,
        "full_val_ceiling": ceiling,
        **decision,
        "savings": {**savings, "screen_cost_usd": screen_cost_usd},
        "promote_to": ("full-val evaluate + gate_check.py"
                       if decision["decision"] == "promote" else None),
        "note": ("A screen is TRIAGE. It may kill; it may never accept. Only "
                 "gate_check.py on FULL val (Δ̄ > k·SE and no regression) accepts."),
    }
    screens = run_dir.root / "screens"
    screens.mkdir(parents=True, exist_ok=True)
    (screens / f"{screen_tag}.json").write_text(json.dumps(payload, indent=2),
                                                encoding="utf-8")
    run_dir.log_event("screen", tag=tag, tier=args.tier, ids=sub["ids"],
                      fired=fired, decision=decision["decision"],
                      mean_delta=decision["mean_delta"], se=decision["se"],
                      n=decision["n"], inconclusive=decision["inconclusive"],
                      net_rollouts=savings["net_rollouts"], rationale=sub["rationale"])
    print(json.dumps(payload, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""spend — ONE call that answers "may I spend, and am I done?".

``references/algorithm.md`` says the agent re-reads spend against the project's
free-text ``stop_condition`` every few rounds. Re-reads is the operative word: a running
total carried in an agent's context is how a $6.00 cap becomes $6.01. Everything here is
read from the RUN DIR (``state.json`` spend, ``events.jsonl`` timestamps, the persisted
rollouts) — never from anything remembered.

It prints, in one JSON object:

  * ``best_id`` + the current best's **full-val** mean/stderr/coverage;
  * every recorded ``spent`` field, the ``budget``, and ``RunDir.budget_exhausted()``
    as ``stop``/``stop_reason`` (the exact rule the deterministic loops stop on);
  * ``wallclock_seconds`` — measured from the first event in ``events.jsonl``;
  * ``constraints`` — the free-text ``stop_condition`` parsed into concrete predicates
    (:mod:`cap_evolve.constraints`), each with its measured actual and satisfied/violated
    state, the original prose verbatim, an ``ambiguous`` list for anything the parser
    would have had to guess at, and one ``recommendation``:
    ``stop`` | ``continue`` | ``narrow_scope``;
  * ``afford`` — with ``--n-siblings N``, whether N full-val evaluations fit in what is
    left, using the run's own **measured** $/rollout. Check this BEFORE fanning out N
    proposers, not after: N candidates can blow a budget that had room for one.

``recommendation`` combines both halves: a violated ``budget_exhausted()`` is a ``stop``
even when the prose says nothing about money.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

# Imported for its side effect ONLY: seeds sys.path so `cap_evolve` resolves when
# this script is run standalone (`python <this-file>`). Must precede the
# cap_evolve imports below; not "unused" — deleting it breaks standalone runs.
import _bootstrap  # noqa: F401  # side-effect import, see above

from cap_evolve import RunDir, harness
from cap_evolve.constraints import check_constraints, parse_constraints
from cap_evolve.loop import has_valid_trials
from cap_evolve.specfile import spec_for_run


def _wallclock(run_dir: RunDir) -> float:
    """Seconds since the run's FIRST recorded event (measured, not remembered)."""
    import time
    try:
        with run_dir.events_path.open(encoding="utf-8") as f:
            first = json.loads(f.readline())
        return max(0.0, time.time() - float(first.get("t") or 0.0))
    except Exception:  # noqa: BLE001
        try:
            return max(0.0, time.time() - run_dir.state_path.stat().st_mtime)
        except Exception:  # noqa: BLE001
            return 0.0


def _regressed_vs_seed(run_dir: RunDir, best_id: str | None) -> list:
    """Val tasks the SEED measured-and-passed that the current best now scores lower.

    This is what a "don't regress task X" clause is checked against. Tasks either side
    failed to measure are skipped: missing data is not a regression.
    """
    if not best_id or best_id == "seed":
        return []
    seed = harness.split_result_from_rollouts(run_dir, "seed", "val")
    best = harness.split_result_from_rollouts(run_dir, best_id, "val")
    s = {pt["task_id"]: pt for pt in (seed.per_task or []) if has_valid_trials(pt)}
    b = {pt["task_id"]: pt for pt in (best.per_task or []) if has_valid_trials(pt)}
    # The move must clear 2*SE of its own per-task measurement (harness.move_is_resolved —
    # the ONE bar every broke/fixed claim in the framework uses). A constraint clause is
    # checked against this list, so a task that only wobbled by one flipped rollout must not
    # report a violated promise.
    return sorted(str(t) for t in s if t in b
                  and (b[t].get("reward", 0.0) or 0.0) < (s[t].get("reward", 0.0) or 0.0)
                  and harness.move_is_resolved(
                      s[t].get("reward", 0.0) or 0.0, b[t].get("reward", 0.0) or 0.0,
                      s[t].get("stderr") or 0.0, b[t].get("stderr") or 0.0))


def _afford(run_dir: RunDir, spec: dict, n_siblings: int, n_trials: int) -> dict:
    """Can N full-val evals be paid for? Uses the run's MEASURED $/rollout.

    ``usd_per_rollout`` is ``spent.usd / spent.metric_calls`` — an observed average from
    this run's own rollouts, so the answer gets more accurate as the run proceeds and is
    honestly ``null`` before any rollout has been paid for (in which case only the
    rollout-count ceilings can be checked, and that is said out loud).
    """
    spent, budget = run_dir.spent, run_dir.budget
    val_n = len(run_dir.read_splits().ids("val"))
    per_eval = val_n * max(1, n_trials)
    need = per_eval * max(0, n_siblings)
    # A measured rate of EXACTLY $0 after real rollouts is not "free" — it is
    # UNMETERED. It happens whenever the serving path returns no cost (the IBM litellm
    # proxy does exactly this: litellm logs "model isn't mapped yet" and reports 0.0).
    # Treating it as 0.0 made `need_usd` 0.0, so the max_usd ceiling could never
    # block anything and ANY fan-out came back affordable: true. Unknown, not zero.
    metered = bool(spent.metric_calls) and spent.usd > 0.0
    upr = (spent.usd / spent.metric_calls) if metered else None
    unmetered = bool(spent.metric_calls) and not metered
    need_usd = (upr * need) if upr is not None else None

    blockers: list = []
    if budget.max_metric_calls:
        left = budget.max_metric_calls - spent.metric_calls
        if need > left:
            blockers.append(f"needs {need} rollouts, {left} left under max_metric_calls")
    if budget.max_usd and need_usd is not None:
        left_usd = budget.max_usd - spent.total_usd
        if need_usd > left_usd:
            blockers.append(f"needs ~${need_usd:.2f} of runner spend (measured "
                            f"${upr:.4f}/rollout), ${left_usd:.2f} left under max_usd")
    return {
        "n_siblings": n_siblings,
        "val_n": val_n,
        "n_trials": n_trials,
        "rollouts_per_full_val_eval": per_eval,
        "rollouts_needed": need,
        "usd_per_rollout_measured": upr,
        "usd_needed_estimate": need_usd,
        "affordable": not blockers,
        "blockers": blockers,
        "runner_spend_metered": (None if not spent.metric_calls else metered),
        "caveat": ("usd_per_rollout is this run's measured average and excludes the "
                   "proposer's own cost — record that with commit.py --optimizer-usd"
                   if upr is not None else
                   (f"{spent.metric_calls} rollouts are recorded but runner usd is still "
                    "0.0, so this serving path does NOT meter cost. The $ ceiling cannot "
                    "be enforced from measurements — bound the run with max_metric_calls "
                    "/ max_iterations instead, and report rollout counts, not dollars."
                    if unmetered else
                    "no rollout has been paid for yet, so only rollout-count ceilings "
                    "could be checked — the $ answer is unknown, not 'yes'")),
    }


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="spend")
    p.add_argument("--run-dir", required=True)
    p.add_argument("--project", default=None,
                   help="project dir, to read stop_condition + num_trials")
    p.add_argument("--n-siblings", type=int, default=0,
                   help="check affordability of N full-val evals BEFORE fanning out")
    p.add_argument("--n-trials", type=int, default=0,
                   help="trials per full-val eval; default = the spec's num_trials")
    p.add_argument("--warn-frac", type=float, default=0.8,
                   help="ceiling consumption at which to recommend narrow_scope")
    p.add_argument("--ceiling-file", default=None,
                   help="JSON file mapping task_id -> highest rate that task can structurally "
                        "reach (e.g. from a diagnose-phase ceiling analysis). When given, a "
                        "target_val_score predicate is COSTED against it before being treated "
                        "as reachable — see cap_evolve.constraints.cost_target.")
    args = p.parse_args(argv)

    run_dir = RunDir.open(Path(args.run_dir))
    project = Path(args.project) if args.project else None
    spec = spec_for_run(run_dir, project)
    stop, reason = run_dir.budget_exhausted()
    best_id = run_dir.best_id
    best = harness.split_result_from_rollouts(run_dir, best_id, "val") if best_id else None
    spent = run_dir.spent
    wall = _wallclock(run_dir)

    per_task_ceiling = None
    if args.ceiling_file:
        per_task_ceiling = {
            str(k): float(v)
            for k, v in json.loads(Path(args.ceiling_file).read_text(encoding="utf-8")).items()
        }

    parsed = parse_constraints(str(spec.get("stop_condition") or ""))
    checked = check_constraints(
        parsed,
        best_val=(best.reward if best else None),
        usd=spent.total_usd, wallclock_seconds=wall,
        iterations=spent.iterations, stall=spent.stall,
        metric_calls=spent.metric_calls,
        regressed_tasks=_regressed_vs_seed(run_dir, best_id),
        warn_frac=args.warn_frac,
        per_task_ceiling=per_task_ceiling,
    )

    # The run dir's own hard stop always wins: a prose condition cannot buy more budget.
    rec = "stop" if stop else checked["recommendation"]
    reasons = ([reason] if stop else []) + list(checked["reasons"])

    n_trials = args.n_trials or int(spec.get("num_trials") or 1)
    out = {
        "best_id": best_id,
        "best_val": ({"reward": best.reward, "stderr": best.stderr,
                      "coverage": best.coverage, "n_scored": best.n_scored,
                      "n_tasks": best.n_tasks} if best else None),
        "spent": spent.to_dict(),
        "budget": run_dir.budget.to_dict(),
        "wallclock_seconds": round(wall, 1),
        "stop": stop,
        "stop_reason": reason,
        "stop_condition": parsed["text"],
        "constraints": checked,
        "recommendation": rec,
        "recommendation_reasons": reasons,
        "test_sealed": not run_dir.read_splits().test_used,
    }
    if args.n_siblings:
        out["afford"] = _afford(run_dir, spec, args.n_siblings, n_trials)
    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Evaluate ONE candidate on a NAMED SUBSET of tasks, with traces. The per-task gradient.

Why this exists. A full-val gate round costs ``val_n * n_trials`` rollouts and returns one bit
per candidate — accept or reject. One task at ``n_trials`` costs ``n_trials`` rollouts and
returns the same bit about the failure that actually exists. At 30 tasks x 10 trials that is
300 rollouts per learning step versus 10, and the defect lives in the task, not in the mean.

The cost is overfitting, and it is real: an edit tuned on one task can break another. That is
what ``--canary`` is for (tasks measured 1.0 at baseline, evaluated in the same call), and why
nothing measured here is evidence until the merged artifact clears a full-val gate against
``ctl_null``. A per-task rate is a TRAINING number — the optimiser tuned on it.

    python taskeval.py --project <dir> <candidate_dir> 7,17 --n 10 \
        --canary 0,3,12 --canary-n 3 --traces /tmp/tr.json [--split val] [--conc 40]

Prints per-task pass RATE (k/n), the distinct failure feedback strings, and with --traces every
agent tool call per failing trial, so the next edit is aimed at an observed decision rather than
a guess. For an UNSTABLE task, diff a failing trial against a passing one: the divergence point
is the ambiguity.
"""

import argparse
import json
import os
import sys
import time
from collections import defaultdict
from pathlib import Path


def component_rates(score, rollout) -> dict:
    """Per-component sub-scores for one rollout, WITHOUT knowing the benchmark.

    Why this is not just ``metadata["<domain>_reward_info"]``: a binary task reward collapses
    "got the database right but missed a required confirmation" and "did nothing" into the same
    0.0, and the per-component means are what separate them. That signal is worth having on every
    benchmark, so the lookup has to be generic.

    Resolution order, first hit wins:
      1. ``Score.raw`` — the adapter's own structured payload, checked under the conventional
         breakdown keys. This is the documented place for it.
      2. ``Score.metrics`` — a list of ``{"name": ..., "value": ...}`` entries.
      3. ``Rollout.metadata`` — any dict-valued key ending in ``_reward_info`` or ``_score_info``
         that itself carries a breakdown. This is the escape hatch for adapters that stash the
         runner's native structure without mapping it, and it is why no benchmark name appears here.

    Returns ``{}`` when nothing is exposed, which is a normal answer, not an error: a scorer with
    one scalar reward has no components and per-task rates remain fully usable without them.
    """
    BREAKDOWN_KEYS = ("reward_breakdown", "component_rates", "components", "breakdown", "subscores")

    def _numeric(d):
        return {str(k): float(v) for k, v in d.items()
                if isinstance(v, (int, float)) and not isinstance(v, bool)}

    raw = getattr(score, "raw", None) or {}
    if isinstance(raw, dict):
        for key in BREAKDOWN_KEYS:
            got = raw.get(key)
            if isinstance(got, dict) and _numeric(got):
                return _numeric(got)

    metrics = getattr(score, "metrics", None) or []
    named = {str(m.get("name")): float(m.get("value"))
             for m in metrics
             if isinstance(m, dict) and m.get("name") is not None
             and isinstance(m.get("value"), (int, float)) and not isinstance(m.get("value"), bool)}
    if named:
        return named

    meta = getattr(rollout, "metadata", None) or {}
    if isinstance(meta, dict):
        for k, v in meta.items():
            if not (isinstance(k, str) and (k.endswith("_reward_info") or k.endswith("_score_info"))):
                continue
            if not isinstance(v, dict):
                continue
            for key in BREAKDOWN_KEYS:
                got = v.get(key)
                if isinstance(got, dict) and _numeric(got):
                    return _numeric(got)
    return {}


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("candidate_dir")
    ap.add_argument("tasks", help="comma-separated task ids to optimise")
    ap.add_argument("--project", required=True, help="cap-evolve project dir (holds adapters/)")
    ap.add_argument("--split", default="val")
    ap.add_argument("--n", type=int, default=10, help="trials per target task")
    ap.add_argument("--canary", default="", help="comma-separated ids measured 1.0 at baseline")
    ap.add_argument("--canary-n", type=int, default=3)
    ap.add_argument("--conc", type=int, default=0,
                    help="rollout concurrency, if >0. Exported as CAPEVOLVE_MAX_CONCURRENCY (the "
                         "canonical name every adapter should read) AND as any extra names given "
                         "by --conc-env, for runners whose knob predates that convention.")
    ap.add_argument("--conc-env", default="",
                    help="comma-separated EXTRA env var names to set to --conc, e.g. "
                         "the runner's own knob. The canonical CAPEVOLVE_MAX_CONCURRENCY is always "
                         "set, so this is only needed for an adapter that does not read it yet.")
    ap.add_argument("--traces", default="", help="write per-trial agent tool calls here")
    ap.add_argument("--json", dest="json_out", default="")
    ap.add_argument("--run-dir", default="",
                    help="charge these rollouts to the run's budget ledger (strongly advised: "
                         "a fan-out of K optimisers x M iterations is the bulk of a round's "
                         "real spend, and a budget that cannot see it is not a budget)")
    args = ap.parse_args()

    proj = Path(args.project).resolve()
    sys.path.insert(0, str(proj / "adapters"))
    if args.conc > 0:
        # The concurrency knob is named per RUNNER, so the skill cannot hardcode one benchmark's
        # variable and still be general. Canonical name always; extra aliases on request.
        os.environ["CAPEVOLVE_MAX_CONCURRENCY"] = str(args.conc)
        for name in [n.strip() for n in args.conc_env.split(",") if n.strip()]:
            os.environ[name] = str(args.conc)

    from cap_evolve.check import load_adapter

    adapter = load_adapter(proj)
    cand = Path(args.candidate_dir).resolve()

    want = [t.strip() for t in args.tasks.split(",") if t.strip()]
    canary = [t.strip() for t in args.canary.split(",") if t.strip()]
    by_id = {t.id: t for t in adapter.tasks(args.split)}
    missing = [t for t in want + canary if t not in by_id]
    if missing:
        print(f"unknown task ids in split {args.split!r}: {missing}", file=sys.stderr)
        return 2
    if not canary:
        print("WARNING: no --canary. A per-task edit that breaks a working task will not be "
              "visible until the full-val gate.", file=sys.stderr)

    groups = [(want, args.n, "target")]
    if canary:
        groups.append((canary, args.canary_n, "canary"))

    rates: dict[str, list[float]] = defaultdict(list)
    role: dict[str, str] = {}
    fb: dict[str, list[str]] = defaultdict(list)
    traces: list[dict] = []
    infra: dict[str, int] = defaultdict(int)
    comps: dict[str, dict[str, list[float]]] = defaultdict(
        lambda: defaultdict(list))
    t0 = time.time()

    # `adapter.live(cand)` is the documented contract `run_target`'s docstring points to
    # ("ctx is whatever live() yielded (default: the candidate dir Path)") — NOT
    # `adapter.materialize(cand)`, whose default implementation writes `edits` (there are
    # none here) and returns None. Calling it directly silently made `ctx` None for every
    # adapter using the standard materialize/live split, so every rollout below raised
    # inside run_trials_pool and was counted as an infra drop rather than scored — the
    # reason this script, like integrate.py which calls it, was measured (#434/#438) to
    # have never actually produced a per-task result on a real run.
    with adapter.live(cand) as ctx:
        for ids, n, kind in groups:
            tasks = [by_id[i] for i in ids]
            for i in ids:
                role[i] = kind
            out = adapter.run_trials(tasks, ctx, n_trials=n, base_seed=0)
            for tid, rolls in sorted(out.items()):
                for k, roll in enumerate(rolls or []):
                    if roll is None or getattr(roll, "error", None):
                        infra[tid] += 1          # missing data, NOT a zero
                        continue
                    s = adapter.score(by_id[tid], roll)
                    rates[tid].append(float(s.reward))
                    for comp, v in component_rates(s, roll).items():
                        comps[tid][comp].append(v)
                    if s.reward < 1.0:
                        if s.feedback:
                            fb[tid].append(s.feedback)
                        if args.traces:
                            traces.append({
                                "task": tid, "trial": k, "reward": s.reward,
                                "feedback": s.feedback,
                                "tool_calls": [
                                    {"name": c.get("name"), "arguments": c.get("arguments")}
                                    for c in (getattr(roll, "tool_calls", None) or [])
                                ],
                                "trace": [
                                    {"role": m.get("role"),
                                     "content": str(m.get("content") or "")[:900]}
                                    for m in (getattr(roll, "trace", None) or [])
                                ],
                            })

    per_task = {
        tid: {
            "role": role[tid],
            "rate": round(sum(v) / len(v), 3) if v else None,
            "trials": len(v),
            "infra_dropped": infra.get(tid, 0),
            # Partial credit, when the scorer exposes any (see component_rates). A BINARY task
            # reward collapses "satisfied most of the contract" and "did nothing" into the same
            # 0.0; per-component means separate them, which is the difference between one
            # actionable number and thirty useless ones. Empty when the scorer has no components.
            "component_rates": {c: round(sum(v) / len(v), 3)
                                for c, v in sorted(comps[tid].items()) if v},
            "distinct_feedback": sorted({f for f in fb[tid]})[:4],
        }
        for tid, v in sorted(rates.items(), key=lambda kv: (role[kv[0]], kv[0]))
    }
    tgt = [v["rate"] for v in per_task.values() if v["role"] == "target" and v["rate"] is not None]
    can = [v["rate"] for v in per_task.values() if v["role"] == "canary" and v["rate"] is not None]
    result = {
        "candidate": str(cand),
        "wall_seconds": round(time.time() - t0, 1),
        "target_mean": round(sum(tgt) / len(tgt), 3) if tgt else None,
        "canary_mean": round(sum(can) / len(can), 3) if can else None,
        "reminder": "target_mean is a TRAINING number; only the full-val gate is evidence",
        "per_task": per_task,
    }
    if args.traces:
        Path(args.traces).write_text(json.dumps(traces, indent=2))
        result["traces_written"] = f"{args.traces} ({len(traces)} failing trials)"
    charged = sum(len(v) for v in rates.values()) + sum(infra.values())
    result["rollouts_spent"] = charged
    if args.run_dir:
        from cap_evolve import RunDir

        rd = RunDir.open(Path(args.run_dir))
        rd.update_spent(metric_calls=charged)
        result["charged_to_budget"] = str(Path(args.run_dir))
    else:
        result["charged_to_budget"] = None
        print(f"NOTE: {charged} rollouts NOT charged to any budget (no --run-dir)",
              file=sys.stderr)

    text = json.dumps(result, indent=2)
    if args.json_out:
        Path(args.json_out).write_text(text)
    print(text)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

"""watchdog.py — notice when ``host.py`` itself died, and relaunch it.

``host.py``'s own docstring covers every way the AGENT it launches can stop short —
turn budget, a backgrounded wait, a transient CLI crash — because in each of those cases
host.py is still alive to diagnose and (for a transient crash) retry. None of that helps
when host.py's OS process dies: the machine sleeps, the terminal running it closes, the
CI runner is preempted. The process just stops, mid tool-call, with no exit code anyone
sees and no ``final.json``. Re-running host.py against the same run dir is already safe
(``commit.py`` refuses to double-book a decided candidate; ``_seal`` is idempotent) — this
script only automates *noticing* that it needs to happen and doing it, so a run does not
sit silently stalled until a human happens to look.

Run it periodically (cron, a loop, CI) alongside host.py, once per run dir being watched.
It does ONE check per invocation and exits — no internal sleep loop, no daemon.

What it is not: a way to resume a hung turn inside the agent's own conversation. host.py's
briefing already states that invariant (delegate the work, never the waiting) because
ending a turn with work outstanding ends the process with nothing left to resume from
the inside. This script only restarts the OUTER process from the outside, which then
starts a fresh turn against the same run dir.
"""

from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path

HERE = Path(__file__).resolve().parent
HOST_PY = HERE / "host.py"

#: Heartbeat age past which host.py is presumed dead if its pid is also gone. Several
#: multiples of `host.py`'s HEARTBEAT_INTERVAL_SECONDS (60s), so one missed write from a
#: slow disk or a GC pause never reads as a crash.
DEFAULT_STALE_SECONDS = 10 * 60


def _read_json(path: Path) -> dict | None:
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError):
        return None


def _pid_alive(pid: int | None) -> bool:
    if not pid:
        return False
    try:
        os.kill(pid, 0)
    except ProcessLookupError:
        return False
    except PermissionError:
        return True  # exists, just owned by someone else — still alive
    except OSError:
        return False
    return True


def _default_relaunch(cmd: list[str], log_path: Path) -> None:
    """Detached, output redirected to `log_path` — the file this run's launch was missing,
    which is what made the original stall so hard to see after the fact.
    """
    with log_path.open("a", encoding="utf-8") as log:
        subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT, start_new_session=True)


def check(run_dir: Path, *, stale_seconds: int = DEFAULT_STALE_SECONDS,
          host_py: Path = HOST_PY, relaunch=_default_relaunch, dry_run: bool = False) -> dict:
    """One staleness check + (maybe) relaunch. Returns a JSON-serializable report.

    ``relaunch`` is a seam for tests: a callable ``(cmd, log_path) -> None``, never invoked
    for real when ``dry_run`` is set.
    """
    run_dir = Path(run_dir).resolve()
    if not run_dir.is_dir():
        return {"run_dir": str(run_dir), "action": "error", "reason": "run dir not found"}

    if (run_dir / "final.json").exists():
        return {"run_dir": str(run_dir), "action": "none",
                 "reason": "already sealed (final.json present)"}

    heartbeat = _read_json(run_dir / "host" / "heartbeat.json")
    if heartbeat is None:
        return {"run_dir": str(run_dir), "action": "none",
                 "reason": "no host/heartbeat.json — host.py has not started an agent "
                           "invocation yet, or this run predates the heartbeat"}

    age = time.time() - float(heartbeat.get("ts") or 0.0)
    pid = heartbeat.get("pid")
    if _pid_alive(pid):
        return {"run_dir": str(run_dir), "action": "none",
                 "reason": f"pid {pid} is alive", "age_seconds": round(age, 1)}

    if age < stale_seconds:
        return {"run_dir": str(run_dir), "action": "none",
                 "reason": f"heartbeat is {age:.0f}s old, under the {stale_seconds}s "
                           "threshold — pid is gone but may just be between attempts",
                 "age_seconds": round(age, 1)}

    launch = _read_json(run_dir / "host" / "launch_args.json")
    if launch is None:
        return {"run_dir": str(run_dir), "action": "error",
                 "reason": "heartbeat is stale and its pid is gone, but "
                           "host/launch_args.json is missing — cannot reconstruct the "
                           "original command line to relaunch it",
                 "age_seconds": round(age, 1)}

    cmd = [str(launch.get("python") or sys.executable), str(host_py),
           *[str(a) for a in (launch.get("argv") or [])]]
    log_path = run_dir / "host_launch.log"
    report = {"run_dir": str(run_dir), "cmd": cmd, "age_seconds": round(age, 1),
              "log": str(log_path)}
    if dry_run:
        report["action"] = "would_relaunch"
        return report
    relaunch(cmd, log_path)
    report["action"] = "relaunched"
    return report


def main(argv=None) -> int:
    p = argparse.ArgumentParser(
        prog="watchdog.py",
        description="Check a cap-evolve run dir's host.py heartbeat; relaunch it if stale.")
    p.add_argument("--run-dir", required=True, help="run dir host.py was launched against")
    p.add_argument("--stale-minutes", type=float, default=DEFAULT_STALE_SECONDS / 60,
                   help=f"heartbeat age, in minutes, before host.py is presumed dead "
                        f"(default {DEFAULT_STALE_SECONDS / 60:g})")
    p.add_argument("--dry-run", action="store_true",
                   help="report what would happen without actually relaunching")
    args = p.parse_args(argv)

    out = check(Path(args.run_dir), stale_seconds=args.stale_minutes * 60, dry_run=args.dry_run)
    print(json.dumps(out, indent=2))
    return 1 if out["action"] == "error" else 0


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())