gate
Apply the acceptance decision that keeps optimization honest — always on the val split, by default requiring the improvement to exceed the significance bar (Δ > k·SE) so noise is not mistaken for progress. Use to inspect or reproduce a single accept/reject decision; the algorithm
Install
npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/gate
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
git clone https://github.com/skillberry-ai/cap-evolve.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
gate — accept only real improvements, on val
The gate is where dishonest optimization is prevented. Search is a noise
amplifier: try enough candidates and some will look better by chance alone
(the more candidates you screen, the larger the expected best-of-noise). The gate
is the rule that keeps a lucky draw from being promoted to "the new best". It
refuses any split but val, and by default accepts a candidate only when its val
reward beats the current best by more than k standard errors.
Inputs / outputs (manifest tokens)
- needs:
scores— the candidate's and current best's val reward andstderr(fromevaluate). The SE is not optional: significance is meaningless without it. - provides:
decision—{accept, reason, delta, threshold}, the audit record of why a candidate was kept or rejected.
The significance rule
paired (the default): accept ⟺ mean(Δ[t]) > k · SE(Δ) over the SAME val tasks
significant (fallback): accept ⟺ Δ = cand − curr > k · sqrt(cand_se² + curr_se²)
The bar is Δ > k·SE and not Δ > 0 because search is a noise amplifier:
screen enough candidates and the best-looking one is best by luck, so Δ > 0
banks noise as progress and the val curve climbs while nothing improved. Clearing
k standard errors of the measurement's own error is what makes an accept mean
something — turn this down and the run's numbers stop being evidence. k=1 is
lenient (~1σ); raise it to be stricter. It is the textual-optimization analogue of
Koehn's bootstrap significance test for metric differences.
paired is stronger because both sides were scored on the same val tasks, so
per-task difficulty cancels and only the paired variance counts; significant
treats the two means as independent samples and is only correct when they are.
Single-trial scores report stderr=0, collapsing k·SE to 0 — then
significant silently degrades to strict and accepts any positive blip. If you
run the significance gate, score with multiple trials (see evaluate).
Modes
paired(the default):mean(per-task Δ) > k·SE(Δ). The loop selects it whenever per-task val data exists (harness.py:1524-1526,gepa.py:741-743) andcapevolve.yamlshipsgate_mode: paired.significant:Δ > k·SE_combined— the unpaired fallback, used when the two sides aren't aligned per task.decide()'s ownmode=parameter defaults here for bare callers with no per-task data; that is not the default of a real run.threshold:Δ > T— a flat margin (use when you have a domain minimum worthwhile gain, e.g. "don't bother unless +2pp").strict:Δ > 0— any improvement. Only safe with a near-zero-variance scorer (deterministic, single correct answer).
Anything else raises. There is no simplicity/size mode: it was unreachable dead
code (nothing ever supplied a size) so it silently behaved as strict, and it has
been removed rather than documented.
No-regression (the second gate)
A mean can rise while previously-passing tasks silently break. Pair the
significance gate with a no-regression check: reject a candidate that improves
the aggregate but drops any task that the current best passed. This is the same
dual-gate discipline SWE-bench-style harnesses use (a patch must pass the new
tests and not break the existing ones — FAIL_TO_PASS and PASS_TO_PASS).
diagnose provides kept_good (the currently-passing tasks) precisely so this
check has something to protect.
Dual-mode
This phase runs two ways from the same SKILL.md: standalone as the slash command /cap-evolve:gate (the argument-hint shows its run.py args), and orchestrator-callable — cap-evolve run / the orchestrate skill invokes the same scripts/run.py headlessly and threads the run dir between phases.
How to run
python scripts/run.py --current 0.50 --candidate 0.62 \
--mode significant --k-se 1.0 --candidate-stderr 0.03 --current-stderr 0.03
Algorithms call the gate internally every iteration via the harness; this skill exists so a human or agent can reproduce and understand a single decision.
What good vs bad looks like
- Good:
pairedmode (the default) with real multi-trial SEs; a no-regression check on top; every accept/reject logged with itsreason. - Bad: gating on
train(the tool refuses this — it overfits the optimizer to the data it edits against);strictmode on a noisy agent (accepts noise); raising the mean while quietly regressing tasks because no-regression was off.
References
references/concepts.md— the difference-of-means SE, choosingk, the multiple-comparisons motivation, the dual-gate / no-regression rationale, and why gating on val (never train, never test) is the honest split, with sources.
Files (cap-evolve)
-
references
-
concepts.md 5 KB
Loading…
-
-
scripts
-
abstract.py 161 B
Loading… -
check.py 1.7 KB
Loading… -
run.py 4.7 KB
Loading… -
_bootstrap.py 3.6 KB
Loading…
-
-
meta.yaml 301 B
Loading… -
SKILL.md 5.3 KB
--- name: gate description: Apply the acceptance decision that keeps optimization honest — always on the val split, by default requiring the improvement to exceed the significance bar (Δ > k·SE) so noise is not mistaken for progress. Use to inspect or reproduce a single accept/reject decision; the algorithms apply it internally every iteration. component: phase argument-hint: "--current R --candidate R --mode significant --k-se 1.0" allowed-tools: Bash provides: [decision] needs: [scores] sources: [] --- # gate — accept only real improvements, on val The gate is where dishonest optimization is prevented. Search is a noise amplifier: try enough candidates and some will *look* better by chance alone (the more candidates you screen, the larger the expected best-of-noise). The gate is the rule that keeps a lucky draw from being promoted to "the new best". It refuses any split but `val`, and by default accepts a candidate only when its val reward beats the current best by **more than `k` standard errors**. ## Inputs / outputs (manifest tokens) - **needs:** `scores` — the candidate's and current best's val reward *and* `stderr` (from `evaluate`). The SE is not optional: significance is meaningless without it. - **provides:** `decision` — `{accept, reason, delta, threshold}`, the audit record of why a candidate was kept or rejected. ## The significance rule ``` paired (the default): accept ⟺ mean(Δ[t]) > k · SE(Δ) over the SAME val tasks significant (fallback): accept ⟺ Δ = cand − curr > k · sqrt(cand_se² + curr_se²) ``` The bar is `Δ > k·SE` and **not `Δ > 0`** because search is a noise amplifier: screen enough candidates and the best-looking one is best by *luck*, so `Δ > 0` banks noise as progress and the val curve climbs while nothing improved. Clearing `k` standard errors of the measurement's own error is what makes an accept mean something — turn this down and the run's numbers stop being evidence. `k=1` is lenient (~1σ); raise it to be stricter. It is the textual-optimization analogue of Koehn's bootstrap significance test for metric differences. `paired` is stronger because both sides were scored on the *same* val tasks, so per-task difficulty cancels and only the paired variance counts; `significant` treats the two means as independent samples and is only correct when they are. **Single-trial scores report `stderr=0`, collapsing `k·SE` to 0** — then `significant` silently degrades to `strict` and accepts any positive blip. If you run the significance gate, score with multiple trials (see `evaluate`). ## Modes - `paired` (**the default**): `mean(per-task Δ) > k·SE(Δ)`. The loop selects it whenever per-task val data exists (`harness.py:1524-1526`, `gepa.py:741-743`) and `capevolve.yaml` ships `gate_mode: paired`. - `significant`: `Δ > k·SE_combined` — the **unpaired fallback**, used when the two sides aren't aligned per task. `decide()`'s own `mode=` parameter defaults here for bare callers with no per-task data; that is not the default of a real run. - `threshold`: `Δ > T` — a flat margin (use when you have a domain minimum worthwhile gain, e.g. "don't bother unless +2pp"). - `strict`: `Δ > 0` — any improvement. Only safe with a near-zero-variance scorer (deterministic, single correct answer). Anything else raises. There is no simplicity/size mode: it was unreachable dead code (nothing ever supplied a size) so it silently behaved as `strict`, and it has been removed rather than documented. ## No-regression (the second gate) A mean can rise while previously-passing tasks silently break. Pair the significance gate with a **no-regression** check: reject a candidate that improves the aggregate but *drops* any task that the current best passed. This is the same dual-gate discipline SWE-bench-style harnesses use (a patch must pass the new tests **and** not break the existing ones — FAIL_TO_PASS *and* PASS_TO_PASS). `diagnose` provides `kept_good` (the currently-passing tasks) precisely so this check has something to protect. ## Dual-mode This phase runs two ways from the **same** SKILL.md: standalone as the slash command `/cap-evolve:gate` (the `argument-hint` shows its run.py args), and orchestrator-callable — `cap-evolve run` / the `orchestrate` skill invokes the same `scripts/run.py` headlessly and threads the run dir between phases. ## How to run ``` python scripts/run.py --current 0.50 --candidate 0.62 \ --mode significant --k-se 1.0 --candidate-stderr 0.03 --current-stderr 0.03 ``` Algorithms call the gate internally every iteration via the harness; this skill exists so a human or agent can reproduce and *understand* a single decision. ## What good vs bad looks like - **Good:** `paired` mode (the default) with real multi-trial SEs; a no-regression check on top; every accept/reject logged with its `reason`. - **Bad:** gating on `train` (the tool refuses this — it overfits the optimizer to the data it edits against); `strict` mode on a noisy agent (accepts noise); raising the mean while quietly regressing tasks because no-regression was off. ## References - `references/concepts.md` — the difference-of-means SE, choosing `k`, the multiple-comparisons motivation, the dual-gate / no-regression rationale, and why gating on val (never train, never test) is the honest split, with sources.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.