The gate is where dishonest optimization is prevented. Search is a noise\namplifier: try enough candidates and some will look better by chance alone\n(the more candidates you screen, the larger the expected best-of-noise). The gate\nis the rule that keeps a lucky draw from being promoted to \"the new best\". It\nrefuses any split but val, and by default accepts a candidate only when its val\nreward beats the current best by more than k standard errors.
scores — the candidate's and current best's val reward and\nstderr (from evaluate). The SE is not optional: significance is meaningless\nwithout it.decision — {accept, reason, delta, threshold}, the audit\nrecord of why a candidate was kept or rejected.paired (the default): accept ⟺ mean(Δ[t]) > k · SE(Δ) over the SAME val tasks\nsignificant (fallback): accept ⟺ Δ = cand − curr > k · sqrt(cand_se² + curr_se²)\nThe bar is Δ > k·SE and not Δ > 0 because search is a noise amplifier:\nscreen enough candidates and the best-looking one is best by luck, so Δ > 0\nbanks noise as progress and the val curve climbs while nothing improved. Clearing\nk standard errors of the measurement's own error is what makes an accept mean\nsomething — turn this down and the run's numbers stop being evidence. k=1 is\nlenient (~1σ); raise it to be stricter. It is the textual-optimization analogue of\nKoehn's bootstrap significance test for metric differences.
paired is stronger because both sides were scored on the same val tasks, so\nper-task difficulty cancels and only the paired variance counts; significant\ntreats the two means as independent samples and is only correct when they are.
Single-trial scores report stderr=0, collapsing k·SE to 0 — then\nsignificant silently degrades to strict and accepts any positive blip. If you\nrun the significance gate, score with multiple trials (see evaluate).
paired (the default): mean(per-task Δ) > k·SE(Δ). The loop selects it\nwhenever per-task val data exists (harness.py:1524-1526, gepa.py:741-743)\nand capevolve.yaml ships gate_mode: paired.significant: Δ > k·SE_combined — the unpaired fallback, used when the two\nsides aren't aligned per task. decide()'s own mode= parameter defaults here\nfor bare callers with no per-task data; that is not the default of a real run.threshold: Δ > T — a flat margin (use when you have a domain minimum\nworthwhile gain, e.g. \"don't bother unless +2pp\").strict: Δ > 0 — any improvement. Only safe with a near-zero-variance scorer\n(deterministic, single correct answer).Anything else raises. There is no simplicity/size mode: it was unreachable dead\ncode (nothing ever supplied a size) so it silently behaved as strict, and it has\nbeen removed rather than documented.
A mean can rise while previously-passing tasks silently break. Pair the\nsignificance gate with a no-regression check: reject a candidate that improves\nthe aggregate but drops any task that the current best passed. This is the same\ndual-gate discipline SWE-bench-style harnesses use (a patch must pass the new\ntests and not break the existing ones — FAIL_TO_PASS and PASS_TO_PASS).\ndiagnose provides kept_good (the currently-passing tasks) precisely so this\ncheck has something to protect.
This phase runs two ways from the same SKILL.md: standalone as the slash command /cap-evolve:gate (the argument-hint shows its run.py args), and orchestrator-callable — cap-evolve run / the orchestrate skill invokes the same scripts/run.py headlessly and threads the run dir between phases.
python scripts/run.py --current 0.50 --candidate 0.62 \\\n --mode significant --k-se 1.0 --candidate-stderr 0.03 --current-stderr 0.03\nAlgorithms call the gate internally every iteration via the harness; this skill\nexists so a human or agent can reproduce and understand a single decision.
\npaired mode (the default) with real multi-trial SEs; a no-regression check on\ntop; every accept/reject logged with its reason.train (the tool refuses this — it overfits the optimizer to\nthe data it edits against); strict mode on a noisy agent (accepts noise);\nraising the mean while quietly regressing tasks because no-regression was off.references/concepts.md — the difference-of-means SE, choosing k, the\nmultiple-comparisons motivation, the dual-gate / no-regression rationale, and\nwhy gating on val (never train, never test) is the honest split, with sources.