Claude Skill

agent-optimize

Free-form optimization algorithm for agent orchestration mode: the conversational agent owns the whole search — proposing capability edits itself, screening them cheaply, gating each on full val, and sealing test once. Use when orchestration_mode is agent and algorithm_skill is a

LLM Mart · 0 points · 18 views 0 listing impressions 0 install-command copies
Download skillberry-ai-cap-evolve-skills_algorithms_agent-optimize-2e823be.zip · 326 KB

Install

skills CLI npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/algorithms/agent-optimize
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
Git git clone https://github.com/skillberry-ai/cap-evolve.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

agent-optimize — the loop you own

You are the optimizer. cap-evolve run (agent mode) does check -> baseline and hands you R. Everything else is six act.py verbs and one digest.py; the scripts decide, you judge.

Honest statistics (read twice)

  1. Noise sd of a full-val mean is ~0.05 (30 tasks x 3 trials): only big effects are resolvable. Never re-test a within-noise delta; make a bigger edit instead.
  2. Never view test traces. Test is sealed, scored ONCE at finalize, and development-exposed after use.

Details only when needed: references/stats.md.

Setup

The loop REQUIRES the new engine. Put this in capevolve.yaml (or export CAPEVOLVE_<NAME>=1):

optimizer:
  ablation: {dag_parallel: true, active_eval: true, smart_merge: true, cost_gating: true,
             failure_clustering: true, pregate: true, context_digest: true}

Stock default has active_eval OFF: probe --auto then silently runs FULL val. The digest prints !! CONFIG: ... at the top (after any INFRA line) when any switch is off; fix it before looping. Switches: references/ablation.md.

R="<run_dir from the handoff>"; P="<project dir>"
S="${CAPEVOLVE_SKILLS_DIR:?set to the skills/ dir}"; A="$S/algorithms/agent-optimize/scripts"
mkdir -p "$R/work"
python "$A/spend.py" --run-dir "$R" --project "$P"      # parses stop_condition; ask the user if constraints.ambiguous
python "$A/digest.py" --run-dir "$R" --project "$P"     # first line must NOT start with "!! CONFIG"
BEST=$(python "$A/digest.py" --run-dir "$R" --project "$P" --format json | python -c 'import json,sys;print(json.load(sys.stdin)["champion"]["id"])')

BEST is the digest's champion (re-read it after every promote). Read once: PROJECT.md, capevolve.yaml, the adapter, the files under capability_path, optimizer/INSTRUCTIONS.md if present (benchmark facts only; ignore its "one edit" advice).

The loop

python "$S/phases/diagnose/scripts/run.py" --run-dir "$R" --tag "$BEST" --split val --cluster v2 > /dev/null
python "$A/digest.py" --run-dir "$R" --project "$P"

Diagnose writes a SLIM $R/clusters.json (ids, labels, exemplar trace ids) and the heavy clusters_full.json. Never cat/Read clusters_full.json (MBs of embedded traces).

  1. Digest. One screen: budget, noise, clusters (with hypothesis status), branch tips (posterior, coverage, P(beat parent/champion)), merge opportunities, pregate warnings, suggestions. Act on it.
  2. Read RAW evidence for the chosen cluster's exemplar trials: full tool-call args and the full conversation, never clips. Split deterministic tool bugs (fix in code) from policy failures. references/forensics.md has the dump command and subagent hand-off.
  3. Pick a hypothesis covering >= 2 tasks (or headroom >= 0.06), or a deterministic tool bug. Write it to a JSON file (references/clusters.md for the schema).
  4. Make a BIG coherent edit under work/TAG: policy and tools together when the failure needs both. General rules only, never a task id or answer. No single-task edits (their ceiling is below noise). Never hide write calls inside a composite tool. Form by failure type: references/tool-edit-lessons.md.
  5. Propose, probe, decide:
python "$A/act.py" propose cand_1 --run-dir "$R" --project "$P" --parent "$BEST"     # 1st call: creates work/cand_1
python "$A/act.py" propose cand_1 --run-dir "$R" --project "$P" --parent "$BEST" --hypothesis-file "$R/work/h1.json"   # 2nd: rule + pre-gate + record
python "$A/act.py" probe cand_1 --run-dir "$R" --project "$P" --auto --plan          # missing cells, spends nothing
python "$A/act.py" probe cand_1 --run-dir "$R" --project "$P" --auto --n 3 --budget 60
python "$A/act.py" promote cand_1 --run-dir "$R" --project "$P"                      # needs full-val coverage
python "$A/act.py" prune cand_1 --run-dir "$R" --project "$P" --reason "<why>"

Pre-gate failure: fix the edit (the text says what), propose again. Probe runs only cells the ledger lacks, on the hypothesis tasks plus sentinels (--tasks a,b to choose). The digest posterior says promote (P(beat champion) high at full coverage), prune (clearly worse) or continue (probe more). promote that does not accept leaves the node alive; best_id does not move. Full-val probes need --yes. When the budget is EXHAUSTED even propose refuses (--over-budget REASON is recorded). Before each promote/prune append ## Iteration TAG to work/TAG/JOURNAL.md: what changed (file -> cluster), expected effect, what the numbers said, next focus. 6. Merge complementary branches (digest merge_opps):

python "$A/act.py" merge cand_1 cand_2 --run-dir "$R" --project "$P" --tag cand_m

then probe and promote cand_m like any node. references/merge.md. 7. Repeat from the digest until it says stop (budget, goal met, or stop_condition). Then finalize.

Conductor engine (opt-in: ao_engine: conductor)

With ao_engine: conductor in orchestration_mode: agent, cap-evolve run starts scripts/conductor.py supervise instead of handing the loop to a conversational agent. Code runs a tick loop (reap, promote, merge or rebase, rungs, propose, stop); the LLM only runs bounded one-shot jobs (propose, merge-resolve) with a per-job token cap, and never polls or waits. Evaluations are chunked (ao_chunk, default 20 rollouts) in a shared slot pool, with promotion_rule: P2 going through the shared P2 evaluator. State is graph.jsonl plus conductor.json; a killed run resumes and finalize is idempotent. Stopping is structured only (rollouts, tokens, wall deadline_utc, stop_at_reward, plateau with a floor); prose in stop_condition is never parsed into a cap. A cost objective (minimize) adds cost per success: promote on reward with cost per success at most eps (default 0.10) worse, or on cost per success at least eps lower with reward non-inferior. Any other gating objective is refused (exit 2). ao_focus: true judges a candidate only on the champion's failing tasks plus ao_focus_canary passing ones (1-3 trials), with a stratified full-set gain; it promotes directly (no full-val screen or fresh confirm), gives proposers the parent's pass rates and failing traces, merges near misses, and runs one fresh full-val estimate at finalize. With K=1, rungs off and merge off it makes the same decisions as hill-climb. When PR-3 helper checks are importable and helpers.enabled, each proposal gets lint_candidate and run_smoke before any rollout. Without the flag, nothing here runs and the loop below is unchanged.

If the digest says X, do Y

digest says do
cluster open, >= 2 tasks or headroom >= 0.06 hypothesis + big edit + propose
cluster stuck (2 failed attempts) re-read raw full traces (never test), change the edit FORM, not wording
same tool-error signature across tasks fix it in tool code; no multi-task argument needed
pregate warning (tool fixture fails, policy growth) fix before any probe; it is free
same procedure fails or is redone by hand in >= 2 distinct traces, surface allows scripts codify it into a helper (references/helpers.md); 1 trace never justifies one
tip at full coverage, P(beat champion) high promote
tip P(beat parent) low after full probe prune with the numbers
tip partial coverage, undecided probe --auto for more cells
gate verdict inconclusive (low coverage / unstable) not a reject: probe more cells (legacy commit.py --decision inconclusive); never prune on it
delta within noise (noise.sd_est) do not re-test; make a bigger or different edit
noise.stale buy a seed duplicate probe, then re-read noise
merge_opps listed merge, probe the union of wins, promote
budget EXHAUSTED stop; finalize (spending verbs refuse; --over-budget REASON is recorded)
infra warnings read them; --strict makes them refusals
objectives / cost constraints references/objectives-cost.md
need to turn a switch off or run legacy references/ablation.md

Finalize (once)

python "$A/act.py" finalize --run-dir "$R" --project "$P" --n-confirm 3
python "$S/phases/report/scripts/run.py" --run-dir "$R"

Confirms the champion on full val with fresh seeds, tests it against the seed, seals test ONCE, writes final.json. If the confirm does not accept, finalize refuses to seal (--seal-anyway REASON records "no accepted change"). Report refusals unsoftened: best_id == seed is a null result with a diagnosed cause; an empty split is empty, not 0.0; a no-holdout spec is a FIT metric. No finalize, no result.

Invariants

Core enforces the split seal, val-only gating and the tamper guard. Yours: never hand a subset result to gate_check.py; every round leaves run-dir artifacts (JOURNAL.md entry in work/TAG, DIAGNOSIS.json); report a broken framework file, do not work around it. Wait for finalize to exit or the seal is wasted.

References

One level deep; load only when the digest points there.

  • references/forensics.md — raw-trace dump, subagent diagnosis, composite-tool pitfall. Load before your first edit and when stuck.
  • references/clusters.md — hypothesis schema, per-task rates. Load when picking or writing a hypothesis.
  • references/merge.md — merge flow and conflicts. Load when merge_opps is non-empty.
  • references/objectives-cost.md — cost gating, budget, stop_condition. Load when objectives are set or budget is tight.
  • references/ablation.md — the seven switches. Load before disabling any.
  • references/tool-edit-lessons.md — edit forms, guard closure. Load before editing a surface the first time or after a stuck cluster.
  • references/helpers.md — when to codify a procedure into a deterministic helper, and the lint/smoke it must pass. Load when the surface allows scripts and a procedure recurs.
  • references/stats.md — what the numbers resolve, sign test. Load when a result surprises you.
  • references/legacy.md — all switches off: pointer to the archived round-loop docs. Load only in legacy mode (old SKILL sections live in docs/archive/agent-optimize-legacy/legacy-SKILL.md).
Files (cap-evolve)
  • references
    • ablation.md 823 B
      Loading…
    • clusters.md 1.1 KB
      Loading…
    • forensics.md 1.5 KB
      Loading…
    • helpers.md 2.2 KB
      Loading…
    • legacy.md 466 B
      Loading…
    • merge.md 1.6 KB
      Loading…
    • objectives-cost.md 923 B
      Loading…
    • stats.md 1.1 KB
      Loading…
    • tool-edit-lessons.md 1.6 KB
      Loading…
  • scripts
    • ablate.py 34.8 KB
      Loading…
    • abstract.py 826 B
      Loading…
    • act.py 51.2 KB
      Loading…
    • check.py 71.2 KB
      Loading…
    • commit.py 64.4 KB
      Loading…
    • conductor.py 67.6 KB
      Loading…
    • conductor_backend.py 35.6 KB
      Loading…
    • detect_loop_patterns.py 5.3 KB
      Loading…
    • digest.py 25 KB
      Loading…
    • funcmerge.py 25.5 KB
      Loading…
    • gate_check.py 21.5 KB
      Loading…
    • grow.py 10.2 KB
      Loading…
    • host.py 86.2 KB
      Loading…
    • integrate.py 14.5 KB
      Loading…
    • linkcheck.py 2.3 KB
      Loading…
    • measure.py 16 KB
      Loading…
    • mechanisms.py 7.4 KB
      Loading…
    • merge.py 19 KB
      Loading…
    • merge_n.py 14.7 KB
      Loading…
    • merge_rejects.py 17.8 KB
      Loading…
    • merge_search.py 27.2 KB
      Loading…
    • merge_taskopt.py 7.8 KB
      Loading…
    • merge_verify.py 14.5 KB
      Loading…
    • meter.py 6.3 KB
      Loading…
    • microcase.py 16.9 KB
      Loading…
    • multirep.py 2.8 KB
      Loading…
    • plan_round.py 16.7 KB
      Loading…
    • policy.py 14 KB
      Loading…
    • posterior_check.py 2.1 KB
      Loading…
    • pregate.py 25 KB
      Loading…
    • prepare_candidate.py 8 KB
      Loading…
    • proposer.py 11.6 KB
      Loading…
    • round.py 123.2 KB
      Loading…
    • run.py 2.2 KB
      Loading…
    • screen.py 18.6 KB
      Loading…
    • spend.py 10.8 KB
      Loading…
    • taskeval.py 11.7 KB
      Loading…
    • watchdog.py 6 KB
      Loading…
    • _bootstrap.py 3.6 KB
      Loading…
  • meta.yaml 946 B
    Loading…
  • SKILL.md 11.3 KB
    ---
    name: agent-optimize
    description: 'Free-form optimization algorithm for agent orchestration mode: the conversational agent owns the whole search — reading a digest, proposing big coherent capability edits against failure-cluster hypotheses, probing only missing cells, promoting or pruning on a posterior, merging complementary branches, and sealing test once. Use when orchestration_mode is agent and algorithm_skill is agent-optimize. For a deterministic loop use hill-climb, gepa or skillopt instead.'
    component: algorithm
    argument-hint: "agent-mode only — set orchestration_mode: agent + algorithm_skill: agent-optimize"
    allowed-tools: Read, Write, Edit, Bash, Task
    provides: [candidate]
    needs: [scores, traces, candidate]
    ---
    
    # agent-optimize — the loop you own
    
    You are the optimizer. `cap-evolve run` (agent mode) does check -> baseline and hands you `R`. Everything
    else is six `act.py` verbs and one `digest.py`; the scripts decide, you judge.
    
    ## Honest statistics (read twice)
    
    1. Noise sd of a full-val mean is ~0.05 (30 tasks x 3 trials): only big effects are resolvable. Never
       re-test a within-noise delta; make a bigger edit instead.
    2. Never view test traces. Test is sealed, scored ONCE at finalize, and development-exposed after use.
    
    Details only when needed: [`references/stats.md`](references/stats.md).
    
    ## Setup
    
    The loop REQUIRES the new engine. Put this in `capevolve.yaml` (or export `CAPEVOLVE_<NAME>=1`):
    
    ```yaml
    optimizer:
      ablation: {dag_parallel: true, active_eval: true, smart_merge: true, cost_gating: true,
                 failure_clustering: true, pregate: true, context_digest: true}
    ```
    
    Stock default has `active_eval` OFF: `probe --auto` then silently runs FULL val. The digest prints
    `!! CONFIG: ...` at the top (after any INFRA line) when any switch is off; fix it before looping. Switches: [`references/ablation.md`](references/ablation.md).
    
    ```bash
    R="<run_dir from the handoff>"; P="<project dir>"
    S="${CAPEVOLVE_SKILLS_DIR:?set to the skills/ dir}"; A="$S/algorithms/agent-optimize/scripts"
    mkdir -p "$R/work"
    python "$A/spend.py" --run-dir "$R" --project "$P"      # parses stop_condition; ask the user if constraints.ambiguous
    python "$A/digest.py" --run-dir "$R" --project "$P"     # first line must NOT start with "!! CONFIG"
    BEST=$(python "$A/digest.py" --run-dir "$R" --project "$P" --format json | python -c 'import json,sys;print(json.load(sys.stdin)["champion"]["id"])')
    ```
    
    `BEST` is the digest's champion (re-read it after every promote). Read once: `PROJECT.md`,
    `capevolve.yaml`, the adapter, the files under `capability_path`, `optimizer/INSTRUCTIONS.md` if
    present (benchmark facts only; ignore its "one edit" advice).
    
    ## The loop
    
    ```bash
    python "$S/phases/diagnose/scripts/run.py" --run-dir "$R" --tag "$BEST" --split val --cluster v2 > /dev/null
    python "$A/digest.py" --run-dir "$R" --project "$P"
    ```
    
    Diagnose writes a SLIM `$R/clusters.json` (ids, labels, exemplar trace ids) and the heavy
    `clusters_full.json`. Never `cat`/Read `clusters_full.json` (MBs of embedded traces).
    
    1. **Digest.** One screen: budget, noise, clusters (with hypothesis status), branch tips (posterior,
       coverage, P(beat parent/champion)), merge opportunities, pregate warnings, suggestions. Act on it.
    2. **Read RAW evidence** for the chosen cluster's exemplar trials: full tool-call args and the full
       conversation, never clips. Split deterministic tool bugs (fix in code) from policy failures.
       [`references/forensics.md`](references/forensics.md) has the dump command and subagent hand-off.
    3. **Pick a hypothesis** covering >= 2 tasks (or headroom >= 0.06), or a deterministic tool bug. Write
       it to a JSON file ([`references/clusters.md`](references/clusters.md) for the schema).
    4. **Make a BIG coherent edit** under `work/TAG`: policy and tools together when the failure needs both.
       General rules only, never a task id or answer. No single-task edits (their ceiling is below noise).
       Never hide write calls inside a composite tool. Form by failure type:
       [`references/tool-edit-lessons.md`](references/tool-edit-lessons.md).
    5. **Propose, probe, decide:**
    
    ```bash
    python "$A/act.py" propose cand_1 --run-dir "$R" --project "$P" --parent "$BEST"     # 1st call: creates work/cand_1
    python "$A/act.py" propose cand_1 --run-dir "$R" --project "$P" --parent "$BEST" --hypothesis-file "$R/work/h1.json"   # 2nd: rule + pre-gate + record
    python "$A/act.py" probe cand_1 --run-dir "$R" --project "$P" --auto --plan          # missing cells, spends nothing
    python "$A/act.py" probe cand_1 --run-dir "$R" --project "$P" --auto --n 3 --budget 60
    python "$A/act.py" promote cand_1 --run-dir "$R" --project "$P"                      # needs full-val coverage
    python "$A/act.py" prune cand_1 --run-dir "$R" --project "$P" --reason "<why>"
    ```
    
       Pre-gate failure: fix the edit (the text says what), propose again. Probe runs only cells the ledger
       lacks, on the hypothesis tasks plus sentinels (`--tasks a,b` to choose). The digest posterior says
       promote (P(beat champion) high at full coverage), prune (clearly worse) or continue (probe more).
       `promote` that does not accept leaves the node `alive`; best_id does not move. Full-val probes need `--yes`.
       When the budget is EXHAUSTED even `propose` refuses (`--over-budget REASON` is recorded).
       Before each promote/prune append `## Iteration TAG` to `work/TAG/JOURNAL.md`: what changed (file ->
       cluster), expected effect, what the numbers said, next focus.
    6. **Merge** complementary branches (digest `merge_opps`):
    
    ```bash
    python "$A/act.py" merge cand_1 cand_2 --run-dir "$R" --project "$P" --tag cand_m
    ```
    
       then probe and promote `cand_m` like any node. [`references/merge.md`](references/merge.md).
    7. **Repeat** from the digest until it says stop (budget, goal met, or `stop_condition`). Then finalize.
    
    ## Conductor engine (opt-in: `ao_engine: conductor`)
    
    With `ao_engine: conductor` in `orchestration_mode: agent`, `cap-evolve run` starts
    `scripts/conductor.py supervise` instead of handing the loop to a conversational agent. Code runs a tick
    loop (reap, promote, merge or rebase, rungs, propose, stop); the LLM only runs bounded one-shot jobs
    (propose, merge-resolve) with a per-job token cap, and never polls or waits. Evaluations are chunked
    (`ao_chunk`, default 20 rollouts) in a shared slot pool, with `promotion_rule: P2` going through the shared
    P2 evaluator. State is `graph.jsonl` plus `conductor.json`; a killed run resumes and `finalize` is
    idempotent. Stopping is structured only (rollouts, tokens, wall `deadline_utc`, `stop_at_reward`, plateau
    with a floor); prose in `stop_condition` is never parsed into a cap. A `cost` objective (minimize) adds cost per
    success: promote on reward with cost per success at most `eps` (default 0.10) worse, or on cost per
    success at least `eps` lower with reward non-inferior. Any other gating objective is refused (exit 2). `ao_focus: true` judges a
    candidate only on the champion's failing tasks plus `ao_focus_canary` passing ones (1-3 trials), with a
    stratified full-set gain; it promotes directly (no full-val screen or fresh confirm), gives proposers the
    parent's pass rates and failing traces, merges near misses, and runs one fresh full-val estimate at finalize. With K=1, rungs off and merge off it makes the same decisions as hill-climb. When PR-3 helper
    checks are importable and `helpers.enabled`, each proposal gets `lint_candidate` and `run_smoke` before any
    rollout. Without the flag, nothing here runs and the loop below is unchanged.
    
    ## If the digest says X, do Y
    
    | digest says | do |
    | --- | --- |
    | cluster open, >= 2 tasks or headroom >= 0.06 | hypothesis + big edit + `propose` |
    | cluster stuck (2 failed attempts) | re-read raw full traces (never test), change the edit FORM, not wording |
    | same tool-error signature across tasks | fix it in tool code; no multi-task argument needed |
    | pregate warning (tool fixture fails, policy growth) | fix before any probe; it is free |
    | same procedure fails or is redone by hand in >= 2 distinct traces, surface allows scripts | codify it into a helper ([`references/helpers.md`](references/helpers.md)); 1 trace never justifies one |
    | tip at full coverage, P(beat champion) high | `promote` |
    | tip P(beat parent) low after full probe | `prune` with the numbers |
    | tip partial coverage, undecided | `probe --auto` for more cells |
    | gate verdict inconclusive (low coverage / unstable) | not a reject: `probe` more cells (legacy `commit.py --decision inconclusive`); never `prune` on it |
    | delta within noise (`noise.sd_est`) | do not re-test; make a bigger or different edit |
    | `noise.stale` | buy a seed duplicate probe, then re-read noise |
    | merge_opps listed | `merge`, probe the union of wins, `promote` |
    | budget EXHAUSTED | stop; `finalize` (spending verbs refuse; `--over-budget REASON` is recorded) |
    | infra warnings | read them; `--strict` makes them refusals |
    | objectives / cost constraints | [`references/objectives-cost.md`](references/objectives-cost.md) |
    | need to turn a switch off or run legacy | [`references/ablation.md`](references/ablation.md) |
    
    ## Finalize (once)
    
    ```bash
    python "$A/act.py" finalize --run-dir "$R" --project "$P" --n-confirm 3
    python "$S/phases/report/scripts/run.py" --run-dir "$R"
    ```
    
    Confirms the champion on full val with fresh seeds, tests it against the seed, seals test ONCE, writes
    `final.json`. If the confirm does not accept, finalize refuses to seal (`--seal-anyway REASON` records
    "no accepted change"). Report refusals unsoftened: `best_id == seed` is a null result with a diagnosed
    cause; an empty split is `empty`, not 0.0; a no-holdout spec is a FIT metric. No finalize, no result.
    
    ## Invariants
    
    Core enforces the split seal, val-only gating and the tamper guard. Yours: never hand a subset result to
    `gate_check.py`; every round leaves run-dir artifacts (JOURNAL.md entry in `work/TAG`, DIAGNOSIS.json);
    report a broken framework file, do not work around it. Wait for finalize to exit or the seal is wasted.
    
    ## References
    
    One level deep; load only when the digest points there.
    
    - [`references/forensics.md`](references/forensics.md) — raw-trace dump, subagent diagnosis, composite-tool pitfall. **Load** before your first edit and when stuck.
    - [`references/clusters.md`](references/clusters.md) — hypothesis schema, per-task rates. **Load** when picking or writing a hypothesis.
    - [`references/merge.md`](references/merge.md) — merge flow and conflicts. **Load** when `merge_opps` is non-empty.
    - [`references/objectives-cost.md`](references/objectives-cost.md) — cost gating, budget, stop_condition. **Load** when objectives are set or budget is tight.
    - [`references/ablation.md`](references/ablation.md) — the seven switches. **Load** before disabling any.
    - [`references/tool-edit-lessons.md`](references/tool-edit-lessons.md) — edit forms, guard closure. **Load** before editing a surface the first time or after a stuck cluster.
    - [`references/helpers.md`](references/helpers.md) — when to codify a procedure into a deterministic helper, and the lint/smoke it must pass. **Load** when the surface allows scripts and a procedure recurs.
    - [`references/stats.md`](references/stats.md) — what the numbers resolve, sign test. **Load** when a result surprises you.
    - [`references/legacy.md`](references/legacy.md) — all switches off: pointer to the archived round-loop docs. **Load** only in legacy mode (old SKILL sections live in docs/archive/agent-optimize-legacy/legacy-SKILL.md).
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.