using-cap-evolve
Front door for cap-evolve: routes an optimization request to the right pipeline phase. Use when someone wants an agent, skill, system prompt, tool surface, or MCP toolset to score higher on an eval, benchmark, or task suite — "optimize my skill", "raise the pass rate on these tas
Install
npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/orchestrate/using-cap-evolve
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
git clone https://github.com/skillberry-ai/cap-evolve.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
using-cap-evolve — the router
The front door: it works out where the user is and hands off, running no phase and
editing nothing. Boundary: this router picks the door, orchestrate drives the run.
Routing decision
Run from the user's project dir; S is the absolute path of the directory you loaded this
SKILL.md from — the one location always known here (no env var is set for a plugin install):
S=<this skill's own directory>; python "$S/scripts/run.py" --base .capevolve
Follow next; pass reason on to the user. Two things the JSON cannot say for itself:
- On a fresh request go through
intake, and if an input it needs is missing, ask the user for it rather than inventing one (intakeowns that rule). - An existing run is never restarted from zero: interrupted →
cap-evolve run --resume; sealed and the user wants another attempt →cap-evolve run --reuse-baseline <run dir>.
Three ways to run — orchestrate has the detail
- Phase chain —
/cap-evolve:<phase>turn by turn, so each step is inspected. - Deterministic —
cap-evolve run --spec .capevolve/project/capevolve.yamlsequences the check gate → baseline → algorithm → finalize → report. It presumes intake already happened; it does not run intake. - Agent handoff — with
orchestration_mode: agent,cap-evolve runstops after baseline and hands the loop back to you; no sealed-test number until you finalize.
No plugin, or a non-Claude host: follow RUN.md step by step. Same engine, same rules.
Files (cap-evolve)
-
scripts
-
abstract.py 231 B
"""using-cap-evolve is a pure router; it declares no adapter abstract methods. It only inspects on-disk project state and recommends the next command. check.py verifies it can resolve that state and that run.py exposes main().""" -
check.py 4.5 KB
"""Behavioral check for using-cap-evolve. Asserts the router actually routes, and that the three misroutes issue #339 reproduced stay fixed: * fresh (empty base) -> `fresh` / intake; a scaffolded base is not `fresh`. * a run dir with no `state.json` is NOT reported as `scaffolded` (it would send the agent to implement an adapter that already exists). * a newer live run is never masked by an older sealed one (which would claim a sealed headline number while an optimization is mid-flight). * `running` routes to `--resume` and `finalized` to `--reuse-baseline` — report continues nothing. """ from __future__ import annotations import json import sys import tempfile from pathlib import Path import _bootstrap # noqa: F401 def main() -> int: rep = {"ok": False, "problems": [], "notes": []} try: import run if not hasattr(run, "main") or not hasattr(run, "resolve_state"): rep["problems"].append("run.py missing main()/resolve_state()") else: with tempfile.TemporaryDirectory() as td: base = Path(td) / ".capevolve" base.mkdir(parents=True) fresh = run.resolve_state(base) if fresh.get("state") != "fresh": rep["problems"].append(f"empty base should be 'fresh', got {fresh.get('state')!r}") if "intake" not in (fresh.get("next") or ""): rep["problems"].append("fresh state must route to intake") # scaffolded: project dir + capevolve.yaml, no run, no adapter proj = base / "project" proj.mkdir(parents=True) (proj / "capevolve.yaml").write_text("capability_path: seed\n", encoding="utf-8") scaffolded = run.resolve_state(base) if scaffolded.get("state") == "fresh": rep["problems"].append("base with capevolve.yaml should not be 'fresh'") rep["notes"].append(f"fresh -> {fresh.get('next')}; " f"scaffolded -> {scaffolded.get('next')}") # case A: a run dir mid-create (no state.json) must not read as scaffolded. (base / "run_20260101_000000").mkdir() started = run.resolve_state(base) if started.get("state") == "scaffolded": rep["problems"].append("a run dir with no state.json must not resolve to " "'scaffolded' (sends the agent to re-implement)") if "--resume" not in (started.get("next") or ""): rep["problems"].append("a half-created run must route to `run --resume`") # case C: newer live run must not be masked by an older sealed one. sealed = base / "run_20260101_000000" (sealed / "state.json").write_text("{}", encoding="utf-8") (sealed / "splits.json").write_text('{"test_used": true}', encoding="utf-8") fin = run.resolve_state(base) if fin.get("state") != "finalized": rep["problems"].append(f"sealed run should be 'finalized', got {fin.get('state')!r}") if "--reuse-baseline" not in (fin.get("next") or ""): rep["problems"].append("'finalized' must offer a new run with --reuse-baseline, " "not just a report") live = base / "run_20260202_000000" live.mkdir() (live / "state.json").write_text("{}", encoding="utf-8") (live / "splits.json").write_text('{"test_used": false}', encoding="utf-8") mid = run.resolve_state(base) if mid.get("state") != "running": rep["problems"].append(f"a newer unsealed run must win over an older sealed " f"one; got {mid.get('state')!r}") if "--resume" not in (mid.get("next") or ""): rep["problems"].append("'running' must route to `cap-evolve run --resume`") rep["notes"].append(f"run_started -> {started.get('next')}; " f"finalized -> {fin.get('state')}; running -> {mid.get('next')}") except Exception as e: # noqa: BLE001 rep["problems"].append(f"import/exec failed: {e}") rep["ok"] = not rep["problems"] print(json.dumps(rep, indent=2)) return 0 if rep["ok"] else 1 if __name__ == "__main__": sys.exit(main()) -
run.py 6.7 KB
"""using-cap-evolve — resolve where the user is and recommend the next command. This router does NOT optimize. It inspects the on-disk state of a cap-evolve project and prints a routing decision the agent (or the orchestrator) acts on: {state, next, sequence, reason, intent} States (a simple, deterministic state machine over the project dir): fresh — no .capevolve/project/, or no capevolve.yaml -> next: /cap-evolve:intake scaffolded — capevolve.yaml exists, check not yet green -> next: /cap-evolve:implement-and-check ready — `cap-evolve check` is green, no run yet -> next: baseline / `cap-evolve run` run_started — a run_* dir with no state.json (a torn/partial create — see RunDir.create's exist_ok contract in core/cap_evolve/rundir.py) -> next: `cap-evolve run --resume --run-ts` running — a run_* dir, test split not used -> next: `cap-evolve run --resume` finalized — splits.json test_used -> next: a NEW run with --reuse-baseline (report only inspects the sealed one) Runs are globbed unconditionally. Requiring state.json would hide a run that is mid-create behind an older sealed one and report it as `finalized` while an optimization is in flight — the worst answer this router can give. The check is best-effort: if core isn't importable we fall back to "is there a capevolve.yaml" rather than failing. Pure stdlib + cap_evolve (optional). """ from __future__ import annotations import argparse import json import sys from pathlib import Path import _bootstrap # noqa: F401 PHASE_CHAIN = ["intake", "implement-and-check", "baseline", "<algorithm>", "finalize", "report"] def _latest_run(base: Path) -> Path | None: runs = sorted((r for r in base.glob("run_*") if r.is_dir()), key=lambda r: r.name) return runs[-1] if runs else None def _check_green(project: Path) -> bool | None: """True/False if we can run the real check, None if core unavailable.""" try: from cap_evolve.check import run_check except Exception: return None try: return run_check(project).ok except Exception: return False def _run_state(run: Path) -> dict: """Classify the newest run dir. Never claims a seal it cannot read.""" ts = run.name[len("run_"):] sp = run / "splits.json" if not (run / "state.json").exists() and not sp.exists(): return {"state": "run_started", "next": f"cap-evolve run --resume --run-ts {ts}", "run": str(run), "reason": "a run dir exists but has no state.json — a torn/partial create. " "Resume it by ts (plain --resume skips a run with no state.json); " "the adapter is already implemented, do not re-run intake."} if sp.exists(): try: spd = json.loads(sp.read_text(encoding="utf-8")) except Exception as e: # noqa: BLE001 return {"state": "running", "next": f"cap-evolve run --resume --run-ts {ts}", "run": str(run), "reason": f"splits.json is unreadable ({e}) — the split is an " "honesty-critical artifact, so the seal cannot be confirmed " "either way. Inspect it before spending more budget."} if spd.get("test_used"): return {"state": "finalized", "next": ("cap-evolve run --spec .capevolve/project/capevolve.yaml " f"--reuse-baseline {run}"), "run": str(run), "reason": "test is sealed/used — this run's headline number is recorded " "and cannot be re-scored. For another attempt start a new run " "reusing this baseline; `/cap-evolve:report` only inspects it."} return {"state": "running", "next": f"cap-evolve run --resume --run-ts {ts}", "run": str(run), "reason": "a run is in progress and the test split is still sealed — resume it " "at iteration N+1; `/cap-evolve:report` shows status without continuing."} def resolve_state(base: Path) -> dict: base = Path(base) project = base / "project" yaml = project / "capevolve.yaml" if not project.is_dir(): return {"state": "fresh", "next": "/cap-evolve:intake", "reason": "no .capevolve/project/ — start with intake (Phase 1)."} run = _latest_run(base) if run is not None: return _run_state(run) if not yaml.exists(): return {"state": "fresh", "next": "/cap-evolve:intake", "reason": "project dir exists but no capevolve.yaml — run intake. Warn the " "user first if the dir is non-empty: intake scaffolds into it."} green = _check_green(project) if green is True: return {"state": "ready", "next": "/cap-evolve:baseline", "reason": "cap-evolve check is green — baseline then the algorithm, " "or `cap-evolve run --spec` for the automatic path."} if green is False: return {"state": "scaffolded", "next": "/cap-evolve:implement-and-check", "reason": "capevolve.yaml present but `cap-evolve check` is not green — " "implement the adapter and pass the hard gate first."} # green is None: core not importable here — recommend the gate step conservatively. return {"state": "scaffolded", "next": "/cap-evolve:implement-and-check", "reason": "capevolve.yaml present; could not run check here — verify the " "hard gate via implement-and-check before baseline."} def main(argv=None) -> int: p = argparse.ArgumentParser(prog="using-cap-evolve") p.add_argument("intent", nargs="*", help="free-text 'what to optimize' (echoed back)") p.add_argument("--base", default=".capevolve") args = p.parse_args(argv) decision = resolve_state(Path(args.base)) decision["sequence"] = PHASE_CHAIN decision["intent"] = " ".join(args.intent) or None decision["run_modes"] = { "standalone": "drive /cap-evolve:<phase> turn by turn", "automatic": "cap-evolve run --spec .capevolve/project/capevolve.yaml", "agent_handoff": "orchestration_mode: agent — run stops after baseline, you drive", "host_agnostic": "follow RUN.md (no plugin / non-Claude host)", } print(json.dumps(decision, indent=2)) return 0 if __name__ == "__main__": sys.exit(main()) -
_bootstrap.py 3.6 KB
"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap. Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim only has to find that package, which means a minimal upward walk for ``core/`` — the single bit of bootstrapping that genuinely must run before cap_evolve is importable. Everything else delegates. """ from __future__ import annotations import os import sys from pathlib import Path def _seed_path() -> None: """Minimal: put a dir containing the cap_evolve package on sys.path. ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that it imports the checkout you are standing in. Deferring to the ambient package here made an explicit override unreachable, and the symptom was a stale core silently answering for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that predates that module). An explicit env var wins. """ env = os.environ.get("CAPEVOLVE_CORE") want = Path(env).resolve() if env else None if want and (want / "cap_evolve" / "__init__.py").exists(): loaded = sys.modules.get("cap_evolve") already = getattr(loaded, "__file__", None) if already and Path(already).resolve().parent.parent == want: return # right checkout already imported: touch nothing p = str(want) if p in sys.path: sys.path.remove(p) sys.path.insert(0, p) if loaded is not None: # Evicting a module makes a re-import yield a DIFFERENT object, so anything # already holding a reference fails an `is` check. Only ever do it when the # loaded package really is the wrong checkout — otherwise this "fix" becomes # the bug (it broke two identity assertions in core/tests exactly once). for name in [m for m in sys.modules if m == "cap_evolve" or m.startswith("cap_evolve.")]: sys.modules.pop(name, None) for finder in list(sys.meta_path): if "cap_evolve" in getattr(finder, "MAPPING", {}): sys.meta_path.remove(finder) return # A checkout's own core outranks an ambient install. Without this, a skill script run # from checkout X silently executed against checkout Y's cap_evolve (an editable install # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was # missing modules — or, worse, a green result measured against the wrong tree. here = Path(__file__).resolve() own = next((p / "core" for p in here.parents if (p / "core" / "cap_evolve" / "__init__.py").exists()), None) if own is not None: os.environ.setdefault("CAPEVOLVE_CORE", str(own)) return _seed_path() try: import cap_evolve # noqa: F401 return except Exception: pass cands = [] for parent in here.parents: cands.append(parent / "core") cands.append(parent) for c in cands: if (c / "cap_evolve" / "__init__.py").exists(): p = str(c) if p not in sys.path: sys.path.insert(0, p) return _seed_path() from cap_evolve._bootstrap import ensure_core # noqa: E402 # Anchor the upward walk at THIS skill script's location (not the core module's). ensure_core(Path(__file__).resolve())
-
-
meta.yaml 456 B
component: orchestrate name: using-cap-evolve summary: Session-start router — turns an "optimize this" request into the cap-evolve pipeline; resolves the on-disk state and names the next command (intake, the check gate, a resumed run, or a new run reusing a sealed baseline). entry: scripts/run.py abstract: scripts/abstract.py check: scripts/check.py needs: [] provides: [] compatible_with: capabilities: ["*"] optimizers: ["*"] algorithms: ["*"] -
SKILL.md 2.5 KB
--- name: using-cap-evolve description: 'Front door for cap-evolve: routes an optimization request to the right pipeline phase. Use when someone wants an agent, skill, system prompt, tool surface, or MCP toolset to score higher on an eval, benchmark, or task suite — "optimize my skill", "raise the pass rate on these tasks", "my agent keeps failing these cases", "get this prompt''s accuracy up on my evals" — even when they never say "optimize", and whenever a .capevolve/ project or an unfinished run is in the tree. Routes to intake, the check gate, a resumed run, or the report; optimizes nothing itself. Not for making code or a query faster, and not for rewording one prompt with no eval to score it against. When the user names a phase or algorithm outright (baseline, gate, hill-climb, gepa), use that skill directly.' component: orchestrate argument-hint: "[what to optimize] [--base .capevolve]" allowed-tools: Read, Bash provides: [] needs: [] sources: [evo, superpowers] --- # using-cap-evolve — the router The front door: it works out *where the user is* and hands off, running no phase and editing nothing. Boundary: this router picks the door, `orchestrate` drives the run. ## Routing decision Run from the user's project dir; `S` is the absolute path of the directory you loaded this SKILL.md from — the one location always known here (no env var is set for a plugin install): ```bash S=<this skill's own directory>; python "$S/scripts/run.py" --base .capevolve ``` Follow `next`; pass `reason` on to the user. Two things the JSON cannot say for itself: - On a fresh request go through `intake`, and if an input it needs is missing, ask the user for it rather than inventing one (`intake` owns that rule). - An existing run is never restarted from zero: interrupted → `cap-evolve run --resume`; sealed and the user wants another attempt → `cap-evolve run --reuse-baseline <run dir>`. ## Three ways to run — `orchestrate` has the detail 1. **Phase chain** — `/cap-evolve:<phase>` turn by turn, so each step is inspected. 2. **Deterministic** — `cap-evolve run --spec .capevolve/project/capevolve.yaml` sequences the check gate → baseline → algorithm → finalize → report. It presumes intake already happened; it does not run intake. 3. **Agent handoff** — with `orchestration_mode: agent`, `cap-evolve run` stops after baseline and hands the loop back to you; no sealed-test number until you finalize. No plugin, or a non-Claude host: follow `RUN.md` step by step. Same engine, same rules.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.