Claude Skill

intake

Starts a cap-evolve optimization run. Interviews the user to decide what capability to optimize, which runner/optimizer/algorithm to use, and where the tasks and the scoring source live, then scaffolds .capevolve/project/ (adapter stub, capevolve.yaml, PROJECT.md). Use when someo

LLM Mart · 0 points · 13 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download skillberry-ai-cap-evolve-skills_phases_intake-1431b31.zip · 17 KB
Part of skillberry-ai/cap-evolve — 22 skills

Install

skills CLI npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/intake
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
Git git clone https://github.com/skillberry-ai/cap-evolve.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

intake — collect inputs, scaffold the project

Turn a vague wish ("make this agent better at X") into a runnable project: a filled capevolve.yaml, an adapter ready to implement, and every NEEDED input resolved before any budget is spent. Intake is cheap; an unresolved input found three phases later is a wasted run and a meaningless number.

Ask, never fabricate — the core discipline of this phase

inputs/INPUTS.md classifies every input NEEDED or RECOMMENDED. For each NEEDED input that is not already present, do not proceed:

  • Interactive / chat mode — ASK THE USER and wait. Quote all three, they are in INPUTS.md per input: (a) the exact path the input is expected at, (b) the command or option that produces it, (c) the alternatives. Say what breaks without it.
  • Non-interactive (cap-evolve run / the orchestrate skill, nobody to ask) — write BLOCKED: <input> — why it is needed — how to provide it into PROJECT.md and exit non-zero. A blocked-but-honest stop is correct; a green run on a guessed input is not.

A fabricated dataset, scorer, trajectories path or gold answer does not unblock the run — it produces a number that measures nothing and hides that fact. A missing tasks file is a question for the user, not a gap for you to paper over.

RECOMMENDED inputs may take their default, but log every default in PROJECT.md with its honesty cost (e.g. "num_trials=1 — single-trial scores, so the significance gate will correctly reject marginal gains"), so the cost is visible at report time.

Step 0 — mine, then inspect, then ask once

  1. Mine the conversation first. Anything the user already said is an answer you must not re-ask — "optimize my airline policy on the flight-change tasks" already fixed the capability, the artifact and the task subset. Harvest that, and any correction the user made, before asking anything.
  2. Run the miner. python scripts/run.py --base .capevolve --workdir <repo-root> scaffolds and returns discovered — task files, capability artifacts, existing adapters. Reuse what it found; never re-author it.
  3. Inspect what discovered leaves open: the entrypoint, how one eval runs, where traces and scores land, candidate metrics, a natural train/val/test split, cost caps. Run gh auth status. Fan subagents out over the benchmark repo (entrypoint, scorer, trace dir, task schema) while the user answers instead of serializing — come prepared, so the user carries as little of the research as possible.
  4. Then ask the FEWEST questions, as ONE numbered batch, each with the detected value pre-filled as a default plus a free-text escape — including the ones only a human can answer: which metric gates accept/reject and each shown metric's direction, GitHub mirroring, deterministic vs agent orchestration (plus stop_condition in agent mode), splits, trials, budget, and memory_skill (default md-files; offer wiki — the weakness-graph format, see inputs/INPUTS.md — when the user wants weaknesses tracked as a persistent graph rather than an append-only journal). inputs/INPUTS.md → RECOMMENDED is the authority on each key; SKILL.md only fixes when to ask. Define jargon in a clause before using it ("pass^k — how often it succeeds on all k tries"); the user may be a domain expert, not an ML one.
  5. Confirm before scaffolding. Echo the resolved spec back as one block — capability, optimizer, algorithm, dataset, splits, budget, every RECOMMENDED input you are defaulting — and get a yes. A misread is cheapest to fix here.

What it does

The interview settles the capability skill (what is optimized), the optimizer (which coding agent proposes edits), the algorithm (the search loop), dataset, splits, budget.

  1. Scaffold .capevolve/project/ via scripts/run.py: adapter stub, inputs/, capevolve.yaml, PROJECT.md, and optimizer/INSTRUCTIONS.md. The whole templates/project/ tree is copytree'd verbatim — confirm the files landed.
  2. Resolve inputs per inputs/INPUTS.md, honoring the ask-never-fabricate rule.
  3. Record the resolved trajectories path and the scoring source in PROJECT.md, so implement-and-check wires trajectories() and score() against real inputs rather than guesses. inputs/INPUTS.md → scorer specifies exactly what the feedback must be (argument-level, gold-safe) and that score() must be deterministic — follow it literally, that feedback is the learning signal. Note in PROJECT.md if you deliberately return None from trajectories() (cap-evolve then falls back to its own per-rollout JSON).
  4. Customize the scaffolded optimizer/INSTRUCTIONS.md for THIS benchmark. The shipped template already carries the depth mandate, the non-overfitting guardrail, the STEP-0 reading mandate and the cross-iteration file protocol — do not re-author any of them. Your three jobs:
    1. keep every {{...}} placeholder intact ({{FOCUS_SUMMARY}}, {{FAILURES}}, {{CAP_BRIEF}}, {{ALGO_BRIEF}}, {{BENCH_REPO}} — the harness fills them per iteration; implement-and-check's pipeline self-test fails if one is deleted, and rendering must leave no {{ behind);
    2. scope it to the selected capabilities — delete the sections for capabilities not listed in capevolve.yaml: capabilities, so a run never presents an artifact as editable that this run does not own. Point the optimizer at ./guidance/<cap>/SKILL.md for each selected capability's own edit space, and at ./guidance/diagnose/SKILL.md for the failure taxonomy — both are materialized into its working dir. When a selected capability ships one, also point at ./guidance/<cap>/references/optimizer-playbook.md;
    3. add the benchmark-specific facts the template cannot know: where the runner writes traces, what the scoring source is, which data-model files the capability's code imports.
  5. Set the spec keys in capevolve.yaml — runner_repo_path, optimizer_instructions_file, capability_sources (the module(s) a selected capability's code imports, copied into the optimizer's ./guidance/sources/), target_model. inputs/INPUTS.md defines each one.
    • Caution (issue #252): a relative optimizer_instructions_file resolves project-relative under check but cwd-relative under run, which then silently falls back to the generic template. Write it absolute, or verify run actually picks up the customized file — intake authors it, so intake is the cheapest place to get it right.

How to run

python scripts/run.py --base .capevolve --workdir .   # mine, then scaffold

The script is purely mechanical; the judgment — interviewing, choosing components, the ask-if-missing loop — is yours. Then implement adapters/adapter.py, fill capevolve.yaml, and hand off to implement-and-check: together the two phases are the full integration (scaffold → the 3 required adapter methods → cap-evolve check green) and no budget is spent until that gate passes.

Onboarding transcript (one example): examples/tau2_airline/setup.sh clones and installs a benchmark and wires the adapter until cap-evolve check is green, and its run.sh runs the optimization. Read it only when onboarding a benchmark you have not integrated before.

Good vs bad intake

  • Good: every NEEDED input resolved to a real path or "adapter"; splits and budget chosen deliberately; each defaulted RECOMMENDED input logged; spec confirmed by the user.
  • Bad: a synthesized tasks file that "looked plausible"; a scorer that leaks the gold answer into feedback; test == train with no note; a budget too small to find a gain; the run proceeded past a missing NEEDED input "to keep moving".

References

  • inputs/INPUTS.md — the binding contract: every input classified NEEDED vs RECOMMENDED with the path / how-to-retrieve / alternatives you must quote, plus the meaning and default of every spec key. Read it during the interview.
  • references/concepts.md — why the contract is shaped this way, the 3 required adapter methods, split/trial/budget guidance with sources. Read it if this phase is new to you.
Files (cap-evolve)
  • inputs
    • INPUTS.md 13.1 KB
      # Inputs for a cap-evolve run (collected by `intake`)
      
      For every **NEEDED** input that is missing, ASK THE USER — quote the expected
      path, how to obtain it, and the alternatives. Never invent a NEEDED input.
      **RECOMMENDED** inputs have sane defaults; note any you skip in `PROJECT.md`.
      
      ## NEEDED  (the run cannot proceed without these)
      
      - **tasks dataset**: the evaluation tasks (each with an id and a gold/criterion).
        - where: `examples/<bench>/tasks.jsonl` or your benchmark's export
        - how to get it: export from your benchmark, or return them from the adapter's
          `tasks(split)`; one JSON object per line `{"id","input","target",...}`
        - options: a `.jsonl` file | a directory of json | `"adapter"` (tasks() builds them)
      
      - **target agent (RUNNER)**: the agent under test + how to run it on a task.
        - where: implemented in `.capevolve/project/adapters/adapter.py::run_target`
        - how to get it: wire your agent's entrypoint (CLI/SDK/HTTP) inside `run_target`;
          capture output + trace into a `Rollout`
        - options: in-process call | subprocess | a benchmark's own runner (`run_batch`)
        - **runner model + credentials**: which model(s) the runner uses and the env vars /
          `.env` keys it needs (e.g. `OPENAI_API_KEY`, `WATSONX_*`, `RITS_API_KEY`). For an
          OpenAI-compatible / custom endpoint (vLLM, IBM RITS, a gateway), capture the
          `api_base` + any custom auth header and pass them through the runner's LLM config
          (most benchmarks forward extra kwargs to litellm) — prefer per-call config over
          monkeypatching. ASK the user for missing credentials; never hardcode a secret.
        - **benchmark repo (if the runner IS a benchmark)**: where to get it (a local path
          or git URL) and how to install it (e.g. `pip install -e ../<bench>`). Record the
          resolved commit so the run is reproducible.
      
      - **scorer**: how a rollout becomes a reward in [0,1] + ARGUMENT-LEVEL feedback.
        - where: `adapter.py::score`
        - how to get it: exact-match / state-check / rubric for the reward.
        - **feedback is the learning signal — make it ARGUMENT-LEVEL and gold-SAFE.**
          A tool-name-only signal ("action X was wrong") is too coarse for the optimizer to
          localize a fix — it can only pattern-match to prose rules and plateaus. For EACH
          failing check, the feedback MUST localize the defect:
          - name the wrong ARGUMENT key and the **AGENT'S OWN wrong value** (NOT the gold
            value) — e.g. `"<tool>: arg <key>=<agent's value> is invalid"`;
          - name the wrong TARGET id when a write acted on the wrong entity — e.g.
            `"<tool>: called on <agent's target> but the task targets a different one"`;
          - for communication / omission misses, name the value or field the agent FAILED
            to state **when it is derivable from the agent's own state** (e.g. a computed
            total it could have summed from its own observed amounts).
        - **gold-SAFE (the hard constraint):** never read or print the gold/expected
          value. Derive everything from the AGENT'S OWN messages/tool-calls/observed state
          (and the user's own profile/db state the agent saw). Use the gold record ONLY to
          learn WHICH check/argument failed (key names are safe; values are not). If a piece
          is not safely derivable, fall back to the coarser tool-name message.
        - **deterministic:** `score()` must be deterministic on a fixed rollout (the
          `cap-evolve check` gate enforces this) — derive feedback from the rollout, do not
          call out to an LLM or use randomness.
      
      - **metric extraction / scoring source**: WHERE the objective metric lives, so
        `score()` can be implemented AND verified against the benchmark's own number.
        - where: a reference to the benchmark's scoring implementation (file/function/CLI)
          OR a precise description of how the metric is read out of one trajectory (which
          field/file in a native trace holds pass/fail or the graded reward)
        - how to get it: point at the runner's scorer (`<bench>/.../score.py`, a results
          `metrics.json` key, a rubric spec) or describe the read path ("trajectory's
          `reward` field", "the `outcome=="success"` line of the result json")
        - why: without this the intake agent cannot write a faithful `score()` — a guessed
          scorer produces a number that does not match the benchmark and the run is wasted
      
      - **trajectories path**: the DIRECTORY the runner writes its native traces/results
        to for an eval (any structure, any format — JSON, logs, per-task subdirs).
        - where: returned by the intake-authored `adapter.trajectories(split)`; the path
          itself comes from your runner config (e.g. the runner's `--output-dir`/log dir)
        - how to get it: run one eval and note where the runner dumps its traces; return
          that `Path` from `trajectories(split)` (return `None` to fall back to cap-evolve's
          own per-rollout JSON)
        - why: cap-evolve copies this directory **verbatim** into the optimizer's working
          dir as `./trajectories/`, so the optimizer reads the FULL, unmodified traces (not
          a lossy summary) when proposing edits. This is the optimizer's ground truth.
      
      - **capability artifact**: the thing being optimized (a copy is edited).
        - where: a dir/file, e.g. `policy/policy.md`, `tools.json`, a skill package dir
        - capability skill: `system-prompt | tools | mcp-tool | skill-package | …`
      
      ## RECOMMENDED  (defaults shown; override in capevolve.yaml)
      
      - **splits** — `train` / `val` / `test`.
        - default: seeded ratio split `0.5 / 0.25 / 0.25` (`split_seed`, `split_train/val/test`)
        - pin explicitly: `split_ids_file` → JSON `{"train":[],"val":[],"test":[]}`
          (use a benchmark's official split, or set all three equal to fit the whole set
          with **no holdout** — the report will flag the test number as a fit metric)
        - guidance: enough tasks to split three ways; **test is sealed** (scored once).
      
      - **num_trials** (default 1): trials per task. Use ≥3–4 for stochastic agents —
        single-trial scores are noisy and the significance gate will (correctly) reject
        marginal gains. Enables pass^k / pass@k.
      
      - **budget**: `max_iterations` (default 10), `stall` (stop after N rejects),
        `max_metric_calls` (0 = unlimited), `max_usd` (0 = unlimited; total cap over
        runner + optimizer + intake), `max_optimizer_usd` (cumulative optimizer-only cap),
        `optimizer_max_turns` (per-iteration WORK cap passed to the agent CLI, e.g.
        claude-code `--max-turns N`), and `optimizer_usd_per_iter` (per-iteration DOLLAR cap
        passed to the agent CLI and enforced by it where supported, e.g. claude-code
        `--max-budget-usd N`; optimizers without a native $ cap, e.g. ibm-bob, ignore it and
        rely on `optimizer_max_turns` / `max_optimizer_usd`). Write all of these into
        `capevolve.yaml` — the template has slots for each. Suggest the user run
        `cap-evolve estimate --spec capevolve.yaml` to preview call counts and a $ range
        before the first run.
      
      - **optimizer + model**: `optimizer_skill` is the optimizer NAME, resolved by the
        `run-optimizer` skill against `optimizers/registry.yaml` (run `run-optimizer --list`
        to see the available names); `optimizer_model` is the backend-specific model id.
      
      - **memory_skill** (default `md-files`, ask alongside algorithm/optimizer): which
        cross-iteration memory scheme the optimizer reads/writes, selected the same way as
        `algorithm_skill`/`optimizer_skill` (`harness.OptimizerContext` threads it through
        every algorithm's `run.py` via `--memory-skill`; resolved off `MEMORY_SKILLS` in
        `core/cap_evolve/harness.py`). Two options today:
        - `md-files` (default) — `harness.py`'s built-in LEDGER/JOURNAL/PROCESS/INSIGHTS/
          META_INSIGHTS/FRAMEWORK_IMPROVEMENTS scheme, append-only prose read fresh each
          iteration.
        - `wiki` — the weakness-graph format extracted from the deprecated `evograph`
          algorithm (`skills/memory/wiki/SKILL.md`): persistent weakness nodes + solution
          cards under `<run_dir>/wiki/`, rendered by the dashboard's Weakness-graph tab.
          Offer this when the user wants to inspect known weaknesses across iterations as a
          graph rather than scroll a journal, or is migrating off `evograph`.
        Note the choice in `PROJECT.md` either way.
      
      - **target_model** (default `""` = profile-agnostic): the runtime/CONSUMING LLM the
        agent reads these capabilities with — DISTINCT from `optimizer_model`, which proposes
        the edits. Give a concrete model id (e.g. `gpt-oss-120b`) or a capability tier
        (`frontier | strong | mid | weak`). cap-evolve steers the optimizer prompt and the
        capability guidance to optimize FOR this reader (a weaker reader gets more explicit
        rules, worked examples, and code enforcement; a frontier reader gets leaner prose that
        explains the *why*). ASK the user which model the agent runs at runtime; if unknown,
        leave blank and note it in `PROJECT.md`. Optionally set `target_profile_file` to a raw
        text/markdown file to override the resolved tier's built-in brief.
      
      - **runner_repo_path** (default `""`): the benchmark/runner SOURCE (a local path or
        checkout), surfaced to the optimizer as READ-ONLY context so it can consult the
        runner's tools / scoring / task structure while proposing edits. Set it when the
        runner is a benchmark repo; leave empty if there is no separate source to read.
      
      - **capability_sources** (default `[]`): extra source files — the benchmark's
        data-model / types module(s) that a selected capability's code imports — copied
        VERBATIM into the optimizer's `./guidance/sources/` so it can write correct code
        against the real types. Resolved relative to the project dir (or capability dir).
        - how to get it: look at what the seed capability's code imports (e.g. the tools
          file's `from <bench>.data_model import ...`) and list those module paths.
        - set it whenever a selected capability edits code against a shared types module;
          leave `[]` when there is no such source.
      
      - **optimizer_instructions_file** (default `optimizer/INSTRUCTIONS.md`): the
        per-iteration optimizer-prompt TEMPLATE. The scaffold already copies a generic default
        to `project/optimizer/INSTRUCTIONS.md` — the agent CUSTOMIZES that file rather than
        authoring one from scratch, and points this key at it. Three jobs, no re-authoring of
        what the template already says (depth mandate, non-overfitting guardrail, STEP-0
        reading mandate, cross-iteration file protocol):
        - keep every `{{...}}` placeholder intact — the harness fills them per iteration, and
          `implement-and-check`'s pipeline self-test fails if one is deleted;
        - **scope it to the SELECTED capabilities** — include guidance, skill references and
          editable artifacts only for the caps in `capevolve.yaml: capabilities`, so no run
          presents as editable an artifact it does not own. Each capability's own edit space
          lives in its `./guidance/<cap>/SKILL.md`; the failure taxonomy lives in
          `./guidance/diagnose/SKILL.md`; load `./guidance/<cap>/references/optimizer-playbook.md`
          for any selected capability that ships one;
        - add the benchmark facts the template cannot know: where the runner writes traces,
          what the scoring source is, which data-model files the capability's code imports.
        - **caution (issue #252):** a *relative* value here resolves project-relative under
          `cap-evolve check` but cwd-relative under `cap-evolve run`, which then silently falls
          back to the generic template. Write it absolute, or verify `run` picks up the
          customized file.
      - **gate**: `gate_mode` (**paired** recommended — per-task paired SE on the same tasks
        both sides, ~2-3x smaller than combined-SE `significant`, so real 1-task gains bank;
        also: significant|strict|threshold), `gate_k_se` (default 1.0; the
        examples use 0.2). Add `--no-regression` to forbid breaking passing tasks.
      
      - **metrics (display)**: which numbers to surface and which one GATES.
        - `metric_primary`: the single metric that decides accept/reject (= the scalar reward). Blank = use the reward directly.
        - `metrics_display` + `metric_directions`: extra SHOWN-ONLY metrics and each one's direction (`higher`|`lower`). These never affect the gate — display only.
      - **github_integration** (default `false`): if `true`, intake runs `gh auth status`; when authed, cap-evolve may mirror the algorithm's work items as issues and ship the winner as a PR (`Closes #n`). WHAT gets mirrored is algorithm-specific — the chosen `algorithm_skill` defines it (e.g. evograph mirrors *weaknesses*; a candidate-based algorithm might mirror candidates/iterations). GitHub is NEVER the source of truth — the run dir is. If unauthed, intake offers `gh auth login` or skip.
      - **orchestration_mode** (default `deterministic`): `deterministic` = cap-evolve sequences the loop (code-enforced honesty). `agent` = the coding agent drives the loop via cap-evolve primitives and seals with the finalize phase script (`skills/phases/finalize/scripts/run.py`). Agent mode also uses `stop_condition`.
      - **stop_condition** (default empty): agent-mode free-text halt rule, re-read each round. Deterministic mode ignores it and uses the budget knobs.
      
      - **baseline traces** (optional): prior rollouts to seed diagnosis. Default: none
        (the baseline phase produces them on the first val eval).
      
      ## Notes
      - The intake script scaffolds `.capevolve/project/` from the template; fill the
        adapter + `capevolve.yaml`, then run `cap-evolve check` (the hard gate).
      - Paths are relative to the project working dir unless absolute.
      
  • references
    • concepts.md 5.5 KB
      # Concepts — intake and the inputs contract
      
      > intake turns "make this agent better at X" into a runnable project. The hard
      > part is not scaffolding files; it is refusing to proceed on a fabricated input.
      > This note explains the contract `inputs/INPUTS.md` encodes and why it is shaped
      > that way.
      
      ## The adapter is the whole interface
      
      Everything cap-evolve measures flows through **3 required methods** the user implements in
      `.capevolve/project/adapters/adapter.py` (plus defaulted hooks). intake's job is to make
      sure each one *can* be implemented from real inputs:
      
      | method                             | question it answers                              | NEEDED input behind it |
      |------------------------------------|--------------------------------------------------|------------------------|
      | `tasks(split)`                     | what problems do we evaluate on?                 | tasks dataset          |
      | `run_target(task, ctx, *, seed=0)` | how do we run the agent under test?              | target agent / runner  |
      | `score(task, rollout)`             | how does a rollout become a reward in [0,1] + feedback? | scorer          |
      
      The defaulted hooks `materialize(candidate_dir, edits=None)` / `live(candidate_dir)` /
      `apply(candidate_dir, edits=None)` answer "how is a proposed edit materialized and made
      live?". Their defaults already work for file-shaped capabilities, but they still need a
      real **capability artifact** to write into — so that is a NEEDED input too.
      
      If any of these cannot be filled from a real input, the optimization cannot
      produce a meaningful number. That is why the tasks dataset, the runner, the scorer and
      the capability artifact are NEEDED, not RECOMMENDED (`inputs/INPUTS.md` lists two more
      NEEDED inputs — metric-extraction source and trajectories path — which back the
      same three methods rather than adding new ones).
      
      ## NEEDED vs RECOMMENDED — and why the split exists
      
      **NEEDED** = the run is meaningless without it, and there is no honest default.
      The correct action when one is missing is to **ask the user** — quoting the
      expected path, the command that produces it, and the alternatives — then wait.
      
      **RECOMMENDED** = a defensible default exists. You may proceed on the default,
      but you must **log the choice in `PROJECT.md`** so its honesty cost is visible.
      
      This is the central anti-pattern guard. "Auto-optimize my agent" tools fail when
      they treat a missing input as a gap to backfill: they synthesize a plausible
      dataset or a lenient scorer, run green, and report a number that measures
      nothing. Encoding inputs as a contract makes the only legitimate
      proceed-without-input path an *explicit, recorded default* — never silent
      fabrication. The model has good judgment; the contract exists so that judgment is
      applied to *which question to ask*, not to *what to invent*.
      
      ### Feedback must not leak the gold
      
      A subtle NEEDED-input rule: the scorer's textual `feedback` is what `diagnose`
      turns into the learning signal. It must describe *why* a rollout failed without
      quoting the gold answer. A scorer that echoes the target answer into feedback
      turns the optimizer into a memorizer of the eval set — the agent "improves" on
      val/test by being told the answers, and the held-out number becomes a lie. Keep
      feedback general (what was wrong, what class of error), never the solution.
      
      ## Splits, trials, and budget (RECOMMENDED, but consequential)
      
      - **Splits** — `train` / `val` / `test`. The default is a seeded ratio split
        (0.5 / 0.25 / 0.25). You may pin an official benchmark split via
        `split_ids_file`. You may also set all three equal to fit the whole set with
        **no holdout** — but then there is no honest test number, and the report must
        flag it as a *fit* metric. **train** is what the optimizer edits against;
        **val** is what the gate accepts on; **test** is sealed and scored once at
        finalize. This train/val/test discipline is the same one that keeps supervised
        ML honest — the test set must never inform a decision made during search.
      
      - **num_trials** — trials per task. Default 1, but stochastic agents need ≥3–4:
        a single trial hides variance, so the significance gate cannot tell a real gain
        from noise, and multi-trial reliability metrics (pass^k / pass@k) are
        undefined. tau-bench introduced pass^k precisely because single-run success
        overstates how dependable an agent is.
      
      - **budget** — `max_iterations`, `stall` (stop after N consecutive rejects),
        `max_metric_calls`, `max_usd` (total cap: runner + optimizer + intake),
        `max_optimizer_usd` (optimizer-only cap), `optimizer_max_turns` (per-iteration
        agent-CLI cap). All are hard stops at 0=unlimited; `cap-evolve estimate` previews
        the spend before a run. A budget too small to plausibly find a gain is itself a
        misconfiguration worth flagging at intake.
      
      ## Sources
      - τ-bench (Yao, Shinn, Razavi, Narasimhan, 2024) — pass^k as a *reliability*
        metric; why single-run success overstates dependability:
        https://arxiv.org/abs/2406.12045
      - Koehn, "Statistical Significance Tests for Machine Translation Evaluation"
        (EMNLP 2004) — bootstrap resampling for deciding whether a score difference is
        real, the statistical backbone of the gate: https://aclanthology.org/W04-3250/
      - GEPA: Reflective Prompt Evolution (Agrawal et al., 2025) — natural-language
        feedback as the learning signal the scorer must produce honestly:
        https://arxiv.org/abs/2507.19457
      - Hastie, Tibshirani, Friedman, *The Elements of Statistical Learning* — the
        train/validation/test protocol and why the test set must stay sealed:
        https://hastie.su.domains/ElemStatLearn/
      
  • scripts
    • abstract.py 163 B
      """The 'intake' phase composes the project adapter + shared harness; it declares no
      abstract methods of its own. check.py verifies the wiring instead of stubs."""
      
    • check.py 1.6 KB
      """Contract: intake scaffolds a project from the template and mines reusable
      artifacts (task files / capability files) from the working dir before scaffolding.
      """
      
      from __future__ import annotations
      
      import sys
      import tempfile
      from pathlib import Path
      
      import _bootstrap  # noqa: F401
      
      from cap_evolve.skillcheck import Checker, import_run, quiet
      
      
      def main() -> int:
          c = Checker("intake")
          run = import_run()
          c.require_main(run)
      
          with tempfile.TemporaryDirectory() as d:
              wd = Path(d)
              (wd / "tasks.jsonl").write_text('{"id": "1"}\n', encoding="utf-8")
              cap = wd / "seed" / "prompt.txt"
              cap.parent.mkdir(parents=True)
              cap.write_text("you are an agent", encoding="utf-8")
      
              found = run.mine_artifacts(wd)
              c.check("tasks.jsonl" in found["task_files"],
                      f"did not mine the task file: {found}",
                      note="mines existing task files")
              c.check(any(p.endswith("prompt.txt") for p in found["capability_artifacts"]),
                      f"did not mine the capability artifact: {found}",
                      note="mines existing capability artifacts")
      
              # scaffold writes the adapter stub + spec under .capevolve/project
              with quiet():
                  rc = run.main(["--base", str(wd / ".capevolve"), "--workdir", str(wd)])
              c.check(rc == 0, "intake scaffold returned nonzero")
              proj = wd / ".capevolve" / "project"
              c.check((proj / "adapters" / "adapter.py").exists(),
                      "scaffold missing adapters/adapter.py",
                      note="scaffolds the adapter stub + spec")
      
          return c.emit()
      
      
      if __name__ == "__main__":
          sys.exit(main())
      
    • run.py 3.9 KB
      """intake — Phase 1: mine existing artifacts, then scaffold the cap-evolve project.
      
      What the SCRIPT does (this file):
        1. **capture-intent / mine first** — scan the working dir for artifacts a run can
           reuse (task files, an existing capability/prompt/tools surface, an existing
           adapter, a benchmark runtime) so the agent doesn't re-author what already
           exists. The findings are reported under ``discovered``.
        2. **scaffold** — copy ``templates/project`` into ``.capevolve/project`` (adapter
           stub, inputs/, capevolve.yaml, PROJECT.md).
      
      What the OPTIMIZER AGENT does (driven by SKILL.md, not this script): decide the
      capability/optimizer/algorithm, **implement the 3 required adapter methods**, and fill the
      spec. What the USER does: supply NEEDED inputs the agent cannot infer.
      
      Advancing past intake is gated by ``implement-and-check`` (runs ``cap-evolve
      check`` and refuses to proceed until the adapter contract is green).
      """
      
      from __future__ import annotations
      
      import argparse
      import json
      import shutil
      import sys
      from pathlib import Path
      
      # Filename hints used to mine reusable artifacts from the working directory.
      _TASK_HINTS = ("tasks.jsonl", "tasks.json", "dataset.jsonl", "data.jsonl")
      _CAP_HINTS = ("prompt.txt", "policy.md", "tools.json", "SKILL.md", "system_prompt.txt")
      
      
      def find_templates() -> Path:
          here = Path(__file__).resolve()
          for parent in here.parents:
              t = parent / "templates" / "project"
              if t.is_dir():
                  return t
          raise FileNotFoundError("templates/project not found; run from the repo or set --templates")
      
      
      def mine_artifacts(workdir: Path) -> dict:
          """Best-effort scan for things a run can reuse (mine existing work first)."""
          workdir = Path(workdir)
          skip = {".git", "__pycache__", ".capevolve", "node_modules", ".venv"}
      
          def _walk(pred):
              out = []
              for p in workdir.rglob("*"):
                  if any(part in skip for part in p.parts):
                      continue
                  if p.is_file() and pred(p):
                      out.append(str(p.relative_to(workdir)))
              return sorted(out)[:50]
      
          return {
              "task_files": _walk(lambda p: p.name in _TASK_HINTS),
              "capability_artifacts": _walk(lambda p: p.name in _CAP_HINTS),
              "existing_adapters": _walk(lambda p: p.name == "adapter.py"
                                         and ".capevolve" not in p.parts),
          }
      
      
      def main(argv=None) -> int:
          p = argparse.ArgumentParser(prog="intake")
          p.add_argument("--base", default=".capevolve")
          p.add_argument("--workdir", default=".", help="dir to mine for reusable artifacts")
          p.add_argument("--templates", default=None)
          p.add_argument("--force", action="store_true")
          args = p.parse_args(argv)
      
          discovered = mine_artifacts(Path(args.workdir))
      
          tmpl = Path(args.templates) if args.templates else find_templates()
          project = Path(args.base) / "project"
          if project.exists() and not args.force:
              print(json.dumps({"project": str(project), "status": "exists",
                                "discovered": discovered,
                                "note": "use --force to overwrite"}, indent=2))
              return 0
          if project.exists():
              shutil.rmtree(project)
          shutil.copytree(tmpl, project)
      
          created = sorted(str(q.relative_to(project)) for q in project.rglob("*") if q.is_file())
          print(json.dumps({
              "project": str(project),
              "status": "scaffolded",
              "discovered": discovered,
              "created": created,
              "next": [
                  "reuse the discovered artifacts where possible (don't re-author them)",
                  "implement the 3 required methods in adapters/adapter.py",
                  "fill capevolve.yaml (capability / optimizer / algorithm / budget)",
                  "resolve NEEDED inputs (ask the user for any that are missing)",
                  "run: cap-evolve check " + str(project) + "  (implement-and-check gates this)",
              ],
          }, indent=2))
          return 0
      
      
      if __name__ == "__main__":
          sys.exit(main())
      
    • _bootstrap.py 3.6 KB
      """Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.
      
      Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
      ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
      only has to find that package, which means a minimal upward walk for ``core/`` —
      the single bit of bootstrapping that genuinely must run before cap_evolve is
      importable. Everything else delegates.
      """
      
      from __future__ import annotations
      
      import os
      import sys
      from pathlib import Path
      
      
      def _seed_path() -> None:
          """Minimal: put a dir containing the cap_evolve package on sys.path.
      
          ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
          *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
          both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
          it imports the checkout you are standing in. Deferring to the ambient package here made
          an explicit override unreachable, and the symptom was a stale core silently answering
          for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
          predates that module). An explicit env var wins.
          """
          env = os.environ.get("CAPEVOLVE_CORE")
          want = Path(env).resolve() if env else None
          if want and (want / "cap_evolve" / "__init__.py").exists():
              loaded = sys.modules.get("cap_evolve")
              already = getattr(loaded, "__file__", None)
              if already and Path(already).resolve().parent.parent == want:
                  return                      # right checkout already imported: touch nothing
              p = str(want)
              if p in sys.path:
                  sys.path.remove(p)
              sys.path.insert(0, p)
              if loaded is not None:
                  # Evicting a module makes a re-import yield a DIFFERENT object, so anything
                  # already holding a reference fails an `is` check. Only ever do it when the
                  # loaded package really is the wrong checkout — otherwise this "fix" becomes
                  # the bug (it broke two identity assertions in core/tests exactly once).
                  for name in [m for m in sys.modules
                               if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                      sys.modules.pop(name, None)
              for finder in list(sys.meta_path):
                  if "cap_evolve" in getattr(finder, "MAPPING", {}):
                      sys.meta_path.remove(finder)
              return
          # A checkout's own core outranks an ambient install. Without this, a skill script run
          # from checkout X silently executed against checkout Y's cap_evolve (an editable install
          # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
          # missing modules — or, worse, a green result measured against the wrong tree.
          here = Path(__file__).resolve()
          own = next((p / "core" for p in here.parents
                      if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
          if own is not None:
              os.environ.setdefault("CAPEVOLVE_CORE", str(own))
              return _seed_path()
          try:
              import cap_evolve  # noqa: F401
              return
          except Exception:
              pass
          cands = []
          for parent in here.parents:
              cands.append(parent / "core")
              cands.append(parent)
          for c in cands:
              if (c / "cap_evolve" / "__init__.py").exists():
                  p = str(c)
                  if p not in sys.path:
                      sys.path.insert(0, p)
                  return
      
      
      _seed_path()
      from cap_evolve._bootstrap import ensure_core  # noqa: E402
      
      # Anchor the upward walk at THIS skill script's location (not the core module's).
      ensure_core(Path(__file__).resolve())
      
  • meta.yaml 360 B
    component: phase
    name: intake
    summary: Phase-1 input collection: scaffold .capevolve/project, gather inputs, ask user for missing NEEDED ones.
    entry: scripts/run.py
    abstract: scripts/abstract.py
    check: scripts/check.py
    inputs: inputs/INPUTS.md
    needs: []
    provides: [project, tasks]
    compatible_with:
      capabilities: ["*"]
      optimizers: ["*"]
      algorithms: ["*"]
    
  • SKILL.md 9.2 KB
    ---
    name: intake
    description: Starts a cap-evolve optimization run. Interviews the user to decide what capability to optimize, which runner/optimizer/algorithm to use, and where the tasks and the scoring source live, then scaffolds .capevolve/project/ (adapter stub, capevolve.yaml, PROJECT.md). Use when someone asks to optimize or improve an agent capability against an eval and no project exists yet — "set up a run", "start optimizing X", "make X score higher on this benchmark". This is Phase 1 of the pipeline. For every NEEDED input that is missing it asks the user — quoting the expected path, how to retrieve it, and the alternatives — instead of fabricating it. Not for a project that already exists: when .capevolve/project/ is present, go to implement-and-check or the algorithm loop instead.
    component: phase
    argument-hint: "--base .capevolve --workdir ."
    allowed-tools: Read, Write, Edit, Bash
    provides: [project, tasks]
    needs: []
    sources: []
    ---
    
    # intake — collect inputs, scaffold the project
    
    Turn a vague wish ("make this agent better at X") into a runnable project: a filled
    `capevolve.yaml`, an adapter ready to implement, and **every NEEDED input resolved
    before any budget is spent**. Intake is cheap; an unresolved input found three phases
    later is a wasted run and a meaningless number.
    
    ## Ask, never fabricate — the core discipline of this phase
    
    `inputs/INPUTS.md` classifies every input **NEEDED** or **RECOMMENDED**. For each
    **NEEDED** input that is not already present, do not proceed:
    - **Interactive / chat mode — ASK THE USER and wait.** Quote all three, they are in
      `INPUTS.md` per input: (a) the exact path the input is expected at, (b) the command
      or option that produces it, (c) the alternatives. Say what breaks without it.
    - **Non-interactive** (`cap-evolve run` / the `orchestrate` skill, nobody to ask) —
      write `BLOCKED: <input> — why it is needed — how to provide it` into `PROJECT.md`
      and exit non-zero. A blocked-but-honest stop is correct; a green run on a guessed
      input is not.
    
    A fabricated dataset, scorer, trajectories path or gold answer does not unblock the run
    — it produces a number that measures nothing and hides that fact. A missing tasks file
    is a *question for the user*, not a gap for you to paper over.
    
    **RECOMMENDED** inputs may take their default, but log every default in `PROJECT.md`
    with its honesty cost (e.g. "num_trials=1 — single-trial scores, so the significance
    gate will correctly reject marginal gains"), so the cost is visible at report time.
    
    ## Step 0 — mine, then inspect, then ask once
    1. **Mine the conversation first.** Anything the user already said is an answer you
       must not re-ask — "optimize my airline policy on the flight-change tasks" already
       fixed the capability, the artifact and the task subset. Harvest that, and any
       correction the user made, before asking anything.
    2. **Run the miner.** `python scripts/run.py --base .capevolve --workdir <repo-root>`
       scaffolds and returns `discovered` — task files, capability artifacts, existing
       adapters. Reuse what it found; never re-author it.
    3. **Inspect what `discovered` leaves open**: the entrypoint, how one eval runs, where
       traces and scores land, candidate metrics, a natural train/val/test split, cost caps.
       Run `gh auth status`. Fan subagents out over the benchmark repo (entrypoint, scorer,
       trace dir, task schema) *while* the user answers instead of serializing — come
       prepared, so the user carries as little of the research as possible.
    4. **Then ask the FEWEST questions, as ONE numbered batch**, each with the detected
       value pre-filled as a default plus a free-text escape — including the ones only a
       human can answer: which metric gates accept/reject and each shown metric's
       direction, GitHub mirroring, deterministic vs agent orchestration (plus
       `stop_condition` in agent mode), splits, trials, budget, and `memory_skill`
       (default `md-files`; offer `wiki` — the weakness-graph format, see
       `inputs/INPUTS.md` — when the user wants weaknesses tracked as a persistent graph
       rather than an append-only journal). `inputs/INPUTS.md` → RECOMMENDED is the
       authority on each key; SKILL.md only fixes *when* to ask. Define jargon in a
       clause before using it ("pass^k — how often it succeeds on all k tries"); the user
       may be a domain expert, not an ML one.
    5. **Confirm before scaffolding.** Echo the resolved spec back as one block —
       capability, optimizer, algorithm, dataset, splits, budget, every RECOMMENDED input
       you are defaulting — and get a yes. A misread is cheapest to fix here.
    
    ## What it does
    The interview settles the capability skill (*what* is optimized), the optimizer (*which*
    coding agent proposes edits), the algorithm (*the search loop*), dataset, splits, budget.
    1. **Scaffold** `.capevolve/project/` via `scripts/run.py`: adapter stub, `inputs/`,
       `capevolve.yaml`, `PROJECT.md`, and `optimizer/INSTRUCTIONS.md`. The whole
       `templates/project/` tree is copytree'd verbatim — confirm the files landed.
    2. **Resolve inputs** per `inputs/INPUTS.md`, honoring the ask-never-fabricate rule.
    3. **Record** the resolved trajectories path and the scoring source in `PROJECT.md`, so
       `implement-and-check` wires `trajectories()` and `score()` against real inputs rather
       than guesses. `inputs/INPUTS.md` → **scorer** specifies exactly what the feedback must
       be (argument-level, gold-safe) and that `score()` must be deterministic — follow it
       literally, that feedback is the learning signal. Note in `PROJECT.md` if you
       deliberately return `None` from `trajectories()` (cap-evolve then falls back to its
       own per-rollout JSON).
    4. **Customize the scaffolded `optimizer/INSTRUCTIONS.md`** for THIS benchmark. The
       shipped template already carries the depth mandate, the non-overfitting guardrail,
       the STEP-0 reading mandate and the cross-iteration file protocol — do not
       re-author any of them. Your three jobs:
       a. keep every `{{...}}` placeholder intact (`{{FOCUS_SUMMARY}}`, `{{FAILURES}}`,
          `{{CAP_BRIEF}}`, `{{ALGO_BRIEF}}`, `{{BENCH_REPO}}` — the harness fills them per
          iteration; `implement-and-check`'s pipeline self-test fails if one is deleted,
          and rendering must leave no `{{` behind);
       b. **scope it to the selected capabilities** — delete the sections for capabilities
          not listed in `capevolve.yaml: capabilities`, so a run never presents an
          artifact as editable that this run does not own. Point the optimizer at
          `./guidance/<cap>/SKILL.md` for each selected capability's own edit space, and
          at `./guidance/diagnose/SKILL.md` for the failure taxonomy — both are
          materialized into its working dir. When a selected capability ships one, also
          point at `./guidance/<cap>/references/optimizer-playbook.md`;
       c. add the benchmark-specific facts the template cannot know: where the runner
          writes traces, what the scoring source is, which data-model files the
          capability's code imports.
    5. **Set the spec keys** in `capevolve.yaml` — `runner_repo_path`,
       `optimizer_instructions_file`, `capability_sources` (the module(s) a selected
       capability's code imports, copied into the optimizer's `./guidance/sources/`),
       `target_model`. `inputs/INPUTS.md` defines each one.
       - **Caution (issue #252):** a *relative* `optimizer_instructions_file` resolves
         project-relative under `check` but cwd-relative under `run`, which then silently
         falls back to the generic template. Write it absolute, or verify `run` actually
         picks up the customized file — intake authors it, so intake is the cheapest place
         to get it right.
    
    ## How to run
    ```
    python scripts/run.py --base .capevolve --workdir .   # mine, then scaffold
    ```
    The script is purely mechanical; the *judgment* — interviewing, choosing components, the
    ask-if-missing loop — is yours. Then implement `adapters/adapter.py`, fill
    `capevolve.yaml`, and hand off to `implement-and-check`: together the two phases are the
    *full integration* (scaffold → the 3 required adapter methods → `cap-evolve check` green)
    and no budget is spent until that gate passes.
    
    > **Onboarding transcript (one example):** `examples/tau2_airline/setup.sh` clones and
    > installs a benchmark and wires the adapter until `cap-evolve check` is green, and its
    > `run.sh` runs the optimization. Read it only when onboarding a benchmark you have not
    > integrated before.
    
    ## Good vs bad intake
    - **Good:** every NEEDED input resolved to a real path or `"adapter"`; splits and budget
      chosen deliberately; each defaulted RECOMMENDED input logged; spec confirmed by the user.
    - **Bad:** a synthesized tasks file that "looked plausible"; a scorer that leaks the gold
      answer into feedback; test == train with no note; a budget too small to find a gain;
      the run proceeded past a missing NEEDED input "to keep moving".
    
    ## References
    - `inputs/INPUTS.md` — the binding contract: every input classified NEEDED vs RECOMMENDED
      with the path / how-to-retrieve / alternatives you must quote, plus the meaning and
      default of every spec key. Read it during the interview.
    - `references/concepts.md` — why the contract is shaped this way, the 3 required adapter
      methods, split/trial/budget guidance with sources. Read it if this phase is new to you.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related