Claude Skill

baseline

Establish the starting point. Use after implement-and-check and before any algorithm. Creates the run directory, freezes the seeded train/val/test split (written once), scores the unmodified seed capability on val, and records it as the candidate every algorithm must beat. Report

LLM Mart · 0 points · 12 views 0 listing impressions 0 install-command copies
Download skillberry-ai-cap-evolve-skills_phases_baseline-0fc6924.zip · 13 KB

Install

skills CLI npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/baseline
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
Git git clone https://github.com/skillberry-ai/cap-evolve.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

baseline — freeze splits, score the seed

baseline is the first phase that touches data, so it owns the run's one irreversible decision: the split. It writes splits.json once (seeded), scores the unmodified seed capability on val, and records that score as the bar every algorithm must beat.

Run implement-and-check first. baseline re-runs that check itself and exits non-zero before creating a run dir if it is red — a split frozen against a broken adapter poisons every number measured afterwards.

Why it matters

  • Fair comparison point. Every algorithm hill-climbs against the baseline val score; a candidate that does not beat it is not progress.
  • Headroom. The printed JSON carries headroom (1 - val) and headroom_verdict: saturated means the seed is already at the ceiling and further iterations buy noise — stop; floor (val at 0) usually means a mis-wired adapter rather than a hard task — re-check before spending budget; ok means proceed. The same verdict is logged as a headroom event so the orchestrator can stop on it with no human reading the number.

Splitting choices

  • Seeded ratio split (default 0.5 / 0.25 / 0.25): deterministic given --seed. Reproducible runs partition identically.
  • Pinned split (--split-ids): a JSON {train,val,test} of ids — use a benchmark's official split, or set all three equal to fit the whole set with no holdout (the test number is then a fit metric, not a held-out result; the run dir records a splits_warning saying so).
  • A ratio split that leaves val or test empty is refused — the gate would have nothing to decide on and the sealed test number would cover no tasks. Below 5 val tasks baseline warns: the gate's bar is optimistic at that n, and a candidate that improves exactly one val task cannot reliably clear it at all (issue #351), so size val with the decisions it has to make in mind.

Reusing a prior baseline (--reuse-baseline PRIOR_RUN_DIR)

Re-scoring the seed is wasteful when the split + seed are unchanged. --reuse-baseline <prior run_* dir> (spec key reuse_baseline) copies that run's splits.json, baseline.json, seed snapshot and seed val and train rollouts into the fresh run dir and skips the baseline eval; the copied test_used flag is reset so this run can still finalize on test exactly once. The prior run's sealed seed test score is carried over as a result (reused_seed_test.json, never as test rollouts), and finalize uses it instead of re-scoring the seed while the seed bytes are unchanged. --resume is the same-run variant: reopen an existing run dir, skip the eval when baseline.json is already there. Budget flags (--max-iterations, --stall, --max-usd, …) are accepted here because the run dir owns the budget and later phases read it from there.

Runs standalone (/cap-evolve:baseline) or headlessly via cap-evolve run; same scripts/run.py either way.

How to run

python scripts/run.py --base .capevolve --project .capevolve/project \
    --capability seed_capability --seed 0 --ratios 0.5,0.25,0.25 \
    --max-iterations 10 --stall 2

Prints the run-dir path (used by the algorithm + finalize), the split sizes, the baseline val and the headroom verdict. Use --n-trials ≥ 3 for stochastic targets so the baseline carries a real stderr rather than 0.

The one failure mode nothing later can repair is re-splitting after this phase: a task migrating out of test leaks the held-out set, and every later number — including the sealed test score — becomes unfalsifiable.

References

  • references/concepts.md — why the split is frozen once and seeded, how to read the headroom verdict, and why no-holdout runs are fit metrics, with sources.
Files (cap-evolve)
  • references
    • concepts.md 3.4 KB
      Loading…
  • scripts
    • abstract.py 165 B
      Loading…
    • check.py 5.3 KB
      Loading…
    • run.py 14.5 KB
      Loading…
    • _bootstrap.py 3.6 KB
      Loading…
  • meta.yaml 352 B
    Loading…
  • SKILL.md 4.4 KB
    ---
    name: baseline
    description: Establish the starting point. Use after implement-and-check and before any algorithm. Creates the run directory, freezes the seeded train/val/test split (written once), scores the unmodified seed capability on val, and records it as the candidate every algorithm must beat. Reports the remaining headroom so a saturated seed stops the run before it spends budget.
    component: phase
    argument-hint: "--base .capevolve --project DIR --capability DIR [--seed N] [--ratios a,b,c] [--n-trials N] [--split-ids FILE] [--resume] [--reuse-baseline DIR]"
    allowed-tools: Read, Write, Bash
    provides: [splits, baseline, candidate, scores, traces]
    needs: [project, tasks]
    sources: []
    ---
    
    # baseline — freeze splits, score the seed
    
    baseline is the first phase that touches data, so it owns the run's one
    irreversible decision: **the split**. It writes `splits.json` once (seeded), scores
    the *unmodified* seed capability on val, and records that score as the bar every
    algorithm must beat.
    
    Run `implement-and-check` first. baseline re-runs that check itself and exits
    non-zero before creating a run dir if it is red — a split frozen against a broken
    adapter poisons every number measured afterwards.
    
    ## Why it matters
    - **Fair comparison point.** Every algorithm hill-climbs *against* the baseline
      val score; a candidate that does not beat it is not progress.
    - **Headroom.** The printed JSON carries `headroom` (`1 - val`) and
      `headroom_verdict`: `saturated` means the seed is already at the ceiling and
      further iterations buy noise — stop; `floor` (val at 0) usually means a
      mis-wired adapter rather than a hard task — re-check before spending budget;
      `ok` means proceed. The same verdict is logged as a `headroom` event so the
      orchestrator can stop on it with no human reading the number.
    
    ## Splitting choices
    - **Seeded ratio split** (default `0.5 / 0.25 / 0.25`): deterministic given
      `--seed`. Reproducible runs partition identically.
    - **Pinned split** (`--split-ids`): a JSON `{train,val,test}` of ids — use a
      benchmark's official split, or set all three equal to fit the whole set with
      **no holdout** (the test number is then a *fit* metric, not a held-out result;
      the run dir records a `splits_warning` saying so).
    - A ratio split that leaves val or test empty is **refused** — the gate would
      have nothing to decide on and the sealed test number would cover no tasks.
      Below 5 val tasks baseline warns: the gate's bar is optimistic at that `n`, and
      a candidate that improves exactly one val task cannot reliably clear it at all
      (issue #351), so size val with the decisions it has to make in mind.
    
    ## Reusing a prior baseline (`--reuse-baseline PRIOR_RUN_DIR`)
    Re-scoring the seed is wasteful when the split + seed are unchanged.
    `--reuse-baseline <prior run_* dir>` (spec key `reuse_baseline`) copies that run's
    `splits.json`, `baseline.json`, seed snapshot and seed val and train rollouts into the
    fresh run dir and skips the baseline eval; the copied `test_used` flag is reset so this
    run can still finalize on test exactly once. The prior run's sealed seed **test** score
    is carried over as a result (`reused_seed_test.json`, never as test rollouts), and
    finalize uses it instead of re-scoring the seed while the seed bytes are unchanged. `--resume` is the same-run variant:
    reopen an existing run dir, skip the eval when `baseline.json` is already there.
    Budget flags (`--max-iterations`, `--stall`, `--max-usd`, …) are accepted here
    because the run dir owns the budget and later phases read it from there.
    
    Runs standalone (`/cap-evolve:baseline`) or headlessly via `cap-evolve run`; same
    `scripts/run.py` either way.
    
    ## How to run
    ```
    python scripts/run.py --base .capevolve --project .capevolve/project \
        --capability seed_capability --seed 0 --ratios 0.5,0.25,0.25 \
        --max-iterations 10 --stall 2
    ```
    Prints the run-dir path (used by the algorithm + finalize), the split sizes, the
    baseline val and the headroom verdict. Use `--n-trials ≥ 3` for stochastic
    targets so the baseline carries a real `stderr` rather than 0.
    
    The one failure mode nothing later can repair is re-splitting after this phase: a
    task migrating out of test leaks the held-out set, and every later number —
    including the sealed test score — becomes unfalsifiable.
    
    ## References
    - `references/concepts.md` — why the split is frozen once and seeded, how to read
      the headroom verdict, and why no-holdout runs are fit metrics, with sources.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.