evograph
Deprecated agent-mode algorithm (evo-graph port): a weakness-graph search that dispatched one solver agent per failure cluster and reverted a whole round on regression. Do not start new runs with it — its per-weakness fan-out is already `agent-optimize`'s sibling fan-out, done be
Install
npx skills add https://github.com/skillberry-ai/cap-evolve/tree/main/skills/algorithms/evograph
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install skillberry-ai-cap-evolve@llmmart
git clone https://github.com/skillberry-ai/cap-evolve.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole skillberry-ai/cap-evolve collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
evograph — DEPRECATED
Do not select algorithm_skill: evograph for a new run. Use agent-optimize (agent mode) or
hill-climb / gepa / skillopt (deterministic) with memory_skill: wiki if you want THIS run's
weakness-graph format. This file stays so an existing evograph run dir is still readable.
Why it is deprecated
evograph advertised one distinctive capability — a collaborative weakness graph with one solver agent per weakness, merged into a shared candidate each round. Measured against its four siblings, that capability is not distinctive and the part that was distinctive was a defect:
- The fan-out already exists, gated properly.
agent-optimizefans out N sibling candidates from the same parent, one diagnosed failure cluster each, every sibling in its own working copy (a git worktree when the capability is in git), gated one at a time with a re-gate after each accept so several fixes accumulate into one lineage honestly. That is evograph's round, minus the flaws below. The clustering itself isphases/diagnose's job in both cases. - Acceptance was never held out. evograph kept a merge on a raw delta over a frozen 3-task
subset of train, self-reported by the solver subagent that made the edit — no val split, no
standard error, no
Δ > k·SE. Betweenbaselineandfinalizean evograph run took no held-out measurement at all, so the sealed test number was the first honest signal anyone saw. Whole-round revert existed only as a one-round-late substitute for the gate it lacked; wire the real gate and there is nothing left for it to catch. - What remains unique is an output format, not a search strategy. The run-dir
wiki/is genuinely useful, but the dashboard renders the Weakness-graph tab fromwiki/presence alone, for any algorithm that writes the format (core/cap_evolve/dashboard.py). An output contract does not earn a second agent-mode algorithm that users must choose between — it has since moved toskills/memory/wiki/SKILL.md, a standalonememory_skillany algorithm can select (#400, #404), so it no longer needs evograph to stay alive.
There is no deterministic engine
There never was one, and scripts/run.py is a tombstone, not a stub: invoked deterministically it
exits 2 with an agent-mode only payload rather than faking a loop. So evograph is also the one
algorithm that could not be routed through the shared per-iteration record — see
the hill-climb skill's references/run-step.md, which owns the shared iteration mechanics
(parent selection, val gate, commit, iteration record) every other algorithm routes through. Read
it if you are reconstructing what an evograph round should have done.
Reading an existing evograph run
The run dir is authoritative. <run_dir>/wiki/ holds the weakness nodes, solution cards and
per-round results; <run_dir>/runs/round-<N>/agents/<slug>.log holds solver progress. The formats
are in references/dashboard.md — load it if you need to write or parse
that wiki. Treat any per-weakness "kept / new record" number in it as a train-subset self-report,
not a gated result; only finalize's sealed test number and any gate decision recorded in
events.jsonl are honest.
scripts/now.py is the one-clock timestamp stamper those formats require; it is correct and still
used by anything writing the wiki.
Removal is a separate decision
Deprecation is reversible; removal is not. The wiki format contract now has a real owner
(skills/memory/wiki/SKILL.md), so what's left to decide (maintainer): stop dashboard.py
inferring algorithm = "evograph" from wiki/ presence alone (any memory_skill: wiki run now
writes it too), then delete this directory.
References
- skills/memory/wiki/SKILL.md — the wiki format contract's current
home: weakness-node/solution-card schemas, generalized for any algorithm via
memory_skill: wiki. - references/dashboard.md — the same file formats, kept here for reading an existing evograph run dir's branch/round-revert specifics the generalized skill dropped.
- references/clustering.md — weakness-node schema and the
affected_tasksfreeze rule, kept for reading historical run dirs. - references/graph.md — solution-card schema, branch layout, whole-round revert, kept for the same reason.
Files (cap-evolve)
-
references
-
clustering.md 5.6 KB
# Building the weakness graph (clustering) The weakness wiki is the team's **shared state**. Builders write it **directly** — there are no intermediate draft files. This is the "parallel cartography on a shared graph" idea: many agents grow one graph at once. ## The flow 1. After the round's eval, the failed tasks (score < max) are the starting point. **Distribute them across however many builders makes sense** for the failure volume and the harness — your call. Just split **by whole task id** (never split one task's trajectory across builders) so each builder sees full context per task. **Don't look at failures only** — builders should also read the **successful** trajectories (if bench has trials-per-task a task often passes on some trials and fails on others); contrasting pass vs fail for the same task is what reveals the cause. 2. Each builder reads its slice + the existing weaknesses, then for each failure that hurts the primary metric: - **Search** `wiki/weaknesses/` for a matching pattern. - **Match** → append/extend that weakness file (broaden "What fails", add a trajectory excerpt and a reference). Add tasks **only if it's that weakness's discovery round** (freeze rule below). - **No match** → create a new `wiki/weaknesses/<slug>.md` with `status: open`. 3. The lead does a light **dedup pass — only over weaknesses newly coined this round**: if two builders created near-duplicate slugs for the same pattern, merge them (pick a canonical slug, fold the other's tasks/excerpts in). Leave established weaknesses (those already carrying history/solutions/frozen tasks) alone. A weakness is *anything* that keeps the primary metric below its max — including **inconsistency**: a task that sometimes passes and sometimes doesn't is a real weakness (a consistency problem), not a pass. ## Freeze rule `affected_tasks` may grow **only in the weakness's discovery round**. After that, it's frozen: solutions have already been scored against that exact task set, and changing it would invalidate those comparisons and the "new record" signal. In later rounds you may only change `status`: `open`/`completed`/`reverted` → `in-progress` when re-attacked → `completed` (shipped an improvement, stopped) or `solved` (tasks now all ~perfect); a recurring `solved` weakness can go back to `in-progress`; a whole-round revert flips every weakness attacked that round to `reverted`. Add each attack round to `attacked_in_rounds`. ## Related weaknesses When a weakness clearly relates to another (same root cause, overlapping fix), add it to `related` with a one-line `why`. The dashboard draws these as edges, so the graph reflects how failures cluster. If two `related` weaknesses look like the *same* problem, you may **merge them — but only while both are in their discovery round** (folding one's tasks into the other). After that the freeze rule applies: their task sets are pinned to existing solutions, so leave them linked rather than merged. ## Concurrency (light, no heavy locks) Builders write different files almost always. To avoid two builders editing the *same* file at once, announce the slug you're about to touch on the shared task list before writing it; if someone else holds it, hand them your finding instead. The lead's dedup pass cleans up the rare collision. Don't build a locking protocol — keep it light. ## Canonical weakness file ```markdown --- slug: tool-call-arg-mismatch status: in-progress # open | in-progress | completed | solved | reverted tags: [tool-calling, type-error] discovered_in_round: 1 attacked_in_rounds: [1, 2] solved_in_round: null reverted_in_rounds: [] # rounds whose merged fix was rolled back by a whole-round revert branch: evograph/w/tool-call-arg-mismatch # the weakness's worktree branch (set when first attacked) affected_tasks: [task_007, task_011, task_023] # FROZEN after discovery round related: # optional — graph edges; also the fix's blast-radius watchlist - slug: schema-drift-after-retry why: both corrupt the tool-call payload; candidates to merge solutions: - "[[tool-call-arg-mismatch-r1-h1]]" --- # Tool call arg mismatch ## What fails The agent's tool-call planner passes a dict where the tool expects a string → `ToolError`. ## Tasks (found on) - task_007 — search query sent as JSON object — `runs/round-1/...:42` - task_011 — ... ## Trajectory excerpts > Tool call: search(query={"q": "..."}) → ToolError: expected str, got dict > *— task_007* ## References - `agent/planner.py:88` — builds the args dict; no type coercion before dispatch. ## Suggested directions - Validate arg types in the planner before dispatch. ## Rejected Store Memory (RSM) (Empty at discovery; the solver appends dead-end attempts here so future rounds don't retry them.) ``` ### RSM entry format (append-only, inside the weakness md) ```markdown ### Round N · `<rejected-direction-slug>` - **Thesis**: <one line> - **Change**: <files touched, summarized> - **Metrics (weakness tasks)**: <primary> <value>[, <secondary> <value>, …] # always record the attempt's measured metrics - **Why rejected**: <dead end → no gain (≤ baseline) · or reverted → round regressed, rolled back> - **Branch**: <the unmerged branch the attempt lives on> ``` Include the metric **value** the attempt reached in the `Result` line (e.g. `reward 0.48`): the dashboard parses it and shows that number on the red timeline node, so a reader sees *how far it dropped*, not just that it was rejected. Read the full RSM before proposing a fix; treat "the same idea but stricter" as a re-propose and pick a genuinely different angle. -
dashboard.md 5.6 KB
# Dashboard — the agent ↔ UI contract This is a **file-format contract**: agents only ever *write* the files described below into the run dir, in the formats given. **Agents never call any backend** — they write these files and the viewer reflects them within a couple of seconds. The renderer is the **cap-evolve dashboard itself**: its reducer reads `wiki/` straight out of the run dir and shows a **Weakness graph** tab. There is no separate server, no port to pick, and no registration step — the tab appears because the files exist, in the live dashboard and in the self-contained static export alike. So: *anything you want the user to see, write into one of these files in this format.* ## Files agents write ### Per-round metrics → `wiki/results/round-<N>.json` (and `final-test.json`) ```json { "round": 1, "split": "train", "started_at": "2026-06-28T14:00:00+03:00", "completed_at": "2026-06-28T14:04:12+03:00", "num_tasks": 30, "metrics": { "reward": { "value": 0.62, "primary": true, "direction": "higher" }, "avg_steps":{ "value": 14.3, "primary": false, "direction": "lower" } }, "per_task": [ { "task_id": "0", "reward": 1.0, "avg_steps": 9 } ], "extra": { "num_trials": 3, "concurrency": 4, "note": "anything else worth keeping on the record" } } ``` The UI builds the metric-over-rounds timeline from the **round** files. Write **one file per round** (`round-1.json`, `round-2.json`, …) as that round completes — a missing round file is a gap on the timeline. `final-test.json` is the same shape with `"split": "test"` and `"round": "final"`. The held-out test is rendered in its **own Final-test panel**, not plotted on the rounds line — so put it **only** in `final-test.json`, never in a `round-<N>.json`, or it will look like just another round. - **`metrics`** — every metric **must** be the wrapped object `{ value, primary, direction }`, never a flat number. The timeline reads each metric's `.value`, so a bare `"reward": 0.62` charts as an empty graph. Exactly one metric is `"primary": true`; `direction` is `"higher"` or `"lower"`. - **`started_at` / `completed_at`** — round-start eval time and round-end time; their difference is the round's duration (the UI shows per-round bars + the total). Stamp both from the shared clock script so every file agrees on the time **and** timezone (the user's PC local time): `python scripts/now.py`. `completed_at` is absent while the round is still running (the UI shows it as "running"). (`timestamp` is still accepted as a legacy alias for `started_at`.) - **`cost_usd`** (optional) — **one** number for the whole optimization (the agents' conversation cost, not the benchmark eval spend), recorded on **`final-test.json`** at the end, READ FROM cap-evolve's run-dir spend (never per-round, and never hand-totalled). The UI shows it next to the total time; the source of truth is cap-evolve's cost accounting (the Cost tab / `RunDir` spend). - **`extra`** (optional, free-form) — a catch-all object for anything you want recorded but that the UI doesn't render: the run-params actually used (trials-per-task, concurrency), seeds, notes, links, whatever. **The dashboard ignores fields it doesn't know**, so adding your own keys here (or anywhere in these files) is always safe. ### Weakness nodes → `wiki/weaknesses/<slug>.md` Front-matter drives the graph: `slug, status (open|in-progress|completed|solved|reverted), tags, discovered_in_round, attacked_in_rounds, solved_in_round, reverted_in_rounds, branch, affected_tasks, solutions, related`. Each `related` entry is a `slug` + a one-line `why` — those are the graph's edges. `status` is one of `open | in-progress | completed | solved | reverted`, and `affected_tasks` is frozen after the weakness's discovery round. ### Solutions → `wiki/solutions/<weakness-slug>/<sol-id>/{solution.md,changes.diff}` Front-matter drives the solution cards (`outcome, primary_metric, secondary_metrics, new_record, timestamp, weakness, round, attempt_index, branch, tags`) and `changes.diff` drives the diff tabs. The `sol-id` is `r<N>-h<M>`: round N, hypothesis M for that weakness. ### Live progress → `runs/round-<N>/agents/<weakness-slug>.log` Append-only, one line per step (free text). The solver assigned to a weakness appends here as it works; the UI streams it live in the weakness detail panel. Just append — no format ceremony. If a line doesn't already start with a `HH:MM[:SS]` time, the dashboard prefixes it with the **local time of the machine running the dashboard** (i.e. your timezone), so timestamps stay consistent — the same clock as `scripts/now.py`. ## What the dashboard renders from these files (for reference; agents only write files) - **Weakness graph tab** — one row per `wiki/weaknesses/<slug>.md`, showing `status`, `tags`, `discovered_in_round` / `solved_in_round`, `affected_tasks`, the `related` edges, and how many solution dirs exist under `wiki/solutions/<slug>/`. - **Primary metric over rounds** — one bar per `wiki/results/round-<N>.json`, read from the metric marked `"primary": true`. A round with no `completed_at` is labelled *running*. - **Final-test panel** — `wiki/results/final-test.json`, shown apart from the rounds so a number scored once on sealed data is never mistaken for another round. - Everything else in these files is preserved and ignored, so extra keys are always safe. ## What this buys the agents No registration, no build step, no API client. Write a markdown file or append a log line and the user sees it. The wiki stays the single source of truth for what an evograph run recorded — but see SKILL.md: its per-weakness numbers were never val-gated. -
graph.md 7.2 KB
# The two graphs, branches, and solutions EvoGraph keeps two mutually-linked graphs under `<run_dir>/wiki/`: - **Weaknesses** (`wiki/weaknesses/<slug>.md`) — what's broken. Persistent across rounds; shrinks (in active count) as weaknesses get stamped `solved`. - **Solutions** (`wiki/solutions/<weakness-slug>/<sol-id>/`) — a kept improvement (it raised the weakness's task average). Grows monotonically. Dead ends never become solutions — they go to the weakness's RSM. Every solution `[[wikilink]]`s back to its weakness; every weakness lists its solutions. The dashboard graph shows **weaknesses only** (connected by their `related` links, declared in the weakness node's front matter); solutions appear inside a weakness's detail panel. ## Absolute wiki path (most important rule) Every teammate gets the wiki as an **absolute** path: `<run_dir>/wiki/`. Solvers run in their own worktrees — a relative `./<run_dir>/wiki/` would point at the worktree's private copy, invisible to everyone else and to the dashboard. Always write to the absolute path. ## Branch model - The lead creates the `evograph` branch up front **and a root worktree at `<run_dir>/root`** checked out on it — *all* of EvoGraph's work happens there, so the user's original checkout is never modified. All accepted fixes land on `evograph` (never on the user's branch unless they ask). - When a weakness is first attacked, give it a **weakness branch** off `evograph` (e.g. `evograph/w/<slug>`); record it in the weakness md `branch:` field. - A solver works in its own **worktree** (under `<run_dir>/worktrees/`) on a solution branch off the weakness branch. The shared wiki stays in `<run_dir>/wiki/` (absolute path), outside every worktree. ## Solution layout `wiki/solutions/<weakness-slug>/<sol-id>/`, where **sol-id = `r<N>-h<M>`**: `r<N>` is the **round** the attempt was made in, and `h<M>` is the **hypothesis (attempt) index** within that weakness — `h1` is the first fix tried, `h2` the second, and so on. So `r2-h3` = the third hypothesis tried for this weakness, made in round 2. - `solution.md` - `changes.diff` — captured **before** requesting merge: `git diff <weakness-branch-base>..<solution-HEAD>`. Snapshotting here keeps the UI's diff stable even after later commits land on the weakness branch. Example `solution.md` (front-matter + body) — this is a **solution** file, not the weakness node: ```markdown --- weakness: "[[tool-call-arg-mismatch]]" round: 1 attempt_index: 1 # the h<M> branch: evograph/w/tool-call-arg-mismatch/h1 timestamp: 2026-06-28T14:03:00+03:00 # from `python scripts/now.py` — never hand-written outcome: kept # pending while in-flight → kept once re-eval confirms it improved (dead ends aren't solutions) tags: [tool-calling] primary_metric: { name: reward, value: 0.74 } secondary_metrics: [ { name: avg_steps, value: 12.1 } ] new_record: true # set true if this beat the weakness's previous best on its tasks --- # Validate tool-call arg types in the planner ## Thesis One line: the idea for resolving the weakness. ## Reasoning / approach A paragraph: why this should work, given what the trajectories show. ## Change list - `agent/planner.py` — coerce arg types against the tool schema before dispatch. ## Per-task metric delta (weakness tasks) | task | before | after | Δ | |---------|--------|-------|------| | task_007| 0.0 | 1.0 | +1.0 | | task_011| 0.4 | 0.4 | 0.0 | ## Baseline comparison Kept iff the **average primary metric across all the weakness's `affected_tasks`** rises vs the prior best (equivalently, net Δ > 0 over that fixed task set — not just one task improving). With trials-per-task, each task's score is itself the mean over its trials. ## References - `agent/planner.py:88` See also [[tool-call-arg-mismatch]]. ``` Pre-write `solution.md` with `outcome: pending` before editing code, so intent survives a crash; finalize `outcome` + metrics + `new_record` after the verified re-eval. ## Don't break the neighbors — use the graph edges The graph's **edges live on the weakness node, not on the solution.** Each `wiki/weaknesses/<slug>.md` lists `related:` neighbors (`slug` + `why`) — those are exactly the edges the dashboard draws. A solution file only links *up* to its own weakness via `weakness: [[…]]`. A weakness's `related` neighbors are the fix's **blast radius — stay aware of them, don't re-run them.** A fix kept on W's own task average can quietly regress a connected weakness V, so avoid changes that obviously undermine W's neighbors and note any likely cross-weakness impact in the solution body. The **round-start eval** is the backstop for a real regression. ## PR / merge - **Only the lead merges.** The lead **trusts the solver's reported result** — the solver already re-evaluated the fix on its **own weakness's tasks** during research — so it does *not* re-run the eval per fix; it merges into `evograph` one weakness at a time and resolves conflicts. The whole-train **round-start eval** (see below) is the objective backstop that catches anything wrong. - The route is set by the **`github_integration`** choice captured at setup by cap-evolve `intake` and recorded in the project spec (`capevolve.yaml`): - **`github_integration: true`** (GitHub CLI authenticated and the user opted in) → GitHub **mirrors** the wiki, which stays the source of truth (it's what the UI reads). At PR time the solver **syncs the weakness's GitHub issue to match its weakness md**, then opens a **PR** explaining the auto-research + measured gain, with `Closes #<n>`; the lead reviews/merges. - **`github_integration: false`** (not authenticated, or the user opted to keep it local) → no GitHub issue/PR; the solver asks the lead to merge the branch directly. ## Per-round tags + whole-round revert The lead drives the round loop, and round-over-round it guards against regressions: - **Tag the round's starting tip.** At the start of round N, *before* that round's merges land, the lead tags the current evograph tip: `git -C <run_dir>/root tag -f evograph-round-<N>-start`. This tip is the state the round's start-eval measures (everything merged through round N−1). - **Detect regression.** The start-of-round-N eval reflects round N−1's merges. If round N's **primary metric** is below round N−1's, round N−1's combined merges regressed the suite. - **Revert the whole round.** `git -C <run_dir>/root reset --hard evograph-round-<N-1>-start` rolls back to the tip *before* round N−1's merges, then re-eval to confirm the baseline is restored. Whole-round revert is deliberately coarse: even a single bad merge rolls back all of that round's fixes, and the reverted weaknesses get re-attacked next round. - **Sign the graph.** Every weakness in round N−1's `attacked_in_rounds` flips to status `reverted` (dashboard renders it distinctly; it is eligible to be attacked again), gains the round in `reverted_in_rounds`, and gets an RSM note: `round N−1 combined fixes regressed primary X→Y — rolled back`. Their round-N−1 solution files are **demoted into RSM with their metrics + branch** — a reverted attempt is not a kept solution, so it must not stay under `wiki/solutions/`.
-
-
scripts
-
abstract.py 535 B
"""evograph carries no per-skill abstract methods. Like the other algorithm skills, it composes the project adapter contract (tasks/run_target/score) via cap-evolve's primitives. evograph is AGENT MODE ONLY: its loop is prose the coding agent runs (see SKILL.md "Step 2 — Round loop"), not a deterministic engine — so there is no policy to materialize here. """ from __future__ import annotations from pathlib import Path DEFAULT_POLICY: dict = {} def materialize(capability_dir: Path) -> dict: # noqa: ARG001 return {} -
check.py 2.5 KB
"""Behavioral contract for evograph (DEPRECATED, agent mode only). evograph never had a deterministic loop, and it is now deprecated, so there is still nothing to run offline. This check pins what must stay true of the deprecated skill: 1. its ``run.py`` REFUSES a deterministic invocation (exit 2 + an "agent-mode only" directive) rather than faking a deterministic loop; 2. the SKILL.md marks itself DEPRECATED and names the replacement (``agent-optimize``), so nobody starts a new run on it by accident; 3. every referenced doc exists — above all the wiki-format contract the dashboard's Weakness-graph tab reads, which is the reason this directory still exists. """ from __future__ import annotations import io import json import sys from contextlib import redirect_stdout from pathlib import Path import _bootstrap # noqa: F401 from cap_evolve.skillcheck import Checker, import_run SKILL_DIR = Path(__file__).resolve().parents[1] def main() -> int: c = Checker("evograph") run = import_run() c.require_main(run) # 1: deterministic invocation is refused loudly (agent-mode only). buf = io.StringIO() with redirect_stdout(buf): rc = run.main(["--run-dir", "x", "--project", "y", "--optimizer", "mock"]) out = buf.getvalue() c.check(rc == 2, f"deterministic run.py should exit 2, got {rc}", note="run.py refuses deterministic mode") try: payload = json.loads(out) except Exception: payload = {} c.check("agent-mode only" in payload.get("error", ""), "run.py did not emit the agent-mode-only error", note="clear agent-mode directive emitted") # 2: SKILL.md is honestly marked deprecated and points somewhere better. skill = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8") for needle, label in [ ("DEPRECATED", "itself deprecated"), ("agent-optimize", "the agent-mode replacement"), ("hill-climb", "the deterministic replacements"), ("no deterministic engine", "that it has no deterministic engine"), ]: c.check(needle in skill, f"SKILL.md missing: {label!r} ({needle!r})", note=f"SKILL.md declares {label}") # 3: the referenced wiki-format docs exist. for ref in ("clustering.md", "graph.md", "dashboard.md"): c.check((SKILL_DIR / "references" / ref).exists(), f"missing reference: references/{ref}", note=f"references/{ref} present") return c.emit() if __name__ == "__main__": sys.exit(main()) -
now.py 682 B
#!/usr/bin/env python3 """Print the current time on the user's machine, in EvoGraph's canonical timestamp format. Every agent (lead, builders, solvers, PR Manager) calls this whenever it needs a timestamp — results JSON, solution front-matter, anywhere a time is written — instead of inventing one. That way every timestamp across the whole run shares one clock and one timezone: the user's PC local time. ISO-8601 with the local UTC offset, second precision, e.g. 2026-06-28T14:00:00+03:00. python skills/algorithms/evograph/scripts/now.py """ from datetime import datetime if __name__ == "__main__": print(datetime.now().astimezone().isoformat(timespec="seconds")) -
run.py 2 KB
"""evograph — DEPRECATED, and never had a deterministic engine. evograph is deprecated (see SKILL.md): use ``agent-optimize`` for agent-mode search, or ``hill-climb`` / ``gepa`` / ``skillopt`` for a deterministic loop. It also never had a deterministic engine — its weakness-graph loop was agent-driven, so under ``orchestration_mode: agent`` cap-evolve ran intake → check → baseline and then HANDED OFF to the agent (see cli.py); this run.py was never invoked for a real evograph run. If it IS invoked — i.e. someone selected ``algorithm_skill: evograph`` with ``orchestration_mode: deterministic`` — fail loudly with a clear directive rather than pretending to run a deterministic loop. This keeps the honesty contract explicit: there is no fake deterministic evograph, and now no reason to want one. """ from __future__ import annotations import argparse import json import sys import _bootstrap # noqa: F401 ALGO = "evograph" def main(argv=None) -> int: p = argparse.ArgumentParser(prog=ALGO) # Accept (and ignore) the standard algorithm flags so a mis-configured # deterministic invocation reaches our clear error instead of an argparse crash. p.add_argument("--run-dir") p.add_argument("--project") p.add_argument("--optimizer") p.add_argument("--max-iterations") p.add_argument("--n-trials") p.add_argument("--gate-mode") p.add_argument("--k-se") p.add_argument("--store") p.parse_known_args(argv) print(json.dumps({ "algorithm": ALGO, "error": "evograph is agent-mode only, and is DEPRECATED", "detail": ( "evograph has no deterministic engine and is deprecated. Set " "`algorithm_skill: agent-optimize` (with `orchestration_mode: agent`) for the " "same per-cluster fan-out behind the val significance gate, or " "`hill-climb` | `gepa` | `skillopt` for a deterministic loop. See " "skills/algorithms/evograph/SKILL.md for why." ), })) return 2 if __name__ == "__main__": sys.exit(main()) -
_bootstrap.py 3.6 KB
"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap. Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim only has to find that package, which means a minimal upward walk for ``core/`` — the single bit of bootstrapping that genuinely must run before cap_evolve is importable. Everything else delegates. """ from __future__ import annotations import os import sys from pathlib import Path def _seed_path() -> None: """Minimal: put a dir containing the cap_evolve package on sys.path. ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that it imports the checkout you are standing in. Deferring to the ambient package here made an explicit override unreachable, and the symptom was a stale core silently answering for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that predates that module). An explicit env var wins. """ env = os.environ.get("CAPEVOLVE_CORE") want = Path(env).resolve() if env else None if want and (want / "cap_evolve" / "__init__.py").exists(): loaded = sys.modules.get("cap_evolve") already = getattr(loaded, "__file__", None) if already and Path(already).resolve().parent.parent == want: return # right checkout already imported: touch nothing p = str(want) if p in sys.path: sys.path.remove(p) sys.path.insert(0, p) if loaded is not None: # Evicting a module makes a re-import yield a DIFFERENT object, so anything # already holding a reference fails an `is` check. Only ever do it when the # loaded package really is the wrong checkout — otherwise this "fix" becomes # the bug (it broke two identity assertions in core/tests exactly once). for name in [m for m in sys.modules if m == "cap_evolve" or m.startswith("cap_evolve.")]: sys.modules.pop(name, None) for finder in list(sys.meta_path): if "cap_evolve" in getattr(finder, "MAPPING", {}): sys.meta_path.remove(finder) return # A checkout's own core outranks an ambient install. Without this, a skill script run # from checkout X silently executed against checkout Y's cap_evolve (an editable install # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was # missing modules — or, worse, a green result measured against the wrong tree. here = Path(__file__).resolve() own = next((p / "core" for p in here.parents if (p / "core" / "cap_evolve" / "__init__.py").exists()), None) if own is not None: os.environ.setdefault("CAPEVOLVE_CORE", str(own)) return _seed_path() try: import cap_evolve # noqa: F401 return except Exception: pass cands = [] for parent in here.parents: cands.append(parent / "core") cands.append(parent) for c in cands: if (c / "cap_evolve" / "__init__.py").exists(): p = str(c) if p not in sys.path: sys.path.insert(0, p) return _seed_path() from cap_evolve._bootstrap import ensure_core # noqa: E402 # Anchor the upward walk at THIS skill script's location (not the core module's). ensure_core(Path(__file__).resolve())
-
-
meta.yaml 809 B
component: algorithm name: evograph summary: DEPRECATED (agent mode only) — the evo-graph weakness-graph algorithm. Do not select it for new runs: its one-solver-per-weakness fan-out is agent-optimize's sibling fan-out done behind the val significance gate evograph never applied, and its clustering, rejected-edit memory and stop-condition handling live in agent-optimize + phases/diagnose. Kept so an existing evograph run dir stays readable and the run-dir wiki/ format the dashboard's Weakness-graph tab reads still has an owner. Use agent-optimize (agent mode) or hill-climb | gepa | skillopt (deterministic). entry: scripts/run.py abstract: scripts/abstract.py check: scripts/check.py needs: [scores, traces, candidate] provides: [candidate] compatible_with: capabilities: ["*"] optimizers: ["*"] -
SKILL.md 5.5 KB
--- name: evograph description: >- Deprecated agent-mode algorithm (evo-graph port): a weakness-graph search that dispatched one solver agent per failure cluster and reverted a whole round on regression. Do not start new runs with it — its per-weakness fan-out is already `agent-optimize`'s sibling fan-out, done behind the honest val significance gate that evograph never applied, and everything else it did (failure clustering, rejected-edit memory, budget-aware fan-out, free-text stop condition) lives in `agent-optimize` + `phases/diagnose`. Use when reading or repairing an existing evograph run dir, or when writing the run-dir `wiki/` format the dashboard's Weakness-graph tab reads — and to see what to select instead: `agent-optimize` for agent-mode search, `hill-climb`, `gepa`, or `skillopt` for a deterministic loop. component: algorithm argument-hint: "deprecated — use agent-optimize (agent mode) or hill-climb | gepa | skillopt" allowed-tools: Read, Write, Edit, Bash, Task needs: [scores, traces, candidate] provides: [candidate] sources: [evo-graph] --- # evograph — DEPRECATED **Do not select `algorithm_skill: evograph` for a new run.** Use `agent-optimize` (agent mode) or `hill-climb` / `gepa` / `skillopt` (deterministic) with `memory_skill: wiki` if you want THIS run's weakness-graph format. This file stays so an existing evograph run dir is still readable. ## Why it is deprecated evograph advertised one distinctive capability — a collaborative weakness graph with one solver agent per weakness, merged into a shared candidate each round. Measured against its four siblings, that capability is not distinctive and the part that *was* distinctive was a defect: - **The fan-out already exists, gated properly.** `agent-optimize` fans out N sibling candidates from the same parent, one diagnosed failure cluster each, every sibling in its own working copy (a git worktree when the capability is in git), gated **one at a time with a re-gate after each accept** so several fixes accumulate into one lineage honestly. That is evograph's round, minus the flaws below. The clustering itself is `phases/diagnose`'s job in both cases. - **Acceptance was never held out.** evograph kept a merge on a raw delta over a frozen 3-task subset of *train*, self-reported by the solver subagent that made the edit — no val split, no standard error, no `Δ > k·SE`. Between `baseline` and `finalize` an evograph run took no held-out measurement at all, so the sealed test number was the first honest signal anyone saw. Whole-round revert existed only as a one-round-late substitute for the gate it lacked; wire the real gate and there is nothing left for it to catch. - **What remains unique is an output format, not a search strategy.** The run-dir `wiki/` is genuinely useful, but the dashboard renders the Weakness-graph tab from `wiki/` presence alone, for any algorithm that writes the format (`core/cap_evolve/dashboard.py`). An output contract does not earn a second agent-mode algorithm that users must choose between — it has since moved to `skills/memory/wiki/SKILL.md`, a standalone `memory_skill` any algorithm can select (#400, #404), so it no longer needs evograph to stay alive. ## There is no deterministic engine There never was one, and `scripts/run.py` is a tombstone, not a stub: invoked deterministically it exits 2 with an `agent-mode only` payload rather than faking a loop. So evograph is also the one algorithm that could not be routed through the shared per-iteration record — see the `hill-climb` skill's `references/run-step.md`, which owns the shared iteration mechanics (parent selection, val gate, commit, iteration record) every other algorithm routes through. Read it if you are reconstructing what an evograph round *should* have done. ## Reading an existing evograph run The run dir is authoritative. `<run_dir>/wiki/` holds the weakness nodes, solution cards and per-round results; `<run_dir>/runs/round-<N>/agents/<slug>.log` holds solver progress. The formats are in [references/dashboard.md](references/dashboard.md) — load it if you need to write or parse that wiki. Treat any per-weakness "kept / new record" number in it as a train-subset self-report, not a gated result; only `finalize`'s sealed test number and any `gate` decision recorded in `events.jsonl` are honest. `scripts/now.py` is the one-clock timestamp stamper those formats require; it is correct and still used by anything writing the wiki. ## Removal is a separate decision Deprecation is reversible; removal is not. The wiki format contract now has a real owner (`skills/memory/wiki/SKILL.md`), so what's left to decide (maintainer): stop `dashboard.py` inferring `algorithm = "evograph"` from `wiki/` presence alone (any `memory_skill: wiki` run now writes it too), then delete this directory. ## References - [skills/memory/wiki/SKILL.md](../../memory/wiki/SKILL.md) — the wiki format contract's current home: weakness-node/solution-card schemas, generalized for any algorithm via `memory_skill: wiki`. - [references/dashboard.md](references/dashboard.md) — the same file formats, kept here for reading an existing evograph run dir's branch/round-revert specifics the generalized skill dropped. - [references/clustering.md](references/clustering.md) — weakness-node schema and the `affected_tasks` freeze rule, kept for reading historical run dirs. - [references/graph.md](references/graph.md) — solution-card schema, branch layout, whole-round revert, kept for the same reason.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.