lab:autoresearch
Self-improving loop for plugin skills. Reads program.md, proposes one mutation per iteration, evaluates against deterministic scorer, keeps improvements via git, reverts failures. Targets weakest skill+dimension. Use with /loop for overnight runs.
Install
npx skills add https://github.com/oliver-kriska/claude-elixir-phoenix/tree/main/lab/autoresearch
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oliver-kriska-claude-elixir-phoenix@llmmart
git clone https://github.com/oliver-kriska/claude-elixir-phoenix.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole oliver-kriska/claude-elixir-phoenix collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Autoresearch — Plugin Skill Self-Improvement
Iteratively improve plugin skills via the autoresearch pattern: propose one mutation -> eval -> keep/revert -> repeat.
Usage
/lab:autoresearch # Targeted: attack weakest skill+dimension
/lab:autoresearch --skill review # Focus on one skill
/lab:autoresearch --strategy sweep # Process all skills alphabetically
/lab:autoresearch --dry-run # Show what would change, don't commit
For overnight runs:
/loop 5m /lab:autoresearch --strategy sweep --max-iterations 200
Iron Laws
- ONE mutation per iteration — if description needs "and", split into two
- NEVER mutate read-only files — check program.md before every write
- EVAL is deterministic — always use the wrapper script, never LLM-judge
- REVERT on regression OR checks failure — no exceptions
- LOG every iteration — use
keeporrevertcommand (never skip) - CHECK ideas.md before proposing — don't rediscover known optimizations
Wrapper Script Commands
All eval/git/journal operations go through ONE script. Do NOT run these manually.
# Find the weakest skill+dimension
python3 lab/autoresearch/scripts/run-iteration.py target --strategy targeted
# Score a skill (before mutation, to get baseline)
python3 lab/autoresearch/scripts/run-iteration.py score <skill-name>
# After mutation: score + checks + compare → verdict (KEEP or REVERT)
python3 lab/autoresearch/scripts/run-iteration.py eval <skill-name>
# Act on verdict:
python3 lab/autoresearch/scripts/run-iteration.py keep <skill> <dim> <old> <new> \
--desc "what changed" --asi '{"hypothesis": "why", "mechanism": "how"}'
python3 lab/autoresearch/scripts/run-iteration.py revert <skill> <dim> <old> <new> \
--desc "what was attempted" --asi '{"hypothesis": "why", "regression": "what broke", "avoid": "do not retry this"}'
# Check overall progress
python3 lab/autoresearch/scripts/run-iteration.py status
Core Loop (ONE iteration)
Step 1: Read State
- Read
lab/autoresearch/program.md(goals, mutable surface, rules) - Read
lab/autoresearch/ideas.mdif it exists (deferred optimizations) - Run:
python3 lab/autoresearch/scripts/run-iteration.py status
Step 2: Select Target
Run: python3 lab/autoresearch/scripts/run-iteration.py target --strategy targeted
Parse the JSON: skill, dimension, failing_checks. If all_perfect → STOP.
Step 3: Read + Propose
- Read target SKILL.md and its references/ listing
- Read eval definition from
lab/eval/evals/{skill}.json - Check
ideas.mdfor deferred ideas about this skill - Check recent journal entries for prior failures on this skill (avoid repeats)
- Consult
${CLAUDE_SKILL_DIR}/references/mutation-strategies.md - Propose exactly ONE change targeting the failing checks
Step 4: Apply + Evaluate
- Apply the mutation via Edit tool
- Run:
python3 lab/autoresearch/scripts/run-iteration.py eval <skill-name> - Parse JSON → check
verdictfield
Step 5: Keep or Revert
If verdict is KEEP:
python3 lab/autoresearch/scripts/run-iteration.py keep <skill> <dim> <old> <new> \
--desc "..." --asi '{"hypothesis": "...", "mechanism": "..."}'
If verdict is REVERT:
python3 lab/autoresearch/scripts/run-iteration.py revert <skill> <dim> <old> <new> \
--desc "..." --asi '{"hypothesis": "...", "regression": "...", "avoid": "..."}'
Step 6: Ideas Backlog
If during analysis you discovered a promising optimization you can't act on now:
- Append it to
lab/autoresearch/ideas.mdas a bullet - On next resume: prune stale/tried ideas, experiment with the rest
Step 7: Continue or Stop
- All targets >= 0.95? Print "AUTORESEARCH_COMPLETE"
- Max iterations reached? Print "AUTORESEARCH_COMPLETE"
- 50 consecutive discards? Print "AUTORESEARCH_STUCK"
- Otherwise: immediately start Step 1 again
References
${CLAUDE_SKILL_DIR}/references/mutation-strategies.md— mutation type catalog${CLAUDE_SKILL_DIR}/references/state-management.md— git protocol, journalinglab/autoresearch/program.md— research agenda (read every iteration)
Files (claude-elixir-phoenix)
-
references
-
mutation-strategies.md 2.4 KB
Loading… -
state-management.md 1.1 KB
Loading…
-
-
scripts
-
checks.sh 3.2 KB
Loading… -
protected_sections.py 3.4 KB
Loading… -
run-iteration.py 25 KB
Loading… -
score-skill.py 1.2 KB
Loading…
-
-
tests
-
test_deviation_dispatch.py 3.1 KB
Loading… -
test_protected_sections.py 3.3 KB
Loading… -
__init__.py 0 B
Loading…
-
-
.gitignore 130 B · in bundle
-
program.md 3.9 KB
Loading… -
retention.py 5.7 KB
Loading… -
SKILL.md 4.6 KB
--- name: lab:autoresearch description: > Self-improving loop for plugin skills. Reads program.md, proposes one mutation per iteration, evaluates against deterministic scorer, keeps improvements via git, reverts failures. Targets weakest skill+dimension. Use with /loop for overnight runs. effort: high argument-hint: "[--skill NAME] [--strategy targeted|sweep|random] [--dry-run] [--max-iterations N]" disable-model-invocation: true --- # Autoresearch — Plugin Skill Self-Improvement Iteratively improve plugin skills via the autoresearch pattern: propose one mutation -> eval -> keep/revert -> repeat. ## Usage ``` /lab:autoresearch # Targeted: attack weakest skill+dimension /lab:autoresearch --skill review # Focus on one skill /lab:autoresearch --strategy sweep # Process all skills alphabetically /lab:autoresearch --dry-run # Show what would change, don't commit ``` For overnight runs: ``` /loop 5m /lab:autoresearch --strategy sweep --max-iterations 200 ``` ## Iron Laws 1. **ONE mutation per iteration** — if description needs "and", split into two 2. **NEVER mutate read-only files** — check program.md before every write 3. **EVAL is deterministic** — always use the wrapper script, never LLM-judge 4. **REVERT on regression OR checks failure** — no exceptions 5. **LOG every iteration** — use `keep` or `revert` command (never skip) 6. **CHECK ideas.md before proposing** — don't rediscover known optimizations ## Wrapper Script Commands All eval/git/journal operations go through ONE script. Do NOT run these manually. ```bash # Find the weakest skill+dimension python3 lab/autoresearch/scripts/run-iteration.py target --strategy targeted # Score a skill (before mutation, to get baseline) python3 lab/autoresearch/scripts/run-iteration.py score <skill-name> # After mutation: score + checks + compare → verdict (KEEP or REVERT) python3 lab/autoresearch/scripts/run-iteration.py eval <skill-name> # Act on verdict: python3 lab/autoresearch/scripts/run-iteration.py keep <skill> <dim> <old> <new> \ --desc "what changed" --asi '{"hypothesis": "why", "mechanism": "how"}' python3 lab/autoresearch/scripts/run-iteration.py revert <skill> <dim> <old> <new> \ --desc "what was attempted" --asi '{"hypothesis": "why", "regression": "what broke", "avoid": "do not retry this"}' # Check overall progress python3 lab/autoresearch/scripts/run-iteration.py status ``` ## Core Loop (ONE iteration) ### Step 1: Read State 1. Read `lab/autoresearch/program.md` (goals, mutable surface, rules) 2. Read `lab/autoresearch/ideas.md` if it exists (deferred optimizations) 3. Run: `python3 lab/autoresearch/scripts/run-iteration.py status` ### Step 2: Select Target Run: `python3 lab/autoresearch/scripts/run-iteration.py target --strategy targeted` Parse the JSON: `skill`, `dimension`, `failing_checks`. If `all_perfect` → STOP. ### Step 3: Read + Propose 1. Read target SKILL.md and its references/ listing 2. Read eval definition from `lab/eval/evals/{skill}.json` 3. Check `ideas.md` for deferred ideas about this skill 4. Check recent journal entries for prior failures on this skill (avoid repeats) 5. Consult `${CLAUDE_SKILL_DIR}/references/mutation-strategies.md` 6. Propose exactly ONE change targeting the failing checks ### Step 4: Apply + Evaluate 1. Apply the mutation via Edit tool 2. Run: `python3 lab/autoresearch/scripts/run-iteration.py eval <skill-name>` 3. Parse JSON → check `verdict` field ### Step 5: Keep or Revert **If verdict is KEEP**: ```bash python3 lab/autoresearch/scripts/run-iteration.py keep <skill> <dim> <old> <new> \ --desc "..." --asi '{"hypothesis": "...", "mechanism": "..."}' ``` **If verdict is REVERT**: ```bash python3 lab/autoresearch/scripts/run-iteration.py revert <skill> <dim> <old> <new> \ --desc "..." --asi '{"hypothesis": "...", "regression": "...", "avoid": "..."}' ``` ### Step 6: Ideas Backlog If during analysis you discovered a promising optimization you can't act on now: - Append it to `lab/autoresearch/ideas.md` as a bullet - On next resume: prune stale/tried ideas, experiment with the rest ### Step 7: Continue or Stop - All targets >= 0.95? Print "AUTORESEARCH_COMPLETE" - Max iterations reached? Print "AUTORESEARCH_COMPLETE" - 50 consecutive discards? Print "AUTORESEARCH_STUCK" - Otherwise: immediately start Step 1 again ## References - `${CLAUDE_SKILL_DIR}/references/mutation-strategies.md` — mutation type catalog - `${CLAUDE_SKILL_DIR}/references/state-management.md` — git protocol, journaling - `lab/autoresearch/program.md` — research agenda (read every iteration)
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.