Claude Skill

autoresearch

Autonomous iteration loop: modify, verify, keep/discard against any metric

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download mxyhi-ok-skills-autoresearch-7933c15.zip · 56 KB

Install

skills CLI npx skills add https://github.com/mxyhi/ok-skills/tree/main/autoresearch
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mxyhi-ok-skills@llmmart
Git git clone https://github.com/mxyhi/ok-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mxyhi/ok-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Autoresearch — Autonomous Goal-directed Iteration

Safety Invariants (all subcommands)

  • Never push, publish, or deploy without explicit user approval.
  • Bounded by default. Override with Iterations: unlimited.
  • All results logged to autoresearch/{subcommand}-{YYMMDD}-{HHMM}/ directory.
  • Chain handoff via handoff.json. Evals reads *-results.tsv.

Dispatch (bare $autoresearch)

Parse the invocation in this order:

Condition Mode
Metric: or Verify: present Classic — existing metric loop, unchanged
Free-form natural-language goal, no metric/verify Orchestrator — see Orchestrator section
Nothing Setup wizard — interactive config builder
--classic flag Force Classic regardless of goal text
--auto flag Force Orchestrator regardless of goal text

Print a banner on every invocation: [autoresearch] mode: classic | orchestrator | wizard.

Subcommands

Command Does Default Iterations
$autoresearch Iterate against a metric: modify → verify → keep/discard 25
$autoresearch plan Convert a goal into validated Scope, Metric, Verify config N/A
$autoresearch debug Hunt bugs: hypothesize → test → falsify → repeat 15
$autoresearch fix Crush errors one-by-one until zero remain 20
$autoresearch security STRIDE + OWASP audit with red-team personas 15
$autoresearch ship Ship through 8 phases: checklist → dry-run → deploy → verify N/A
$autoresearch scenario Generate edge cases across 12 dimensions 20
$autoresearch predict 5 expert personas debate before implementation N/A
$autoresearch learn Scout codebase → generate docs or wiki → validate → fix loop 10
$autoresearch reason Adversarial debate with blind judges until convergence 8
$autoresearch probe 8 personas interrogate requirements until saturation 15
$autoresearch improve Research ICP challenges, discover improvements, generate PRDs 15
$autoresearch evals Analyze iteration results: trends, plateaus, regressions N/A
$autoresearch regression Regression stability gate: baseline vs candidate, verdict STABLE/UNSTABLE N/A

Universal Flags

Flag Applies To Purpose
Iterations: N All looping Set iteration count
Iterations: unlimited All looping Opt-in unbounded
--evals All looping Mid-loop checkpoints + final summary
--evals-interval N All looping Override checkpoint frequency
--chain <targets> All Sequential handoff after completion
--<subcommand> All Shorthand for --chain <subcommand>
--dry-run Orchestrator Print derived config + planned pipeline; no execution
--max-cycles N Orchestrator Hard ceiling on orchestration cycles (default 50)
--classic Bare $autoresearch Force Classic metric-loop mode
--auto Bare $autoresearch Force Orchestrator mode

Orchestrator

Activated when a plain-language goal is given without Metric:/Verify:. Classifies the goal into a Goal archetype — see references/orchestrator-routing.md for the archetype table and router decision table.

Resolve every scripts/... path below relative to this installed skill directory, never relative to the caller's working directory.

Two modes based on archetype:

  • Orchestration loop — predicate-bearing archetypes (ship-ready, optimize-metric, fix-broken, harden, build-feature, explore). Goal has a mechanical Success predicate; the loop runs until that predicate is met.
  • Single-pass dispatch — subjective/terminal archetypes (document, what-to-build, decide-design). Routes once to the fitting subcommand (learn / improve / reason), lets it self-terminate, then reports. No loop, no Plateau, no ship gate.

Orchestration Loop Steps

Backed by scripts/orchestrate.sh (deterministic seam — all routing logic lives there). Subcommands exposed: classify, next-hop, units, plateau, screen-cmd, verdict, validate-state, screen-state-predicate.

  1. Classify — scripts/orchestrate.sh classify "<goal>" → archetype label + mode.
  2. Derive predicate — reuse plan logic to produce a concrete Success predicate: exact shell command + expected output. For optimize-metric, run the full plan/wizard derivation internally.
  3. Confirm — ONE request_user_input showing: archetype, mode, concrete predicate (command + expected output), terminal choice (stop-at-verified vs proceed-to-ship). Misclassifications are caught here, not mid-run.
  4. Round-0 dry-run — prove the predicate command runs and returns a value; safety-screen every derived command via screen-cmd; print projected cycle budget. Stop here if --dry-run.
  5. Loop until predicate satisfied:
    1. Assess state via cheap signals (last handoff.json, regression verdict, error count) + affected-test verify.
    2. scripts/orchestrate.sh next-hop orchestrator-state.json → next subcommand.
    3. Run subcommand (its own bounded inner loop).
    4. Record per-hop outcome ∈ {progressed, no-op, failed, blocked}.
    5. Fold hop's handoff.json into orchestrator-state.json.
    6. scripts/orchestrate.sh units → recompute Units remaining.
  6. Stop conditions (checked after each hop):
    • Predicate met → ship gate (only if ship is in the pipeline) else CONVERGED.
    • scripts/orchestrate.sh plateau orchestrator-state.json → true → stop + report PLATEAU.
    • Cycles > ceiling (default 50, override --max-cycles N) → stop + report CEILING.
    • Hop outcome blocked/failed with no alternative route → checkpoint + stop + report BLOCKED.

Orchestrator State

orchestrator-state.json — orchestrator-owned, additive. Tracks: goal, archetype, predicate, terminal-choice, units_remaining history, cycle count, per-hop pipeline log with outcomes, current incumbent. Each hop's handoff.json is unchanged (single-hop bridge); the orchestrator reads it and folds it in. Two clearly-owned state objects, no overlap.

Orchestrator Safety Invariants

  • Never auto-approve ship/deploy/push. The orchestrator never passes --auto to ship; deploy always requires explicit user approval.
  • Data-migration behind anchored DB-URL allowlist. Reuses regression's allowlist — host must be localhost/127.0.0.1/container hostname, or database name carries _test/_ci suffix. Bare substring match does not qualify. Anything else refused.
  • screen-cmd on every derived command — run before the loop starts AND on every command read from a persisted state file on resume. Persisted commands are never trusted; resume re-screens the pinned predicate via screen-state-predicate and refuses on refuse.
  • No un-screened commands mid-loop. The autonomous loop cannot introduce new shell commands that bypass screen-cmd.
  • Predicate pinned, not re-derived. Round-0 writes the derived Success predicate verbatim into orchestrator-state.json; every cycle and every resume reuses that exact string so "done" is reproducible across runs.
  • Validate the ledger before routing. validate-state gates orchestrator-state.json (required fields + coarse types); a malformed ledger is not trusted to route from.
  • Independent verify before convergence. High-impact changes accepted on the working signal set pending_verify; next-hop routes to a verify hop (held-out / adversarial check) before DONE or ship. The verify hop never auto-approves ship.
  • Unknown-units cycles excluded from Plateau counter. A cycle where units returns unknown (e.g. runner crash) is not counted as zero-progress; repeated unknown routes to BLOCKED.
Files (ok-skills)
  • agents
    • openai.yaml 250 B
      Loading…
  • references
    • orchestrator-routing.md 6.8 KB
      Loading…
    • predict-personas.md 3.1 KB
      Loading…
    • reason-judge-protocol.md 3.4 KB
      Loading…
    • security-checklist.md 3.4 KB
      Loading…
  • scripts
    • orchestrate.sh 18.5 KB
      Loading…
    • score-regression.sh 7.5 KB
      Loading…
  • autoresearch.md 4.8 KB
    Loading…
  • debug.md 4 KB
    Loading…
  • evals.md 5 KB
    Loading…
  • fix.md 4.3 KB
    Loading…
  • improve.md 6 KB
    Loading…
  • learn.md 7.9 KB
    Loading…
  • LICENSE 1 KB · in bundle
  • plan.md 2.8 KB
    Loading…
  • predict.md 3.7 KB
    Loading…
  • probe.md 4.8 KB
    Loading…
  • reason.md 4.7 KB
    Loading…
  • regression.md 9.3 KB
    Loading…
  • scenario.md 4.1 KB
    Loading…
  • security.md 4.6 KB
    Loading…
  • ship.md 3.8 KB
    Loading…
  • SKILL.md 7.8 KB
    ---
    name: autoresearch
    description: "Autonomous iteration loop: modify, verify, keep/discard against any metric"
    metadata:
      version: "2.2.2"
    ---
    
    # Autoresearch — Autonomous Goal-directed Iteration
    
    ## Safety Invariants (all subcommands)
    - Never push, publish, or deploy without explicit user approval.
    - Bounded by default. Override with `Iterations: unlimited`.
    - All results logged to `autoresearch/{subcommand}-{YYMMDD}-{HHMM}/` directory.
    - Chain handoff via `handoff.json`. Evals reads `*-results.tsv`.
    
    ## Dispatch (bare `$autoresearch`)
    
    Parse the invocation in this order:
    
    | Condition | Mode |
    |---|---|
    | `Metric:` or `Verify:` present | **Classic** — existing metric loop, unchanged |
    | Free-form natural-language goal, no metric/verify | **Orchestrator** — see Orchestrator section |
    | Nothing | **Setup wizard** — interactive config builder |
    | `--classic` flag | Force Classic regardless of goal text |
    | `--auto` flag | Force Orchestrator regardless of goal text |
    
    Print a banner on every invocation: `[autoresearch] mode: classic | orchestrator | wizard`.
    
    ## Subcommands
    
    | Command | Does | Default Iterations |
    |---|---|---|
    | `$autoresearch` | Iterate against a metric: modify → verify → keep/discard | 25 |
    | `$autoresearch plan` | Convert a goal into validated Scope, Metric, Verify config | N/A |
    | `$autoresearch debug` | Hunt bugs: hypothesize → test → falsify → repeat | 15 |
    | `$autoresearch fix` | Crush errors one-by-one until zero remain | 20 |
    | `$autoresearch security` | STRIDE + OWASP audit with red-team personas | 15 |
    | `$autoresearch ship` | Ship through 8 phases: checklist → dry-run → deploy → verify | N/A |
    | `$autoresearch scenario` | Generate edge cases across 12 dimensions | 20 |
    | `$autoresearch predict` | 5 expert personas debate before implementation | N/A |
    | `$autoresearch learn` | Scout codebase → generate docs or wiki → validate → fix loop | 10 |
    | `$autoresearch reason` | Adversarial debate with blind judges until convergence | 8 |
    | `$autoresearch probe` | 8 personas interrogate requirements until saturation | 15 |
    | `$autoresearch improve` | Research ICP challenges, discover improvements, generate PRDs | 15 |
    | `$autoresearch evals` | Analyze iteration results: trends, plateaus, regressions | N/A |
    | `$autoresearch regression` | Regression stability gate: baseline vs candidate, verdict STABLE/UNSTABLE | N/A |
    
    ## Universal Flags
    
    | Flag | Applies To | Purpose |
    |---|---|---|
    | `Iterations: N` | All looping | Set iteration count |
    | `Iterations: unlimited` | All looping | Opt-in unbounded |
    | `--evals` | All looping | Mid-loop checkpoints + final summary |
    | `--evals-interval N` | All looping | Override checkpoint frequency |
    | `--chain <targets>` | All | Sequential handoff after completion |
    | `--<subcommand>` | All | Shorthand for `--chain <subcommand>` |
    | `--dry-run` | Orchestrator | Print derived config + planned pipeline; no execution |
    | `--max-cycles N` | Orchestrator | Hard ceiling on orchestration cycles (default 50) |
    | `--classic` | Bare `$autoresearch` | Force Classic metric-loop mode |
    | `--auto` | Bare `$autoresearch` | Force Orchestrator mode |
    
    ## Orchestrator
    
    Activated when a plain-language goal is given without `Metric:`/`Verify:`. Classifies the goal into a **Goal archetype** — see `references/orchestrator-routing.md` for the archetype table and router decision table.
    
    Resolve every `scripts/...` path below relative to this installed skill directory, never relative to the caller's working directory.
    
    **Two modes based on archetype:**
    - **Orchestration loop** — predicate-bearing archetypes (ship-ready, optimize-metric, fix-broken, harden, build-feature, explore). Goal has a mechanical Success predicate; the loop runs until that predicate is met.
    - **Single-pass dispatch** — subjective/terminal archetypes (document, what-to-build, decide-design). Routes once to the fitting subcommand (learn / improve / reason), lets it self-terminate, then reports. No loop, no Plateau, no ship gate.
    
    ### Orchestration Loop Steps
    
    Backed by `scripts/orchestrate.sh` (deterministic seam — all routing logic lives there). Subcommands exposed: `classify`, `next-hop`, `units`, `plateau`, `screen-cmd`, `verdict`, `validate-state`, `screen-state-predicate`.
    
    1. **Classify** — `scripts/orchestrate.sh classify "<goal>"` → archetype label + mode.
    2. **Derive predicate** — reuse `plan` logic to produce a concrete Success predicate: exact shell command + expected output. For `optimize-metric`, run the full plan/wizard derivation internally.
    3. **Confirm** — ONE `request_user_input` showing: archetype, mode, concrete predicate (command + expected output), terminal choice (stop-at-verified vs proceed-to-ship). Misclassifications are caught here, not mid-run.
    4. **Round-0 dry-run** — prove the predicate command runs and returns a value; safety-screen every derived command via `screen-cmd`; print projected cycle budget. Stop here if `--dry-run`.
    5. **Loop** until predicate satisfied:
       a. Assess state via cheap signals (last `handoff.json`, regression verdict, error count) + affected-test verify.
       b. `scripts/orchestrate.sh next-hop orchestrator-state.json` → next subcommand.
       c. Run subcommand (its own bounded inner loop).
       d. Record per-hop outcome ∈ {progressed, no-op, failed, blocked}.
       e. Fold hop's `handoff.json` into `orchestrator-state.json`.
       f. `scripts/orchestrate.sh units` → recompute **Units remaining**.
    6. **Stop conditions** (checked after each hop):
       - Predicate met → ship gate (only if ship is in the pipeline) else `CONVERGED`.
       - `scripts/orchestrate.sh plateau orchestrator-state.json` → true → stop + report `PLATEAU`.
       - Cycles > ceiling (default 50, override `--max-cycles N`) → stop + report `CEILING`.
       - Hop outcome `blocked`/`failed` with no alternative route → checkpoint + stop + report `BLOCKED`.
    
    ### Orchestrator State
    
    `orchestrator-state.json` — orchestrator-owned, additive. Tracks: goal, archetype, predicate, terminal-choice, `units_remaining` history, cycle count, per-hop pipeline log with outcomes, current incumbent. Each hop's `handoff.json` is unchanged (single-hop bridge); the orchestrator reads it and folds it in. Two clearly-owned state objects, no overlap.
    
    ### Orchestrator Safety Invariants
    
    - **Never auto-approve ship/deploy/push.** The orchestrator never passes `--auto` to `ship`; deploy always requires explicit user approval.
    - **Data-migration behind anchored DB-URL allowlist.** Reuses regression's allowlist — host must be `localhost`/`127.0.0.1`/container hostname, or database name carries `_test`/`_ci` suffix. Bare substring match does not qualify. Anything else refused.
    - **screen-cmd on every derived command** — run before the loop starts AND on every command read from a persisted state file on resume. Persisted commands are never trusted; resume re-screens the pinned predicate via `screen-state-predicate` and refuses on `refuse`.
    - **No un-screened commands mid-loop.** The autonomous loop cannot introduce new shell commands that bypass `screen-cmd`.
    - **Predicate pinned, not re-derived.** Round-0 writes the derived Success predicate verbatim into `orchestrator-state.json`; every cycle and every resume reuses that exact string so "done" is reproducible across runs.
    - **Validate the ledger before routing.** `validate-state` gates `orchestrator-state.json` (required fields + coarse types); a malformed ledger is not trusted to route from.
    - **Independent verify before convergence.** High-impact changes accepted on the working signal set `pending_verify`; `next-hop` routes to a `verify` hop (held-out / adversarial check) before `DONE` or ship. The verify hop never auto-approves ship.
    - **Unknown-units cycles excluded from Plateau counter.** A cycle where `units` returns `unknown` (e.g. runner crash) is not counted as zero-progress; repeated `unknown` routes to `BLOCKED`.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.