Claude Skill

autoresearch

Autonomous iteration loop: modify, verify, keep/discard against any metric

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download mxyhi-ok-skills-autoresearch-7933c15.zip · 56 KB
mxyhi/ok-skills 490 46 forks Apache-2.0 Updated 5d ago
Part of mxyhi/ok-skills — 37 skills

Install

skills CLI npx skills add https://github.com/mxyhi/ok-skills/tree/main/autoresearch
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mxyhi-ok-skills@llmmart
Git git clone https://github.com/mxyhi/ok-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mxyhi/ok-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Autoresearch — Autonomous Goal-directed Iteration

Safety Invariants (all subcommands)

  • Never push, publish, or deploy without explicit user approval.
  • Bounded by default. Override with Iterations: unlimited.
  • All results logged to autoresearch/{subcommand}-{YYMMDD}-{HHMM}/ directory.
  • Chain handoff via handoff.json. Evals reads *-results.tsv.

Dispatch (bare $autoresearch)

Parse the invocation in this order:

Condition Mode
Metric: or Verify: present Classic — existing metric loop, unchanged
Free-form natural-language goal, no metric/verify Orchestrator — see Orchestrator section
Nothing Setup wizard — interactive config builder
--classic flag Force Classic regardless of goal text
--auto flag Force Orchestrator regardless of goal text

Print a banner on every invocation: [autoresearch] mode: classic | orchestrator | wizard.

Subcommands

Command Does Default Iterations
$autoresearch Iterate against a metric: modify → verify → keep/discard 25
$autoresearch plan Convert a goal into validated Scope, Metric, Verify config N/A
$autoresearch debug Hunt bugs: hypothesize → test → falsify → repeat 15
$autoresearch fix Crush errors one-by-one until zero remain 20
$autoresearch security STRIDE + OWASP audit with red-team personas 15
$autoresearch ship Ship through 8 phases: checklist → dry-run → deploy → verify N/A
$autoresearch scenario Generate edge cases across 12 dimensions 20
$autoresearch predict 5 expert personas debate before implementation N/A
$autoresearch learn Scout codebase → generate docs or wiki → validate → fix loop 10
$autoresearch reason Adversarial debate with blind judges until convergence 8
$autoresearch probe 8 personas interrogate requirements until saturation 15
$autoresearch improve Research ICP challenges, discover improvements, generate PRDs 15
$autoresearch evals Analyze iteration results: trends, plateaus, regressions N/A
$autoresearch regression Regression stability gate: baseline vs candidate, verdict STABLE/UNSTABLE N/A

Universal Flags

Flag Applies To Purpose
Iterations: N All looping Set iteration count
Iterations: unlimited All looping Opt-in unbounded
--evals All looping Mid-loop checkpoints + final summary
--evals-interval N All looping Override checkpoint frequency
--chain <targets> All Sequential handoff after completion
--<subcommand> All Shorthand for --chain <subcommand>
--dry-run Orchestrator Print derived config + planned pipeline; no execution
--max-cycles N Orchestrator Hard ceiling on orchestration cycles (default 50)
--classic Bare $autoresearch Force Classic metric-loop mode
--auto Bare $autoresearch Force Orchestrator mode

Orchestrator

Activated when a plain-language goal is given without Metric:/Verify:. Classifies the goal into a Goal archetype — see references/orchestrator-routing.md for the archetype table and router decision table.

Resolve every scripts/... path below relative to this installed skill directory, never relative to the caller's working directory.

Two modes based on archetype:

  • Orchestration loop — predicate-bearing archetypes (ship-ready, optimize-metric, fix-broken, harden, build-feature, explore). Goal has a mechanical Success predicate; the loop runs until that predicate is met.
  • Single-pass dispatch — subjective/terminal archetypes (document, what-to-build, decide-design). Routes once to the fitting subcommand (learn / improve / reason), lets it self-terminate, then reports. No loop, no Plateau, no ship gate.

Orchestration Loop Steps

Backed by scripts/orchestrate.sh (deterministic seam — all routing logic lives there). Subcommands exposed: classify, next-hop, units, plateau, screen-cmd, verdict, validate-state, screen-state-predicate.

  1. Classify — scripts/orchestrate.sh classify "<goal>" → archetype label + mode.
  2. Derive predicate — reuse plan logic to produce a concrete Success predicate: exact shell command + expected output. For optimize-metric, run the full plan/wizard derivation internally.
  3. Confirm — ONE request_user_input showing: archetype, mode, concrete predicate (command + expected output), terminal choice (stop-at-verified vs proceed-to-ship). Misclassifications are caught here, not mid-run.
  4. Round-0 dry-run — prove the predicate command runs and returns a value; safety-screen every derived command via screen-cmd; print projected cycle budget. Stop here if --dry-run.
  5. Loop until predicate satisfied:
    1. Assess state via cheap signals (last handoff.json, regression verdict, error count) + affected-test verify.
    2. scripts/orchestrate.sh next-hop orchestrator-state.json → next subcommand.
    3. Run subcommand (its own bounded inner loop).
    4. Record per-hop outcome ∈ {progressed, no-op, failed, blocked}.
    5. Fold hop's handoff.json into orchestrator-state.json.
    6. scripts/orchestrate.sh units → recompute Units remaining.
  6. Stop conditions (checked after each hop):
    • Predicate met → ship gate (only if ship is in the pipeline) else CONVERGED.
    • scripts/orchestrate.sh plateau orchestrator-state.json → true → stop + report PLATEAU.
    • Cycles > ceiling (default 50, override --max-cycles N) → stop + report CEILING.
    • Hop outcome blocked/failed with no alternative route → checkpoint + stop + report BLOCKED.

Orchestrator State

orchestrator-state.json — orchestrator-owned, additive. Tracks: goal, archetype, predicate, terminal-choice, units_remaining history, cycle count, per-hop pipeline log with outcomes, current incumbent. Each hop's handoff.json is unchanged (single-hop bridge); the orchestrator reads it and folds it in. Two clearly-owned state objects, no overlap.

Orchestrator Safety Invariants

  • Never auto-approve ship/deploy/push. The orchestrator never passes --auto to ship; deploy always requires explicit user approval.
  • Data-migration behind anchored DB-URL allowlist. Reuses regression's allowlist — host must be localhost/127.0.0.1/container hostname, or database name carries _test/_ci suffix. Bare substring match does not qualify. Anything else refused.
  • screen-cmd on every derived command — run before the loop starts AND on every command read from a persisted state file on resume. Persisted commands are never trusted; resume re-screens the pinned predicate via screen-state-predicate and refuses on refuse.
  • No un-screened commands mid-loop. The autonomous loop cannot introduce new shell commands that bypass screen-cmd.
  • Predicate pinned, not re-derived. Round-0 writes the derived Success predicate verbatim into orchestrator-state.json; every cycle and every resume reuses that exact string so "done" is reproducible across runs.
  • Validate the ledger before routing. validate-state gates orchestrator-state.json (required fields + coarse types); a malformed ledger is not trusted to route from.
  • Independent verify before convergence. High-impact changes accepted on the working signal set pending_verify; next-hop routes to a verify hop (held-out / adversarial check) before DONE or ship. The verify hop never auto-approves ship.
  • Unknown-units cycles excluded from Plateau counter. A cycle where units returns unknown (e.g. runner crash) is not counted as zero-progress; repeated unknown routes to BLOCKED.
Files (ok-skills)
  • agents
    • openai.yaml 250 B
      interface:
        display_name: "Autoresearch"
        short_description: "Autonomous goal-directed iteration engine"
        brand_color: "#7C3AED"
        default_prompt: "Set a goal, define a metric, let Codex loop until done"
      
      policy:
        allow_implicit_invocation: true
      
  • references
    • orchestrator-routing.md 6.8 KB
      # Orchestrator Routing
      
      ## Goal Archetypes
      
      | Archetype | Trigger Keywords | Mode | Preset Pipeline |
      |---|---|---|---|
      | `ship-ready` | ship, release, deploy, publish, production-ready, merge | loop | probe, debug, fix, regression, ship |
      | `optimize-metric` | improve, optimize, increase, reduce, faster, smaller, coverage, score | loop | plan, (classic loop), evals |
      | `fix-broken` | fix, broken, failing, error, crash, bug, can't run, tests fail | loop | debug, fix, regression |
      | `harden` | security, vulnerability, audit, OWASP, CVE, harden, lock down | loop | security, fix, security |
      | `build-feature` | build, add, implement, create, new feature, acceptance test | loop | (acceptance-test derive), debug, fix, regression |
      | `explore` | understand, explore, investigate, what does, how does, edge cases | loop | probe, scenario, plan |
      | `document` | document, wiki, generate docs, explain codebase, write guide | dispatch | learn |
      | `what-to-build` | what should I build, ideas, improvements, PRD, roadmap | dispatch | improve |
      | `decide-design` | which approach, compare options, design decision, architecture choice | dispatch | reason |
      
      Keyword matching is fuzzy — partial matches and synonyms qualify. When a goal matches multiple archetypes, prefer the more specific one (fix-broken over explore; ship-ready over fix-broken if "ship" is explicit). When ambiguous, show the top two candidates in the upfront confirm and let the user choose.
      
      ## Router Decision Table
      
      The `next-hop` subcommand of `scripts/orchestrate.sh` reads `orchestrator-state.json` and applies these rules in order. First match wins.
      
      | State Signal | Source | Next Hop |
      |---|---|---|
      | `errors > 0` in last handoff | handoff.json `findings` | `fix` |
      | regression verdict `UNSTABLE` | handoff.json `verdict` | `regression` |
      | `untested_gaps` flagged | handoff.json or units output | `debug` |
      | `pending_verify` true | orchestrator-state.json | `verify` (fresh independent acceptance check) |
      | predicate met | Success predicate command exit/output | `DONE` (exit loop) |
      | hop outcome `blocked` or `failed`, no retry route | orchestrator-state.json | `BLOCKED` (checkpoint + stop) |
      | plateau detected | `scripts/orchestrate.sh plateau` | `PLATEAU` (stop + report) |
      | archetype pipeline has remaining steps | preset pipeline sequence | next preset step |
      | all preset steps exhausted, predicate not met | — | `regression` (convergence re-check) |
      
      State signals are cheap reads — last `handoff.json` plus the regression verdict field and error count. No re-run of the full suite just to route.
      
      ## Independent Verify & Overfit Guard
      
      The orchestrator must not optimize and accept against the same signal — that lets a
      change game its own metric. For `optimize-metric` and `build-feature`, the acceptance
      check runs on a **held-out** set (a fresh scenario set or holdout assertions), separate
      from the `units` signal used to choose the change. When a high-impact change is accepted
      on the working signal, the orchestrator sets `pending_verify` in `orchestrator-state.json`;
      `next-hop` then routes to a **verify** hop (dispatched to `reason` or `predict` as an
      independent adversarial check) before declaring `DONE` or shipping. The verify hop is
      advisory input to convergence — it never auto-approves ship, which stays human-gated.
      
      ## Two-Mode Split
      
      **Orchestration loop** — used when the goal has an external, mechanical Success predicate: a shell command that returns a value the orchestrator can compare across cycles. Progress is objective (Units remaining falls), plateau is well-defined, and the loop terminates on convergence or a safety backstop. Archetypes: ship-ready, optimize-metric, fix-broken, harden, build-feature, explore.
      
      **Single-pass dispatch** — used when no mechanical predicate exists. The goal is subjective or the subcommand is internally-converging (reason runs its own adversarial loop) or a one-shot terminal emitter (learn, improve produce a document and stop). The orchestrator routes once, the subcommand self-terminates, and the orchestrator reports the result. No Units remaining, no Plateau counter, no ship gate. Archetypes: document, what-to-build, decide-design.
      
      The criterion is: "Can the orchestrator independently verify done without re-running the subcommand?" If yes → loop. If no → dispatch.
      
      ## Build-Feature: TDD Ladder
      
      The `build-feature` archetype has no pre-existing metric, so progress is reframed as `green-assertion-count` (monotone integer, higher-is-better). A change that turns a red sub-test green is kept; a change that regresses a green sub-test is reverted. A floor-guard prevents reverting scaffolding commits that compile and add no new failures but pass zero new tests. Large net-new scope (greenfield with no existing test suite) is detected and the orchestrator advises handing off to a dedicated build command rather than grinding cycles.
      
      ## Preset Pipelines (Reference)
      
      | Archetype | Step 1 | Step 2 | Step 3 | Step 4 | Step 5 |
      |---|---|---|---|---|---|
      | ship-ready | probe | debug | fix | regression | ship |
      | optimize-metric | plan | (classic loop) | holdout-verify | evals | — |
      | fix-broken | debug | fix | regression | — | — |
      | harden | security | fix | security | — | — |
      | build-feature | (acceptance-test derive) | debug | fix | regression | — |
      | explore | probe | scenario | plan | — | — |
      | document | learn | — | — | — | — |
      | what-to-build | improve | — | — | — | — |
      | decide-design | reason | — | — | — | — |
      
      Presets are starting pipelines. The router adapts per cycle from observed state — it may skip, repeat, or reorder steps based on the decision table above. The preset is a prior, not a fixed schedule.
      
      ## Glossary
      
      Terms used consistently across this file, SKILL.md, and orchestrator-state.json. Definitions live in CONTEXT.md.
      
      | Term | Short meaning |
      |---|---|
      | Goal archetype | Classification of the user's natural-language goal into one of the 9 categories above |
      | Success predicate | Exact shell command + expected output that defines "done" for Orchestration loop goals |
      | Units remaining | Scalar measure of open gaps (failing tests, errors, metric delta); lower-is-better; computed by `scripts/orchestrate.sh units` |
      | Plateau | Units remaining flat or worse for N consecutive computed cycles (default 5); oscillation that nets zero also qualifies |
      | Orchestration loop | The cycle-bounded assess→route→run→record loop used for predicate-bearing archetypes |
      | Single-pass dispatch | One-shot routing to a self-terminating subcommand; no loop, Plateau, ceiling, or ship gate |
      | Independent verify hop | A `verify` routing step (reason/predict) that checks an accepted high-impact change against a fresh signal before DONE/ship; gated by `pending_verify` |
      | Holdout-verify | Acceptance check run on a held-out set, separate from the `units` signal used to choose the change, to prevent overfitting the metric |
      
    • predict-personas.md 3.1 KB
      # Predict Personas
      
      ## Default Persona Set (5 personas)
      
      ### 1. Software Architect
      - **Focus:** System design, component boundaries, data flow, scalability
      - **Questions:** Does this design scale? Are boundaries clean? Is coupling minimized? Will this survive 10x growth?
      - **Evidence required:** file:line citations, dependency graphs, coupling metrics
      - **Red flags:** God classes, circular dependencies, leaky abstractions, shared mutable state
      
      ### 2. Security Analyst
      - **Focus:** Attack surfaces, auth/authz, data protection, injection vectors
      - **Questions:** Can this be exploited? Are trust boundaries enforced? Is data sanitized? Are secrets protected?
      - **Evidence required:** file:line citations, attack scenarios, data flow through trust boundaries
      - **Red flags:** Raw SQL, missing authz, hardcoded secrets, unsanitized user input
      
      ### 3. Performance Engineer
      - **Focus:** Latency, throughput, resource usage, algorithmic complexity
      - **Questions:** Will this be fast enough? What's the worst case? Where are the bottlenecks? Is caching effective?
      - **Evidence required:** file:line citations, complexity analysis, resource estimates
      - **Red flags:** N+1 queries, unbounded loops, missing indexes, synchronous I/O in hot paths
      
      ### 4. Reliability Engineer
      - **Focus:** Error handling, failure modes, observability, recovery
      - **Questions:** What happens when this fails? Can we detect it? Can we recover? Is it observable?
      - **Evidence required:** file:line citations, failure scenarios, recovery paths
      - **Red flags:** Swallowed errors, missing retries, no circuit breakers, silent failures
      
      ### 5. Devil's Advocate
      - **Focus:** Assumptions, edge cases, hidden complexity, maintainability
      - **Questions:** What assumptions are wrong? What's the simplest thing that breaks this? Is this over-engineered?
      - **Evidence required:** Concrete counter-examples, edge case scenarios
      - **Red flags:** Happy-path-only design, untested assumptions, complexity without justification
      
      ## Adversarial Persona Set (activated with --adversarial)
      
      Replace default personas with hostile reviewers:
      1. **The Breaker** — tries to crash/corrupt the system
      2. **The Cheater** — finds ways to bypass rules and abuse features
      3. **The Scaler** — imagines 1000x load and finds what breaks
      4. **The Newbie** — misuses every API and expects it to work
      5. **The Malicious Insider** — has credentials, wants to exfiltrate
      
      ## Debate Protocol
      
      1. Each persona analyzes independently (no shared context between personas)
      2. Findings reported with confidence score (0-100%)
      3. Cross-examination: personas challenge each other's findings
      4. Synthesizer aggregates, removes duplicates, resolves conflicts
      5. Anti-herd check: if all personas agree, synthesizer must find at least 1 counter-argument
      6. Final consensus: ranked findings with persona attribution
      
      ## Output Format
      
      Each persona produces:
      ```
      ### [Persona Name] — [N findings]
      | # | Finding | Severity | Confidence | File:Line | Recommendation |
      ```
      
      Synthesizer produces:
      ```
      ### Consensus — [N findings after dedup]
      | # | Finding | Severity | Agreement | Source Personas | Action |
      ```
      
    • reason-judge-protocol.md 3.4 KB
      # Reason Judge Protocol
      
      ## Adversarial Refinement Loop
      
      ```
      Round N:
        1. Author-A generates candidate (or incumbent from previous round)
        2. Critic attacks candidate — MUST find weaknesses (forced adversarial)
        3. Author-B reads task + candidate-A + critique → produces candidate-B
        4. Synthesizer reads A + B → produces hybrid candidate-AB
        5. Judge panel receives 3 candidates with randomized labels → picks winner
        6. Winner becomes incumbent for round N+1
      ```
      
      ## Agent Isolation Rules
      
      - Each agent (Author-A, Critic, Author-B, Synthesizer, Judges) runs COLD START
      - No shared session state between agents — prevents sycophancy
      - Agents receive ONLY: task description + relevant candidate(s) + critique
      - Judges receive candidates with randomized labels (Label-X, Label-Y, Label-Z)
      - Judges MUST compare and rank — "all are good" is not a valid verdict
      
      ## Critic Protocol
      
      The critic MUST:
      1. Identify at least 3 specific weaknesses in the candidate
      2. Provide concrete evidence for each weakness
      3. Suggest what a superior candidate would do differently
      4. Rate candidate on domain-specific criteria (1-10 scale)
      5. Never compliment the candidate — role is purely adversarial
      
      ## Judge Protocol
      
      Each judge receives:
      - Task description (identical for all judges)
      - 3 candidates with randomized labels (Label-X, Label-Y, Label-Z)
      - Evaluation criteria relevant to the domain
      
      Each judge MUST:
      1. Evaluate each candidate independently on all criteria
      2. Produce a ranking (1st, 2nd, 3rd) with reasoning
      3. Select a winner with one-paragraph justification
      4. Label randomization prevents position bias
      
      Verdict: majority vote. Tie → synthesized candidate (Label-Z) wins.
      
      ## Convergence Detection
      
      | Mode | Stop Condition |
      |---|---|
      | Convergent (default) | Same incumbent wins N consecutive rounds (default N=3) |
      | Creative | Never auto-stops; runs until iteration limit |
      | Debate | Same as convergent but no synthesis step |
      
      ## Oscillation Guard
      
      If the incumbent changes more than 5 times in the last 8 rounds → recommend early stop. The candidates are not converging — further rounds waste context.
      
      ## Domain-Specific Judge Criteria
      
      | Domain | Criteria |
      |---|---|
      | Software architecture | Scalability, maintainability, performance, security, simplicity |
      | Product strategy | Market fit, feasibility, differentiation, risk, timeline |
      | Business decision | ROI, risk, alignment, resource requirements, reversibility |
      | Security approach | Coverage, false positive rate, practicality, compliance |
      | Research hypothesis | Testability, novelty, evidence support, explanatory power |
      | Content/writing | Clarity, accuracy, engagement, completeness, actionability |
      
      ## Output Files
      
      | File | Content |
      |---|---|
      | `reason-results.tsv` | Per-round: round, candidate_label, judge_verdict, convergence_count, description |
      | `lineage.md` | Full history of all candidates + critiques + judge reasoning |
      | `summary.md` | Final winner, convergence trajectory, key insights |
      | `handoff.json` | Chain handoff with winner as primary finding |
      
      ## TSV Schema
      
      ```
      round	timestamp	candidate_label	judge_verdict	convergence_count	description
      1	2026-05-19T00:00:00Z	Candidate-A	winner	1	Event sourcing with CQRS
      2	2026-05-19T00:05:00Z	Candidate-AB	winner	1	Hybrid: event sourcing for writes, read projections
      3	2026-05-19T00:10:00Z	Candidate-AB	winner	2	Refined hybrid with materialized views
      4	2026-05-19T00:15:00Z	Candidate-AB	winner	3	CONVERGED — same approach refined
      ```
      
    • security-checklist.md 3.4 KB
      # Security Audit Checklist
      
      ## STRIDE Threat Categories
      
      | Category | Threat | Look For |
      |---|---|---|
      | Spoofing | Identity impersonation | Weak auth, token prediction, session fixation |
      | Tampering | Data modification | Unvalidated input, missing integrity checks, SQL injection |
      | Repudiation | Deniable actions | Missing audit logs, unsigned transactions |
      | Info Disclosure | Data leaks | Error messages with stack traces, verbose logging, exposed env vars |
      | Denial of Service | Availability attacks | Unbounded queries, missing rate limits, regex DoS |
      | Elevation of Privilege | Unauthorized access | Missing authz checks, IDOR, privilege escalation paths |
      
      ## OWASP Top 10 (2021) Checklist
      
      | # | Category | Key Checks |
      |---|---|---|
      | A01 | Broken Access Control | IDOR, missing function-level authz, CORS misconfiguration, path traversal |
      | A02 | Cryptographic Failures | Plaintext secrets, weak algorithms, missing TLS, hardcoded keys |
      | A03 | Injection | SQL, NoSQL, OS command, LDAP, XSS (stored/reflected/DOM) |
      | A04 | Insecure Design | Missing threat model, no rate limiting, no abuse prevention |
      | A05 | Security Misconfiguration | Default credentials, unnecessary features enabled, missing headers |
      | A06 | Vulnerable Components | Known CVEs in dependencies, outdated packages, unmaintained libs |
      | A07 | Auth Failures | Credential stuffing, brute force, weak passwords, missing MFA |
      | A08 | Data Integrity Failures | Unsigned updates, insecure deserialization, CI/CD poisoning |
      | A09 | Logging Failures | Missing security events, insufficient monitoring, no alerting |
      | A10 | SSRF | Unvalidated URLs, internal service access, cloud metadata exposure |
      
      ## Red-Team Personas
      
      | Persona | Focus | Mindset |
      |---|---|---|
      | Security Adversary | Auth, crypto, injection | External attacker with browser + Burp Suite |
      | Supply Chain Attacker | Dependencies, CI/CD, build pipeline | Compromise through third-party code |
      | Insider Threat | Data access, privilege abuse, exfiltration | Authenticated user with malicious intent |
      | Infrastructure Attacker | Network, cloud config, containers | Target infrastructure misconfigurations |
      
      ## Severity Classification
      
      | Severity | Criteria | Examples |
      |---|---|---|
      | Critical | Remote exploitation, no auth required, data breach | RCE, SQL injection, auth bypass |
      | High | Requires some access, significant impact | Stored XSS, IDOR, privilege escalation |
      | Medium | Limited impact or requires interaction | CSRF, reflected XSS, info disclosure |
      | Low | Minimal impact, informational | Missing headers, verbose errors |
      | Info | Best practice recommendation | Hardening suggestions, defense in depth |
      
      ## Composite Metric Formula
      
      ```
      score = (owasp_categories_tested / 10) * 50
            + (stride_categories_tested / 6) * 30
            + min(unique_findings, 20)
      ```
      
      Higher is better. Perfect score = 100 (all OWASP tested + all STRIDE tested + 20 findings).
      
      ## Coverage Tracking
      
      Print coverage summary every 5 iterations:
      ```
      OWASP: [A01✓ A02✓ A03✗ A04✗ A05✓ A06✗ A07✓ A08✗ A09✗ A10✗] 4/10
      STRIDE: [S✓ T✓ R✗ I✓ D✗ E✗] 3/6
      Score: 48.3 | Findings: 7
      ```
      
      ## Finding Format
      
      Every finding requires:
      1. **Title** — one-line summary
      2. **Severity** — Critical/High/Medium/Low/Info
      3. **OWASP** — A01-A10 category
      4. **STRIDE** — S/T/R/I/D/E category
      5. **Evidence** — file:line + attack scenario (no theoretical fluff)
      6. **Reproduction** — steps to trigger
      7. **Mitigation** — concrete fix recommendation
      
  • scripts
    • orchestrate.sh 18.5 KB
      #!/usr/bin/env bash
      # orchestrate.sh — deterministic seam for the autoresearch orchestrator loop.
      #
      #   classify   <goal-string>   → Goal archetype label (keyword heuristics)
      #   next-hop   <state.json>    → Next subcommand from router decision table
      #   units      <results.json>  → Units-remaining scalar (lower_is_better)
      #   plateau    <history.txt>   → Exit 0 if last N computed values are flat-or-worse
      #   screen-cmd <shell-string>  → "ok" exit 0 | "refuse" exit 1 safety gate
      #   verdict    <state.json>    → CONVERGED|PLATEAU|CEILING|BLOCKED + ship-gate
      #
      # All subcommands are pure and CI-usable via exit codes.
      set -uo pipefail
      
      # ---------------------------------------------------------------------------
      # classify: map a goal string to one of the 9 Goal archetype labels.
      # Priority order matters: higher-stakes archetypes checked first so that
      # "fix and add the broken feature" → fix-broken, not build-feature.
      # ---------------------------------------------------------------------------
      classify() {
        local goal="${1:?usage: classify <goal-string>}"
        local g
        g=$(printf '%s' "$goal" | tr '[:upper:]' '[:lower:]')
      
        # Security/hardening — above build because "secure" is higher stakes than "add"
        if printf '%s' "$g" | grep -qE '(secure|harden|vuln)'; then
          echo "harden"; return 0
        fi
      
        # Broken/bugfix
        if printf '%s' "$g" | grep -qE '(fix|bug|broken)'; then
          echo "fix-broken"; return 0
        fi
      
        # Ship/release/deploy
        if printf '%s' "$g" | grep -qE '(ship|release|deploy)'; then
          echo "ship-ready"; return 0
        fi
      
        # Product direction — requires a "what …" question so bare "next"/"build" in a
        # build-feature goal (e.g. "build the next-gen parser") doesn't mis-route here.
        if printf '%s' "$g" | grep -qE '(what.*build|what.*next)'; then
          echo "what-to-build"; return 0
        fi
      
        # Build/implement/add — "feature" alone is insufficient; any of these words qualify
        if printf '%s' "$g" | grep -qE '(build|implement|add)'; then
          echo "build-feature"; return 0
        fi
      
        # Metric optimization
        if printf '%s' "$g" | grep -qE '(faster|smaller|reduce|optimize|coverage)'; then
          echo "optimize-metric"; return 0
        fi
      
        # Documentation
        if printf '%s' "$g" | grep -qE '(document|docs)'; then
          echo "document"; return 0
        fi
      
        # Design decision — "should we" / "decide" / "approach"
        if printf '%s' "$g" | grep -qE '(should we|decide|approach)'; then
          echo "decide-design"; return 0
        fi
      
        # Default: open-ended investigation
        echo "explore"
      }
      
      # ---------------------------------------------------------------------------
      # next-hop: cheap router over fields in a state JSON file.
      # Decision order: errors → regression → untested gaps → ship/DONE.
      # ---------------------------------------------------------------------------
      next-hop() {
        local state_file="${1:?usage: next-hop <state.json>}"
        if [[ ! -f "$state_file" ]]; then
          echo "ERROR: missing state file" >&2; return 2
        fi
      
        # Parse with sed/grep — no jq dependency (score-regression.sh doesn't use jq)
        local errors regression gaps archetype
        errors=$(grep -o '"errors_remaining"[[:space:]]*:[[:space:]]*[0-9]*' "$state_file" \
                   | grep -o '[0-9]*$')
        regression=$(grep -o '"regression_verdict"[[:space:]]*:[[:space:]]*"[^"]*"' "$state_file" \
                       | grep -o '"[^"]*"$' | tr -d '"')
        gaps=$(grep -o '"untested_gaps"[[:space:]]*:[[:space:]]*[0-9]*' "$state_file" \
                 | grep -o '[0-9]*$')
        archetype=$(grep -o '"archetype"[[:space:]]*:[[:space:]]*"[^"]*"' "$state_file" \
                      | grep -o '"[^"]*"$' | tr -d '"')
      
        # Optional: pending_verify gates an independent acceptance check before DONE/ship.
        # Absent (or false) → routing is identical to prior behavior.
        local pending
        pending=$(grep -o '"pending_verify"[[:space:]]*:[[:space:]]*[a-z]*' "$state_file" \
                    | grep -o '[a-z]*$')
      
        # Guard: missing required fields
        if [[ -z "$errors" || -z "$regression" || -z "$gaps" ]]; then
          echo "ERROR: malformed state file" >&2; return 2
        fi
      
        if [[ "$errors" -gt 0 ]]; then
          echo "fix"; return 0
        fi
      
        if [[ "$regression" == "UNSTABLE" ]]; then
          echo "regression"; return 0
        fi
      
        if [[ "$gaps" -gt 0 ]]; then
          echo "debug"; return 0
        fi
      
        # Gaps clear but an accepted high-impact change still needs a fresh, independent
        # acceptance check (separate from the signal used to choose it) → verify first.
        if [[ "$pending" == "true" ]]; then
          echo "verify"; return 0
        fi
      
        # All clear: ship if archetype has ship in the pipeline, else DONE
        if [[ "$archetype" == "ship-ready" ]]; then
          echo "ship"; return 0
        fi
      
        echo "DONE"
      }
      
      # ---------------------------------------------------------------------------
      # units: compute Units-remaining scalar from a results JSON file.
      # Formula: failing_tests + open_hard_regressions + (metric_delta / metric_target)
      # Prints "unknown" and exits 2 when inputs are missing or uncomputable.
      # ---------------------------------------------------------------------------
      units() {
        local results_file="${1:?usage: units <results.json>}"
        if [[ ! -f "$results_file" ]]; then
          echo "unknown"; return 2
        fi
      
        local ft regressions delta target
        ft=$(grep -o '"failing_tests"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \
               | grep -o '[0-9.]*$')
        regressions=$(grep -o '"open_hard_regressions"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \
                        | grep -o '[0-9.]*$')
        delta=$(grep -o '"metric_delta"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \
                  | grep -o '[0-9.]*$')
        target=$(grep -o '"metric_target"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \
                   | grep -o '[0-9.]*$')
      
        if [[ -z "$ft" || -z "$regressions" || -z "$delta" || -z "$target" ]]; then
          echo "unknown"; return 2
        fi
      
        # Integer check: metric_target must be non-zero to avoid divide-by-zero
        if [[ "$target" == "0" || "$target" == "0.0" ]]; then
          echo "unknown"; return 2
        fi
      
        # awk handles floating point; strip trailing .0 for clean integer output
        awk -v ft="$ft" -v r="$regressions" -v d="$delta" -v t="$target" '
          BEGIN {
            val = ft + r + (d / t)
            # Strip unnecessary trailing zeros (e.g. 4.500 → 4.5, 0.000 → 0)
            if (val == int(val)) printf "%d\n", val
            else printf "%g\n", val
          }
        '
      }
      
      # ---------------------------------------------------------------------------
      # plateau: read newline list of unit values; determine if progress has stalled.
      # Skips interleaved "unknown" cycles; N=5 consecutive trailing unknowns = BLOCKED.
      # Exit 0 = plateau (no net progress); exit 1 = still improving; exit 3 = BLOCKED.
      # ---------------------------------------------------------------------------
      plateau() {
        local history_file="${1:?usage: plateau <history.txt>}"
        local n=5
      
        if [[ ! -f "$history_file" ]]; then
          echo "BLOCKED"; return 3
        fi
      
        awk -v n="$n" '
          {
            line = $0
            gsub(/^[[:space:]]+|[[:space:]]+$/, "", line)
            if (line == "unknown") { trailing_unknown++; next }            # crash/uncomputable cycle
            if (line ~ /^[0-9]/)   { vals[++count] = line + 0; trailing_unknown = 0 }
          }
          END {
            # A runner stuck emitting "unknown" must not read as progress: n consecutive
            # trailing unknowns (or no computed value at all) → BLOCKED, not "improving".
            if (trailing_unknown >= n) { print "BLOCKED"; exit 3 }
            if (count == 0)            { print "BLOCKED"; exit 3 }
      
            # Need at least n computed values before a plateau call.
            if (count < n) { exit 1 }
      
            # Net progress over the window = last value strictly below the first
            # (lower_is_better). Any oscillation that nets flat-or-worse is a plateau,
            # so a thrashing loop stops instead of running to the ceiling.
            start = count - n + 1
            if (vals[count] < vals[start]) { exit 1 }   # net improvement → still working
            exit 0                                       # flat or worse → plateau
          }
        ' "$history_file"
      }
      
      # ---------------------------------------------------------------------------
      # screen-cmd: safety gate for shell strings before execution.
      # Prints "ok" / "refuse". Anchored DB-host allowlist: only localhost,
      # 127.0.0.1, or a plain hostname (no dots) with a _test or _ci dbname suffix.
      # Bare substring "test" inside words like "latest" or "precision" must NOT qualify.
      # ---------------------------------------------------------------------------
      screen-cmd() {
        local cmd="${1:?usage: screen-cmd <shell-string>}"
      
        # rm with recursive AND force, in any flag arrangement: bundled (-rf/-Rf/-fr),
        # separate (-r -f), or long (--recursive --force). Both flags must be present.
        # The optional path prefix catches path-qualified invocations (/bin/rm, ./rm,
        # /usr/local/bin/rm) that a bare command-name anchor would miss.
        if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?rm([[:space:]]|$)'; then
          local rm_rec=0 rm_force=0
          printf '%s' "$cmd" | grep -qE -- '(^|[[:space:]])-[a-zA-Z]*[rR]|--recursive' && rm_rec=1
          printf '%s' "$cmd" | grep -qE -- '(^|[[:space:]])-[a-zA-Z]*[fF]|--force'     && rm_force=1
          if [[ "$rm_rec" -eq 1 && "$rm_force" -eq 1 ]]; then
            echo "refuse"; return 1
          fi
        fi
      
        # curl/wget piped to an interpreter (sh/bash/zsh/dash/fish/ksh/python/perl/ruby/
        # node/php), including a path-qualified one (| /bin/bash). Enumerated interpreters
        # rather than "refuse any curl pipe" so a legitimate derived predicate that pipes
        # curl output to a parser (jq/grep/awk) is not falsely refused.
        if printf '%s' "$cmd" | grep -qE '(curl|wget)[^|]*\|[[:space:]]*([^[:space:]]*/)?(sh|bash|zsh|dash|fish|ksh|python[0-9.]*|perl|ruby|node|php)([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # curl/wget routed through xargs into an interpreter. The xargs wrapper sidesteps the
        # direct pipe matcher above, so a remote payload still reaches a shell.
        if printf '%s' "$cmd" | grep -qE '(curl|wget)[^|]*\|.*xargs.*[[:space:]]([^[:space:]]*/)?(sh|bash|zsh|dash|ksh|python[0-9.]*|perl|ruby|node|php)([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # Output piped to netcat exfiltrates data off-host.
        if printf '%s' "$cmd" | grep -qE '\|[[:space:]]*([^[:space:]]*/)?(nc|ncat|netcat)([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # Raw block-device write — dd target or shell redirect onto a disk device wipes it.
        # Scoped to real device families (incl. SD/eMMC mmcblk, mdadm md, device-mapper dm-)
        # so dd/redirect to /dev/null or a regular file stays ok.
        if printf '%s' "$cmd" | grep -qE '(of=|>[[:space:]]*)/dev/(sd|hd|vd|nvme|disk|mapper|loop|xvd|mmcblk|md|dm-)'; then
          echo "refuse"; return 1
        fi
      
        # Filesystem format destroys everything on a partition. Optional path prefix catches a
        # path-qualified invocation (/sbin/mkfs.ext4) that a bare-name anchor would miss.
        if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?(mkfs|mke2fs)'; then
          echo "refuse"; return 1
        fi
      
        # find ... -delete mass-removes matched files. Both tokens required so a plain find
        # search (no -delete) is not refused; optional path prefix catches /usr/bin/find.
        if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?find([[:space:]]|$)' \
           && printf '%s' "$cmd" | grep -qE '[[:space:]]-delete([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # shred overwrites then unlinks — irrecoverable.
        if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?shred([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # truncate to zero size destroys file contents in place. Non-zero sizes are allowed.
        # Optional path prefix catches /usr/bin/truncate; size matcher covers -s 0, -s0,
        # --size 0, and --size=0.
        if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?truncate([[:space:]]|$)' \
           && printf '%s' "$cmd" | grep -qE '(-s[[:space:]]*0|--size[[:space:]]*=?[[:space:]]*0)([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # Recursive chmod to a zero mode locks an entire tree out of access. Scoped to the
        # zero lock-out (000/00/0 octal short forms) so ordinary recursive permission changes
        # are not refused; optional path prefix catches /bin/chmod.
        if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?chmod([[:space:]]|$)' \
           && printf '%s' "$cmd" | grep -qE '(-R|--recursive)([[:space:]]|$)' \
           && printf '%s' "$cmd" | grep -qE '(^|[[:space:]])(000|00|0)([[:space:]]|$)'; then
          echo "refuse"; return 1
        fi
      
        # Fork bomb pattern
        if printf '%s' "$cmd" | grep -qF ':(){ :|:'; then
          echo "refuse"; return 1
        fi
        if printf '%s' "$cmd" | grep -qE ':\(\)\{'; then
          echo "refuse"; return 1
        fi
      
        # AWS credential patterns (key IDs start with AKIA, secret keys are 40-char base64)
        if printf '%s' "$cmd" | grep -qE 'AKIA[0-9A-Z]{16}'; then
          echo "refuse"; return 1
        fi
      
        # PASSWORD= credential pattern
        if printf '%s' "$cmd" | grep -qE 'PASSWORD[[:space:]]*='; then
          echo "refuse"; return 1
        fi
      
        # Private key headers — pattern starts with dashes so pass -- to avoid flag misparse
        if printf '%s' "$cmd" | grep -qE -- 'BEGIN (RSA |EC |OPENSSH |DSA )?PRIVATE KEY'; then
          echo "refuse"; return 1
        fi
      
        # Database URL safety: extract host and dbname from postgres:// or postgresql:// URIs
        # Pattern: postgres(ql)://user:pass@HOST/DBNAME or postgres(ql)://HOST/DBNAME
        if printf '%s' "$cmd" | grep -qE 'postgres(ql)?://'; then
          # Extract the host portion (after @ or after ://)
          local db_host db_name
          db_host=$(printf '%s' "$cmd" \
            | grep -oE 'postgres(ql)?://[^[:space:]]+' \
            | sed -E 's|postgres(ql)?://([^@]+@)?([^/:]+)[:/].*|\3|')
          db_name=$(printf '%s' "$cmd" \
            | grep -oE 'postgres(ql)?://[^[:space:]]+' \
            | sed -E 's|postgres(ql)?://[^/]*/([^?[:space:]]+).*|\2|')
      
          # Allowed hosts: localhost, 127.0.0.1, or a single-label hostname (no dots = container)
          local host_ok=0
          if [[ "$db_host" == "localhost" || "$db_host" == "127.0.0.1" ]]; then
            host_ok=1
          elif printf '%s' "$db_host" | grep -qvE '\.'; then
            # No dots = plain container hostname → allowed
            host_ok=1
          fi
      
          if [[ "$host_ok" -eq 0 ]]; then
            # Non-allowlisted host: dbname must end with _test or _ci (anchored suffix, not substring)
            if printf '%s' "$db_name" | grep -qE '_test$|_ci$'; then
              echo "ok"; return 0
            fi
            echo "refuse"; return 1
          fi
        fi
      
        echo "ok"; return 0
      }
      
      # ---------------------------------------------------------------------------
      # verdict: synthesize a convergence verdict from state JSON.
      # Reads: units, plateau, ceiling fields. Prints verdict + ship-gate line.
      # Exit 0 = CONVERGED; exit 1 = not converged; exit 2 = error.
      # ---------------------------------------------------------------------------
      verdict() {
        local state_file="${1:?usage: verdict <state.json>}"
        if [[ ! -f "$state_file" ]]; then
          echo "BLOCKED"; echo "ship=no"; return 2
        fi
      
        local units_val plateau_val ceiling_val
        units_val=$(grep -o '"units"[[:space:]]*:[[:space:]]*[0-9.]*' "$state_file" \
                      | grep -o '[0-9.]*$')
        plateau_val=$(grep -o '"plateau"[[:space:]]*:[[:space:]]*[a-z]*' "$state_file" \
                        | grep -o '[a-z]*$')
        ceiling_val=$(grep -o '"ceiling"[[:space:]]*:[[:space:]]*[a-z]*' "$state_file" \
                        | grep -o '[a-z]*$')
      
        if [[ -z "$units_val" ]]; then
          echo "BLOCKED"; echo "ship=no"; return 2
        fi
      
        if [[ "$plateau_val" == "true" ]]; then
          echo "PLATEAU"; echo "ship=no"; return 1
        fi
      
        if [[ "$ceiling_val" == "true" ]]; then
          echo "CEILING"; echo "ship=no"; return 1
        fi
      
        # units==0 with no plateau/ceiling → converged
        if awk -v u="$units_val" 'BEGIN { exit (u == 0 ? 0 : 1) }'; then
          echo "CONVERGED"; echo "ship=yes"; return 0
        fi
      
        # units > 0, no plateau/ceiling signal yet → still running
        echo "BLOCKED"; echo "ship=no"; return 1
      }
      
      # ---------------------------------------------------------------------------
      # validate-state: schema gate for orchestrator-state.json. The ledger is the
      # loop's evidence trail; a malformed one must not be trusted to route from.
      # Prints "valid" exit 0 | "invalid" exit 2. Node is a checked installation
      # prerequisite, so use its JSON parser rather than approximating JSON with grep.
      # ---------------------------------------------------------------------------
      validate-state() {
        local state_file="${1:?usage: validate-state <state.json>}"
        if [[ ! -f "$state_file" ]]; then
          echo "invalid"; return 2
        fi
      
        if ! node -e '
          const fs = require("fs");
          const state = JSON.parse(fs.readFileSync(process.argv[1], "utf8"));
          const strings = ["goal", "archetype", "predicate", "terminal_choice"];
          if (!strings.every((key) => typeof state[key] === "string" && state[key].length > 0)) process.exit(1);
          if (!Number.isInteger(state.cycle) || state.cycle < 0) process.exit(1);
          if (!Array.isArray(state.units_remaining) || !Array.isArray(state.pipeline_log)) process.exit(1);
        ' "$state_file" 2>/dev/null; then
          echo "invalid"; return 2
        fi
      
        echo "valid"; return 0
      }
      
      # ---------------------------------------------------------------------------
      # screen-state-predicate: extract the pinned predicate from a persisted state
      # file and re-run it through screen-cmd. Persisted commands are never trusted —
      # a poisoned state file must not re-enter the loop with an unscreened command.
      # Delegates the verdict (ok/refuse + exit) to screen-cmd; "invalid" exit 2 when
      # the state has no pinned predicate.
      # ---------------------------------------------------------------------------
      screen-state-predicate() {
        local state_file="${1:?usage: screen-state-predicate <state.json>}"
        if [[ ! -f "$state_file" ]]; then
          echo "invalid"; return 2
        fi
      
        local pred
        if ! pred=$(node -e '
          const fs = require("fs");
          const state = JSON.parse(fs.readFileSync(process.argv[1], "utf8"));
          if (typeof state.predicate !== "string" || state.predicate.length === 0 || state.predicate.includes("\0")) process.exit(1);
          process.stdout.write(state.predicate);
        ' "$state_file" 2>/dev/null); then
          echo "invalid"; return 2
        fi
      
        screen-cmd "$pred"
      }
      
      case "${1:-}" in
        classify)               shift; classify               "$@" ;;
        next-hop)               shift; next-hop               "$@" ;;
        units)                  shift; units                  "$@" ;;
        plateau)                shift; plateau                "$@" ;;
        screen-cmd)             shift; screen-cmd             "$@" ;;
        verdict)                shift; verdict                "$@" ;;
        validate-state)         shift; validate-state         "$@" ;;
        screen-state-predicate) shift; screen-state-predicate "$@" ;;
        *) echo "usage: $0 {classify|next-hop|units|plateau|screen-cmd|verdict|validate-state|screen-state-predicate}" >&2; exit 64 ;;
      esac
      
    • score-regression.sh 7.5 KB
      #!/usr/bin/env bash
      # score-regression.sh — scoring backend for autoresearch:regression
      #
      #   rubric  [file]          → grep-rubric quality score of regression.md   → "SCORE: N"
      #   verdict <results.tsv>   → tiered stability verdict from a results TSV
      #
      # verdict logic:
      #   - any HARD row that is a green→red regression (classification=regression-eligible, regressed=true) → UNSTABLE
      #   - else weighted SCORE: per-dim worst subscore, weights renormalized over dims that ran,
      #     STABLE iff stability_score >= threshold (default 95)
      #   - classification in {pre-existing,new-coverage,baseline-unavailable,flaky} never gates
      #   - exit 0 STABLE / 1 UNSTABLE / 2 ERROR   (CI-usable)   · score math → stderr
      #
      # Overridable env: REG_THRESHOLD, REG_W_FLAKINESS, REG_W_PERFORMANCE, REG_W_RESOURCE, REG_W_VISUAL
      set -euo pipefail
      
      SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
      REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
      
      resolve_default_spec() {
        local candidate
        for candidate in \
          "$SCRIPT_DIR/../regression.md" \
          "$SCRIPT_DIR/../../../commands/autoresearch/regression.md" \
          "$SCRIPT_DIR/../../../commands/autoresearch_regression.md" \
          "$REPO_ROOT/.claude/commands/autoresearch/regression.md" \
          "$REPO_ROOT/claude-plugin/commands/autoresearch/regression.md"; do
          [[ -f "$candidate" ]] && { printf '%s\n' "$candidate"; return; }
        done
        printf '%s\n' "$SCRIPT_DIR/../regression.md"
      }
      
      SPEC_DEFAULT="$(resolve_default_spec)"
      
      REG_THRESHOLD="${REG_THRESHOLD:-95}"
      REG_W_FLAKINESS="${REG_W_FLAKINESS:-0.30}"
      REG_W_PERFORMANCE="${REG_W_PERFORMANCE:-0.30}"
      REG_W_RESOURCE="${REG_W_RESOURCE:-0.20}"
      REG_W_VISUAL="${REG_W_VISUAL:-0.20}"
      
      # ---------------------------------------------------------------------------
      # rubric: grep the protocol spec for required invariants/sections/flags.
      # Each pattern matched = +1. Prints "SCORE: N" only.
      # ---------------------------------------------------------------------------
      rubric() {
        local file="${1:-$SPEC_DEFAULT}"
        if [[ ! -f "$file" ]]; then echo "SCORE: 0"; return 0; fi
      
        local checks=(
          "classification"
          "green.{0,3}red"
          "regression.eligible|[^a-z]eligible"
          "pre-existing"
          "new-coverage"
          "baseline.unavailable|BASELINE_UNAVAILABLE"
          "functional"
          "api-contract"
          "data-migration"
          "integration-e2e"
          "flakiness"
          "performance"
          "resource"
          "visual"
          "HARD"
          "SCORE"
          "worktree"
          "--detach|detach"
          "submodule"
          "baseline.cache|--baseline-cache"
          "--select"
          "findRelatedTests|nx affected|affected"
          "Mann.?Whitney"
          "independent.process"
          "effect.size|median delta"
          "SSIM|maxDiffPixelRatio|pixel.ratio"
          "samples"
          "noise-band"
          "forward-only"
          "allowlist"
          "fix-cycle|--fix-cycles"
          "probe"
          "auto-skip"
          "--max-runs"
          "handoff"
          "verdict"
          "STABLE"
          "UNSTABLE"
        )
      
        local score=0 pat
        for pat in "${checks[@]}"; do
          if grep -qiE -- "$pat" "$file"; then score=$((score + 1)); fi
        done
        echo "SCORE: $score"
      }
      
      # ---------------------------------------------------------------------------
      # verdict: reduce a results TSV to STABLE|UNSTABLE + stability score.
      # ---------------------------------------------------------------------------
      verdict() {
        local tsv="${1:?usage: verdict <results.tsv>}"
        if [[ ! -f "$tsv" ]]; then
          echo "VERDICT: ERROR"; echo "score=0.0"; echo "blocking=missing-tsv"
          return 2
        fi
      
        awk -v FS='\t' \
            -v wF="$REG_W_FLAKINESS" -v wP="$REG_W_PERFORMANCE" \
            -v wR="$REG_W_RESOURCE"  -v wV="$REG_W_VISUAL" \
            -v thr="$REG_THRESHOLD" '
          /^#/      { next }              # comment (e.g. metric_direction)
          $1=="iteration" { next }        # header
          NF != 15  { invalid_schema=1; next }
          {
            dim=$3; tier=$5; cls=$6; regressed=$10; subv=$11+0;
            valid_dim = (dim=="functional" || dim=="api-contract" || dim=="data-migration" || dim=="integration-e2e" || dim=="flakiness" || dim=="performance" || dim=="resource" || dim=="visual-ui");
            expected_tier = (dim=="functional" || dim=="api-contract" || dim=="data-migration" || dim=="integration-e2e") ? "HARD" : "SCORE";
            if (!valid_dim || tier!=expected_tier || (regressed!="true" && regressed!="false") || $11 !~ /^([0-9]+([.][0-9]+)?|[.][0-9]+)$/ || subv<0 || subv>100) {
              invalid_schema=1;
              next;
            }
            if (cls!="regression-eligible" && cls!="pre-existing" && cls!="new-coverage" && cls!="baseline-unavailable" && cls!="flaky") {
              invalid_classification=cls;
              next;
            }
            if (cls!="baseline-unavailable") {
              nrows++;
              present[dim]=1;
            }
            if (tier=="HARD" && regressed=="true" && cls=="regression-eligible") hardset[dim]=1;
            if (tier=="SCORE" && cls!="baseline-unavailable") {
              if (!(dim in dmin) || subv < dmin[dim]) dmin[dim]=subv;   # per-dim worst case
            }
          }
          END {
            if (invalid_schema) {
              printf "VERDICT: ERROR\n";
              printf "score=0.00\n";
              printf "blocking=invalid-schema\n";
              exit 2;
            }
            if (invalid_classification!="") {
              printf "VERDICT: ERROR\n";
              printf "score=0.00\n";
              printf "blocking=invalid-classification\n";
              exit 2;
            }
            # No measurable data rows ran at all → nothing to gate on. Must NOT read as a
            # green ship signal: an empty/header-only TSV or all-dims-unavailable run is
            # advisory, not STABLE. Emit BASELINE_UNAVAILABLE + non-zero exit.
            if (nrows==0) {
              printf "VERDICT: BASELINE_UNAVAILABLE\n";
              printf "score=0.00\n";
              printf "blocking=no-dims-ran\n";
              printf "dims_ran=none\n";
              printf "dims_unavailable=functional,api-contract,data-migration,integration-e2e,flakiness,performance,resource,visual-ui\n";
              exit 2;
            }
      
            hb="";
            for (d in hardset) hb = hb (hb==""?"":",") d;
      
            wt["flakiness"]=wF; wt["performance"]=wP; wt["resource"]=wR; wt["visual-ui"]=wV;
            num=0; den=0;
            for (d in dmin) { w=(d in wt)?wt[d]:0; if (w>0){ num+=w*dmin[d]; den+=w; } }
            score = (den>0) ? num/den : 100;
      
            split("functional,api-contract,data-migration,integration-e2e,flakiness,performance,resource,visual-ui", reg, ",");
            ran=""; unavail="";
            for (i=1;i<=8;i++){
              if (reg[i] in present) ran = ran (ran==""?"":",") reg[i];
              else                   unavail = unavail (unavail==""?"":",") reg[i];
            }
      
            unstable = (hb!="") || (score < thr);
            verdict  = unstable ? "UNSTABLE" : "STABLE";
            if      (hb!="" && score<thr) blocking = hb ",score";
            else if (hb!="")              blocking = hb;
            else if (score<thr)           blocking = "score";
            else                          blocking = "none";
      
            # Display floors to 2 decimals (never rounds up): a true 94.9999 must print as
            # 94.99 — not 95.00 — so the shown number cannot read >= threshold while UNSTABLE.
            # The gate itself (above) compares full precision.
            disp = int(score*100)/100;
      
            printf "VERDICT: %s\n", verdict;
            printf "score=%.2f\n", disp;
            printf "blocking=%s\n", blocking;
            printf "dims_ran=%s\n", (ran==""?"none":ran);
            printf "dims_unavailable=%s\n", (unavail==""?"none":unavail);
      
            for (d in dmin) printf "  %-12s subscore=%.2f weight=%s\n", d, int(dmin[d]*100)/100, ((d in wt)?wt[d]:"0") > "/dev/stderr";
            printf "  stability_score=%.2f threshold=%s\n", disp, thr > "/dev/stderr";
      
            exit (unstable ? 1 : 0);
          }
        ' "$tsv"
      }
      
      case "${1:-}" in
        rubric)  shift; rubric  "$@" ;;
        verdict) shift; verdict "$@" ;;
        *) echo "usage: $0 {rubric [file] | verdict <results.tsv>}" >&2; exit 64 ;;
      esac
      
  • autoresearch.md 4.8 KB
    ---
    name: autoresearch
    description: "Autonomous iteration loop: modify, verify, keep/discard against any metric"
    argument-hint: "[Goal: <text>] [Scope: <glob>] [Metric: <text>] [Verify: <cmd>] [Guard: <cmd>] [Iterations: N] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY — do not deliberate before reading this protocol.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Goal:` — what to improve
    - `Scope:` or `--scope` — file globs
    - `Metric:` — what to measure
    - `Direction:` — higher_is_better (default) or lower_is_better
    - `Verify:` — shell command that outputs a number
    - `Guard:` — optional safety command (must always pass)
    - `Iterations:` or `--iterations` — integer N for bounded mode (default: 25). "unlimited" for unbounded.
    - `--evals` — enable mid-loop checkpoints
    - `--evals-interval N` — checkpoint frequency override
    - `--chain <targets>` — comma-separated downstream commands
    
    ## Setup (if required context missing)
    
    If Goal, Scope, Metric, or Verify missing → use request_user_input (single batched call):
      Q1 (Goal): "What do you want to improve?"
      Q2 (Scope): "Which files?" — suggest globs from project
      Q3 (Metric+Verify): "How to measure? Provide a shell command that outputs a number"
      Q4 (Guard): "Safety command that must always pass?" — options: test cmd, build cmd, skip
    If ALL provided inline → skip setup, proceed directly.
    
    ## Precondition Checks
    
    1. Verify git repo exists (`git rev-parse --git-dir`)
    2. Check clean working tree (`git status --porcelain`) — warn if dirty
    3. Check for stale lock files, detached HEAD
    4. If Guard set → run Guard to establish guard baseline
    5. Fail fast on any critical issue. Warn on non-critical.
    
    ## Verify Safety Screen
    
    Before first dry-run, screen Verify command for: rm -rf, fork bombs, curl|sh, embedded credentials, outbound writes. Block dangerous commands.
    
    ## Establish Baseline (Iteration 0)
    
    1. Run Verify command → extract numeric metric
    2. Record as iteration 0 in TSV: `0\t{timestamp}\t{commit}\t{metric}\t0.0\t{guard}\t-\tbaseline\tinitial state`
    3. Create output directory: `autoresearch/loop-{YYMMDD}-{HHMM}/`
    4. Write TSV header: `# metric_direction: {direction}\niteration\ttimestamp\tcommit\tmetric\tdelta\tguard\tguard-metric\tstatus\tdescription`
    
    ## Iteration Loop
    
    For each iteration (1 to max_iterations, or unbounded):
    
    ### Phase 1: Review (read git history as memory)
    - Read last 10-20 lines of results TSV
    - Run `git log --oneline -20` — see what worked/failed
    - If last iteration was "keep" → run `git diff HEAD~1` to see what improved metric
    - Identify: what worked, what failed, what's untried
    
    ### Phase 2: Modify
    - Based on review, make ONE focused change to improve the metric
    - Change must be atomic — one logical unit of work
    
    ### Phase 3: Commit
    - Stage and commit with `experiment: {description}` prefix
    - Record commit SHA
    
    ### Phase 4: Verify
    - Run Verify command → extract new metric value
    - Calculate delta from previous iteration
    - Metric improved (correct direction) → candidate for keep
    
    ### Phase 5: Guard (if configured)
    - Run Guard command. If fails → revert regardless of metric improvement
    
    ### Phase 6: Decide
    - **keep** — metric improved, guard passed → commit stays
    - **discard** — metric worsened → `git revert HEAD --no-edit`
    - **crash** — verify/guard command failed → `git revert HEAD --no-edit`
    - **no-op** — no change made this iteration
    - **hook-blocked** — git hook blocked the commit
    - **metric-error** — verify output not a valid number → `git revert HEAD --no-edit`
    
    ### Phase 7: Log
    Append row to TSV: iteration, timestamp, commit/-, metric, delta, guard status, guard-metric, status, description
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint analysis.
    
    ### Bounded Check
    If bounded: current_iteration >= max_iterations → exit loop, print summary.
    
    ## Summary (after loop ends)
    
    Print: total iterations, kept/discarded counts, starting metric → final metric, improvement %, top 3 most effective changes.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N.
    - Every {interval} iterations, pause and analyze current results TSV.
    - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nMetric: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down}\n{one-line recommendation}\n---`
    - If plateau 3+ checkpoints → recommend early stop.
    - At loop end → full evals summary to evals-summary.md in output directory.
    
    ## Chain Handoff
    
    After completion, write handoff.json to output directory: version "2.1.0", source "loop", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings[], config{goal, scope, metric, direction, verify}.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • debug.md 4 KB
    ---
    name: autoresearch:debug
    description: "Hunt bugs with scientific method: hypothesize, test, falsify, repeat"
    argument-hint: "[Scope: <glob>] [Symptom: <text>] [Iterations: N] [--fix] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Scope:` or `--scope` — file globs to investigate
    - `Symptom:` or `--symptom` — error message or behavior description
    - `Iterations:` or `--iterations` — default 15. "unlimited" for unbounded.
    - `--fix` — shorthand for `--chain fix`
    - `--severity` — filter: critical, high, medium, low
    - `--technique` — force specific technique
    - `--evals`, `--evals-interval N`, `--chain`
    
    ## Setup (if required context missing)
    
    If Scope and Symptom both missing:
    1. Auto-scan: run tests, lint, typecheck to detect existing failures
    2. request_user_input (single batch):
       Q1 (Issue): "What's the problem?" — hunt all bugs, specific error, failing tests, CI failure, performance
       Q2 (Scope): "Which files?" — suggested globs + entire codebase
       Q3 (Depth): "How deep?" — quick (5), standard (15), deep (30+), unlimited
       Q4 (After): "When bugs found?" — report only, find and fix (--chain fix), chain to other, ask each time
    If all provided → skip.
    
    ## Investigation Techniques
    
    | Technique | When to Use |
    |---|---|
    | Binary search | Know when it worked, find when it broke |
    | Differential | Compare working vs broken state |
    | Minimal reproduction | Simplify to smallest failing case |
    | Trace | Follow execution path through code |
    | Pattern search | Grep for known anti-patterns |
    | Working backwards | Start from error, trace to root cause |
    
    ## Establish Baseline (before loop)
    
    1. Auto-scan for failures if no symptom provided
    2. Create output directory: `autoresearch/debug-{YYMMDD}-{HHMM}/`
    3. TSV header: `# metric_direction: higher_is_better\niteration\ttimestamp\thypothesis\tstatus\ttechnique\tevidence\tfile_line`
    4. Metric = cumulative confirmed findings count
    
    ## Iteration Loop
    
    ### Phase 1: Review Context
    - Read results TSV (past findings)
    - Assess: what's been tested, what vectors remain
    - If no hypotheses left → early stop
    
    ### Phase 2: Hypothesize
    - Form ONE specific, falsifiable hypothesis
    - Format: "I hypothesize that {X} because {evidence}. Test by {Y}."
    - Hypothesis must be testable and different from all previous
    
    ### Phase 3: Investigate
    - Apply appropriate technique for this hypothesis
    - Read relevant code, run targeted tests, check logs
    - Collect evidence (file:line references required)
    
    ### Phase 4: Classify
    - **confirmed** — hypothesis correct, bug found with evidence
    - **disproven** — hypothesis wrong, evidence against it
    - **inconclusive** — can't prove or disprove, needs different approach
    
    ### Phase 5: Log
    Append to TSV: iteration, timestamp, hypothesis, status, technique, evidence, file_line
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint analysis.
    
    ### Bounded Check
    If bounded: current_iteration >= max_iterations → exit loop, print summary.
    
    ## Summary
    
    Print: total hypotheses tested, confirmed/disproven/inconclusive counts, all confirmed bugs with severity and file:line.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N.
    - Every {interval} iterations, pause and analyze current results TSV.
    - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nFindings: {confirmed} confirmed | Trend: {up/flat/down}\n{one-line recommendation}\n---`
    - If plateau 3+ checkpoints (no new confirmed) → recommend early stop.
    - At loop end → full evals summary to evals-summary.md in output directory.
    
    ## Chain Handoff
    
    After completion, write handoff.json to output directory: version "2.1.0", source "debug", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = confirmed bugs with severity + file:line, config{scope, symptom}.
    If --fix flag → chain to fix automatically.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • evals.md 5 KB
    ---
    name: autoresearch:evals
    description: "Analyze iteration results: trends, plateaus, regressions, recommendations"
    argument-hint: "[path/to/results.tsv] [--format text|json|md]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - Positional path to a specific TSV file
    - `--format` — output format: text (default console), json, md (markdown file)
    - `--compare <path>` — (v2.2.0 placeholder, not yet implemented)
    
    ## Input Discovery
    
    1. If path provided → use that TSV directly
    2. If no path → scan current directory + `autoresearch/*/` for `*-results.tsv` files
    3. If multiple found → request_user_input: "Which results to analyze?" — list found files
    4. If none found → request_user_input: "Provide path to results TSV"
    5. Also scan project root for v2.0.03 legacy TSV files (backward compat)
    
    ## Parse TSV
    
    1. Read line 1: extract `# metric_direction: higher_is_better|lower_is_better` comment
       - If missing → infer from column names (metric/error_count → guess, or ask user)
    2. Read line 2: header row → detect available columns
    3. Read remaining lines: data rows
    4. Handle missing `timestamp` column gracefully (v2.0.03 compat)
    
    ## Column Detection & Analysis
    
    Activate analysis based on columns present in header:
    
    | Column | Analysis |
    |---|---|
    | `metric` | Trend direction, plateau detection (3+ flat iterations), diminishing returns, biggest single-iteration jumps |
    | `delta` | Per-iteration efficiency, cumulative improvement, effort-to-gain ratio |
    | `status` | Keep/discard rate, crash frequency, success streaks, failure clusters, longest winning streak |
    | `guard` + `guard-metric` | Guard failure rate, metric-improved-but-guard-failed analysis |
    | `severity` | Severity distribution (critical/high/medium/low/info), critical discovery rate per iteration |
    | `hypothesis` + `status` | Confirmation rate, investigation efficiency, most productive techniques |
    | `commit` | File hotspot analysis (cross-ref with `git diff` for kept commits), change size correlation |
    | `technique` | Technique effectiveness ranking |
    | `dimension` | Dimension coverage completeness (X/12) |
    | `candidate_label` + `judge_verdict` | Convergence speed, oscillation count |
    | `error_type` | Error category distribution, fix rate per category |
    | `classification` | New vs extension vs duplicate ratio, saturation curve |
    | `convergence_count` | Convergence trajectory |
    
    Unknown columns: report presence but skip analysis. Forward-compatible with future subcommands.
    
    ## Report Structure
    
    ```
    ## Evals Summary — {subcommand} ({N} iterations)
    
    ### Key Metrics
    - Total iterations: N | Kept: X | Reverted: Y | Revert rate: Z%
    - Starting metric: A | Final metric: B | Improvement: C%
    
    ### Trend Analysis
    - Metric progression: [description of trajectory]
    - Plateau detected at iteration N (metric stable for M iterations)
    - Biggest win: iteration X (+delta, description)
    - Biggest loss: iteration Y (-delta, description)
    - Diminishing returns: [after iteration N, average delta dropped below threshold]
    
    ### Patterns
    - What types of changes succeeded: [extracted from descriptions of kept iterations]
    - What types of changes failed: [extracted from descriptions of discarded iterations]
    - File hotspots: [files changed most in kept iterations, if commit data available]
    - Technique effectiveness: [ranked by confirmation rate, if technique column present]
    
    ### Recommendation
    - [continue / stop / change strategy — based on trend, plateau, revert rate]
    - [specific actionable suggestion based on pattern analysis]
    ```
    
    ## Output
    
    - Console: structured report (30-50 lines)
    - If `--format md` → write `evals-summary.md` in same directory as input TSV
    - If `--format json` → write `evals-summary.json` with structured data
    
    ## Mid-Loop Checkpoint Protocol (for --evals flag in other commands)
    
    This section documents the checkpoint protocol that looping commands embed:
    
    - **Adaptive interval:** `floor(max_iterations / 3)`, minimum 1. Fixed 10 for unbounded. Override: `--evals-interval N`.
    - **Checkpoint format (5 lines max):**
      ```
      --- Eval Checkpoint (iterations {X}-{Y}) ---
      Metric: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down}
      {one-line recommendation}
      ---
      ```
    - **Early stop recommendation:** if plateau detected for 3+ consecutive checkpoints
    - **Final summary:** at loop end, produce full evals report to console + evals-summary.md
    
    ### Adaptive Interval Examples
    
    | Subcommand | Default Iterations | Interval | Checkpoints At |
    |---|---|---|---|
    | reason | 8 | 2 | 2, 4, 6, 8 |
    | learn | 10 | 3 | 3, 6, 9, final |
    | debug/security | 15 | 5 | 5, 10, 15 |
    | fix/scenario | 20 | 6 | 6, 12, 18, final |
    | core | 25 | 8 | 8, 16, 24, final |
    | unbounded | unlimited | 10 | every 10 |
    
    ## Backward Compatibility
    
    - v2.0.03 TSV files: column names preserved, `timestamp` absence handled gracefully
    - Fuzzy column matching: `metric_value` → `metric`, `error_count` → `metric`
    - Files in project root (not `autoresearch/` subdirectory) → discovered during scan
    - v2.0.03 status values all supported: baseline, keep, keep (reworked), discard, crash, no-op, hook-blocked, metric-error
    
  • fix.md 4.3 KB
    ---
    name: autoresearch:fix
    description: "Crush errors one-by-one until zero remain: tests, types, lint, build"
    argument-hint: "[Target: <cmd>] [Scope: <glob>] [Guard: <cmd>] [Iterations: N] [--evals] [--from-debug]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Target:` or `--target` — command that shows errors (e.g., `npm test`, `tsc --noEmit`)
    - `Scope:` or `--scope` — file globs to modify
    - `Guard:` or `--guard` — safety command (must always pass)
    - `Iterations:` or `--iterations` — default 20. "unlimited" for unbounded.
    - `--from-debug` — read handoff.json from previous debug run
    - `--category` — filter: test, type, lint, build
    - `--evals`, `--evals-interval N`, `--chain`
    
    ## Setup (if required context missing)
    
    If Target and Scope both missing:
    1. Auto-detect failures: run test suite, type checker, linter, build
    2. Present results via request_user_input (single batched call):
       Q1 (Fix What): "Found [N] test failures, [M] type errors, [K] lint errors. Fix what?" — everything, only tests, only types, only lint
       Q2 (Guard): "Safety command that must always pass?" — npm test, tsc, npm run build, skip
       Q3 (Scope): "Which files can I modify?" — suggested globs from error locations + all
       Q4 (Launch): "Ready?" — fix until zero, fix with limit, cancel
    If all provided → skip setup.
    If --from-debug → read handoff.json for scope and findings.
    
    ## Precondition Checks
    
    Verify: git repo exists, clean working tree, no lock files, no detached HEAD. Fail fast on critical issues.
    
    ## Establish Baseline (Iteration 0)
    
    1. Run Target command → count errors (metric = error count, direction = lower_is_better)
    2. Record baseline in TSV
    3. Create output directory: `autoresearch/fix-{YYMMDD}-{HHMM}/`
    4. TSV header: `# metric_direction: lower_is_better\niteration\ttimestamp\terror_type\terror_fixed\tcommit\tmetric\tdelta\tguard\tstatus\tdescription`
    
    ## Iteration Loop (until zero errors or max_iterations)
    
    ### Phase 1: Review
    - Read results TSV + git log
    - Run Target to get current error list
    - If error count == 0 → exit loop (SUCCESS)
    
    ### Phase 2: Prioritize
    Order: crash/fatal → test failures → type errors → lint → warnings.
    Within category: easiest first (single-file fixes before cross-file).
    
    ### Phase 3: Fix ONE Thing
    - Pick the highest-priority error
    - Make ONE focused fix (atomic — addresses exactly one error)
    - Record error type and which error was fixed
    
    ### Phase 4: Commit
    - Stage and commit: `experiment: fix {error_type} — {description}`
    
    ### Phase 5: Verify
    - Run Target → count errors → compute delta
    - Expected: error count decreased by 1 or more
    
    ### Phase 6: Guard
    - If Guard set → run Guard. If fails → revert.
    
    ### Phase 7: Decide
    - **keep** — error count decreased AND guard passes
    - **keep (reworked)** — fix needed adjustment, second attempt worked
    - **discard** — error count same/increased → `git revert HEAD --no-edit`
    - **crash** — target/guard command failed → revert
    - **hook-blocked** — git hook blocked the commit
    - **metric-error** — target output not parseable → revert
    
    ### Phase 8: Log
    Append row: iteration, timestamp, error_type, error_fixed, commit/-, metric (error count), delta, guard, status, description
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint analysis.
    
    ### Bounded Check
    If bounded: current_iteration >= max_iterations → exit loop, print summary.
    
    ## Summary
    
    Print: total errors fixed, remaining errors, error types distribution, fix success rate.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N.
    - Every {interval} iterations, pause and analyze current results TSV.
    - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nErrors: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down}\n{one-line recommendation}\n---`
    - If plateau 3+ checkpoints → recommend early stop.
    - At loop end → full evals summary to evals-summary.md in output directory.
    
    ## Chain Handoff
    
    After completion, write handoff.json to output directory: version "2.1.0", source "fix", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = unfixed errors, config{target, scope, guard}.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • improve.md 6 KB
    ---
    name: autoresearch:improve
    description: "Research ICP challenges, discover improvements, generate PRDs"
    argument-hint: "[Goal: <text>] [--icp <text>] [--discover] [--no-discover] [--seeds <categories>] [--depth shallow|standard|deep] [Iterations: N] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Goal:` — product area to improve (or full $ARGUMENTS if no keyword)
    - `--icp` or `ICP:` — ideal customer profile description
    - `--discover` — force inline codebase scan even when context exists
    - `--no-discover` — skip auto-discover, warn instead
    - `--seeds <categories>` — override default research category seeds
    - `--depth` — shallow (5 iterations), standard (15), deep (30)
    - `--features` — comma-separated feature names to pre-select for PRD generation
    - `Iterations:` or `--iterations` — default 15. "unlimited" for unbounded.
    - `--evals`, `--evals-interval N`
    
    If upstream `handoff.json` exists in CWD → read it. Map source findings to default seed categories:
    - probe → ICP challenges, UX & experience
    - predict → Competitor gaps, Revenue & growth
    - debug/security → Competitor gaps, ICP challenges
    - Override with `--seeds`.
    
    ## Setup (if Goal or ICP missing)
    
    request_user_input (single batch):
      Q1 (Goal): "What product area to improve?" — open text
      Q2 (ICP): "Who is your ideal customer?" — open text describing target buyer/user
      Q3 (Pain points): "Top 3 pain points your customers face?" — open text
      Q4 (Competitors): "Key competitors?" — open text, or "skip"
      Q5 (Depth): "How deep?" — shallow (5 iterations, quick scan), standard (15, recommended), deep (30+, exhaustive)
    If all provided inline → skip.
    
    ## Phase 1: Product Context
    
    Resolve product context (priority chain):
    1. Learn summary (`autoresearch/learn-*/summary.md`, most recent) → read it
    2. README.md (≥500 chars, non-boilerplate) → extract product description
    3. `package.json` / `pyproject.toml` / `Cargo.toml` description (≥10 chars) → use it
    4. If ALL above absent AND NOT `--no-discover` → auto-discover: scan 10 key files (manifest, routes, models, config), cap 1500 tokens
    5. If `--discover` → force scan regardless of above
    6. If nothing found → warn: "No product context. Run `$autoresearch learn --mode summarize` for better results."
    
    ## Phase 2: Research Loop
    
    Create output directory: `autoresearch/improve-{YYMMDD}-{HHMM}/`
    TSV header: `# metric_direction: higher_is_better`
    Columns: `iteration|timestamp|category|research_question|status|source|insight_problem|insight_mechanism|confidence|classification`
    
    **5 research categories:**
    1. ICP challenges — pain points, jobs-to-be-done, unmet needs
    2. Competitor gaps — weaknesses, missing features, technical differentiators
    3. Market trends — timing signals, emerging patterns, regulatory shifts
    4. UX & experience — interaction models, onboarding, retention mechanics
    5. Revenue & growth — pricing, acquisition, monetization, upsell/expansion
    
    **Iteration protocol:**
    - Reserve first 5 iterations: one per category (forced breadth)
    - Remaining iterations: target categories with richest signal
    - Per iteration: form research question → WebSearch → synthesize → normalize to canonical insight schema → classify (new/extension/duplicate) → tag confidence (HIGH: 3+ sources, MEDIUM: 2, LOW: 1) → cross-check against codebase → log
    - **Saturation:** net-new insights < 2 for 3 consecutive non-reserved iterations → SATURATED, exit loop
    - Hard ceiling (Iterations flag) as infinite-loop guard
    
    **Insight schema:** `{problem: 10-word canonical form, affected_persona: ICP segment, proposed_mechanism: how to address, expected_outcome: what success looks like}`
    **Classification:** New = novel {problem, persona} pair. Extension = same pair, different mechanism. Duplicate = same pair + mechanism → skip.
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint.
    Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nInsights: {total} (+{new}) | Categories: {covered}/5 | Saturation: {window}/3\n{recommendation}\n---`
    
    ## Phase 3: Feature Ranking + Selection
    
    1. **ICP binary gate** — filter insights not serving the stated ICP
    2. **3-tier bucketing** — Must-have / Nice-to-have / Moonshot
    3. **Pairwise ranking** within Must-have tier only (cap 7-10 items)
    4. **2-sentence rationale** per item citing research evidence
    5. **Confidence indicator** per item (HIGH / MEDIUM / LOW)
    
    Write `improvement-plan.md` with full tiered ranking.
    
    request_user_input (multi-select): present tiered list, user selects which features become PRDs.
    If `--features` provided → pre-select matching items, still show for confirmation.
    
    ## Phase 4: PRD Generation
    
    Per selected feature, write `prd-{feature-slug}.md`:
    - Top disclaimer: "Auto-generated from research findings. DECISION NEEDED items and LOW-confidence sections require your judgment."
    - Problem statement (from research evidence chain)
    - User stories (from ICP + persona data)
    - Requirements (functional + non-functional, MoSCoW from tier)
    - Acceptance criteria
    - Technical approach (from codebase context, framed as "suggested starting points")
    - Risks + confidence (evidence tiers: primary = codebase, secondary = web research)
    - Success metrics
    - `DECISION NEEDED` markers for unresolvable tradeoffs
    - `Open Questions` section
    
    Write `research-findings.md` — all insights with citations + confidence.
    Write `summary.md` — overview, research stats, category coverage, saturation status.
    
    ## Summary
    
    Print: total iterations, insights discovered (new/extension), categories covered, saturation status, PRDs generated, output directory path.
    
    ## Eval Summary (--evals flag)
    
    If --evals: write `evals-summary.md` to output directory with full analysis.
    
    ## Handoff
    
    Write `handoff.json`: version "2.1.0", source "improve", timestamp, status (COMPLETE|SATURATED|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = improvements with tier + confidence + prd_path, config{goal, icp, depth, categories_explored, insights_total, prds_generated}.
    Improve is a terminal emitter — no downstream chain invocation.
    
  • learn.md 7.9 KB
    ---
    name: autoresearch:learn
    description: "Scout codebase and auto-generate docs — or a navigable wiki knowledge base — with validation-fix loop"
    argument-hint: "[Mode: <init|update|check|summarize|wiki>] [Scope: <glob>] [Iterations: N] [--depth <level>] [--modules <list>] [--force] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Mode:` or `--mode` — init (create from scratch), update (refresh existing), check (validate), summarize (brief overview), wiki (navigable knowledge base)
    - `Scope:` or `--scope` — file globs to document
    - `Depth:` or `--depth` — overview, standard, comprehensive
    - `--file <path>` — specific file to document
    - `--scan` — force fresh codebase scout
    - `--topics` — comma-separated focus topics
    - `--modules <list>` — wiki mode: comma-separated module names/paths overriding auto-detection
    - `--force` — wiki mode: regenerate all pages from scratch, ignore existing manifest
    - `--no-fix` — validate only, don't auto-fix issues
    - `--format` — markdown (default), json, rst
    - `Iterations:` or `--iterations` — default 10. "unlimited" for unbounded.
    - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>`
    
    ## Setup (if Mode or Scope missing)
    
    request_user_input (single batch):
      Q1 (Mode): "What to do?" — init (generate docs), update (refresh), check (validate), summarize (overview), wiki (knowledge base)
      Q2 (Scope): "Which files?" — suggested globs + entire codebase
      Q3 (Depth): "How detailed?" — overview only, standard, comprehensive
      Q4 (Topics): "Focus on?" — architecture, API, database, testing, all
    If all provided → skip.
    
    ## Establish Baseline
    
    1. Scout codebase: file tree, imports/exports, existing docs
    2. Identify documentation gaps (undocumented files, outdated docs, missing READMEs)
    3. Create output directory: `autoresearch/learn-{YYMMDD}-{HHMM}/`
    4. TSV header: `# metric_direction: higher_is_better\niteration\ttimestamp\tfile_documented\tvalidation_status\tissues_found\tissues_fixed\tdescription`
    5. Metric = files with valid documentation (higher is better)
    
    ## Summarize Mode (no loop)
    
    If mode == summarize:
    - One-shot: scan codebase → produce structured summary
    - Write summary.md to output directory
    - Skip iteration loop entirely
    
    ## Wiki Mode (no per-file loop)
    
    If mode == wiki: reuse Scout (Phase 1) + Analyze output, then generate a navigable `wiki/` knowledge base. Skip the init/update/check loop. Metric = `pages_generated / pages_planned × 100` (from manifest); size target 300 lines/page.
    
    ### Module Discovery (priority order)
    1. `--modules` flag (explicit override, always wins; every path must resolve inside project root — reject escapes)
    2. Monorepo workspaces (`workspaces` in package.json, Cargo workspace members, `pnpm-workspace.yaml`)
    3. Per-directory project files (`pyproject.toml`, `Cargo.toml`, `go.mod`, `*.csproj`)
    4. Heuristic: dirs with 3+ source files (code extensions only — .ts/.py/.go/.rs/.java/.rb/.swift/.kt/.c/.cpp/.cs; tests count, config/markdown don't); nested dirs roll up to nearest module ancestor
    
    Cap 10 modules. If >10, group by top-level dir; if a group has >5 sub-modules, expand and take 10 largest by file count.
    
    ### Plan (write-ahead)
    1. `mkdir -p wiki/modules/`; append `wiki-manifest.json` to `.gitignore` if absent
    2. Write `wiki-manifest.json` BEFORE generating — `{version:"1", generated_at, generation_status:"in_progress", modules_detected:[…], pages_planned:N, pages:{ "wiki/architecture.md":{status:"pending",type:"architecture"}, "wiki/modules/<name>.md":{…"module"}, "wiki/glossary.md":{…}, "wiki/onboarding.md":{…}, "wiki/index.md":{…} }}`
    3. Write stub `wiki/index.md` listing every planned page as `[pending]` (navigation survives interruption)
    4. Resume: if valid manifest exists → `--force` deletes it and regenerates all, else skip `"generated"` pages and only do `"pending"`. Corrupted manifest (invalid JSON, or missing `version`/`pages`) without `--force` → error directing user to `--force`.
    
    ### Generate (priority-first, one agent call per page, bounded context)
    1. `architecture.md` — system overview from scout context. Up to 5 Mermaid diagrams, 3 types only (`graph TD`, `sequenceDiagram`, `classDiagram`); include one canonical example of each in the prompt; pick by signal.
    2. Module pages (alphabetical) — per-module agent gets: (a) file listing, (b) first 50 lines of ≤10 key files (entry points → largest → alphabetical), (c) Phase 2 overview. Required sections: Overview, Key Files; optional: Patterns/API/Dependencies/Getting Started.
    3. `glossary.md` — domain terms from class names, exports, types, comments; filter language keywords + stdlib; soft cap ~60-80, prioritize terms in 3+ files.
    4. `onboarding.md` — reading order, env setup, first-contribution workflow, gotchas. Sources: dir structure, README/docs, manifests, entry-point sampling, `git log --since='6 months ago'` directory frequency (skip with note if not a git repo or >10s).
    5. `index.md` — final pass: replace stub with real page descriptions + reading order.
    
    ### Per-page contract
    - `generated_by: autoresearch` in YAML frontmatter
    - ~300 lines/page (soft); Mermaid ≤15 nodes/diagram; forward-only cross-links, ≤10 per page
    
    ### Safety
    - **Secrets (2-layer):** (1) prompt instructs "summarize config, never include verbatim values from .env/credentials or strings matching key/secret/token/password; extract env var *names* not *values*"; (2) post-gen, `grep -rlE '(AKIA[0-9A-Z]{16}|sk-[a-zA-Z0-9]{20,}|ghp_[a-zA-Z0-9]{36}|password\s*[:=]\s*\S+|mongodb(\+srv)?://\S+|postgres(ql)?://\S+)' wiki/` and warn (non-blocking) in the report.
    - **Name collision:** before overwriting a page, check for `generated_by: autoresearch` frontmatter; if absent (user-created) skip with warning. `--force` overrides.
    
    ### Finish
    After each page is written, flip its manifest entry `pending`→`generated` (interrupt-safe). When all done, set `generation_status:"complete"`. Then run Phase 3 (Validate) with wiki path swap: replace `docs/` with `wiki/` (`ls wiki/*.md wiki/modules/*.md 2>/dev/null`), use maxLoc 300. Output: `✓ Wiki: [N] modules, [M] pages generated`.
    
    ## Iteration Loop (init/update/check modes)
    
    ### Phase 1: Scout
    - Scan for documentation gaps
    - Prioritize: no docs → outdated docs → incomplete docs
    - If no gaps remain → early stop (SUCCESS)
    
    ### Phase 2: Generate/Update
    - Pick highest-priority gap
    - Write or update documentation for ONE file/module
    - Follow project conventions for doc format and location
    
    ### Phase 3: Validate
    - Check generated docs against code: descriptions accurate? Examples valid? Links work?
    - Run doc linters if available
    - Record: validation_status (pass/fail), issues found
    
    ### Phase 4: Fix (unless --no-fix)
    - If validation finds issues → fix the doc
    - Commit clean doc: `docs: document {file/module}`
    
    ### Phase 5: Log
    Append to TSV: iteration, timestamp, file_documented, validation_status, issues_found, issues_fixed, description
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint.
    
    ### Bounded Check
    If bounded: current_iteration >= max_iterations → exit loop.
    
    ## Output
    
    - `learn-results.tsv`
    - `summary.md` — documentation overview
    - `validation-report.md` — issues found/fixed
    
    ## Summary
    
    Print: files documented, validation pass rate, issues found/fixed, remaining gaps.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded.
    - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nDocs written: {n} | Validation: {pass}/{total} | Gaps remaining: {m}\n{recommendation}\n---`
    - If 3+ checkpoints with no new docs → recommend early stop.
    - At loop end → full evals summary to evals-summary.md.
    
    ## Chain Handoff
    
    After completion, write handoff.json: version "2.1.0", source "learn", timestamp, status, results_tsv path, findings = documentation gaps remaining, config{mode, scope, depth}.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • LICENSE 1 KB · in bundle
  • plan.md 2.8 KB
    ---
    name: autoresearch:plan
    description: "Convert a goal into validated Scope, Metric, Direction, Verify config"
    argument-hint: "[Goal: <text>] [--chain <targets>]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Goal:` — text after keyword, or full $ARGUMENTS if no keyword
    - `--chain <targets>` — comma-separated downstream commands
    - `--<subcommand>` — chain shorthand
    
    Remaining text = goal description.
    
    ## Setup (if Goal missing)
    
    request_user_input (single batch):
      Q1 (Goal): "What do you want to achieve?" — open text
      Q2 (Type): "What kind of goal?" — improve a metric, fix errors, audit security, explore edge cases, document code, ship something
    If Goal provided → skip.
    
    ## Phase 1: Analyze Goal
    
    Parse the goal to determine:
    - Is it measurable? (metric-driven vs subjective)
    - What's the natural scope? (files, modules, entire codebase)
    - What subcommand fits best? (core loop, fix, debug, security, etc.)
    
    ## Phase 2: Derive Scope
    
    1. Scan project structure
    2. Identify files relevant to the goal
    3. Propose file globs
    4. If ambiguous → ask user to confirm
    
    ## Phase 3: Derive Metric + Direction
    
    For metric-driven goals:
    - Identify what to measure (test coverage, error count, bundle size, latency, etc.)
    - Determine direction: higher_is_better or lower_is_better
    - Propose metric name and description
    
    For subjective goals:
    - Suggest proxy metrics where possible
    - Or recommend $autoresearch reason for non-measurable goals
    
    ## Phase 4: Derive Verify Command
    
    1. Identify how to extract the metric as a number from a shell command
    2. Propose Verify command (e.g., `npm test -- --coverage | grep "All files" | awk '{print $10}'`)
    3. **Safety screen:** check proposed command for rm -rf, fork bombs, curl|sh, credentials
    4. Dry-run the Verify command → confirm it outputs a valid number
    5. If dry-run fails → adjust command and retry
    
    ## Phase 5: Derive Guard (optional)
    
    Propose a Guard command if applicable:
    - Test suite: `npm test` / `pytest` / `go test ./...`
    - Type check: `tsc --noEmit` / `mypy`
    - Build: `npm run build`
    - None if not applicable
    
    ## Phase 6: Suggest Iterations
    
    Based on goal complexity:
    - Simple metric improvement → 10-15
    - Moderate refactoring → 20-25
    - Complex multi-file changes → 30+
    - Recommend bounded default, mention `Iterations: unlimited` option
    
    ## Phase 7: Present Config
    
    Output a ready-to-run autoresearch config block:
    
    ```
    $autoresearch
    Goal: {derived goal}
    Scope: {derived globs}
    Metric: {derived metric}
    Direction: {higher_is_better|lower_is_better}
    Verify: {derived command}
    Guard: {derived guard or omit}
    Iterations: {suggested count}
    ```
    
    Ask user: "Run this config now, or adjust?"
    
    ## Chain Handoff
    
    If --chain set:
    - Write handoff.json: version "2.1.0", source "plan", timestamp, status COMPLETE, config = derived config block
    - Invoke next target with the derived config
    
  • predict.md 3.7 KB
    ---
    name: autoresearch:predict
    description: "5 expert personas debate proposed changes before implementation"
    argument-hint: "[Scope: <glob>] [Goal: <text>] [--depth shallow|standard|deep] [--adversarial] [--chain <targets>]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Scope:` or `--scope` — file globs to analyze
    - `Goal:` or `--goal` — focus area for analysis
    - `Depth:` or `--depth` — shallow (3 personas, 1 round), standard (5, 2), deep (8, 3)
    - `--personas N` — override persona count (3-8)
    - `--rounds N` — override debate rounds (1-3)
    - `--adversarial` — use hostile reviewer personas instead of default
    - `--budget N` — max findings across all personas (default 40)
    - `--fail-on <severity>` — CI gate: exit non-zero if findings at/above threshold
    - `--incremental` — reuse existing knowledge files, update only changed files
    - `--chain`, `--<subcommand>`
    
    Remaining text not matching flags = goal description.
    
    ## Setup (if Scope or Goal missing)
    
    request_user_input (single batch):
      Q1 (Scope): "Which files to analyze?" — suggested globs + entire codebase
      Q2 (Goal): "What should personas focus on?" — code quality, security, performance, architecture, all
      Q3 (Depth): "How deep?" — shallow (3 personas, 1 round), standard (5, 2 — recommended), deep (8, 3)
      Q4 (Chain): "After analysis, chain to?" — debug, security, fix, ship, scenario, no chain
    If all provided → skip.
    
    ## Phase 1: Reconnaissance
    
    Scan all in-scope files. Build structured knowledge:
    - File inventory with purpose annotations
    - Dependency graph (imports/exports)
    - API surface (routes, handlers, types)
    - Data flow (inputs → processing → outputs → storage)
    - Existing test coverage map
    
    ## Phase 2: Persona Generation
    
    Load `references/predict-personas.md` for persona definitions.
    
    **Default set (5):** Architect, Security Analyst, Performance Engineer, Reliability Engineer, Devil's Advocate.
    **Adversarial set (--adversarial):** Breaker, Cheater, Scaler, Newbie, Malicious Insider.
    
    Each persona receives: task description + codebase knowledge + their specific evaluation criteria.
    Personas are isolated — no shared context between them.
    
    ## Phase 3: Independent Analysis
    
    Each persona analyzes the codebase independently:
    - Read relevant code through their lens
    - Produce findings with: title, severity, confidence (0-100%), file:line, recommendation
    - Max findings per persona: budget / persona_count
    
    ## Phase 4: Debate (per round)
    
    For each debate round:
    1. Present all personas' findings to each other
    2. Each persona can: challenge findings, raise new issues, change confidence
    3. Cross-examination: personas must respond to challenges with evidence
    4. No persona can dismiss without counter-evidence
    
    ## Phase 5: Consensus
    
    Synthesizer aggregates all findings:
    1. Deduplicate (same file:line + same issue = merge, keep highest severity)
    2. Resolve conflicts (if personas disagree, note dissent)
    3. **Anti-herd check:** if all personas agree on everything, synthesizer MUST find at least 1 counter-argument
    4. Rank by: severity × average confidence × persona agreement count
    
    ## Phase 6: Report
    
    Create output directory: `autoresearch/predict-{YYMMDD}-{HHMM}/`
    
    Write:
    - `summary.md` — top findings, consensus view, risk assessment
    - `debate.md` — full persona analysis + debate transcript
    - Per-persona sections with individual findings
    
    Print to console: top 10 findings ranked by severity × confidence.
    
    ## Phase 7: CI Gate
    
    If `--fail-on` set: check findings against threshold. Exit non-zero if exceeded.
    
    ## Chain Handoff
    
    Write handoff.json: version "2.1.0", source "predict", timestamp, status (COMPLETE|ERROR), findings = consensus findings with severity + confidence + file:line, config{scope, goal, depth}.
    Invoke next target in --chain order.
    
  • probe.md 4.8 KB
    ---
    name: autoresearch:probe
    description: "8 personas interrogate requirements until constraints saturate"
    argument-hint: "[Topic: <text>] [Scope: <glob>] [--depth shallow|standard|deep] [--personas N] [--mode interactive|autonomous] [Iterations: N] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Topic:` — strip keyword, remaining text is topic (or full $ARGUMENTS if no keyword)
    - `Scope:` or `--scope` — file globs for codebase grounding
    - `Depth:` or `--depth` — shallow (5 rounds), standard (15), deep (30)
    - `--personas N` or `Personas:` — active persona count (3-8, default 6)
    - `--saturation-threshold N` — net-new constraints/round below which counts toward saturation (default 2)
    - `--mode` or `Mode:` — interactive (default, uses request_user_input) or autonomous (self-answers from codebase)
    - `--adversarial` — rotate hostile personas to front
    - `Iterations:` or `--iterations` — default 15 rounds. "unlimited" for unbounded.
    - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>`
    
    ## Setup (if Topic missing)
    
    request_user_input (single batch):
      Q1 (Topic): "What to probe?" — open text describing feature, requirement, or design
      Q2 (Scope): "Which files for context?" — suggested globs + entire codebase
      Q3 (Depth): "How deep?" — shallow (5 rounds), standard (15), deep (30), unlimited
      Q4 (Mode): "How to answer persona questions?" — interactive (you answer), autonomous (agent infers from code)
    If all provided → skip.
    
    ## 8 Personas
    
    | # | Persona | Focus |
    |---|---|---|
    | 1 | Domain Expert | Business rules, domain constraints, terminology |
    | 2 | End User | Usability, expectations, error recovery |
    | 3 | Skeptic | Assumptions that might be wrong |
    | 4 | Edge-Case Hunter | Boundary conditions, rare scenarios |
    | 5 | Ops Engineer | Deployment, monitoring, scaling, failure modes |
    | 6 | Security Reviewer | Attack vectors, data protection, auth |
    | 7 | Contradiction Finder | Conflicts between requirements |
    | 8 | Scope Guardian | Feature creep, unnecessary complexity |
    
    If --adversarial: rotate Skeptic + Contradiction Finder + Edge-Case Hunter to front.
    
    ## Phase 1: Seed
    
    - Parse topic into initial constraint set
    - Read codebase context (if --scope provided)
    - Initialize constraint registry (empty)
    
    ## Round Loop
    
    ### Phase 2: Persona Activation
    - Select 2-3 personas for this round (rotate through all 8)
    - Each persona generates 3-5 probing questions from their perspective
    
    ### Phase 3: Codebase Grounding
    - Check questions against existing code for evidence
    - Annotate questions with: relevant file:line, existing behavior, gaps
    
    ### Phase 4: Answer Capture
    - **Interactive mode:** present questions via request_user_input, collect answers
    - **Autonomous mode:** infer answers from codebase context, label confidence (high/medium/low)
    
    ### Phase 5: Constraint Extraction
    - Parse answers into atomic constraints
    - Each constraint: id, source persona, description, confidence, evidence
    - Deduplicate against existing registry
    
    ### Phase 6: Cross-Check
    - Check new constraints against existing for conflicts
    - Flag contradictions for resolution (interactive → ask user, autonomous → note uncertainty)
    
    ### Phase 7: Saturation Check
    - Count net-new constraints this round
    - If net-new < saturation_threshold for 3 consecutive rounds → SATURATED, exit loop
    - Track: total constraints, new this round, saturation window
    
    ### Phase 8: Log
    Append to output: round number, personas active, questions asked, constraints extracted, net-new count
    
    ### Eval Checkpoint
    If --evals: check if current_round % interval == 0 → run checkpoint.
    
    ### Bounded Check
    If bounded: current_round >= max_iterations → exit loop.
    
    ## Phase 9: Synthesize & Output
    
    Create output directory: `autoresearch/probe-{YYMMDD}-{HHMM}/`
    
    1. Write `constraints.md` — full constraint registry organized by category
    2. Write `conflicts.md` — unresolved contradictions
    3. Generate ready-to-run autoresearch config:
       - Derived Goal, Scope, Metric, Verify from constraints
       - Include as code block in summary.md
    
    Print: total rounds, constraints found, saturation status, unresolved conflicts.
    
    ## Summary
    
    Print: total rounds, total constraints, net-new trend, saturation status, top 5 most impactful constraints.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded.
    - Print: `--- Eval Checkpoint (rounds {X}-{Y}) ---\nConstraints: {total} (+{new}) | Saturation: {window_count}/3\n{recommendation}\n---`
    - If saturated 3+ checkpoints → recommend early stop.
    - At loop end → full evals summary to evals-summary.md.
    
    ## Chain Handoff
    
    Write handoff.json: version "2.1.0", source "probe", timestamp, status (COMPLETE|SATURATED|USER_INTERRUPT|BOUNDED|ERROR), findings = constraints, config = derived autoresearch config.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • reason.md 4.7 KB
    ---
    name: autoresearch:reason
    description: "Adversarial debate with blind judges until convergence"
    argument-hint: "[Task: <question>] [Domain: <type>] [--mode convergent|creative|debate] [--judges N] [Iterations: N] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Task:` — question, proposal, design, argument, or claim to refine
    - `Domain:` or `--domain` — software, product, business, security, research, content
    - `Mode:` or `--mode` — convergent (default), creative, debate
    - `--judges N` or `Judges:` — blind judge count (3 default, 5 thorough, 7 deep)
    - `--convergence N` or `Convergence:` — stop when incumbent wins N consecutive rounds (default 3)
    - `Iterations:` or `--iterations` — default 8. "unlimited" for unbounded.
    - `--judge-personas` — custom judge persona overrides
    - `--no-synthesis` — skip synthesis, pure debate only
    - `--temperature` — generation temperature hint
    - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>`
    
    Remaining text not matching flags = task description.
    
    ## Setup (if Task or Domain missing)
    
    request_user_input (single batch):
      Q1 (Task): "What should be reasoned about?" — open text
      Q2 (Domain): "What domain?" — software architecture, product strategy, business decision, security, research, content
      Q3 (Mode): "Refinement mode?" — convergent (stop when winner repeats), creative (never auto-stop), debate (no synthesis)
      Q4 (Judges): "How many blind judges?" — 3 (default), 5 (thorough), 7 (deep)
    If all provided → skip.
    
    ## Setup Phase
    
    1. Load `references/reason-judge-protocol.md` for judge and convergence specs
    2. Parse domain → select domain-specific judge criteria
    3. Create output directory: `autoresearch/reason-{YYMMDD}-{HHMM}/`
    4. TSV header: `round\ttimestamp\tcandidate_label\tjudge_verdict\tconvergence_count\tdescription`
    5. Initialize: incumbent = null, convergence_count = 0
    
    ## Round Loop
    
    ### Phase 1: Generate-A
    - If round 1: Author-A generates first candidate from task description
    - If round N>1: incumbent is Author-A's candidate
    - Cold-start: Author-A sees ONLY task description + domain context
    
    ### Phase 2: Critic
    - Critic receives candidate-A (cold-start, no shared session)
    - MUST find at least 3 specific weaknesses
    - MUST suggest what a superior candidate would do differently
    - Role is purely adversarial — never compliment
    
    ### Phase 3: Generate-B
    - Author-B receives: task + candidate-A + critique (cold-start)
    - Produces candidate-B addressing critique while preserving A's strengths
    
    ### Phase 4: Synthesize (unless --no-synthesis or debate mode)
    - Synthesizer receives: task + A + B (cold-start)
    - Produces hybrid candidate-AB merging best of both
    
    ### Phase 5: Blind Judge Panel
    - Each judge receives 3 candidates with RANDOMIZED labels (Label-X, Label-Y, Label-Z)
    - Judges evaluate independently on domain-specific criteria
    - Each produces ranking + one-paragraph justification
    - Verdict: majority vote. Tie → synthesized candidate wins.
    
    ### Phase 6: Convergence Check
    - If winner == incumbent → convergence_count++
    - If winner != incumbent → convergence_count = 1, winner becomes incumbent
    - **Convergent mode**: convergence_count >= N → CONVERGED, stop
    - **Creative mode**: never auto-stop
    - **Debate mode**: same as convergent, no synthesis
    
    ### Phase 7: Oscillation Guard
    If incumbent changed 5+ times in last 8 rounds → recommend early stop (not converging).
    
    ### Phase 8: Log
    Append to TSV: round, timestamp, winning candidate label, judge verdict, convergence_count, description
    
    ### Eval Checkpoint
    If --evals: check if current_round % interval == 0 → run checkpoint.
    
    ### Bounded Check
    If bounded: current_round >= max_iterations → exit loop.
    
    ## Output
    
    - `reason-results.tsv` — per-round results
    - `lineage.md` — full history of candidates + critiques + judge reasoning
    - `summary.md` — final winner, convergence trajectory, key insights
    
    ## Summary
    
    Print: total rounds, convergence status, final winner summary, judge agreement rate.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded.
    - Print: `--- Eval Checkpoint (rounds {X}-{Y}) ---\nIncumbent: {label} | Convergence: {count}/{target} | Oscillations: {n}\n{recommendation}\n---`
    - If oscillation detected 3+ checkpoints → recommend early stop.
    - At loop end → full evals summary to evals-summary.md.
    
    ## Chain Handoff
    
    After completion, write handoff.json: version "2.1.0", source "reason", timestamp, status (COMPLETE|CONVERGED|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = [{id, type: "recommendation", summary: winner description}], config{task, domain, mode}.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • regression.md 9.3 KB
    ---
    name: autoresearch:regression
    description: "Layered regression stability gate: capture baseline behavior on the base ref, diff the candidate, verdict STABLE/UNSTABLE before you push"
    argument-hint: "[Base: <ref>] [Scope: <glob>] [--select auto|full|affected] [--samples N] [--noise-band %] [--matrix] [--max-runs N] [--baseline-cache] [Baseline: <prebuilt-ref>] [--probe|--no-probe|--probe deep] [--predict --reason --debug --fix --fix-cycles N --evals --evals-interval N --chain]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    A regression is a **green→red transition ONLY**. The gate orchestrates the project's OWN test/bench/snapshot/migrate commands (it is a protocol, not a bundled framework), captures baseline behavior in an isolated git worktree, re-runs the candidate, and reports a tiered ship/no-ship verdict.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Base:` or `--base` — base ref to diff against. Default: `git merge-base HEAD main` (else `main`/`master`).
    - `Scope:` or `--scope` — file globs limiting the change surface.
    - `--select auto|full|affected` — test selection (default `auto`). `auto` = use the detected affected-test mapper if available, else FULL suite. Never a silent subset.
    - `--samples N` — SCORE samples/side (default 7). `--noise-band %` — perf tolerance (default 5%).
    - `--matrix` — opt-in matrix axis (OFF by default). `--max-runs N` — ceiling (default 200).
    - `--baseline-cache` (default on) — reuse `baseline/<full-sha>/` by SHA. `Baseline: <prebuilt-ref>` — bypass capture.
    - `--probe` (default) / `--probe deep` / `--no-probe`.
    - `--predict --reason --debug --fix --fix-cycles N --evals --evals-interval N --chain <targets>` and `--<sub>` shorthand.
    - `Iterations:` — repeat-axis count for `--select`/repeat sweeps.
    
    ## Setup / Probe-on-launch
    
    1. Auto-detect per-dimension verify commands: `package.json` scripts, `Makefile`, `nx`, migrate config, bench/snapshot/size scripts.
    2. request_user_input (single batch) to confirm detected commands + base ref + which dimensions to run.
    3. **Auto-skip probe** when CI / no-TTY / `--mode autonomous` / complete-config / chained-handoff — log the inferred config instead of asking.
    
    ## Classification Phase (first-class, before any differential)
    
    Establish the baseline green-set per dimension, then tag each unit. Match by **test-id first, then path**.
    
    | State | Meaning | Gated? |
    |---|---|---|
    | `regression-eligible` | green on baseline | YES — only green→red counts |
    | `pre-existing` | red→red (already failing) | no — excluded |
    | `new-coverage` | absent→red (brand-new test) | no — new coverage, ungated |
    | `flaky` | nondeterministic on baseline | no — routed to flakiness SCORE |
    | `baseline-unavailable` | dimension never green | no — advisory only |
    
    **Core invariant: red→red, absent→red, and flake→red are NOT regressions.** Run flakiness N× on **both** baseline and candidate; a candidate failure inside the baseline flake-envelope routes to flakiness SCORE, never to a regression. `5/5 green ≠ non-flaky` — detection probability is `1−(1−p)^n` (≈23% at p=5%, n=5); print it.
    
    ## Baseline Capture
    
    `git worktree add --detach <full-sha>` (detached SHA — avoids "branch already checked out" when Base==HEAD) → `baseline/<full-sha>/`; `--baseline-cache` reuses by SHA. Then per worktree: `git submodule update --init` + dependency install (lockfile is SHA-pinned so the cache stays sound). Per-dimension **setup tiers**: api-contract = file-diff, no build; functional / integration-e2e / data-migration = full env. On completion or crash: `git worktree remove` + `git worktree prune`. Warn on concurrent index-lock contention. `Baseline: <prebuilt-ref>` bypasses capture.
    
    ## Dimension Registry (8)
    
    | Dim | Tier | Compare | Key params |
    |---|---|---|---|
    | functional | HARD | baseline green-set vs candidate; new fail = regression | test cmd, globs |
    | api-contract | HARD | schema/exports diff → breaking? | schema cmd, breaking ruleset |
    | data-migration | HARD | default: up applies clean + idempotent re-apply + app boots/schema valid; schema/rowcount roundtrip opt-in | migrate cmds, fixture, allowlisted DB |
    | integration-e2e | HARD | e2e green-set diff | e2e cmd |
    | flakiness | SCORE | run N× on baseline + candidate, count nondeterministic | runs (def 5), flake-threshold |
    | performance | SCORE | K **independent-process** samples/side, Mann-Whitney U AND effect beyond `max(noise-band%, k·stdev)`, report median delta | bench cmd, samples=7, noise-band=5%, k=2 |
    | resource | SCORE | mem/bundle/size delta vs budget | size cmd, budget |
    | visual-ui | SCORE | containerize render; default `maxDiffPixelRatio` + AA-detection; SSIM = per-page escalation | snapshot cmd, diff-threshold, mask regions |
    
    `--select auto` mapper (`jest --findRelatedTests` fed the changed-file list; `nx affected` project-graph) is **best-effort static-import** — blind to dynamic/runtime/global-setup couplings. The report names the mapper + its blind-spot caveat; a HARD STABLE earned on an affected subset prints "run `--select full` for high-stakes". FULL suite is the correctness default.
    
    - **performance independence:** each sample = an independent process launch (warmups discarded), never an in-process iteration — autocorrelation/GC/thermal otherwise violate Mann-Whitney's independence assumption. At n=7 the test detects only ≳1σ regressions; raise `--samples` for tight gates.
    - **data-migration guard:** opt-in. Before any migration the DB URL MUST pass an **anchored** allowlist — host is exactly `localhost` / `127.0.0.1` / a container or service hostname, OR the database name carries a `_test` / `_ci` suffix. A bare substring (e.g. `test` inside `latest`, `ci` inside `precision`) does **not** qualify. Anything else is refused — ephemeral only, never dev/prod — and even an allowlisted URL requires explicit user confirm before applying. Missing/absent down-migration = forward-only advisory, **never a finding**.
    
    ## Differential Loop (per dim × axis × run)
    
    Run candidate verify vs baseline metric → compute `regressed` bool + 0-100 `subscore`. Axes: `diff` (default), `repeat N×`, `full`, `matrix` (opt-in). Log one TSV row per cell.
    
    **--max-runs ceiling:** projected = dims × axes × samples × matrix-cells; if > `--max-runs` (default 200) → warn + require confirm (CI default = abort with message).
    
    ## Verdict
    
    - Any HARD `regressed=true` with `classification=regression-eligible` → **UNSTABLE** (green→red hard-blocks).
    - Else `stability_score = Σ(weight × dim_subscore)` over SCORE dims that ran (flakiness .30 / performance .30 / resource .20 / visual .20, renormalized over present dims). **STABLE iff ≥ 95** (`REG_THRESHOLD`/weights overridable).
    - Print the **score math** (per-dim contribution table) + declare **dims-ran vs UNAVAILABLE** — an UNAVAILABLE dimension is always listed, never silently passed.
    
    Backed by `scripts/score-regression.sh verdict <results.tsv>` relative to the installed Autoresearch skill directory, never the caller's working directory (exit 0 STABLE / 1 UNSTABLE).
    
    ## Hunter (root cause)
    
    On a confirmed HARD regression, auto-engage. Bisect (reuse `debug`) ONLY when the failing case passes a **3/3 reproducibility gate**. SCORE / non-deterministic regressions → differential root-cause + optional `--reason` / `--predict`, no bisect. Non-reproducible → "manual triage" finding.
    
    ## --fix Re-gate
    
    `--fix` repairs blocking regressions, max **3 cycles** (`--fix-cycles N`). Each cycle MUST strictly shrink the blocking-set else STOP "fix not converging". Intermediate re-gate scopes to failing+touched dims; the final cycle runs the full battery. No HARD-gate bypass.
    
    ## Output
    
    `autoresearch/regression-{YYMMDD}-{HHMM}/` → `regression-results.tsv`, `stability-report.md`, `dimensions/<dim>.md`, `baseline/`, `evals-summary.md` (if `--evals`), `handoff.json`.
    
    TSV header: `# metric_direction: higher_is_better` then
    `iteration\ttimestamp\tdimension\taxis\ttier\tclassification\tbaseline\tcandidate\tdelta\tregressed\tsubscore\tseverity\tstatus\tfile_line\tdescription`.
    
    ## Eval Checkpoint (--evals flag)
    
    Interval = floor(max_runs / 3), min 1 (fixed 10 if unbounded); override `--evals-interval N`. Every interval, analyze the results TSV; print trend (up/flat/down) + one-line recommendation. Plateau 3+ checkpoints → recommend early stop. At end → full summary to `evals-summary.md`.
    
    ## Chain Handoff
    
    Write `handoff.json` to the output directory: version "2.1.0", source "regression", timestamp,
    `status` ∈ family enum {COMPLETE, CONVERGED, SATURATED, BOUNDED, USER_INTERRUPT, ERROR} (backward-compat with evals/ship consumers),
    `verdict` ∈ {STABLE, UNSTABLE, BASELINE_UNAVAILABLE} + `regression_state` ∈ {REGRESSION_FOUND, REGRESSION_FIXED, none} — `ship` reads `verdict` for the deploy-gate,
    `results_tsv` path, `findings` = blocking regressions (dim, severity, file_line, classification), `config`{base, scope, dims, axes, verdict-math}.
    
    If `--fix` → chain to fix automatically. Invoke next `--chain` target in order; propagate `--evals`. Canonical combo: `--predict --evals --fix --ship` = predict → gate → (hunter on HARD) → fix(≤3) → re-gate → ship iff STABLE (deploy still needs explicit approval).
    
    ## Safety
    
    Verify-command screen (no `rm -rf` / `curl|sh`); worktree cleanup + prune on crash; data-migration refuses any non-allowlisted DB URL; probe auto-skips non-interactively; chained `ship` never auto-deploys.
    
  • scenario.md 4.1 KB
    ---
    name: autoresearch:scenario
    description: "Generate edge cases across 12 dimensions from a seed scenario"
    argument-hint: "[Scenario: <text>] [Domain: <type>] [Scope: <glob>] [Iterations: N] [--depth <level>] [--focus <area>] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Scenario:` — seed scenario description (or full $ARGUMENTS text if no keyword)
    - `Domain:` or `--domain` — web, mobile, API, CLI, data pipeline, infrastructure
    - `Scope:` or `--scope` — file globs for codebase context
    - `Focus:` or `--focus` — specific dimension to prioritize
    - `--depth` — shallow (10), standard (20), deep (40+)
    - `--format` — markdown (default), json, gherkin
    - `Iterations:` or `--iterations` — default 20. "unlimited" for unbounded.
    - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>`
    
    ## Setup (if Scenario or Domain missing)
    
    request_user_input (single batch):
      Q1 (Scenario): "Describe the feature/flow to explore"
      Q2 (Domain): "What domain?" — web app, mobile app, API, CLI, data pipeline, infrastructure
      Q3 (Scope): "Which files for context?" — suggested globs + entire codebase
      Q4 (Depth): "How deep?" — quick (10), standard (20), deep (40+), unlimited
    If all provided → skip.
    
    ## 12 Dimensions
    
    | # | Dimension | Explores |
    |---|---|---|
    | 1 | Happy path | Normal successful flows |
    | 2 | Validation | Input boundaries, types, formats |
    | 3 | Permissions | Auth, roles, access control |
    | 4 | Concurrency | Race conditions, deadlocks, ordering |
    | 5 | State | Invalid transitions, corruption |
    | 6 | Scale | High volume, large data, many users |
    | 7 | Failure | Network errors, timeouts, partial failures |
    | 8 | Security | Injection, abuse, bypass attempts |
    | 9 | Integration | Third-party failures, API contract violations |
    | 10 | Data | Null, empty, unicode, injection, overflow |
    | 11 | UX | Confusion, misuse, accessibility |
    | 12 | Recovery | Retry, rollback, idempotency |
    
    ## Establish Baseline
    
    1. Read seed scenario + codebase context
    2. Create output directory: `autoresearch/scenario-{YYMMDD}-{HHMM}/`
    3. TSV header: `iteration\ttimestamp\tscenario\tdimension\tclassification\tseverity\tdescription`
    4. No metric_direction comment (exploration, not optimization)
    
    ## Iteration Loop
    
    ### Phase 1: Review
    - Read results TSV, check dimension coverage
    - Identify underexplored dimensions
    - If --focus → prioritize that dimension
    
    ### Phase 2: Generate
    - Pick next dimension (round-robin, or priority if --focus)
    - Generate 3-5 specific scenarios for this dimension
    - Each: title, dimension, classification, severity, description
    
    ### Phase 3: Classify
    - **new** — genuinely novel edge case
    - **extension** — builds on previously found scenario
    - **duplicate** — already covered (skip, don't log)
    
    ### Phase 4: Log
    Append new/extension scenarios to TSV. Skip duplicates.
    Severity: critical/high/medium/low.
    
    ### Phase 5: Saturation Check
    If 3 consecutive iterations produce only duplicates → dimension saturated, move to next.
    If ALL dimensions saturated → early stop.
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint.
    
    ### Bounded Check
    If bounded: current_iteration >= max_iterations → exit loop.
    
    ## Output
    
    - Write `scenarios.md` (organized by dimension, severity-ranked within each)
    - Write `edge-cases.md` (flat severity-ranked list)
    - `scenario-results.tsv`
    
    ## Summary
    
    Print: total scenarios (new/extension/duplicate), dimension coverage (X/12 explored), severity distribution.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded.
    - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nNew scenarios: {n} | Dimensions covered: {x}/12 | Saturation: {status}\n{recommendation}\n---`
    - If 3+ checkpoints with mostly duplicates → recommend early stop.
    - At loop end → full evals summary to evals-summary.md.
    
    ## Chain Handoff
    
    After completion, write handoff.json: version "2.1.0", source "scenario", timestamp, status, results_tsv path, findings = scenarios by severity, config{scenario, domain, scope}.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • security.md 4.6 KB
    ---
    name: autoresearch:security
    description: "STRIDE + OWASP security audit with red-team adversarial personas"
    argument-hint: "[Scope: <glob>] [Focus: <area>] [Iterations: N] [--diff] [--fix] [--fail-on <severity>] [--evals]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Scope:` or `--scope` — file globs to audit
    - `Focus:` — specific area (auth, API, data handling, etc.)
    - `Depth:` or `--depth` — quick (5 iterations), standard (15), deep (30+)
    - `Iterations:` or `--iterations` — default 15. "unlimited" for unbounded.
    - `--diff` — delta mode: only audit files changed since last audit
    - `--fix` — after audit, auto-fix Critical/High findings (chains to fix)
    - `--fail-on <severity>` — exit non-zero if findings at/above threshold (CI gate)
    - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>`
    
    ## Setup (if required context missing)
    
    If Scope missing and no --diff:
    1. Scan codebase for tech stack, frameworks, API routes
    2. request_user_input (single batch):
       Q1 (Scope): "What to audit?" — entire codebase, API + middleware, auth, external-facing
       Q2 (Depth): "How thorough?" — quick (5), standard (15), deep (30+), unlimited
       Q3 (Action): "What to do with findings?" — report only, report + auto-fix, report + CI gate
    If all provided → skip.
    
    ## Setup Phase (once, before loop)
    
    1. **Reconnaissance** — scan: package.json/requirements.txt (deps), .env.example (secrets), Dockerfile (infra), API route files (attack surface), auth/middleware (trust boundaries), DB schemas (data assets), CI/CD configs (supply chain)
    2. **Asset Identification** — catalog data stores, auth systems, external services, user inputs
    3. **Trust Boundary Mapping** — browser↔server, public↔authenticated, user↔admin, CI↔prod
    4. **STRIDE Threat Model** — generate threats per category. Load `references/security-checklist.md` for checklist.
    5. **Attack Surface Map** — entry points, data flows, abuse paths
    6. **Baseline** — count known issues, initialize coverage tracking
    
    Create output directory: `autoresearch/security-{YYMMDD}-{HHMM}/`
    Write: overview.md, threat-model.md, attack-surface-map.md
    TSV header: `# metric_direction: higher_is_better\niteration\ttimestamp\tfinding\tseverity\towasp\tstride\tevidence\tfile_line`
    
    ## Iteration Loop
    
    ### Phase 1: Review
    - Read results TSV + coverage tracking
    - Identify untested attack vectors from threat model
    - Prioritize: untested OWASP categories → untested STRIDE → depth on existing
    
    ### Phase 2: Attack
    - Adopt red-team persona for this vector (rotate: Security Adversary, Supply Chain, Insider Threat, Infra Attacker)
    - Deep-dive into relevant code with adversarial mindset
    - Look for: code paths, input handling, auth checks, data flows
    
    ### Phase 3: Validate
    - Construct proof: file:line + specific attack scenario
    - Every finding MUST have code evidence — no theoretical fluff
    - Classify severity: Critical/High/Medium/Low/Info
    - Map to OWASP (A01-A10) and STRIDE (S/T/R/I/D/E)
    
    ### Phase 4: Log
    - Append finding to TSV
    - Update coverage tracking
    - Print coverage every 5 iterations:
      `OWASP: [A01✓ A02✓ A03✗ ...] X/10 | STRIDE: [S✓ T✓ R✗ ...] Y/6 | Score: Z`
    
    ### Composite Metric
    `score = (owasp_tested/10)*50 + (stride_tested/6)*30 + min(findings, 20)`
    
    ### Eval Checkpoint
    If --evals: check if current_iteration % interval == 0 → run checkpoint.
    
    ### Bounded Check
    If bounded: current_iteration >= max_iterations → exit loop.
    
    ## After Loop
    
    1. Write `findings.md` (severity-ranked)
    2. Write `owasp-coverage.md`
    3. Write `recommendations.md`
    4. If `--fix` → chain to fix with Critical/High findings
    5. If `--fail-on` → check findings against threshold, exit non-zero if exceeded
    
    ## Summary
    
    Print: total findings by severity, OWASP coverage X/10, STRIDE coverage Y/6, composite score.
    
    ## Eval Checkpoint (--evals flag)
    
    If --evals present:
    - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N.
    - Every {interval} iterations, analyze results TSV.
    - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nScore: {start} → {end} | New findings: {n} | Coverage: OWASP {x}/10, STRIDE {y}/6\n{recommendation}\n---`
    - If no new findings 3+ checkpoints → recommend early stop.
    - At loop end → full evals summary to evals-summary.md.
    
    ## Chain Handoff
    
    After completion, write handoff.json to output directory: version "2.1.0", source "security", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = all findings with severity + OWASP + STRIDE + file:line, config{scope, focus, depth}.
    Invoke next target in --chain order. Propagate --evals flag.
    
  • ship.md 3.8 KB
    ---
    name: autoresearch:ship
    description: "Ship anything through 8 phases: checklist, dry-run, deploy, verify"
    argument-hint: "[Target: <what>] [--type <type>] [--dry-run] [--auto] [--force] [--rollback] [--checklist-only] [--monitor N]"
    ---
    
    EXECUTE IMMEDIATELY.
    
    ## Parse Arguments
    
    Extract from $ARGUMENTS:
    - `Target:` or `--target` — what to ship (path, PR, artifact, deployment)
    - `--type <type>` — override auto-detection: code-pr, code-release, deployment, content, docs, package, config
    - `--dry-run` — validate everything but don't ship
    - `--auto` — auto-approve if no errors found
    - `--force` — skip non-critical items (blockers still enforced)
    - `--rollback` — undo last ship action
    - `--monitor N` — post-ship monitoring for N minutes
    - `--checklist-only` — only generate checklist, don't execute
    - `--chain`, `--<subcommand>`
    
    Remaining text = description of what to ship.
    
    ## Setup (if Target or Type unclear)
    
    1. Auto-detect ship type from context:
       - Has uncommitted changes or PR → code-pr
       - Has version bump / changelog → code-release
       - Has Dockerfile / deploy config → deployment
       - Has markdown / content files → content
       - Has package.json version change → package
    2. If still unclear → request_user_input (single batch):
       Q1 (What): "What are you shipping?" — code PR, release, deployment, content, docs, package
       Q2 (Target): "Specific target?" — current branch, specific PR, specific path
       Q3 (Mode): "How to ship?" — full workflow, dry-run only, checklist only
    If all clear → skip.
    
    ## Phase 1: Identify
    
    - Determine ship type (auto-detected or --type override)
    - Identify target artifact(s)
    - Map to domain-specific checklist
    
    ## Phase 2: Inventory
    
    Gather everything that will be shipped:
    - Files changed (git diff)
    - Dependencies affected
    - Config changes
    - Migration files
    - Breaking changes
    
    ## Phase 3: Checklist
    
    Generate domain-specific checklist:
    
    **Code PR:** tests pass, types check, lint clean, no secrets, PR description, reviewers assigned
    **Release:** version bumped, changelog updated, migration tested, rollback plan
    **Deployment:** env vars set, health checks configured, rollback ready, monitoring active
    **Content:** links valid, images optimized, SEO metadata, spell check
    **Package:** version bumped, README updated, breaking changes documented, CI green
    
    If `--checklist-only` → output checklist and stop.
    
    ## Phase 4: Prepare
    
    Execute pre-ship tasks:
    - Run test suite
    - Run type checker
    - Run linter
    - Check for secrets in diff
    - Validate configs
    - Flag blockers (must-fix) vs warnings (can-ship-with)
    
    If blockers found → STOP, report blockers, ask user to fix.
    
    ## Phase 5: Dry-Run
    
    If `--dry-run` or always before actual ship:
    - Simulate the ship action without executing
    - Report what WOULD happen
    - If `--dry-run` → stop here
    
    ## Phase 6: Ship
    
    **REQUIRES EXPLICIT USER APPROVAL** (unless --auto with zero errors).
    
    Execute the ship action:
    - Code PR: create/update PR, request reviewers
    - Release: tag, build, publish
    - Deployment: deploy to target environment
    - Content: publish to CMS/platform
    
    ## Phase 7: Verify
    
    Post-ship verification:
    - Confirm artifact is live/accessible
    - Run smoke tests if available
    - Check monitoring for errors
    - If `--monitor N` → watch for N minutes
    
    ## Phase 8: Log
    
    Create output directory: `autoresearch/ship-{YYMMDD}-{HHMM}/`
    Write:
    - `checklist.md` — completed checklist with pass/fail per item
    - `summary.md` — what was shipped, verification results
    - `ship-log.tsv` — phase-by-phase log
    
    ## Rollback
    
    If `--rollback`:
    - Identify last ship action from most recent ship log
    - Reverse it (revert PR, unpublish, rollback deployment)
    - Verify rollback succeeded
    
    ## Chain Handoff
    
    Write handoff.json: version "2.1.0", source "ship", timestamp, status (COMPLETE|DRY_RUN|ROLLBACK|ERROR), findings = blockers/warnings found during prep.
    Invoke next target in --chain order.
    
  • SKILL.md 7.8 KB
    ---
    name: autoresearch
    description: "Autonomous iteration loop: modify, verify, keep/discard against any metric"
    metadata:
      version: "2.2.2"
    ---
    
    # Autoresearch — Autonomous Goal-directed Iteration
    
    ## Safety Invariants (all subcommands)
    - Never push, publish, or deploy without explicit user approval.
    - Bounded by default. Override with `Iterations: unlimited`.
    - All results logged to `autoresearch/{subcommand}-{YYMMDD}-{HHMM}/` directory.
    - Chain handoff via `handoff.json`. Evals reads `*-results.tsv`.
    
    ## Dispatch (bare `$autoresearch`)
    
    Parse the invocation in this order:
    
    | Condition | Mode |
    |---|---|
    | `Metric:` or `Verify:` present | **Classic** — existing metric loop, unchanged |
    | Free-form natural-language goal, no metric/verify | **Orchestrator** — see Orchestrator section |
    | Nothing | **Setup wizard** — interactive config builder |
    | `--classic` flag | Force Classic regardless of goal text |
    | `--auto` flag | Force Orchestrator regardless of goal text |
    
    Print a banner on every invocation: `[autoresearch] mode: classic | orchestrator | wizard`.
    
    ## Subcommands
    
    | Command | Does | Default Iterations |
    |---|---|---|
    | `$autoresearch` | Iterate against a metric: modify → verify → keep/discard | 25 |
    | `$autoresearch plan` | Convert a goal into validated Scope, Metric, Verify config | N/A |
    | `$autoresearch debug` | Hunt bugs: hypothesize → test → falsify → repeat | 15 |
    | `$autoresearch fix` | Crush errors one-by-one until zero remain | 20 |
    | `$autoresearch security` | STRIDE + OWASP audit with red-team personas | 15 |
    | `$autoresearch ship` | Ship through 8 phases: checklist → dry-run → deploy → verify | N/A |
    | `$autoresearch scenario` | Generate edge cases across 12 dimensions | 20 |
    | `$autoresearch predict` | 5 expert personas debate before implementation | N/A |
    | `$autoresearch learn` | Scout codebase → generate docs or wiki → validate → fix loop | 10 |
    | `$autoresearch reason` | Adversarial debate with blind judges until convergence | 8 |
    | `$autoresearch probe` | 8 personas interrogate requirements until saturation | 15 |
    | `$autoresearch improve` | Research ICP challenges, discover improvements, generate PRDs | 15 |
    | `$autoresearch evals` | Analyze iteration results: trends, plateaus, regressions | N/A |
    | `$autoresearch regression` | Regression stability gate: baseline vs candidate, verdict STABLE/UNSTABLE | N/A |
    
    ## Universal Flags
    
    | Flag | Applies To | Purpose |
    |---|---|---|
    | `Iterations: N` | All looping | Set iteration count |
    | `Iterations: unlimited` | All looping | Opt-in unbounded |
    | `--evals` | All looping | Mid-loop checkpoints + final summary |
    | `--evals-interval N` | All looping | Override checkpoint frequency |
    | `--chain <targets>` | All | Sequential handoff after completion |
    | `--<subcommand>` | All | Shorthand for `--chain <subcommand>` |
    | `--dry-run` | Orchestrator | Print derived config + planned pipeline; no execution |
    | `--max-cycles N` | Orchestrator | Hard ceiling on orchestration cycles (default 50) |
    | `--classic` | Bare `$autoresearch` | Force Classic metric-loop mode |
    | `--auto` | Bare `$autoresearch` | Force Orchestrator mode |
    
    ## Orchestrator
    
    Activated when a plain-language goal is given without `Metric:`/`Verify:`. Classifies the goal into a **Goal archetype** — see `references/orchestrator-routing.md` for the archetype table and router decision table.
    
    Resolve every `scripts/...` path below relative to this installed skill directory, never relative to the caller's working directory.
    
    **Two modes based on archetype:**
    - **Orchestration loop** — predicate-bearing archetypes (ship-ready, optimize-metric, fix-broken, harden, build-feature, explore). Goal has a mechanical Success predicate; the loop runs until that predicate is met.
    - **Single-pass dispatch** — subjective/terminal archetypes (document, what-to-build, decide-design). Routes once to the fitting subcommand (learn / improve / reason), lets it self-terminate, then reports. No loop, no Plateau, no ship gate.
    
    ### Orchestration Loop Steps
    
    Backed by `scripts/orchestrate.sh` (deterministic seam — all routing logic lives there). Subcommands exposed: `classify`, `next-hop`, `units`, `plateau`, `screen-cmd`, `verdict`, `validate-state`, `screen-state-predicate`.
    
    1. **Classify** — `scripts/orchestrate.sh classify "<goal>"` → archetype label + mode.
    2. **Derive predicate** — reuse `plan` logic to produce a concrete Success predicate: exact shell command + expected output. For `optimize-metric`, run the full plan/wizard derivation internally.
    3. **Confirm** — ONE `request_user_input` showing: archetype, mode, concrete predicate (command + expected output), terminal choice (stop-at-verified vs proceed-to-ship). Misclassifications are caught here, not mid-run.
    4. **Round-0 dry-run** — prove the predicate command runs and returns a value; safety-screen every derived command via `screen-cmd`; print projected cycle budget. Stop here if `--dry-run`.
    5. **Loop** until predicate satisfied:
       a. Assess state via cheap signals (last `handoff.json`, regression verdict, error count) + affected-test verify.
       b. `scripts/orchestrate.sh next-hop orchestrator-state.json` → next subcommand.
       c. Run subcommand (its own bounded inner loop).
       d. Record per-hop outcome ∈ {progressed, no-op, failed, blocked}.
       e. Fold hop's `handoff.json` into `orchestrator-state.json`.
       f. `scripts/orchestrate.sh units` → recompute **Units remaining**.
    6. **Stop conditions** (checked after each hop):
       - Predicate met → ship gate (only if ship is in the pipeline) else `CONVERGED`.
       - `scripts/orchestrate.sh plateau orchestrator-state.json` → true → stop + report `PLATEAU`.
       - Cycles > ceiling (default 50, override `--max-cycles N`) → stop + report `CEILING`.
       - Hop outcome `blocked`/`failed` with no alternative route → checkpoint + stop + report `BLOCKED`.
    
    ### Orchestrator State
    
    `orchestrator-state.json` — orchestrator-owned, additive. Tracks: goal, archetype, predicate, terminal-choice, `units_remaining` history, cycle count, per-hop pipeline log with outcomes, current incumbent. Each hop's `handoff.json` is unchanged (single-hop bridge); the orchestrator reads it and folds it in. Two clearly-owned state objects, no overlap.
    
    ### Orchestrator Safety Invariants
    
    - **Never auto-approve ship/deploy/push.** The orchestrator never passes `--auto` to `ship`; deploy always requires explicit user approval.
    - **Data-migration behind anchored DB-URL allowlist.** Reuses regression's allowlist — host must be `localhost`/`127.0.0.1`/container hostname, or database name carries `_test`/`_ci` suffix. Bare substring match does not qualify. Anything else refused.
    - **screen-cmd on every derived command** — run before the loop starts AND on every command read from a persisted state file on resume. Persisted commands are never trusted; resume re-screens the pinned predicate via `screen-state-predicate` and refuses on `refuse`.
    - **No un-screened commands mid-loop.** The autonomous loop cannot introduce new shell commands that bypass `screen-cmd`.
    - **Predicate pinned, not re-derived.** Round-0 writes the derived Success predicate verbatim into `orchestrator-state.json`; every cycle and every resume reuses that exact string so "done" is reproducible across runs.
    - **Validate the ledger before routing.** `validate-state` gates `orchestrator-state.json` (required fields + coarse types); a malformed ledger is not trusted to route from.
    - **Independent verify before convergence.** High-impact changes accepted on the working signal set `pending_verify`; `next-hop` routes to a `verify` hop (held-out / adversarial check) before `DONE` or ship. The verify hop never auto-approves ship.
    - **Unknown-units cycles excluded from Plateau counter.** A cycle where `units` returns `unknown` (e.g. runner crash) is not counted as zero-progress; repeated `unknown` routes to `BLOCKED`.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related