autoresearch
Autonomous iteration loop: modify, verify, keep/discard against any metric
Install
npx skills add https://github.com/mxyhi/ok-skills/tree/main/autoresearch
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mxyhi-ok-skills@llmmart
git clone https://github.com/mxyhi/ok-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mxyhi/ok-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Autoresearch — Autonomous Goal-directed Iteration
Safety Invariants (all subcommands)
- Never push, publish, or deploy without explicit user approval.
- Bounded by default. Override with
Iterations: unlimited. - All results logged to
autoresearch/{subcommand}-{YYMMDD}-{HHMM}/directory. - Chain handoff via
handoff.json. Evals reads*-results.tsv.
Dispatch (bare $autoresearch)
Parse the invocation in this order:
| Condition | Mode |
|---|---|
Metric: or Verify: present |
Classic — existing metric loop, unchanged |
| Free-form natural-language goal, no metric/verify | Orchestrator — see Orchestrator section |
| Nothing | Setup wizard — interactive config builder |
--classic flag |
Force Classic regardless of goal text |
--auto flag |
Force Orchestrator regardless of goal text |
Print a banner on every invocation: [autoresearch] mode: classic | orchestrator | wizard.
Subcommands
| Command | Does | Default Iterations |
|---|---|---|
$autoresearch |
Iterate against a metric: modify → verify → keep/discard | 25 |
$autoresearch plan |
Convert a goal into validated Scope, Metric, Verify config | N/A |
$autoresearch debug |
Hunt bugs: hypothesize → test → falsify → repeat | 15 |
$autoresearch fix |
Crush errors one-by-one until zero remain | 20 |
$autoresearch security |
STRIDE + OWASP audit with red-team personas | 15 |
$autoresearch ship |
Ship through 8 phases: checklist → dry-run → deploy → verify | N/A |
$autoresearch scenario |
Generate edge cases across 12 dimensions | 20 |
$autoresearch predict |
5 expert personas debate before implementation | N/A |
$autoresearch learn |
Scout codebase → generate docs or wiki → validate → fix loop | 10 |
$autoresearch reason |
Adversarial debate with blind judges until convergence | 8 |
$autoresearch probe |
8 personas interrogate requirements until saturation | 15 |
$autoresearch improve |
Research ICP challenges, discover improvements, generate PRDs | 15 |
$autoresearch evals |
Analyze iteration results: trends, plateaus, regressions | N/A |
$autoresearch regression |
Regression stability gate: baseline vs candidate, verdict STABLE/UNSTABLE | N/A |
Universal Flags
| Flag | Applies To | Purpose |
|---|---|---|
Iterations: N |
All looping | Set iteration count |
Iterations: unlimited |
All looping | Opt-in unbounded |
--evals |
All looping | Mid-loop checkpoints + final summary |
--evals-interval N |
All looping | Override checkpoint frequency |
--chain <targets> |
All | Sequential handoff after completion |
--<subcommand> |
All | Shorthand for --chain <subcommand> |
--dry-run |
Orchestrator | Print derived config + planned pipeline; no execution |
--max-cycles N |
Orchestrator | Hard ceiling on orchestration cycles (default 50) |
--classic |
Bare $autoresearch |
Force Classic metric-loop mode |
--auto |
Bare $autoresearch |
Force Orchestrator mode |
Orchestrator
Activated when a plain-language goal is given without Metric:/Verify:. Classifies the goal into a Goal archetype — see references/orchestrator-routing.md for the archetype table and router decision table.
Resolve every scripts/... path below relative to this installed skill directory, never relative to the caller's working directory.
Two modes based on archetype:
- Orchestration loop — predicate-bearing archetypes (ship-ready, optimize-metric, fix-broken, harden, build-feature, explore). Goal has a mechanical Success predicate; the loop runs until that predicate is met.
- Single-pass dispatch — subjective/terminal archetypes (document, what-to-build, decide-design). Routes once to the fitting subcommand (learn / improve / reason), lets it self-terminate, then reports. No loop, no Plateau, no ship gate.
Orchestration Loop Steps
Backed by scripts/orchestrate.sh (deterministic seam — all routing logic lives there). Subcommands exposed: classify, next-hop, units, plateau, screen-cmd, verdict, validate-state, screen-state-predicate.
- Classify —
scripts/orchestrate.sh classify "<goal>"→ archetype label + mode. - Derive predicate — reuse
planlogic to produce a concrete Success predicate: exact shell command + expected output. Foroptimize-metric, run the full plan/wizard derivation internally. - Confirm — ONE
request_user_inputshowing: archetype, mode, concrete predicate (command + expected output), terminal choice (stop-at-verified vs proceed-to-ship). Misclassifications are caught here, not mid-run. - Round-0 dry-run — prove the predicate command runs and returns a value; safety-screen every derived command via
screen-cmd; print projected cycle budget. Stop here if--dry-run. - Loop until predicate satisfied:
- Assess state via cheap signals (last
handoff.json, regression verdict, error count) + affected-test verify. scripts/orchestrate.sh next-hop orchestrator-state.json→ next subcommand.- Run subcommand (its own bounded inner loop).
- Record per-hop outcome ∈ {progressed, no-op, failed, blocked}.
- Fold hop's
handoff.jsonintoorchestrator-state.json. scripts/orchestrate.sh units→ recompute Units remaining.
- Assess state via cheap signals (last
- Stop conditions (checked after each hop):
- Predicate met → ship gate (only if ship is in the pipeline) else
CONVERGED. scripts/orchestrate.sh plateau orchestrator-state.json→ true → stop + reportPLATEAU.- Cycles > ceiling (default 50, override
--max-cycles N) → stop + reportCEILING. - Hop outcome
blocked/failedwith no alternative route → checkpoint + stop + reportBLOCKED.
- Predicate met → ship gate (only if ship is in the pipeline) else
Orchestrator State
orchestrator-state.json — orchestrator-owned, additive. Tracks: goal, archetype, predicate, terminal-choice, units_remaining history, cycle count, per-hop pipeline log with outcomes, current incumbent. Each hop's handoff.json is unchanged (single-hop bridge); the orchestrator reads it and folds it in. Two clearly-owned state objects, no overlap.
Orchestrator Safety Invariants
- Never auto-approve ship/deploy/push. The orchestrator never passes
--autotoship; deploy always requires explicit user approval. - Data-migration behind anchored DB-URL allowlist. Reuses regression's allowlist — host must be
localhost/127.0.0.1/container hostname, or database name carries_test/_cisuffix. Bare substring match does not qualify. Anything else refused. - screen-cmd on every derived command — run before the loop starts AND on every command read from a persisted state file on resume. Persisted commands are never trusted; resume re-screens the pinned predicate via
screen-state-predicateand refuses onrefuse. - No un-screened commands mid-loop. The autonomous loop cannot introduce new shell commands that bypass
screen-cmd. - Predicate pinned, not re-derived. Round-0 writes the derived Success predicate verbatim into
orchestrator-state.json; every cycle and every resume reuses that exact string so "done" is reproducible across runs. - Validate the ledger before routing.
validate-stategatesorchestrator-state.json(required fields + coarse types); a malformed ledger is not trusted to route from. - Independent verify before convergence. High-impact changes accepted on the working signal set
pending_verify;next-hoproutes to averifyhop (held-out / adversarial check) beforeDONEor ship. The verify hop never auto-approves ship. - Unknown-units cycles excluded from Plateau counter. A cycle where
unitsreturnsunknown(e.g. runner crash) is not counted as zero-progress; repeatedunknownroutes toBLOCKED.
Files (ok-skills)
-
agents
-
openai.yaml 250 B
interface: display_name: "Autoresearch" short_description: "Autonomous goal-directed iteration engine" brand_color: "#7C3AED" default_prompt: "Set a goal, define a metric, let Codex loop until done" policy: allow_implicit_invocation: true
-
-
references
-
orchestrator-routing.md 6.8 KB
# Orchestrator Routing ## Goal Archetypes | Archetype | Trigger Keywords | Mode | Preset Pipeline | |---|---|---|---| | `ship-ready` | ship, release, deploy, publish, production-ready, merge | loop | probe, debug, fix, regression, ship | | `optimize-metric` | improve, optimize, increase, reduce, faster, smaller, coverage, score | loop | plan, (classic loop), evals | | `fix-broken` | fix, broken, failing, error, crash, bug, can't run, tests fail | loop | debug, fix, regression | | `harden` | security, vulnerability, audit, OWASP, CVE, harden, lock down | loop | security, fix, security | | `build-feature` | build, add, implement, create, new feature, acceptance test | loop | (acceptance-test derive), debug, fix, regression | | `explore` | understand, explore, investigate, what does, how does, edge cases | loop | probe, scenario, plan | | `document` | document, wiki, generate docs, explain codebase, write guide | dispatch | learn | | `what-to-build` | what should I build, ideas, improvements, PRD, roadmap | dispatch | improve | | `decide-design` | which approach, compare options, design decision, architecture choice | dispatch | reason | Keyword matching is fuzzy — partial matches and synonyms qualify. When a goal matches multiple archetypes, prefer the more specific one (fix-broken over explore; ship-ready over fix-broken if "ship" is explicit). When ambiguous, show the top two candidates in the upfront confirm and let the user choose. ## Router Decision Table The `next-hop` subcommand of `scripts/orchestrate.sh` reads `orchestrator-state.json` and applies these rules in order. First match wins. | State Signal | Source | Next Hop | |---|---|---| | `errors > 0` in last handoff | handoff.json `findings` | `fix` | | regression verdict `UNSTABLE` | handoff.json `verdict` | `regression` | | `untested_gaps` flagged | handoff.json or units output | `debug` | | `pending_verify` true | orchestrator-state.json | `verify` (fresh independent acceptance check) | | predicate met | Success predicate command exit/output | `DONE` (exit loop) | | hop outcome `blocked` or `failed`, no retry route | orchestrator-state.json | `BLOCKED` (checkpoint + stop) | | plateau detected | `scripts/orchestrate.sh plateau` | `PLATEAU` (stop + report) | | archetype pipeline has remaining steps | preset pipeline sequence | next preset step | | all preset steps exhausted, predicate not met | — | `regression` (convergence re-check) | State signals are cheap reads — last `handoff.json` plus the regression verdict field and error count. No re-run of the full suite just to route. ## Independent Verify & Overfit Guard The orchestrator must not optimize and accept against the same signal — that lets a change game its own metric. For `optimize-metric` and `build-feature`, the acceptance check runs on a **held-out** set (a fresh scenario set or holdout assertions), separate from the `units` signal used to choose the change. When a high-impact change is accepted on the working signal, the orchestrator sets `pending_verify` in `orchestrator-state.json`; `next-hop` then routes to a **verify** hop (dispatched to `reason` or `predict` as an independent adversarial check) before declaring `DONE` or shipping. The verify hop is advisory input to convergence — it never auto-approves ship, which stays human-gated. ## Two-Mode Split **Orchestration loop** — used when the goal has an external, mechanical Success predicate: a shell command that returns a value the orchestrator can compare across cycles. Progress is objective (Units remaining falls), plateau is well-defined, and the loop terminates on convergence or a safety backstop. Archetypes: ship-ready, optimize-metric, fix-broken, harden, build-feature, explore. **Single-pass dispatch** — used when no mechanical predicate exists. The goal is subjective or the subcommand is internally-converging (reason runs its own adversarial loop) or a one-shot terminal emitter (learn, improve produce a document and stop). The orchestrator routes once, the subcommand self-terminates, and the orchestrator reports the result. No Units remaining, no Plateau counter, no ship gate. Archetypes: document, what-to-build, decide-design. The criterion is: "Can the orchestrator independently verify done without re-running the subcommand?" If yes → loop. If no → dispatch. ## Build-Feature: TDD Ladder The `build-feature` archetype has no pre-existing metric, so progress is reframed as `green-assertion-count` (monotone integer, higher-is-better). A change that turns a red sub-test green is kept; a change that regresses a green sub-test is reverted. A floor-guard prevents reverting scaffolding commits that compile and add no new failures but pass zero new tests. Large net-new scope (greenfield with no existing test suite) is detected and the orchestrator advises handing off to a dedicated build command rather than grinding cycles. ## Preset Pipelines (Reference) | Archetype | Step 1 | Step 2 | Step 3 | Step 4 | Step 5 | |---|---|---|---|---|---| | ship-ready | probe | debug | fix | regression | ship | | optimize-metric | plan | (classic loop) | holdout-verify | evals | — | | fix-broken | debug | fix | regression | — | — | | harden | security | fix | security | — | — | | build-feature | (acceptance-test derive) | debug | fix | regression | — | | explore | probe | scenario | plan | — | — | | document | learn | — | — | — | — | | what-to-build | improve | — | — | — | — | | decide-design | reason | — | — | — | — | Presets are starting pipelines. The router adapts per cycle from observed state — it may skip, repeat, or reorder steps based on the decision table above. The preset is a prior, not a fixed schedule. ## Glossary Terms used consistently across this file, SKILL.md, and orchestrator-state.json. Definitions live in CONTEXT.md. | Term | Short meaning | |---|---| | Goal archetype | Classification of the user's natural-language goal into one of the 9 categories above | | Success predicate | Exact shell command + expected output that defines "done" for Orchestration loop goals | | Units remaining | Scalar measure of open gaps (failing tests, errors, metric delta); lower-is-better; computed by `scripts/orchestrate.sh units` | | Plateau | Units remaining flat or worse for N consecutive computed cycles (default 5); oscillation that nets zero also qualifies | | Orchestration loop | The cycle-bounded assess→route→run→record loop used for predicate-bearing archetypes | | Single-pass dispatch | One-shot routing to a self-terminating subcommand; no loop, Plateau, ceiling, or ship gate | | Independent verify hop | A `verify` routing step (reason/predict) that checks an accepted high-impact change against a fresh signal before DONE/ship; gated by `pending_verify` | | Holdout-verify | Acceptance check run on a held-out set, separate from the `units` signal used to choose the change, to prevent overfitting the metric | -
predict-personas.md 3.1 KB
# Predict Personas ## Default Persona Set (5 personas) ### 1. Software Architect - **Focus:** System design, component boundaries, data flow, scalability - **Questions:** Does this design scale? Are boundaries clean? Is coupling minimized? Will this survive 10x growth? - **Evidence required:** file:line citations, dependency graphs, coupling metrics - **Red flags:** God classes, circular dependencies, leaky abstractions, shared mutable state ### 2. Security Analyst - **Focus:** Attack surfaces, auth/authz, data protection, injection vectors - **Questions:** Can this be exploited? Are trust boundaries enforced? Is data sanitized? Are secrets protected? - **Evidence required:** file:line citations, attack scenarios, data flow through trust boundaries - **Red flags:** Raw SQL, missing authz, hardcoded secrets, unsanitized user input ### 3. Performance Engineer - **Focus:** Latency, throughput, resource usage, algorithmic complexity - **Questions:** Will this be fast enough? What's the worst case? Where are the bottlenecks? Is caching effective? - **Evidence required:** file:line citations, complexity analysis, resource estimates - **Red flags:** N+1 queries, unbounded loops, missing indexes, synchronous I/O in hot paths ### 4. Reliability Engineer - **Focus:** Error handling, failure modes, observability, recovery - **Questions:** What happens when this fails? Can we detect it? Can we recover? Is it observable? - **Evidence required:** file:line citations, failure scenarios, recovery paths - **Red flags:** Swallowed errors, missing retries, no circuit breakers, silent failures ### 5. Devil's Advocate - **Focus:** Assumptions, edge cases, hidden complexity, maintainability - **Questions:** What assumptions are wrong? What's the simplest thing that breaks this? Is this over-engineered? - **Evidence required:** Concrete counter-examples, edge case scenarios - **Red flags:** Happy-path-only design, untested assumptions, complexity without justification ## Adversarial Persona Set (activated with --adversarial) Replace default personas with hostile reviewers: 1. **The Breaker** — tries to crash/corrupt the system 2. **The Cheater** — finds ways to bypass rules and abuse features 3. **The Scaler** — imagines 1000x load and finds what breaks 4. **The Newbie** — misuses every API and expects it to work 5. **The Malicious Insider** — has credentials, wants to exfiltrate ## Debate Protocol 1. Each persona analyzes independently (no shared context between personas) 2. Findings reported with confidence score (0-100%) 3. Cross-examination: personas challenge each other's findings 4. Synthesizer aggregates, removes duplicates, resolves conflicts 5. Anti-herd check: if all personas agree, synthesizer must find at least 1 counter-argument 6. Final consensus: ranked findings with persona attribution ## Output Format Each persona produces: ``` ### [Persona Name] — [N findings] | # | Finding | Severity | Confidence | File:Line | Recommendation | ``` Synthesizer produces: ``` ### Consensus — [N findings after dedup] | # | Finding | Severity | Agreement | Source Personas | Action | ``` -
reason-judge-protocol.md 3.4 KB
# Reason Judge Protocol ## Adversarial Refinement Loop ``` Round N: 1. Author-A generates candidate (or incumbent from previous round) 2. Critic attacks candidate — MUST find weaknesses (forced adversarial) 3. Author-B reads task + candidate-A + critique → produces candidate-B 4. Synthesizer reads A + B → produces hybrid candidate-AB 5. Judge panel receives 3 candidates with randomized labels → picks winner 6. Winner becomes incumbent for round N+1 ``` ## Agent Isolation Rules - Each agent (Author-A, Critic, Author-B, Synthesizer, Judges) runs COLD START - No shared session state between agents — prevents sycophancy - Agents receive ONLY: task description + relevant candidate(s) + critique - Judges receive candidates with randomized labels (Label-X, Label-Y, Label-Z) - Judges MUST compare and rank — "all are good" is not a valid verdict ## Critic Protocol The critic MUST: 1. Identify at least 3 specific weaknesses in the candidate 2. Provide concrete evidence for each weakness 3. Suggest what a superior candidate would do differently 4. Rate candidate on domain-specific criteria (1-10 scale) 5. Never compliment the candidate — role is purely adversarial ## Judge Protocol Each judge receives: - Task description (identical for all judges) - 3 candidates with randomized labels (Label-X, Label-Y, Label-Z) - Evaluation criteria relevant to the domain Each judge MUST: 1. Evaluate each candidate independently on all criteria 2. Produce a ranking (1st, 2nd, 3rd) with reasoning 3. Select a winner with one-paragraph justification 4. Label randomization prevents position bias Verdict: majority vote. Tie → synthesized candidate (Label-Z) wins. ## Convergence Detection | Mode | Stop Condition | |---|---| | Convergent (default) | Same incumbent wins N consecutive rounds (default N=3) | | Creative | Never auto-stops; runs until iteration limit | | Debate | Same as convergent but no synthesis step | ## Oscillation Guard If the incumbent changes more than 5 times in the last 8 rounds → recommend early stop. The candidates are not converging — further rounds waste context. ## Domain-Specific Judge Criteria | Domain | Criteria | |---|---| | Software architecture | Scalability, maintainability, performance, security, simplicity | | Product strategy | Market fit, feasibility, differentiation, risk, timeline | | Business decision | ROI, risk, alignment, resource requirements, reversibility | | Security approach | Coverage, false positive rate, practicality, compliance | | Research hypothesis | Testability, novelty, evidence support, explanatory power | | Content/writing | Clarity, accuracy, engagement, completeness, actionability | ## Output Files | File | Content | |---|---| | `reason-results.tsv` | Per-round: round, candidate_label, judge_verdict, convergence_count, description | | `lineage.md` | Full history of all candidates + critiques + judge reasoning | | `summary.md` | Final winner, convergence trajectory, key insights | | `handoff.json` | Chain handoff with winner as primary finding | ## TSV Schema ``` round timestamp candidate_label judge_verdict convergence_count description 1 2026-05-19T00:00:00Z Candidate-A winner 1 Event sourcing with CQRS 2 2026-05-19T00:05:00Z Candidate-AB winner 1 Hybrid: event sourcing for writes, read projections 3 2026-05-19T00:10:00Z Candidate-AB winner 2 Refined hybrid with materialized views 4 2026-05-19T00:15:00Z Candidate-AB winner 3 CONVERGED — same approach refined ``` -
security-checklist.md 3.4 KB
# Security Audit Checklist ## STRIDE Threat Categories | Category | Threat | Look For | |---|---|---| | Spoofing | Identity impersonation | Weak auth, token prediction, session fixation | | Tampering | Data modification | Unvalidated input, missing integrity checks, SQL injection | | Repudiation | Deniable actions | Missing audit logs, unsigned transactions | | Info Disclosure | Data leaks | Error messages with stack traces, verbose logging, exposed env vars | | Denial of Service | Availability attacks | Unbounded queries, missing rate limits, regex DoS | | Elevation of Privilege | Unauthorized access | Missing authz checks, IDOR, privilege escalation paths | ## OWASP Top 10 (2021) Checklist | # | Category | Key Checks | |---|---|---| | A01 | Broken Access Control | IDOR, missing function-level authz, CORS misconfiguration, path traversal | | A02 | Cryptographic Failures | Plaintext secrets, weak algorithms, missing TLS, hardcoded keys | | A03 | Injection | SQL, NoSQL, OS command, LDAP, XSS (stored/reflected/DOM) | | A04 | Insecure Design | Missing threat model, no rate limiting, no abuse prevention | | A05 | Security Misconfiguration | Default credentials, unnecessary features enabled, missing headers | | A06 | Vulnerable Components | Known CVEs in dependencies, outdated packages, unmaintained libs | | A07 | Auth Failures | Credential stuffing, brute force, weak passwords, missing MFA | | A08 | Data Integrity Failures | Unsigned updates, insecure deserialization, CI/CD poisoning | | A09 | Logging Failures | Missing security events, insufficient monitoring, no alerting | | A10 | SSRF | Unvalidated URLs, internal service access, cloud metadata exposure | ## Red-Team Personas | Persona | Focus | Mindset | |---|---|---| | Security Adversary | Auth, crypto, injection | External attacker with browser + Burp Suite | | Supply Chain Attacker | Dependencies, CI/CD, build pipeline | Compromise through third-party code | | Insider Threat | Data access, privilege abuse, exfiltration | Authenticated user with malicious intent | | Infrastructure Attacker | Network, cloud config, containers | Target infrastructure misconfigurations | ## Severity Classification | Severity | Criteria | Examples | |---|---|---| | Critical | Remote exploitation, no auth required, data breach | RCE, SQL injection, auth bypass | | High | Requires some access, significant impact | Stored XSS, IDOR, privilege escalation | | Medium | Limited impact or requires interaction | CSRF, reflected XSS, info disclosure | | Low | Minimal impact, informational | Missing headers, verbose errors | | Info | Best practice recommendation | Hardening suggestions, defense in depth | ## Composite Metric Formula ``` score = (owasp_categories_tested / 10) * 50 + (stride_categories_tested / 6) * 30 + min(unique_findings, 20) ``` Higher is better. Perfect score = 100 (all OWASP tested + all STRIDE tested + 20 findings). ## Coverage Tracking Print coverage summary every 5 iterations: ``` OWASP: [A01✓ A02✓ A03✗ A04✗ A05✓ A06✗ A07✓ A08✗ A09✗ A10✗] 4/10 STRIDE: [S✓ T✓ R✗ I✓ D✗ E✗] 3/6 Score: 48.3 | Findings: 7 ``` ## Finding Format Every finding requires: 1. **Title** — one-line summary 2. **Severity** — Critical/High/Medium/Low/Info 3. **OWASP** — A01-A10 category 4. **STRIDE** — S/T/R/I/D/E category 5. **Evidence** — file:line + attack scenario (no theoretical fluff) 6. **Reproduction** — steps to trigger 7. **Mitigation** — concrete fix recommendation
-
-
scripts
-
orchestrate.sh 18.5 KB
#!/usr/bin/env bash # orchestrate.sh — deterministic seam for the autoresearch orchestrator loop. # # classify <goal-string> → Goal archetype label (keyword heuristics) # next-hop <state.json> → Next subcommand from router decision table # units <results.json> → Units-remaining scalar (lower_is_better) # plateau <history.txt> → Exit 0 if last N computed values are flat-or-worse # screen-cmd <shell-string> → "ok" exit 0 | "refuse" exit 1 safety gate # verdict <state.json> → CONVERGED|PLATEAU|CEILING|BLOCKED + ship-gate # # All subcommands are pure and CI-usable via exit codes. set -uo pipefail # --------------------------------------------------------------------------- # classify: map a goal string to one of the 9 Goal archetype labels. # Priority order matters: higher-stakes archetypes checked first so that # "fix and add the broken feature" → fix-broken, not build-feature. # --------------------------------------------------------------------------- classify() { local goal="${1:?usage: classify <goal-string>}" local g g=$(printf '%s' "$goal" | tr '[:upper:]' '[:lower:]') # Security/hardening — above build because "secure" is higher stakes than "add" if printf '%s' "$g" | grep -qE '(secure|harden|vuln)'; then echo "harden"; return 0 fi # Broken/bugfix if printf '%s' "$g" | grep -qE '(fix|bug|broken)'; then echo "fix-broken"; return 0 fi # Ship/release/deploy if printf '%s' "$g" | grep -qE '(ship|release|deploy)'; then echo "ship-ready"; return 0 fi # Product direction — requires a "what …" question so bare "next"/"build" in a # build-feature goal (e.g. "build the next-gen parser") doesn't mis-route here. if printf '%s' "$g" | grep -qE '(what.*build|what.*next)'; then echo "what-to-build"; return 0 fi # Build/implement/add — "feature" alone is insufficient; any of these words qualify if printf '%s' "$g" | grep -qE '(build|implement|add)'; then echo "build-feature"; return 0 fi # Metric optimization if printf '%s' "$g" | grep -qE '(faster|smaller|reduce|optimize|coverage)'; then echo "optimize-metric"; return 0 fi # Documentation if printf '%s' "$g" | grep -qE '(document|docs)'; then echo "document"; return 0 fi # Design decision — "should we" / "decide" / "approach" if printf '%s' "$g" | grep -qE '(should we|decide|approach)'; then echo "decide-design"; return 0 fi # Default: open-ended investigation echo "explore" } # --------------------------------------------------------------------------- # next-hop: cheap router over fields in a state JSON file. # Decision order: errors → regression → untested gaps → ship/DONE. # --------------------------------------------------------------------------- next-hop() { local state_file="${1:?usage: next-hop <state.json>}" if [[ ! -f "$state_file" ]]; then echo "ERROR: missing state file" >&2; return 2 fi # Parse with sed/grep — no jq dependency (score-regression.sh doesn't use jq) local errors regression gaps archetype errors=$(grep -o '"errors_remaining"[[:space:]]*:[[:space:]]*[0-9]*' "$state_file" \ | grep -o '[0-9]*$') regression=$(grep -o '"regression_verdict"[[:space:]]*:[[:space:]]*"[^"]*"' "$state_file" \ | grep -o '"[^"]*"$' | tr -d '"') gaps=$(grep -o '"untested_gaps"[[:space:]]*:[[:space:]]*[0-9]*' "$state_file" \ | grep -o '[0-9]*$') archetype=$(grep -o '"archetype"[[:space:]]*:[[:space:]]*"[^"]*"' "$state_file" \ | grep -o '"[^"]*"$' | tr -d '"') # Optional: pending_verify gates an independent acceptance check before DONE/ship. # Absent (or false) → routing is identical to prior behavior. local pending pending=$(grep -o '"pending_verify"[[:space:]]*:[[:space:]]*[a-z]*' "$state_file" \ | grep -o '[a-z]*$') # Guard: missing required fields if [[ -z "$errors" || -z "$regression" || -z "$gaps" ]]; then echo "ERROR: malformed state file" >&2; return 2 fi if [[ "$errors" -gt 0 ]]; then echo "fix"; return 0 fi if [[ "$regression" == "UNSTABLE" ]]; then echo "regression"; return 0 fi if [[ "$gaps" -gt 0 ]]; then echo "debug"; return 0 fi # Gaps clear but an accepted high-impact change still needs a fresh, independent # acceptance check (separate from the signal used to choose it) → verify first. if [[ "$pending" == "true" ]]; then echo "verify"; return 0 fi # All clear: ship if archetype has ship in the pipeline, else DONE if [[ "$archetype" == "ship-ready" ]]; then echo "ship"; return 0 fi echo "DONE" } # --------------------------------------------------------------------------- # units: compute Units-remaining scalar from a results JSON file. # Formula: failing_tests + open_hard_regressions + (metric_delta / metric_target) # Prints "unknown" and exits 2 when inputs are missing or uncomputable. # --------------------------------------------------------------------------- units() { local results_file="${1:?usage: units <results.json>}" if [[ ! -f "$results_file" ]]; then echo "unknown"; return 2 fi local ft regressions delta target ft=$(grep -o '"failing_tests"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \ | grep -o '[0-9.]*$') regressions=$(grep -o '"open_hard_regressions"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \ | grep -o '[0-9.]*$') delta=$(grep -o '"metric_delta"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \ | grep -o '[0-9.]*$') target=$(grep -o '"metric_target"[[:space:]]*:[[:space:]]*[0-9.]*' "$results_file" \ | grep -o '[0-9.]*$') if [[ -z "$ft" || -z "$regressions" || -z "$delta" || -z "$target" ]]; then echo "unknown"; return 2 fi # Integer check: metric_target must be non-zero to avoid divide-by-zero if [[ "$target" == "0" || "$target" == "0.0" ]]; then echo "unknown"; return 2 fi # awk handles floating point; strip trailing .0 for clean integer output awk -v ft="$ft" -v r="$regressions" -v d="$delta" -v t="$target" ' BEGIN { val = ft + r + (d / t) # Strip unnecessary trailing zeros (e.g. 4.500 → 4.5, 0.000 → 0) if (val == int(val)) printf "%d\n", val else printf "%g\n", val } ' } # --------------------------------------------------------------------------- # plateau: read newline list of unit values; determine if progress has stalled. # Skips interleaved "unknown" cycles; N=5 consecutive trailing unknowns = BLOCKED. # Exit 0 = plateau (no net progress); exit 1 = still improving; exit 3 = BLOCKED. # --------------------------------------------------------------------------- plateau() { local history_file="${1:?usage: plateau <history.txt>}" local n=5 if [[ ! -f "$history_file" ]]; then echo "BLOCKED"; return 3 fi awk -v n="$n" ' { line = $0 gsub(/^[[:space:]]+|[[:space:]]+$/, "", line) if (line == "unknown") { trailing_unknown++; next } # crash/uncomputable cycle if (line ~ /^[0-9]/) { vals[++count] = line + 0; trailing_unknown = 0 } } END { # A runner stuck emitting "unknown" must not read as progress: n consecutive # trailing unknowns (or no computed value at all) → BLOCKED, not "improving". if (trailing_unknown >= n) { print "BLOCKED"; exit 3 } if (count == 0) { print "BLOCKED"; exit 3 } # Need at least n computed values before a plateau call. if (count < n) { exit 1 } # Net progress over the window = last value strictly below the first # (lower_is_better). Any oscillation that nets flat-or-worse is a plateau, # so a thrashing loop stops instead of running to the ceiling. start = count - n + 1 if (vals[count] < vals[start]) { exit 1 } # net improvement → still working exit 0 # flat or worse → plateau } ' "$history_file" } # --------------------------------------------------------------------------- # screen-cmd: safety gate for shell strings before execution. # Prints "ok" / "refuse". Anchored DB-host allowlist: only localhost, # 127.0.0.1, or a plain hostname (no dots) with a _test or _ci dbname suffix. # Bare substring "test" inside words like "latest" or "precision" must NOT qualify. # --------------------------------------------------------------------------- screen-cmd() { local cmd="${1:?usage: screen-cmd <shell-string>}" # rm with recursive AND force, in any flag arrangement: bundled (-rf/-Rf/-fr), # separate (-r -f), or long (--recursive --force). Both flags must be present. # The optional path prefix catches path-qualified invocations (/bin/rm, ./rm, # /usr/local/bin/rm) that a bare command-name anchor would miss. if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?rm([[:space:]]|$)'; then local rm_rec=0 rm_force=0 printf '%s' "$cmd" | grep -qE -- '(^|[[:space:]])-[a-zA-Z]*[rR]|--recursive' && rm_rec=1 printf '%s' "$cmd" | grep -qE -- '(^|[[:space:]])-[a-zA-Z]*[fF]|--force' && rm_force=1 if [[ "$rm_rec" -eq 1 && "$rm_force" -eq 1 ]]; then echo "refuse"; return 1 fi fi # curl/wget piped to an interpreter (sh/bash/zsh/dash/fish/ksh/python/perl/ruby/ # node/php), including a path-qualified one (| /bin/bash). Enumerated interpreters # rather than "refuse any curl pipe" so a legitimate derived predicate that pipes # curl output to a parser (jq/grep/awk) is not falsely refused. if printf '%s' "$cmd" | grep -qE '(curl|wget)[^|]*\|[[:space:]]*([^[:space:]]*/)?(sh|bash|zsh|dash|fish|ksh|python[0-9.]*|perl|ruby|node|php)([[:space:]]|$)'; then echo "refuse"; return 1 fi # curl/wget routed through xargs into an interpreter. The xargs wrapper sidesteps the # direct pipe matcher above, so a remote payload still reaches a shell. if printf '%s' "$cmd" | grep -qE '(curl|wget)[^|]*\|.*xargs.*[[:space:]]([^[:space:]]*/)?(sh|bash|zsh|dash|ksh|python[0-9.]*|perl|ruby|node|php)([[:space:]]|$)'; then echo "refuse"; return 1 fi # Output piped to netcat exfiltrates data off-host. if printf '%s' "$cmd" | grep -qE '\|[[:space:]]*([^[:space:]]*/)?(nc|ncat|netcat)([[:space:]]|$)'; then echo "refuse"; return 1 fi # Raw block-device write — dd target or shell redirect onto a disk device wipes it. # Scoped to real device families (incl. SD/eMMC mmcblk, mdadm md, device-mapper dm-) # so dd/redirect to /dev/null or a regular file stays ok. if printf '%s' "$cmd" | grep -qE '(of=|>[[:space:]]*)/dev/(sd|hd|vd|nvme|disk|mapper|loop|xvd|mmcblk|md|dm-)'; then echo "refuse"; return 1 fi # Filesystem format destroys everything on a partition. Optional path prefix catches a # path-qualified invocation (/sbin/mkfs.ext4) that a bare-name anchor would miss. if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?(mkfs|mke2fs)'; then echo "refuse"; return 1 fi # find ... -delete mass-removes matched files. Both tokens required so a plain find # search (no -delete) is not refused; optional path prefix catches /usr/bin/find. if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?find([[:space:]]|$)' \ && printf '%s' "$cmd" | grep -qE '[[:space:]]-delete([[:space:]]|$)'; then echo "refuse"; return 1 fi # shred overwrites then unlinks — irrecoverable. if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?shred([[:space:]]|$)'; then echo "refuse"; return 1 fi # truncate to zero size destroys file contents in place. Non-zero sizes are allowed. # Optional path prefix catches /usr/bin/truncate; size matcher covers -s 0, -s0, # --size 0, and --size=0. if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?truncate([[:space:]]|$)' \ && printf '%s' "$cmd" | grep -qE '(-s[[:space:]]*0|--size[[:space:]]*=?[[:space:]]*0)([[:space:]]|$)'; then echo "refuse"; return 1 fi # Recursive chmod to a zero mode locks an entire tree out of access. Scoped to the # zero lock-out (000/00/0 octal short forms) so ordinary recursive permission changes # are not refused; optional path prefix catches /bin/chmod. if printf '%s' "$cmd" | grep -qE '(^|[[:space:]])([^[:space:]]*/)?chmod([[:space:]]|$)' \ && printf '%s' "$cmd" | grep -qE '(-R|--recursive)([[:space:]]|$)' \ && printf '%s' "$cmd" | grep -qE '(^|[[:space:]])(000|00|0)([[:space:]]|$)'; then echo "refuse"; return 1 fi # Fork bomb pattern if printf '%s' "$cmd" | grep -qF ':(){ :|:'; then echo "refuse"; return 1 fi if printf '%s' "$cmd" | grep -qE ':\(\)\{'; then echo "refuse"; return 1 fi # AWS credential patterns (key IDs start with AKIA, secret keys are 40-char base64) if printf '%s' "$cmd" | grep -qE 'AKIA[0-9A-Z]{16}'; then echo "refuse"; return 1 fi # PASSWORD= credential pattern if printf '%s' "$cmd" | grep -qE 'PASSWORD[[:space:]]*='; then echo "refuse"; return 1 fi # Private key headers — pattern starts with dashes so pass -- to avoid flag misparse if printf '%s' "$cmd" | grep -qE -- 'BEGIN (RSA |EC |OPENSSH |DSA )?PRIVATE KEY'; then echo "refuse"; return 1 fi # Database URL safety: extract host and dbname from postgres:// or postgresql:// URIs # Pattern: postgres(ql)://user:pass@HOST/DBNAME or postgres(ql)://HOST/DBNAME if printf '%s' "$cmd" | grep -qE 'postgres(ql)?://'; then # Extract the host portion (after @ or after ://) local db_host db_name db_host=$(printf '%s' "$cmd" \ | grep -oE 'postgres(ql)?://[^[:space:]]+' \ | sed -E 's|postgres(ql)?://([^@]+@)?([^/:]+)[:/].*|\3|') db_name=$(printf '%s' "$cmd" \ | grep -oE 'postgres(ql)?://[^[:space:]]+' \ | sed -E 's|postgres(ql)?://[^/]*/([^?[:space:]]+).*|\2|') # Allowed hosts: localhost, 127.0.0.1, or a single-label hostname (no dots = container) local host_ok=0 if [[ "$db_host" == "localhost" || "$db_host" == "127.0.0.1" ]]; then host_ok=1 elif printf '%s' "$db_host" | grep -qvE '\.'; then # No dots = plain container hostname → allowed host_ok=1 fi if [[ "$host_ok" -eq 0 ]]; then # Non-allowlisted host: dbname must end with _test or _ci (anchored suffix, not substring) if printf '%s' "$db_name" | grep -qE '_test$|_ci$'; then echo "ok"; return 0 fi echo "refuse"; return 1 fi fi echo "ok"; return 0 } # --------------------------------------------------------------------------- # verdict: synthesize a convergence verdict from state JSON. # Reads: units, plateau, ceiling fields. Prints verdict + ship-gate line. # Exit 0 = CONVERGED; exit 1 = not converged; exit 2 = error. # --------------------------------------------------------------------------- verdict() { local state_file="${1:?usage: verdict <state.json>}" if [[ ! -f "$state_file" ]]; then echo "BLOCKED"; echo "ship=no"; return 2 fi local units_val plateau_val ceiling_val units_val=$(grep -o '"units"[[:space:]]*:[[:space:]]*[0-9.]*' "$state_file" \ | grep -o '[0-9.]*$') plateau_val=$(grep -o '"plateau"[[:space:]]*:[[:space:]]*[a-z]*' "$state_file" \ | grep -o '[a-z]*$') ceiling_val=$(grep -o '"ceiling"[[:space:]]*:[[:space:]]*[a-z]*' "$state_file" \ | grep -o '[a-z]*$') if [[ -z "$units_val" ]]; then echo "BLOCKED"; echo "ship=no"; return 2 fi if [[ "$plateau_val" == "true" ]]; then echo "PLATEAU"; echo "ship=no"; return 1 fi if [[ "$ceiling_val" == "true" ]]; then echo "CEILING"; echo "ship=no"; return 1 fi # units==0 with no plateau/ceiling → converged if awk -v u="$units_val" 'BEGIN { exit (u == 0 ? 0 : 1) }'; then echo "CONVERGED"; echo "ship=yes"; return 0 fi # units > 0, no plateau/ceiling signal yet → still running echo "BLOCKED"; echo "ship=no"; return 1 } # --------------------------------------------------------------------------- # validate-state: schema gate for orchestrator-state.json. The ledger is the # loop's evidence trail; a malformed one must not be trusted to route from. # Prints "valid" exit 0 | "invalid" exit 2. Node is a checked installation # prerequisite, so use its JSON parser rather than approximating JSON with grep. # --------------------------------------------------------------------------- validate-state() { local state_file="${1:?usage: validate-state <state.json>}" if [[ ! -f "$state_file" ]]; then echo "invalid"; return 2 fi if ! node -e ' const fs = require("fs"); const state = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); const strings = ["goal", "archetype", "predicate", "terminal_choice"]; if (!strings.every((key) => typeof state[key] === "string" && state[key].length > 0)) process.exit(1); if (!Number.isInteger(state.cycle) || state.cycle < 0) process.exit(1); if (!Array.isArray(state.units_remaining) || !Array.isArray(state.pipeline_log)) process.exit(1); ' "$state_file" 2>/dev/null; then echo "invalid"; return 2 fi echo "valid"; return 0 } # --------------------------------------------------------------------------- # screen-state-predicate: extract the pinned predicate from a persisted state # file and re-run it through screen-cmd. Persisted commands are never trusted — # a poisoned state file must not re-enter the loop with an unscreened command. # Delegates the verdict (ok/refuse + exit) to screen-cmd; "invalid" exit 2 when # the state has no pinned predicate. # --------------------------------------------------------------------------- screen-state-predicate() { local state_file="${1:?usage: screen-state-predicate <state.json>}" if [[ ! -f "$state_file" ]]; then echo "invalid"; return 2 fi local pred if ! pred=$(node -e ' const fs = require("fs"); const state = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); if (typeof state.predicate !== "string" || state.predicate.length === 0 || state.predicate.includes("\0")) process.exit(1); process.stdout.write(state.predicate); ' "$state_file" 2>/dev/null); then echo "invalid"; return 2 fi screen-cmd "$pred" } case "${1:-}" in classify) shift; classify "$@" ;; next-hop) shift; next-hop "$@" ;; units) shift; units "$@" ;; plateau) shift; plateau "$@" ;; screen-cmd) shift; screen-cmd "$@" ;; verdict) shift; verdict "$@" ;; validate-state) shift; validate-state "$@" ;; screen-state-predicate) shift; screen-state-predicate "$@" ;; *) echo "usage: $0 {classify|next-hop|units|plateau|screen-cmd|verdict|validate-state|screen-state-predicate}" >&2; exit 64 ;; esac -
score-regression.sh 7.5 KB
#!/usr/bin/env bash # score-regression.sh — scoring backend for autoresearch:regression # # rubric [file] → grep-rubric quality score of regression.md → "SCORE: N" # verdict <results.tsv> → tiered stability verdict from a results TSV # # verdict logic: # - any HARD row that is a green→red regression (classification=regression-eligible, regressed=true) → UNSTABLE # - else weighted SCORE: per-dim worst subscore, weights renormalized over dims that ran, # STABLE iff stability_score >= threshold (default 95) # - classification in {pre-existing,new-coverage,baseline-unavailable,flaky} never gates # - exit 0 STABLE / 1 UNSTABLE / 2 ERROR (CI-usable) · score math → stderr # # Overridable env: REG_THRESHOLD, REG_W_FLAKINESS, REG_W_PERFORMANCE, REG_W_RESOURCE, REG_W_VISUAL set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" resolve_default_spec() { local candidate for candidate in \ "$SCRIPT_DIR/../regression.md" \ "$SCRIPT_DIR/../../../commands/autoresearch/regression.md" \ "$SCRIPT_DIR/../../../commands/autoresearch_regression.md" \ "$REPO_ROOT/.claude/commands/autoresearch/regression.md" \ "$REPO_ROOT/claude-plugin/commands/autoresearch/regression.md"; do [[ -f "$candidate" ]] && { printf '%s\n' "$candidate"; return; } done printf '%s\n' "$SCRIPT_DIR/../regression.md" } SPEC_DEFAULT="$(resolve_default_spec)" REG_THRESHOLD="${REG_THRESHOLD:-95}" REG_W_FLAKINESS="${REG_W_FLAKINESS:-0.30}" REG_W_PERFORMANCE="${REG_W_PERFORMANCE:-0.30}" REG_W_RESOURCE="${REG_W_RESOURCE:-0.20}" REG_W_VISUAL="${REG_W_VISUAL:-0.20}" # --------------------------------------------------------------------------- # rubric: grep the protocol spec for required invariants/sections/flags. # Each pattern matched = +1. Prints "SCORE: N" only. # --------------------------------------------------------------------------- rubric() { local file="${1:-$SPEC_DEFAULT}" if [[ ! -f "$file" ]]; then echo "SCORE: 0"; return 0; fi local checks=( "classification" "green.{0,3}red" "regression.eligible|[^a-z]eligible" "pre-existing" "new-coverage" "baseline.unavailable|BASELINE_UNAVAILABLE" "functional" "api-contract" "data-migration" "integration-e2e" "flakiness" "performance" "resource" "visual" "HARD" "SCORE" "worktree" "--detach|detach" "submodule" "baseline.cache|--baseline-cache" "--select" "findRelatedTests|nx affected|affected" "Mann.?Whitney" "independent.process" "effect.size|median delta" "SSIM|maxDiffPixelRatio|pixel.ratio" "samples" "noise-band" "forward-only" "allowlist" "fix-cycle|--fix-cycles" "probe" "auto-skip" "--max-runs" "handoff" "verdict" "STABLE" "UNSTABLE" ) local score=0 pat for pat in "${checks[@]}"; do if grep -qiE -- "$pat" "$file"; then score=$((score + 1)); fi done echo "SCORE: $score" } # --------------------------------------------------------------------------- # verdict: reduce a results TSV to STABLE|UNSTABLE + stability score. # --------------------------------------------------------------------------- verdict() { local tsv="${1:?usage: verdict <results.tsv>}" if [[ ! -f "$tsv" ]]; then echo "VERDICT: ERROR"; echo "score=0.0"; echo "blocking=missing-tsv" return 2 fi awk -v FS='\t' \ -v wF="$REG_W_FLAKINESS" -v wP="$REG_W_PERFORMANCE" \ -v wR="$REG_W_RESOURCE" -v wV="$REG_W_VISUAL" \ -v thr="$REG_THRESHOLD" ' /^#/ { next } # comment (e.g. metric_direction) $1=="iteration" { next } # header NF != 15 { invalid_schema=1; next } { dim=$3; tier=$5; cls=$6; regressed=$10; subv=$11+0; valid_dim = (dim=="functional" || dim=="api-contract" || dim=="data-migration" || dim=="integration-e2e" || dim=="flakiness" || dim=="performance" || dim=="resource" || dim=="visual-ui"); expected_tier = (dim=="functional" || dim=="api-contract" || dim=="data-migration" || dim=="integration-e2e") ? "HARD" : "SCORE"; if (!valid_dim || tier!=expected_tier || (regressed!="true" && regressed!="false") || $11 !~ /^([0-9]+([.][0-9]+)?|[.][0-9]+)$/ || subv<0 || subv>100) { invalid_schema=1; next; } if (cls!="regression-eligible" && cls!="pre-existing" && cls!="new-coverage" && cls!="baseline-unavailable" && cls!="flaky") { invalid_classification=cls; next; } if (cls!="baseline-unavailable") { nrows++; present[dim]=1; } if (tier=="HARD" && regressed=="true" && cls=="regression-eligible") hardset[dim]=1; if (tier=="SCORE" && cls!="baseline-unavailable") { if (!(dim in dmin) || subv < dmin[dim]) dmin[dim]=subv; # per-dim worst case } } END { if (invalid_schema) { printf "VERDICT: ERROR\n"; printf "score=0.00\n"; printf "blocking=invalid-schema\n"; exit 2; } if (invalid_classification!="") { printf "VERDICT: ERROR\n"; printf "score=0.00\n"; printf "blocking=invalid-classification\n"; exit 2; } # No measurable data rows ran at all → nothing to gate on. Must NOT read as a # green ship signal: an empty/header-only TSV or all-dims-unavailable run is # advisory, not STABLE. Emit BASELINE_UNAVAILABLE + non-zero exit. if (nrows==0) { printf "VERDICT: BASELINE_UNAVAILABLE\n"; printf "score=0.00\n"; printf "blocking=no-dims-ran\n"; printf "dims_ran=none\n"; printf "dims_unavailable=functional,api-contract,data-migration,integration-e2e,flakiness,performance,resource,visual-ui\n"; exit 2; } hb=""; for (d in hardset) hb = hb (hb==""?"":",") d; wt["flakiness"]=wF; wt["performance"]=wP; wt["resource"]=wR; wt["visual-ui"]=wV; num=0; den=0; for (d in dmin) { w=(d in wt)?wt[d]:0; if (w>0){ num+=w*dmin[d]; den+=w; } } score = (den>0) ? num/den : 100; split("functional,api-contract,data-migration,integration-e2e,flakiness,performance,resource,visual-ui", reg, ","); ran=""; unavail=""; for (i=1;i<=8;i++){ if (reg[i] in present) ran = ran (ran==""?"":",") reg[i]; else unavail = unavail (unavail==""?"":",") reg[i]; } unstable = (hb!="") || (score < thr); verdict = unstable ? "UNSTABLE" : "STABLE"; if (hb!="" && score<thr) blocking = hb ",score"; else if (hb!="") blocking = hb; else if (score<thr) blocking = "score"; else blocking = "none"; # Display floors to 2 decimals (never rounds up): a true 94.9999 must print as # 94.99 — not 95.00 — so the shown number cannot read >= threshold while UNSTABLE. # The gate itself (above) compares full precision. disp = int(score*100)/100; printf "VERDICT: %s\n", verdict; printf "score=%.2f\n", disp; printf "blocking=%s\n", blocking; printf "dims_ran=%s\n", (ran==""?"none":ran); printf "dims_unavailable=%s\n", (unavail==""?"none":unavail); for (d in dmin) printf " %-12s subscore=%.2f weight=%s\n", d, int(dmin[d]*100)/100, ((d in wt)?wt[d]:"0") > "/dev/stderr"; printf " stability_score=%.2f threshold=%s\n", disp, thr > "/dev/stderr"; exit (unstable ? 1 : 0); } ' "$tsv" } case "${1:-}" in rubric) shift; rubric "$@" ;; verdict) shift; verdict "$@" ;; *) echo "usage: $0 {rubric [file] | verdict <results.tsv>}" >&2; exit 64 ;; esac
-
-
autoresearch.md 4.8 KB
--- name: autoresearch description: "Autonomous iteration loop: modify, verify, keep/discard against any metric" argument-hint: "[Goal: <text>] [Scope: <glob>] [Metric: <text>] [Verify: <cmd>] [Guard: <cmd>] [Iterations: N] [--evals]" --- EXECUTE IMMEDIATELY — do not deliberate before reading this protocol. ## Parse Arguments Extract from $ARGUMENTS: - `Goal:` — what to improve - `Scope:` or `--scope` — file globs - `Metric:` — what to measure - `Direction:` — higher_is_better (default) or lower_is_better - `Verify:` — shell command that outputs a number - `Guard:` — optional safety command (must always pass) - `Iterations:` or `--iterations` — integer N for bounded mode (default: 25). "unlimited" for unbounded. - `--evals` — enable mid-loop checkpoints - `--evals-interval N` — checkpoint frequency override - `--chain <targets>` — comma-separated downstream commands ## Setup (if required context missing) If Goal, Scope, Metric, or Verify missing → use request_user_input (single batched call): Q1 (Goal): "What do you want to improve?" Q2 (Scope): "Which files?" — suggest globs from project Q3 (Metric+Verify): "How to measure? Provide a shell command that outputs a number" Q4 (Guard): "Safety command that must always pass?" — options: test cmd, build cmd, skip If ALL provided inline → skip setup, proceed directly. ## Precondition Checks 1. Verify git repo exists (`git rev-parse --git-dir`) 2. Check clean working tree (`git status --porcelain`) — warn if dirty 3. Check for stale lock files, detached HEAD 4. If Guard set → run Guard to establish guard baseline 5. Fail fast on any critical issue. Warn on non-critical. ## Verify Safety Screen Before first dry-run, screen Verify command for: rm -rf, fork bombs, curl|sh, embedded credentials, outbound writes. Block dangerous commands. ## Establish Baseline (Iteration 0) 1. Run Verify command → extract numeric metric 2. Record as iteration 0 in TSV: `0\t{timestamp}\t{commit}\t{metric}\t0.0\t{guard}\t-\tbaseline\tinitial state` 3. Create output directory: `autoresearch/loop-{YYMMDD}-{HHMM}/` 4. Write TSV header: `# metric_direction: {direction}\niteration\ttimestamp\tcommit\tmetric\tdelta\tguard\tguard-metric\tstatus\tdescription` ## Iteration Loop For each iteration (1 to max_iterations, or unbounded): ### Phase 1: Review (read git history as memory) - Read last 10-20 lines of results TSV - Run `git log --oneline -20` — see what worked/failed - If last iteration was "keep" → run `git diff HEAD~1` to see what improved metric - Identify: what worked, what failed, what's untried ### Phase 2: Modify - Based on review, make ONE focused change to improve the metric - Change must be atomic — one logical unit of work ### Phase 3: Commit - Stage and commit with `experiment: {description}` prefix - Record commit SHA ### Phase 4: Verify - Run Verify command → extract new metric value - Calculate delta from previous iteration - Metric improved (correct direction) → candidate for keep ### Phase 5: Guard (if configured) - Run Guard command. If fails → revert regardless of metric improvement ### Phase 6: Decide - **keep** — metric improved, guard passed → commit stays - **discard** — metric worsened → `git revert HEAD --no-edit` - **crash** — verify/guard command failed → `git revert HEAD --no-edit` - **no-op** — no change made this iteration - **hook-blocked** — git hook blocked the commit - **metric-error** — verify output not a valid number → `git revert HEAD --no-edit` ### Phase 7: Log Append row to TSV: iteration, timestamp, commit/-, metric, delta, guard status, guard-metric, status, description ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint analysis. ### Bounded Check If bounded: current_iteration >= max_iterations → exit loop, print summary. ## Summary (after loop ends) Print: total iterations, kept/discarded counts, starting metric → final metric, improvement %, top 3 most effective changes. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N. - Every {interval} iterations, pause and analyze current results TSV. - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nMetric: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down}\n{one-line recommendation}\n---` - If plateau 3+ checkpoints → recommend early stop. - At loop end → full evals summary to evals-summary.md in output directory. ## Chain Handoff After completion, write handoff.json to output directory: version "2.1.0", source "loop", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings[], config{goal, scope, metric, direction, verify}. Invoke next target in --chain order. Propagate --evals flag. -
debug.md 4 KB
--- name: autoresearch:debug description: "Hunt bugs with scientific method: hypothesize, test, falsify, repeat" argument-hint: "[Scope: <glob>] [Symptom: <text>] [Iterations: N] [--fix] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Scope:` or `--scope` — file globs to investigate - `Symptom:` or `--symptom` — error message or behavior description - `Iterations:` or `--iterations` — default 15. "unlimited" for unbounded. - `--fix` — shorthand for `--chain fix` - `--severity` — filter: critical, high, medium, low - `--technique` — force specific technique - `--evals`, `--evals-interval N`, `--chain` ## Setup (if required context missing) If Scope and Symptom both missing: 1. Auto-scan: run tests, lint, typecheck to detect existing failures 2. request_user_input (single batch): Q1 (Issue): "What's the problem?" — hunt all bugs, specific error, failing tests, CI failure, performance Q2 (Scope): "Which files?" — suggested globs + entire codebase Q3 (Depth): "How deep?" — quick (5), standard (15), deep (30+), unlimited Q4 (After): "When bugs found?" — report only, find and fix (--chain fix), chain to other, ask each time If all provided → skip. ## Investigation Techniques | Technique | When to Use | |---|---| | Binary search | Know when it worked, find when it broke | | Differential | Compare working vs broken state | | Minimal reproduction | Simplify to smallest failing case | | Trace | Follow execution path through code | | Pattern search | Grep for known anti-patterns | | Working backwards | Start from error, trace to root cause | ## Establish Baseline (before loop) 1. Auto-scan for failures if no symptom provided 2. Create output directory: `autoresearch/debug-{YYMMDD}-{HHMM}/` 3. TSV header: `# metric_direction: higher_is_better\niteration\ttimestamp\thypothesis\tstatus\ttechnique\tevidence\tfile_line` 4. Metric = cumulative confirmed findings count ## Iteration Loop ### Phase 1: Review Context - Read results TSV (past findings) - Assess: what's been tested, what vectors remain - If no hypotheses left → early stop ### Phase 2: Hypothesize - Form ONE specific, falsifiable hypothesis - Format: "I hypothesize that {X} because {evidence}. Test by {Y}." - Hypothesis must be testable and different from all previous ### Phase 3: Investigate - Apply appropriate technique for this hypothesis - Read relevant code, run targeted tests, check logs - Collect evidence (file:line references required) ### Phase 4: Classify - **confirmed** — hypothesis correct, bug found with evidence - **disproven** — hypothesis wrong, evidence against it - **inconclusive** — can't prove or disprove, needs different approach ### Phase 5: Log Append to TSV: iteration, timestamp, hypothesis, status, technique, evidence, file_line ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint analysis. ### Bounded Check If bounded: current_iteration >= max_iterations → exit loop, print summary. ## Summary Print: total hypotheses tested, confirmed/disproven/inconclusive counts, all confirmed bugs with severity and file:line. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N. - Every {interval} iterations, pause and analyze current results TSV. - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nFindings: {confirmed} confirmed | Trend: {up/flat/down}\n{one-line recommendation}\n---` - If plateau 3+ checkpoints (no new confirmed) → recommend early stop. - At loop end → full evals summary to evals-summary.md in output directory. ## Chain Handoff After completion, write handoff.json to output directory: version "2.1.0", source "debug", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = confirmed bugs with severity + file:line, config{scope, symptom}. If --fix flag → chain to fix automatically. Invoke next target in --chain order. Propagate --evals flag. -
evals.md 5 KB
--- name: autoresearch:evals description: "Analyze iteration results: trends, plateaus, regressions, recommendations" argument-hint: "[path/to/results.tsv] [--format text|json|md]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - Positional path to a specific TSV file - `--format` — output format: text (default console), json, md (markdown file) - `--compare <path>` — (v2.2.0 placeholder, not yet implemented) ## Input Discovery 1. If path provided → use that TSV directly 2. If no path → scan current directory + `autoresearch/*/` for `*-results.tsv` files 3. If multiple found → request_user_input: "Which results to analyze?" — list found files 4. If none found → request_user_input: "Provide path to results TSV" 5. Also scan project root for v2.0.03 legacy TSV files (backward compat) ## Parse TSV 1. Read line 1: extract `# metric_direction: higher_is_better|lower_is_better` comment - If missing → infer from column names (metric/error_count → guess, or ask user) 2. Read line 2: header row → detect available columns 3. Read remaining lines: data rows 4. Handle missing `timestamp` column gracefully (v2.0.03 compat) ## Column Detection & Analysis Activate analysis based on columns present in header: | Column | Analysis | |---|---| | `metric` | Trend direction, plateau detection (3+ flat iterations), diminishing returns, biggest single-iteration jumps | | `delta` | Per-iteration efficiency, cumulative improvement, effort-to-gain ratio | | `status` | Keep/discard rate, crash frequency, success streaks, failure clusters, longest winning streak | | `guard` + `guard-metric` | Guard failure rate, metric-improved-but-guard-failed analysis | | `severity` | Severity distribution (critical/high/medium/low/info), critical discovery rate per iteration | | `hypothesis` + `status` | Confirmation rate, investigation efficiency, most productive techniques | | `commit` | File hotspot analysis (cross-ref with `git diff` for kept commits), change size correlation | | `technique` | Technique effectiveness ranking | | `dimension` | Dimension coverage completeness (X/12) | | `candidate_label` + `judge_verdict` | Convergence speed, oscillation count | | `error_type` | Error category distribution, fix rate per category | | `classification` | New vs extension vs duplicate ratio, saturation curve | | `convergence_count` | Convergence trajectory | Unknown columns: report presence but skip analysis. Forward-compatible with future subcommands. ## Report Structure ``` ## Evals Summary — {subcommand} ({N} iterations) ### Key Metrics - Total iterations: N | Kept: X | Reverted: Y | Revert rate: Z% - Starting metric: A | Final metric: B | Improvement: C% ### Trend Analysis - Metric progression: [description of trajectory] - Plateau detected at iteration N (metric stable for M iterations) - Biggest win: iteration X (+delta, description) - Biggest loss: iteration Y (-delta, description) - Diminishing returns: [after iteration N, average delta dropped below threshold] ### Patterns - What types of changes succeeded: [extracted from descriptions of kept iterations] - What types of changes failed: [extracted from descriptions of discarded iterations] - File hotspots: [files changed most in kept iterations, if commit data available] - Technique effectiveness: [ranked by confirmation rate, if technique column present] ### Recommendation - [continue / stop / change strategy — based on trend, plateau, revert rate] - [specific actionable suggestion based on pattern analysis] ``` ## Output - Console: structured report (30-50 lines) - If `--format md` → write `evals-summary.md` in same directory as input TSV - If `--format json` → write `evals-summary.json` with structured data ## Mid-Loop Checkpoint Protocol (for --evals flag in other commands) This section documents the checkpoint protocol that looping commands embed: - **Adaptive interval:** `floor(max_iterations / 3)`, minimum 1. Fixed 10 for unbounded. Override: `--evals-interval N`. - **Checkpoint format (5 lines max):** ``` --- Eval Checkpoint (iterations {X}-{Y}) --- Metric: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down} {one-line recommendation} --- ``` - **Early stop recommendation:** if plateau detected for 3+ consecutive checkpoints - **Final summary:** at loop end, produce full evals report to console + evals-summary.md ### Adaptive Interval Examples | Subcommand | Default Iterations | Interval | Checkpoints At | |---|---|---|---| | reason | 8 | 2 | 2, 4, 6, 8 | | learn | 10 | 3 | 3, 6, 9, final | | debug/security | 15 | 5 | 5, 10, 15 | | fix/scenario | 20 | 6 | 6, 12, 18, final | | core | 25 | 8 | 8, 16, 24, final | | unbounded | unlimited | 10 | every 10 | ## Backward Compatibility - v2.0.03 TSV files: column names preserved, `timestamp` absence handled gracefully - Fuzzy column matching: `metric_value` → `metric`, `error_count` → `metric` - Files in project root (not `autoresearch/` subdirectory) → discovered during scan - v2.0.03 status values all supported: baseline, keep, keep (reworked), discard, crash, no-op, hook-blocked, metric-error -
fix.md 4.3 KB
--- name: autoresearch:fix description: "Crush errors one-by-one until zero remain: tests, types, lint, build" argument-hint: "[Target: <cmd>] [Scope: <glob>] [Guard: <cmd>] [Iterations: N] [--evals] [--from-debug]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Target:` or `--target` — command that shows errors (e.g., `npm test`, `tsc --noEmit`) - `Scope:` or `--scope` — file globs to modify - `Guard:` or `--guard` — safety command (must always pass) - `Iterations:` or `--iterations` — default 20. "unlimited" for unbounded. - `--from-debug` — read handoff.json from previous debug run - `--category` — filter: test, type, lint, build - `--evals`, `--evals-interval N`, `--chain` ## Setup (if required context missing) If Target and Scope both missing: 1. Auto-detect failures: run test suite, type checker, linter, build 2. Present results via request_user_input (single batched call): Q1 (Fix What): "Found [N] test failures, [M] type errors, [K] lint errors. Fix what?" — everything, only tests, only types, only lint Q2 (Guard): "Safety command that must always pass?" — npm test, tsc, npm run build, skip Q3 (Scope): "Which files can I modify?" — suggested globs from error locations + all Q4 (Launch): "Ready?" — fix until zero, fix with limit, cancel If all provided → skip setup. If --from-debug → read handoff.json for scope and findings. ## Precondition Checks Verify: git repo exists, clean working tree, no lock files, no detached HEAD. Fail fast on critical issues. ## Establish Baseline (Iteration 0) 1. Run Target command → count errors (metric = error count, direction = lower_is_better) 2. Record baseline in TSV 3. Create output directory: `autoresearch/fix-{YYMMDD}-{HHMM}/` 4. TSV header: `# metric_direction: lower_is_better\niteration\ttimestamp\terror_type\terror_fixed\tcommit\tmetric\tdelta\tguard\tstatus\tdescription` ## Iteration Loop (until zero errors or max_iterations) ### Phase 1: Review - Read results TSV + git log - Run Target to get current error list - If error count == 0 → exit loop (SUCCESS) ### Phase 2: Prioritize Order: crash/fatal → test failures → type errors → lint → warnings. Within category: easiest first (single-file fixes before cross-file). ### Phase 3: Fix ONE Thing - Pick the highest-priority error - Make ONE focused fix (atomic — addresses exactly one error) - Record error type and which error was fixed ### Phase 4: Commit - Stage and commit: `experiment: fix {error_type} — {description}` ### Phase 5: Verify - Run Target → count errors → compute delta - Expected: error count decreased by 1 or more ### Phase 6: Guard - If Guard set → run Guard. If fails → revert. ### Phase 7: Decide - **keep** — error count decreased AND guard passes - **keep (reworked)** — fix needed adjustment, second attempt worked - **discard** — error count same/increased → `git revert HEAD --no-edit` - **crash** — target/guard command failed → revert - **hook-blocked** — git hook blocked the commit - **metric-error** — target output not parseable → revert ### Phase 8: Log Append row: iteration, timestamp, error_type, error_fixed, commit/-, metric (error count), delta, guard, status, description ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint analysis. ### Bounded Check If bounded: current_iteration >= max_iterations → exit loop, print summary. ## Summary Print: total errors fixed, remaining errors, error types distribution, fix success rate. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N. - Every {interval} iterations, pause and analyze current results TSV. - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nErrors: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down}\n{one-line recommendation}\n---` - If plateau 3+ checkpoints → recommend early stop. - At loop end → full evals summary to evals-summary.md in output directory. ## Chain Handoff After completion, write handoff.json to output directory: version "2.1.0", source "fix", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = unfixed errors, config{target, scope, guard}. Invoke next target in --chain order. Propagate --evals flag. -
improve.md 6 KB
--- name: autoresearch:improve description: "Research ICP challenges, discover improvements, generate PRDs" argument-hint: "[Goal: <text>] [--icp <text>] [--discover] [--no-discover] [--seeds <categories>] [--depth shallow|standard|deep] [Iterations: N] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Goal:` — product area to improve (or full $ARGUMENTS if no keyword) - `--icp` or `ICP:` — ideal customer profile description - `--discover` — force inline codebase scan even when context exists - `--no-discover` — skip auto-discover, warn instead - `--seeds <categories>` — override default research category seeds - `--depth` — shallow (5 iterations), standard (15), deep (30) - `--features` — comma-separated feature names to pre-select for PRD generation - `Iterations:` or `--iterations` — default 15. "unlimited" for unbounded. - `--evals`, `--evals-interval N` If upstream `handoff.json` exists in CWD → read it. Map source findings to default seed categories: - probe → ICP challenges, UX & experience - predict → Competitor gaps, Revenue & growth - debug/security → Competitor gaps, ICP challenges - Override with `--seeds`. ## Setup (if Goal or ICP missing) request_user_input (single batch): Q1 (Goal): "What product area to improve?" — open text Q2 (ICP): "Who is your ideal customer?" — open text describing target buyer/user Q3 (Pain points): "Top 3 pain points your customers face?" — open text Q4 (Competitors): "Key competitors?" — open text, or "skip" Q5 (Depth): "How deep?" — shallow (5 iterations, quick scan), standard (15, recommended), deep (30+, exhaustive) If all provided inline → skip. ## Phase 1: Product Context Resolve product context (priority chain): 1. Learn summary (`autoresearch/learn-*/summary.md`, most recent) → read it 2. README.md (≥500 chars, non-boilerplate) → extract product description 3. `package.json` / `pyproject.toml` / `Cargo.toml` description (≥10 chars) → use it 4. If ALL above absent AND NOT `--no-discover` → auto-discover: scan 10 key files (manifest, routes, models, config), cap 1500 tokens 5. If `--discover` → force scan regardless of above 6. If nothing found → warn: "No product context. Run `$autoresearch learn --mode summarize` for better results." ## Phase 2: Research Loop Create output directory: `autoresearch/improve-{YYMMDD}-{HHMM}/` TSV header: `# metric_direction: higher_is_better` Columns: `iteration|timestamp|category|research_question|status|source|insight_problem|insight_mechanism|confidence|classification` **5 research categories:** 1. ICP challenges — pain points, jobs-to-be-done, unmet needs 2. Competitor gaps — weaknesses, missing features, technical differentiators 3. Market trends — timing signals, emerging patterns, regulatory shifts 4. UX & experience — interaction models, onboarding, retention mechanics 5. Revenue & growth — pricing, acquisition, monetization, upsell/expansion **Iteration protocol:** - Reserve first 5 iterations: one per category (forced breadth) - Remaining iterations: target categories with richest signal - Per iteration: form research question → WebSearch → synthesize → normalize to canonical insight schema → classify (new/extension/duplicate) → tag confidence (HIGH: 3+ sources, MEDIUM: 2, LOW: 1) → cross-check against codebase → log - **Saturation:** net-new insights < 2 for 3 consecutive non-reserved iterations → SATURATED, exit loop - Hard ceiling (Iterations flag) as infinite-loop guard **Insight schema:** `{problem: 10-word canonical form, affected_persona: ICP segment, proposed_mechanism: how to address, expected_outcome: what success looks like}` **Classification:** New = novel {problem, persona} pair. Extension = same pair, different mechanism. Duplicate = same pair + mechanism → skip. ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint. Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nInsights: {total} (+{new}) | Categories: {covered}/5 | Saturation: {window}/3\n{recommendation}\n---` ## Phase 3: Feature Ranking + Selection 1. **ICP binary gate** — filter insights not serving the stated ICP 2. **3-tier bucketing** — Must-have / Nice-to-have / Moonshot 3. **Pairwise ranking** within Must-have tier only (cap 7-10 items) 4. **2-sentence rationale** per item citing research evidence 5. **Confidence indicator** per item (HIGH / MEDIUM / LOW) Write `improvement-plan.md` with full tiered ranking. request_user_input (multi-select): present tiered list, user selects which features become PRDs. If `--features` provided → pre-select matching items, still show for confirmation. ## Phase 4: PRD Generation Per selected feature, write `prd-{feature-slug}.md`: - Top disclaimer: "Auto-generated from research findings. DECISION NEEDED items and LOW-confidence sections require your judgment." - Problem statement (from research evidence chain) - User stories (from ICP + persona data) - Requirements (functional + non-functional, MoSCoW from tier) - Acceptance criteria - Technical approach (from codebase context, framed as "suggested starting points") - Risks + confidence (evidence tiers: primary = codebase, secondary = web research) - Success metrics - `DECISION NEEDED` markers for unresolvable tradeoffs - `Open Questions` section Write `research-findings.md` — all insights with citations + confidence. Write `summary.md` — overview, research stats, category coverage, saturation status. ## Summary Print: total iterations, insights discovered (new/extension), categories covered, saturation status, PRDs generated, output directory path. ## Eval Summary (--evals flag) If --evals: write `evals-summary.md` to output directory with full analysis. ## Handoff Write `handoff.json`: version "2.1.0", source "improve", timestamp, status (COMPLETE|SATURATED|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = improvements with tier + confidence + prd_path, config{goal, icp, depth, categories_explored, insights_total, prds_generated}. Improve is a terminal emitter — no downstream chain invocation. -
learn.md 7.9 KB
--- name: autoresearch:learn description: "Scout codebase and auto-generate docs — or a navigable wiki knowledge base — with validation-fix loop" argument-hint: "[Mode: <init|update|check|summarize|wiki>] [Scope: <glob>] [Iterations: N] [--depth <level>] [--modules <list>] [--force] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Mode:` or `--mode` — init (create from scratch), update (refresh existing), check (validate), summarize (brief overview), wiki (navigable knowledge base) - `Scope:` or `--scope` — file globs to document - `Depth:` or `--depth` — overview, standard, comprehensive - `--file <path>` — specific file to document - `--scan` — force fresh codebase scout - `--topics` — comma-separated focus topics - `--modules <list>` — wiki mode: comma-separated module names/paths overriding auto-detection - `--force` — wiki mode: regenerate all pages from scratch, ignore existing manifest - `--no-fix` — validate only, don't auto-fix issues - `--format` — markdown (default), json, rst - `Iterations:` or `--iterations` — default 10. "unlimited" for unbounded. - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>` ## Setup (if Mode or Scope missing) request_user_input (single batch): Q1 (Mode): "What to do?" — init (generate docs), update (refresh), check (validate), summarize (overview), wiki (knowledge base) Q2 (Scope): "Which files?" — suggested globs + entire codebase Q3 (Depth): "How detailed?" — overview only, standard, comprehensive Q4 (Topics): "Focus on?" — architecture, API, database, testing, all If all provided → skip. ## Establish Baseline 1. Scout codebase: file tree, imports/exports, existing docs 2. Identify documentation gaps (undocumented files, outdated docs, missing READMEs) 3. Create output directory: `autoresearch/learn-{YYMMDD}-{HHMM}/` 4. TSV header: `# metric_direction: higher_is_better\niteration\ttimestamp\tfile_documented\tvalidation_status\tissues_found\tissues_fixed\tdescription` 5. Metric = files with valid documentation (higher is better) ## Summarize Mode (no loop) If mode == summarize: - One-shot: scan codebase → produce structured summary - Write summary.md to output directory - Skip iteration loop entirely ## Wiki Mode (no per-file loop) If mode == wiki: reuse Scout (Phase 1) + Analyze output, then generate a navigable `wiki/` knowledge base. Skip the init/update/check loop. Metric = `pages_generated / pages_planned × 100` (from manifest); size target 300 lines/page. ### Module Discovery (priority order) 1. `--modules` flag (explicit override, always wins; every path must resolve inside project root — reject escapes) 2. Monorepo workspaces (`workspaces` in package.json, Cargo workspace members, `pnpm-workspace.yaml`) 3. Per-directory project files (`pyproject.toml`, `Cargo.toml`, `go.mod`, `*.csproj`) 4. Heuristic: dirs with 3+ source files (code extensions only — .ts/.py/.go/.rs/.java/.rb/.swift/.kt/.c/.cpp/.cs; tests count, config/markdown don't); nested dirs roll up to nearest module ancestor Cap 10 modules. If >10, group by top-level dir; if a group has >5 sub-modules, expand and take 10 largest by file count. ### Plan (write-ahead) 1. `mkdir -p wiki/modules/`; append `wiki-manifest.json` to `.gitignore` if absent 2. Write `wiki-manifest.json` BEFORE generating — `{version:"1", generated_at, generation_status:"in_progress", modules_detected:[…], pages_planned:N, pages:{ "wiki/architecture.md":{status:"pending",type:"architecture"}, "wiki/modules/<name>.md":{…"module"}, "wiki/glossary.md":{…}, "wiki/onboarding.md":{…}, "wiki/index.md":{…} }}` 3. Write stub `wiki/index.md` listing every planned page as `[pending]` (navigation survives interruption) 4. Resume: if valid manifest exists → `--force` deletes it and regenerates all, else skip `"generated"` pages and only do `"pending"`. Corrupted manifest (invalid JSON, or missing `version`/`pages`) without `--force` → error directing user to `--force`. ### Generate (priority-first, one agent call per page, bounded context) 1. `architecture.md` — system overview from scout context. Up to 5 Mermaid diagrams, 3 types only (`graph TD`, `sequenceDiagram`, `classDiagram`); include one canonical example of each in the prompt; pick by signal. 2. Module pages (alphabetical) — per-module agent gets: (a) file listing, (b) first 50 lines of ≤10 key files (entry points → largest → alphabetical), (c) Phase 2 overview. Required sections: Overview, Key Files; optional: Patterns/API/Dependencies/Getting Started. 3. `glossary.md` — domain terms from class names, exports, types, comments; filter language keywords + stdlib; soft cap ~60-80, prioritize terms in 3+ files. 4. `onboarding.md` — reading order, env setup, first-contribution workflow, gotchas. Sources: dir structure, README/docs, manifests, entry-point sampling, `git log --since='6 months ago'` directory frequency (skip with note if not a git repo or >10s). 5. `index.md` — final pass: replace stub with real page descriptions + reading order. ### Per-page contract - `generated_by: autoresearch` in YAML frontmatter - ~300 lines/page (soft); Mermaid ≤15 nodes/diagram; forward-only cross-links, ≤10 per page ### Safety - **Secrets (2-layer):** (1) prompt instructs "summarize config, never include verbatim values from .env/credentials or strings matching key/secret/token/password; extract env var *names* not *values*"; (2) post-gen, `grep -rlE '(AKIA[0-9A-Z]{16}|sk-[a-zA-Z0-9]{20,}|ghp_[a-zA-Z0-9]{36}|password\s*[:=]\s*\S+|mongodb(\+srv)?://\S+|postgres(ql)?://\S+)' wiki/` and warn (non-blocking) in the report. - **Name collision:** before overwriting a page, check for `generated_by: autoresearch` frontmatter; if absent (user-created) skip with warning. `--force` overrides. ### Finish After each page is written, flip its manifest entry `pending`→`generated` (interrupt-safe). When all done, set `generation_status:"complete"`. Then run Phase 3 (Validate) with wiki path swap: replace `docs/` with `wiki/` (`ls wiki/*.md wiki/modules/*.md 2>/dev/null`), use maxLoc 300. Output: `✓ Wiki: [N] modules, [M] pages generated`. ## Iteration Loop (init/update/check modes) ### Phase 1: Scout - Scan for documentation gaps - Prioritize: no docs → outdated docs → incomplete docs - If no gaps remain → early stop (SUCCESS) ### Phase 2: Generate/Update - Pick highest-priority gap - Write or update documentation for ONE file/module - Follow project conventions for doc format and location ### Phase 3: Validate - Check generated docs against code: descriptions accurate? Examples valid? Links work? - Run doc linters if available - Record: validation_status (pass/fail), issues found ### Phase 4: Fix (unless --no-fix) - If validation finds issues → fix the doc - Commit clean doc: `docs: document {file/module}` ### Phase 5: Log Append to TSV: iteration, timestamp, file_documented, validation_status, issues_found, issues_fixed, description ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint. ### Bounded Check If bounded: current_iteration >= max_iterations → exit loop. ## Output - `learn-results.tsv` - `summary.md` — documentation overview - `validation-report.md` — issues found/fixed ## Summary Print: files documented, validation pass rate, issues found/fixed, remaining gaps. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nDocs written: {n} | Validation: {pass}/{total} | Gaps remaining: {m}\n{recommendation}\n---` - If 3+ checkpoints with no new docs → recommend early stop. - At loop end → full evals summary to evals-summary.md. ## Chain Handoff After completion, write handoff.json: version "2.1.0", source "learn", timestamp, status, results_tsv path, findings = documentation gaps remaining, config{mode, scope, depth}. Invoke next target in --chain order. Propagate --evals flag. -
LICENSE 1 KB · in bundle
-
plan.md 2.8 KB
--- name: autoresearch:plan description: "Convert a goal into validated Scope, Metric, Direction, Verify config" argument-hint: "[Goal: <text>] [--chain <targets>]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Goal:` — text after keyword, or full $ARGUMENTS if no keyword - `--chain <targets>` — comma-separated downstream commands - `--<subcommand>` — chain shorthand Remaining text = goal description. ## Setup (if Goal missing) request_user_input (single batch): Q1 (Goal): "What do you want to achieve?" — open text Q2 (Type): "What kind of goal?" — improve a metric, fix errors, audit security, explore edge cases, document code, ship something If Goal provided → skip. ## Phase 1: Analyze Goal Parse the goal to determine: - Is it measurable? (metric-driven vs subjective) - What's the natural scope? (files, modules, entire codebase) - What subcommand fits best? (core loop, fix, debug, security, etc.) ## Phase 2: Derive Scope 1. Scan project structure 2. Identify files relevant to the goal 3. Propose file globs 4. If ambiguous → ask user to confirm ## Phase 3: Derive Metric + Direction For metric-driven goals: - Identify what to measure (test coverage, error count, bundle size, latency, etc.) - Determine direction: higher_is_better or lower_is_better - Propose metric name and description For subjective goals: - Suggest proxy metrics where possible - Or recommend $autoresearch reason for non-measurable goals ## Phase 4: Derive Verify Command 1. Identify how to extract the metric as a number from a shell command 2. Propose Verify command (e.g., `npm test -- --coverage | grep "All files" | awk '{print $10}'`) 3. **Safety screen:** check proposed command for rm -rf, fork bombs, curl|sh, credentials 4. Dry-run the Verify command → confirm it outputs a valid number 5. If dry-run fails → adjust command and retry ## Phase 5: Derive Guard (optional) Propose a Guard command if applicable: - Test suite: `npm test` / `pytest` / `go test ./...` - Type check: `tsc --noEmit` / `mypy` - Build: `npm run build` - None if not applicable ## Phase 6: Suggest Iterations Based on goal complexity: - Simple metric improvement → 10-15 - Moderate refactoring → 20-25 - Complex multi-file changes → 30+ - Recommend bounded default, mention `Iterations: unlimited` option ## Phase 7: Present Config Output a ready-to-run autoresearch config block: ``` $autoresearch Goal: {derived goal} Scope: {derived globs} Metric: {derived metric} Direction: {higher_is_better|lower_is_better} Verify: {derived command} Guard: {derived guard or omit} Iterations: {suggested count} ``` Ask user: "Run this config now, or adjust?" ## Chain Handoff If --chain set: - Write handoff.json: version "2.1.0", source "plan", timestamp, status COMPLETE, config = derived config block - Invoke next target with the derived config -
predict.md 3.7 KB
--- name: autoresearch:predict description: "5 expert personas debate proposed changes before implementation" argument-hint: "[Scope: <glob>] [Goal: <text>] [--depth shallow|standard|deep] [--adversarial] [--chain <targets>]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Scope:` or `--scope` — file globs to analyze - `Goal:` or `--goal` — focus area for analysis - `Depth:` or `--depth` — shallow (3 personas, 1 round), standard (5, 2), deep (8, 3) - `--personas N` — override persona count (3-8) - `--rounds N` — override debate rounds (1-3) - `--adversarial` — use hostile reviewer personas instead of default - `--budget N` — max findings across all personas (default 40) - `--fail-on <severity>` — CI gate: exit non-zero if findings at/above threshold - `--incremental` — reuse existing knowledge files, update only changed files - `--chain`, `--<subcommand>` Remaining text not matching flags = goal description. ## Setup (if Scope or Goal missing) request_user_input (single batch): Q1 (Scope): "Which files to analyze?" — suggested globs + entire codebase Q2 (Goal): "What should personas focus on?" — code quality, security, performance, architecture, all Q3 (Depth): "How deep?" — shallow (3 personas, 1 round), standard (5, 2 — recommended), deep (8, 3) Q4 (Chain): "After analysis, chain to?" — debug, security, fix, ship, scenario, no chain If all provided → skip. ## Phase 1: Reconnaissance Scan all in-scope files. Build structured knowledge: - File inventory with purpose annotations - Dependency graph (imports/exports) - API surface (routes, handlers, types) - Data flow (inputs → processing → outputs → storage) - Existing test coverage map ## Phase 2: Persona Generation Load `references/predict-personas.md` for persona definitions. **Default set (5):** Architect, Security Analyst, Performance Engineer, Reliability Engineer, Devil's Advocate. **Adversarial set (--adversarial):** Breaker, Cheater, Scaler, Newbie, Malicious Insider. Each persona receives: task description + codebase knowledge + their specific evaluation criteria. Personas are isolated — no shared context between them. ## Phase 3: Independent Analysis Each persona analyzes the codebase independently: - Read relevant code through their lens - Produce findings with: title, severity, confidence (0-100%), file:line, recommendation - Max findings per persona: budget / persona_count ## Phase 4: Debate (per round) For each debate round: 1. Present all personas' findings to each other 2. Each persona can: challenge findings, raise new issues, change confidence 3. Cross-examination: personas must respond to challenges with evidence 4. No persona can dismiss without counter-evidence ## Phase 5: Consensus Synthesizer aggregates all findings: 1. Deduplicate (same file:line + same issue = merge, keep highest severity) 2. Resolve conflicts (if personas disagree, note dissent) 3. **Anti-herd check:** if all personas agree on everything, synthesizer MUST find at least 1 counter-argument 4. Rank by: severity × average confidence × persona agreement count ## Phase 6: Report Create output directory: `autoresearch/predict-{YYMMDD}-{HHMM}/` Write: - `summary.md` — top findings, consensus view, risk assessment - `debate.md` — full persona analysis + debate transcript - Per-persona sections with individual findings Print to console: top 10 findings ranked by severity × confidence. ## Phase 7: CI Gate If `--fail-on` set: check findings against threshold. Exit non-zero if exceeded. ## Chain Handoff Write handoff.json: version "2.1.0", source "predict", timestamp, status (COMPLETE|ERROR), findings = consensus findings with severity + confidence + file:line, config{scope, goal, depth}. Invoke next target in --chain order. -
probe.md 4.8 KB
--- name: autoresearch:probe description: "8 personas interrogate requirements until constraints saturate" argument-hint: "[Topic: <text>] [Scope: <glob>] [--depth shallow|standard|deep] [--personas N] [--mode interactive|autonomous] [Iterations: N] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Topic:` — strip keyword, remaining text is topic (or full $ARGUMENTS if no keyword) - `Scope:` or `--scope` — file globs for codebase grounding - `Depth:` or `--depth` — shallow (5 rounds), standard (15), deep (30) - `--personas N` or `Personas:` — active persona count (3-8, default 6) - `--saturation-threshold N` — net-new constraints/round below which counts toward saturation (default 2) - `--mode` or `Mode:` — interactive (default, uses request_user_input) or autonomous (self-answers from codebase) - `--adversarial` — rotate hostile personas to front - `Iterations:` or `--iterations` — default 15 rounds. "unlimited" for unbounded. - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>` ## Setup (if Topic missing) request_user_input (single batch): Q1 (Topic): "What to probe?" — open text describing feature, requirement, or design Q2 (Scope): "Which files for context?" — suggested globs + entire codebase Q3 (Depth): "How deep?" — shallow (5 rounds), standard (15), deep (30), unlimited Q4 (Mode): "How to answer persona questions?" — interactive (you answer), autonomous (agent infers from code) If all provided → skip. ## 8 Personas | # | Persona | Focus | |---|---|---| | 1 | Domain Expert | Business rules, domain constraints, terminology | | 2 | End User | Usability, expectations, error recovery | | 3 | Skeptic | Assumptions that might be wrong | | 4 | Edge-Case Hunter | Boundary conditions, rare scenarios | | 5 | Ops Engineer | Deployment, monitoring, scaling, failure modes | | 6 | Security Reviewer | Attack vectors, data protection, auth | | 7 | Contradiction Finder | Conflicts between requirements | | 8 | Scope Guardian | Feature creep, unnecessary complexity | If --adversarial: rotate Skeptic + Contradiction Finder + Edge-Case Hunter to front. ## Phase 1: Seed - Parse topic into initial constraint set - Read codebase context (if --scope provided) - Initialize constraint registry (empty) ## Round Loop ### Phase 2: Persona Activation - Select 2-3 personas for this round (rotate through all 8) - Each persona generates 3-5 probing questions from their perspective ### Phase 3: Codebase Grounding - Check questions against existing code for evidence - Annotate questions with: relevant file:line, existing behavior, gaps ### Phase 4: Answer Capture - **Interactive mode:** present questions via request_user_input, collect answers - **Autonomous mode:** infer answers from codebase context, label confidence (high/medium/low) ### Phase 5: Constraint Extraction - Parse answers into atomic constraints - Each constraint: id, source persona, description, confidence, evidence - Deduplicate against existing registry ### Phase 6: Cross-Check - Check new constraints against existing for conflicts - Flag contradictions for resolution (interactive → ask user, autonomous → note uncertainty) ### Phase 7: Saturation Check - Count net-new constraints this round - If net-new < saturation_threshold for 3 consecutive rounds → SATURATED, exit loop - Track: total constraints, new this round, saturation window ### Phase 8: Log Append to output: round number, personas active, questions asked, constraints extracted, net-new count ### Eval Checkpoint If --evals: check if current_round % interval == 0 → run checkpoint. ### Bounded Check If bounded: current_round >= max_iterations → exit loop. ## Phase 9: Synthesize & Output Create output directory: `autoresearch/probe-{YYMMDD}-{HHMM}/` 1. Write `constraints.md` — full constraint registry organized by category 2. Write `conflicts.md` — unresolved contradictions 3. Generate ready-to-run autoresearch config: - Derived Goal, Scope, Metric, Verify from constraints - Include as code block in summary.md Print: total rounds, constraints found, saturation status, unresolved conflicts. ## Summary Print: total rounds, total constraints, net-new trend, saturation status, top 5 most impactful constraints. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. - Print: `--- Eval Checkpoint (rounds {X}-{Y}) ---\nConstraints: {total} (+{new}) | Saturation: {window_count}/3\n{recommendation}\n---` - If saturated 3+ checkpoints → recommend early stop. - At loop end → full evals summary to evals-summary.md. ## Chain Handoff Write handoff.json: version "2.1.0", source "probe", timestamp, status (COMPLETE|SATURATED|USER_INTERRUPT|BOUNDED|ERROR), findings = constraints, config = derived autoresearch config. Invoke next target in --chain order. Propagate --evals flag. -
reason.md 4.7 KB
--- name: autoresearch:reason description: "Adversarial debate with blind judges until convergence" argument-hint: "[Task: <question>] [Domain: <type>] [--mode convergent|creative|debate] [--judges N] [Iterations: N] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Task:` — question, proposal, design, argument, or claim to refine - `Domain:` or `--domain` — software, product, business, security, research, content - `Mode:` or `--mode` — convergent (default), creative, debate - `--judges N` or `Judges:` — blind judge count (3 default, 5 thorough, 7 deep) - `--convergence N` or `Convergence:` — stop when incumbent wins N consecutive rounds (default 3) - `Iterations:` or `--iterations` — default 8. "unlimited" for unbounded. - `--judge-personas` — custom judge persona overrides - `--no-synthesis` — skip synthesis, pure debate only - `--temperature` — generation temperature hint - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>` Remaining text not matching flags = task description. ## Setup (if Task or Domain missing) request_user_input (single batch): Q1 (Task): "What should be reasoned about?" — open text Q2 (Domain): "What domain?" — software architecture, product strategy, business decision, security, research, content Q3 (Mode): "Refinement mode?" — convergent (stop when winner repeats), creative (never auto-stop), debate (no synthesis) Q4 (Judges): "How many blind judges?" — 3 (default), 5 (thorough), 7 (deep) If all provided → skip. ## Setup Phase 1. Load `references/reason-judge-protocol.md` for judge and convergence specs 2. Parse domain → select domain-specific judge criteria 3. Create output directory: `autoresearch/reason-{YYMMDD}-{HHMM}/` 4. TSV header: `round\ttimestamp\tcandidate_label\tjudge_verdict\tconvergence_count\tdescription` 5. Initialize: incumbent = null, convergence_count = 0 ## Round Loop ### Phase 1: Generate-A - If round 1: Author-A generates first candidate from task description - If round N>1: incumbent is Author-A's candidate - Cold-start: Author-A sees ONLY task description + domain context ### Phase 2: Critic - Critic receives candidate-A (cold-start, no shared session) - MUST find at least 3 specific weaknesses - MUST suggest what a superior candidate would do differently - Role is purely adversarial — never compliment ### Phase 3: Generate-B - Author-B receives: task + candidate-A + critique (cold-start) - Produces candidate-B addressing critique while preserving A's strengths ### Phase 4: Synthesize (unless --no-synthesis or debate mode) - Synthesizer receives: task + A + B (cold-start) - Produces hybrid candidate-AB merging best of both ### Phase 5: Blind Judge Panel - Each judge receives 3 candidates with RANDOMIZED labels (Label-X, Label-Y, Label-Z) - Judges evaluate independently on domain-specific criteria - Each produces ranking + one-paragraph justification - Verdict: majority vote. Tie → synthesized candidate wins. ### Phase 6: Convergence Check - If winner == incumbent → convergence_count++ - If winner != incumbent → convergence_count = 1, winner becomes incumbent - **Convergent mode**: convergence_count >= N → CONVERGED, stop - **Creative mode**: never auto-stop - **Debate mode**: same as convergent, no synthesis ### Phase 7: Oscillation Guard If incumbent changed 5+ times in last 8 rounds → recommend early stop (not converging). ### Phase 8: Log Append to TSV: round, timestamp, winning candidate label, judge verdict, convergence_count, description ### Eval Checkpoint If --evals: check if current_round % interval == 0 → run checkpoint. ### Bounded Check If bounded: current_round >= max_iterations → exit loop. ## Output - `reason-results.tsv` — per-round results - `lineage.md` — full history of candidates + critiques + judge reasoning - `summary.md` — final winner, convergence trajectory, key insights ## Summary Print: total rounds, convergence status, final winner summary, judge agreement rate. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. - Print: `--- Eval Checkpoint (rounds {X}-{Y}) ---\nIncumbent: {label} | Convergence: {count}/{target} | Oscillations: {n}\n{recommendation}\n---` - If oscillation detected 3+ checkpoints → recommend early stop. - At loop end → full evals summary to evals-summary.md. ## Chain Handoff After completion, write handoff.json: version "2.1.0", source "reason", timestamp, status (COMPLETE|CONVERGED|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = [{id, type: "recommendation", summary: winner description}], config{task, domain, mode}. Invoke next target in --chain order. Propagate --evals flag. -
regression.md 9.3 KB
--- name: autoresearch:regression description: "Layered regression stability gate: capture baseline behavior on the base ref, diff the candidate, verdict STABLE/UNSTABLE before you push" argument-hint: "[Base: <ref>] [Scope: <glob>] [--select auto|full|affected] [--samples N] [--noise-band %] [--matrix] [--max-runs N] [--baseline-cache] [Baseline: <prebuilt-ref>] [--probe|--no-probe|--probe deep] [--predict --reason --debug --fix --fix-cycles N --evals --evals-interval N --chain]" --- EXECUTE IMMEDIATELY. A regression is a **green→red transition ONLY**. The gate orchestrates the project's OWN test/bench/snapshot/migrate commands (it is a protocol, not a bundled framework), captures baseline behavior in an isolated git worktree, re-runs the candidate, and reports a tiered ship/no-ship verdict. ## Parse Arguments Extract from $ARGUMENTS: - `Base:` or `--base` — base ref to diff against. Default: `git merge-base HEAD main` (else `main`/`master`). - `Scope:` or `--scope` — file globs limiting the change surface. - `--select auto|full|affected` — test selection (default `auto`). `auto` = use the detected affected-test mapper if available, else FULL suite. Never a silent subset. - `--samples N` — SCORE samples/side (default 7). `--noise-band %` — perf tolerance (default 5%). - `--matrix` — opt-in matrix axis (OFF by default). `--max-runs N` — ceiling (default 200). - `--baseline-cache` (default on) — reuse `baseline/<full-sha>/` by SHA. `Baseline: <prebuilt-ref>` — bypass capture. - `--probe` (default) / `--probe deep` / `--no-probe`. - `--predict --reason --debug --fix --fix-cycles N --evals --evals-interval N --chain <targets>` and `--<sub>` shorthand. - `Iterations:` — repeat-axis count for `--select`/repeat sweeps. ## Setup / Probe-on-launch 1. Auto-detect per-dimension verify commands: `package.json` scripts, `Makefile`, `nx`, migrate config, bench/snapshot/size scripts. 2. request_user_input (single batch) to confirm detected commands + base ref + which dimensions to run. 3. **Auto-skip probe** when CI / no-TTY / `--mode autonomous` / complete-config / chained-handoff — log the inferred config instead of asking. ## Classification Phase (first-class, before any differential) Establish the baseline green-set per dimension, then tag each unit. Match by **test-id first, then path**. | State | Meaning | Gated? | |---|---|---| | `regression-eligible` | green on baseline | YES — only green→red counts | | `pre-existing` | red→red (already failing) | no — excluded | | `new-coverage` | absent→red (brand-new test) | no — new coverage, ungated | | `flaky` | nondeterministic on baseline | no — routed to flakiness SCORE | | `baseline-unavailable` | dimension never green | no — advisory only | **Core invariant: red→red, absent→red, and flake→red are NOT regressions.** Run flakiness N× on **both** baseline and candidate; a candidate failure inside the baseline flake-envelope routes to flakiness SCORE, never to a regression. `5/5 green ≠ non-flaky` — detection probability is `1−(1−p)^n` (≈23% at p=5%, n=5); print it. ## Baseline Capture `git worktree add --detach <full-sha>` (detached SHA — avoids "branch already checked out" when Base==HEAD) → `baseline/<full-sha>/`; `--baseline-cache` reuses by SHA. Then per worktree: `git submodule update --init` + dependency install (lockfile is SHA-pinned so the cache stays sound). Per-dimension **setup tiers**: api-contract = file-diff, no build; functional / integration-e2e / data-migration = full env. On completion or crash: `git worktree remove` + `git worktree prune`. Warn on concurrent index-lock contention. `Baseline: <prebuilt-ref>` bypasses capture. ## Dimension Registry (8) | Dim | Tier | Compare | Key params | |---|---|---|---| | functional | HARD | baseline green-set vs candidate; new fail = regression | test cmd, globs | | api-contract | HARD | schema/exports diff → breaking? | schema cmd, breaking ruleset | | data-migration | HARD | default: up applies clean + idempotent re-apply + app boots/schema valid; schema/rowcount roundtrip opt-in | migrate cmds, fixture, allowlisted DB | | integration-e2e | HARD | e2e green-set diff | e2e cmd | | flakiness | SCORE | run N× on baseline + candidate, count nondeterministic | runs (def 5), flake-threshold | | performance | SCORE | K **independent-process** samples/side, Mann-Whitney U AND effect beyond `max(noise-band%, k·stdev)`, report median delta | bench cmd, samples=7, noise-band=5%, k=2 | | resource | SCORE | mem/bundle/size delta vs budget | size cmd, budget | | visual-ui | SCORE | containerize render; default `maxDiffPixelRatio` + AA-detection; SSIM = per-page escalation | snapshot cmd, diff-threshold, mask regions | `--select auto` mapper (`jest --findRelatedTests` fed the changed-file list; `nx affected` project-graph) is **best-effort static-import** — blind to dynamic/runtime/global-setup couplings. The report names the mapper + its blind-spot caveat; a HARD STABLE earned on an affected subset prints "run `--select full` for high-stakes". FULL suite is the correctness default. - **performance independence:** each sample = an independent process launch (warmups discarded), never an in-process iteration — autocorrelation/GC/thermal otherwise violate Mann-Whitney's independence assumption. At n=7 the test detects only ≳1σ regressions; raise `--samples` for tight gates. - **data-migration guard:** opt-in. Before any migration the DB URL MUST pass an **anchored** allowlist — host is exactly `localhost` / `127.0.0.1` / a container or service hostname, OR the database name carries a `_test` / `_ci` suffix. A bare substring (e.g. `test` inside `latest`, `ci` inside `precision`) does **not** qualify. Anything else is refused — ephemeral only, never dev/prod — and even an allowlisted URL requires explicit user confirm before applying. Missing/absent down-migration = forward-only advisory, **never a finding**. ## Differential Loop (per dim × axis × run) Run candidate verify vs baseline metric → compute `regressed` bool + 0-100 `subscore`. Axes: `diff` (default), `repeat N×`, `full`, `matrix` (opt-in). Log one TSV row per cell. **--max-runs ceiling:** projected = dims × axes × samples × matrix-cells; if > `--max-runs` (default 200) → warn + require confirm (CI default = abort with message). ## Verdict - Any HARD `regressed=true` with `classification=regression-eligible` → **UNSTABLE** (green→red hard-blocks). - Else `stability_score = Σ(weight × dim_subscore)` over SCORE dims that ran (flakiness .30 / performance .30 / resource .20 / visual .20, renormalized over present dims). **STABLE iff ≥ 95** (`REG_THRESHOLD`/weights overridable). - Print the **score math** (per-dim contribution table) + declare **dims-ran vs UNAVAILABLE** — an UNAVAILABLE dimension is always listed, never silently passed. Backed by `scripts/score-regression.sh verdict <results.tsv>` relative to the installed Autoresearch skill directory, never the caller's working directory (exit 0 STABLE / 1 UNSTABLE). ## Hunter (root cause) On a confirmed HARD regression, auto-engage. Bisect (reuse `debug`) ONLY when the failing case passes a **3/3 reproducibility gate**. SCORE / non-deterministic regressions → differential root-cause + optional `--reason` / `--predict`, no bisect. Non-reproducible → "manual triage" finding. ## --fix Re-gate `--fix` repairs blocking regressions, max **3 cycles** (`--fix-cycles N`). Each cycle MUST strictly shrink the blocking-set else STOP "fix not converging". Intermediate re-gate scopes to failing+touched dims; the final cycle runs the full battery. No HARD-gate bypass. ## Output `autoresearch/regression-{YYMMDD}-{HHMM}/` → `regression-results.tsv`, `stability-report.md`, `dimensions/<dim>.md`, `baseline/`, `evals-summary.md` (if `--evals`), `handoff.json`. TSV header: `# metric_direction: higher_is_better` then `iteration\ttimestamp\tdimension\taxis\ttier\tclassification\tbaseline\tcandidate\tdelta\tregressed\tsubscore\tseverity\tstatus\tfile_line\tdescription`. ## Eval Checkpoint (--evals flag) Interval = floor(max_runs / 3), min 1 (fixed 10 if unbounded); override `--evals-interval N`. Every interval, analyze the results TSV; print trend (up/flat/down) + one-line recommendation. Plateau 3+ checkpoints → recommend early stop. At end → full summary to `evals-summary.md`. ## Chain Handoff Write `handoff.json` to the output directory: version "2.1.0", source "regression", timestamp, `status` ∈ family enum {COMPLETE, CONVERGED, SATURATED, BOUNDED, USER_INTERRUPT, ERROR} (backward-compat with evals/ship consumers), `verdict` ∈ {STABLE, UNSTABLE, BASELINE_UNAVAILABLE} + `regression_state` ∈ {REGRESSION_FOUND, REGRESSION_FIXED, none} — `ship` reads `verdict` for the deploy-gate, `results_tsv` path, `findings` = blocking regressions (dim, severity, file_line, classification), `config`{base, scope, dims, axes, verdict-math}. If `--fix` → chain to fix automatically. Invoke next `--chain` target in order; propagate `--evals`. Canonical combo: `--predict --evals --fix --ship` = predict → gate → (hunter on HARD) → fix(≤3) → re-gate → ship iff STABLE (deploy still needs explicit approval). ## Safety Verify-command screen (no `rm -rf` / `curl|sh`); worktree cleanup + prune on crash; data-migration refuses any non-allowlisted DB URL; probe auto-skips non-interactively; chained `ship` never auto-deploys. -
scenario.md 4.1 KB
--- name: autoresearch:scenario description: "Generate edge cases across 12 dimensions from a seed scenario" argument-hint: "[Scenario: <text>] [Domain: <type>] [Scope: <glob>] [Iterations: N] [--depth <level>] [--focus <area>] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Scenario:` — seed scenario description (or full $ARGUMENTS text if no keyword) - `Domain:` or `--domain` — web, mobile, API, CLI, data pipeline, infrastructure - `Scope:` or `--scope` — file globs for codebase context - `Focus:` or `--focus` — specific dimension to prioritize - `--depth` — shallow (10), standard (20), deep (40+) - `--format` — markdown (default), json, gherkin - `Iterations:` or `--iterations` — default 20. "unlimited" for unbounded. - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>` ## Setup (if Scenario or Domain missing) request_user_input (single batch): Q1 (Scenario): "Describe the feature/flow to explore" Q2 (Domain): "What domain?" — web app, mobile app, API, CLI, data pipeline, infrastructure Q3 (Scope): "Which files for context?" — suggested globs + entire codebase Q4 (Depth): "How deep?" — quick (10), standard (20), deep (40+), unlimited If all provided → skip. ## 12 Dimensions | # | Dimension | Explores | |---|---|---| | 1 | Happy path | Normal successful flows | | 2 | Validation | Input boundaries, types, formats | | 3 | Permissions | Auth, roles, access control | | 4 | Concurrency | Race conditions, deadlocks, ordering | | 5 | State | Invalid transitions, corruption | | 6 | Scale | High volume, large data, many users | | 7 | Failure | Network errors, timeouts, partial failures | | 8 | Security | Injection, abuse, bypass attempts | | 9 | Integration | Third-party failures, API contract violations | | 10 | Data | Null, empty, unicode, injection, overflow | | 11 | UX | Confusion, misuse, accessibility | | 12 | Recovery | Retry, rollback, idempotency | ## Establish Baseline 1. Read seed scenario + codebase context 2. Create output directory: `autoresearch/scenario-{YYMMDD}-{HHMM}/` 3. TSV header: `iteration\ttimestamp\tscenario\tdimension\tclassification\tseverity\tdescription` 4. No metric_direction comment (exploration, not optimization) ## Iteration Loop ### Phase 1: Review - Read results TSV, check dimension coverage - Identify underexplored dimensions - If --focus → prioritize that dimension ### Phase 2: Generate - Pick next dimension (round-robin, or priority if --focus) - Generate 3-5 specific scenarios for this dimension - Each: title, dimension, classification, severity, description ### Phase 3: Classify - **new** — genuinely novel edge case - **extension** — builds on previously found scenario - **duplicate** — already covered (skip, don't log) ### Phase 4: Log Append new/extension scenarios to TSV. Skip duplicates. Severity: critical/high/medium/low. ### Phase 5: Saturation Check If 3 consecutive iterations produce only duplicates → dimension saturated, move to next. If ALL dimensions saturated → early stop. ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint. ### Bounded Check If bounded: current_iteration >= max_iterations → exit loop. ## Output - Write `scenarios.md` (organized by dimension, severity-ranked within each) - Write `edge-cases.md` (flat severity-ranked list) - `scenario-results.tsv` ## Summary Print: total scenarios (new/extension/duplicate), dimension coverage (X/12 explored), severity distribution. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nNew scenarios: {n} | Dimensions covered: {x}/12 | Saturation: {status}\n{recommendation}\n---` - If 3+ checkpoints with mostly duplicates → recommend early stop. - At loop end → full evals summary to evals-summary.md. ## Chain Handoff After completion, write handoff.json: version "2.1.0", source "scenario", timestamp, status, results_tsv path, findings = scenarios by severity, config{scenario, domain, scope}. Invoke next target in --chain order. Propagate --evals flag. -
security.md 4.6 KB
--- name: autoresearch:security description: "STRIDE + OWASP security audit with red-team adversarial personas" argument-hint: "[Scope: <glob>] [Focus: <area>] [Iterations: N] [--diff] [--fix] [--fail-on <severity>] [--evals]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Scope:` or `--scope` — file globs to audit - `Focus:` — specific area (auth, API, data handling, etc.) - `Depth:` or `--depth` — quick (5 iterations), standard (15), deep (30+) - `Iterations:` or `--iterations` — default 15. "unlimited" for unbounded. - `--diff` — delta mode: only audit files changed since last audit - `--fix` — after audit, auto-fix Critical/High findings (chains to fix) - `--fail-on <severity>` — exit non-zero if findings at/above threshold (CI gate) - `--evals`, `--evals-interval N`, `--chain`, `--<subcommand>` ## Setup (if required context missing) If Scope missing and no --diff: 1. Scan codebase for tech stack, frameworks, API routes 2. request_user_input (single batch): Q1 (Scope): "What to audit?" — entire codebase, API + middleware, auth, external-facing Q2 (Depth): "How thorough?" — quick (5), standard (15), deep (30+), unlimited Q3 (Action): "What to do with findings?" — report only, report + auto-fix, report + CI gate If all provided → skip. ## Setup Phase (once, before loop) 1. **Reconnaissance** — scan: package.json/requirements.txt (deps), .env.example (secrets), Dockerfile (infra), API route files (attack surface), auth/middleware (trust boundaries), DB schemas (data assets), CI/CD configs (supply chain) 2. **Asset Identification** — catalog data stores, auth systems, external services, user inputs 3. **Trust Boundary Mapping** — browser↔server, public↔authenticated, user↔admin, CI↔prod 4. **STRIDE Threat Model** — generate threats per category. Load `references/security-checklist.md` for checklist. 5. **Attack Surface Map** — entry points, data flows, abuse paths 6. **Baseline** — count known issues, initialize coverage tracking Create output directory: `autoresearch/security-{YYMMDD}-{HHMM}/` Write: overview.md, threat-model.md, attack-surface-map.md TSV header: `# metric_direction: higher_is_better\niteration\ttimestamp\tfinding\tseverity\towasp\tstride\tevidence\tfile_line` ## Iteration Loop ### Phase 1: Review - Read results TSV + coverage tracking - Identify untested attack vectors from threat model - Prioritize: untested OWASP categories → untested STRIDE → depth on existing ### Phase 2: Attack - Adopt red-team persona for this vector (rotate: Security Adversary, Supply Chain, Insider Threat, Infra Attacker) - Deep-dive into relevant code with adversarial mindset - Look for: code paths, input handling, auth checks, data flows ### Phase 3: Validate - Construct proof: file:line + specific attack scenario - Every finding MUST have code evidence — no theoretical fluff - Classify severity: Critical/High/Medium/Low/Info - Map to OWASP (A01-A10) and STRIDE (S/T/R/I/D/E) ### Phase 4: Log - Append finding to TSV - Update coverage tracking - Print coverage every 5 iterations: `OWASP: [A01✓ A02✓ A03✗ ...] X/10 | STRIDE: [S✓ T✓ R✗ ...] Y/6 | Score: Z` ### Composite Metric `score = (owasp_tested/10)*50 + (stride_tested/6)*30 + min(findings, 20)` ### Eval Checkpoint If --evals: check if current_iteration % interval == 0 → run checkpoint. ### Bounded Check If bounded: current_iteration >= max_iterations → exit loop. ## After Loop 1. Write `findings.md` (severity-ranked) 2. Write `owasp-coverage.md` 3. Write `recommendations.md` 4. If `--fix` → chain to fix with Critical/High findings 5. If `--fail-on` → check findings against threshold, exit non-zero if exceeded ## Summary Print: total findings by severity, OWASP coverage X/10, STRIDE coverage Y/6, composite score. ## Eval Checkpoint (--evals flag) If --evals present: - Compute interval: floor(max_iterations / 3), min 1. Fixed 10 if unbounded. Override: --evals-interval N. - Every {interval} iterations, analyze results TSV. - Print: `--- Eval Checkpoint (iterations {X}-{Y}) ---\nScore: {start} → {end} | New findings: {n} | Coverage: OWASP {x}/10, STRIDE {y}/6\n{recommendation}\n---` - If no new findings 3+ checkpoints → recommend early stop. - At loop end → full evals summary to evals-summary.md. ## Chain Handoff After completion, write handoff.json to output directory: version "2.1.0", source "security", timestamp, status (COMPLETE|USER_INTERRUPT|BOUNDED|ERROR), results_tsv path, findings = all findings with severity + OWASP + STRIDE + file:line, config{scope, focus, depth}. Invoke next target in --chain order. Propagate --evals flag. -
ship.md 3.8 KB
--- name: autoresearch:ship description: "Ship anything through 8 phases: checklist, dry-run, deploy, verify" argument-hint: "[Target: <what>] [--type <type>] [--dry-run] [--auto] [--force] [--rollback] [--checklist-only] [--monitor N]" --- EXECUTE IMMEDIATELY. ## Parse Arguments Extract from $ARGUMENTS: - `Target:` or `--target` — what to ship (path, PR, artifact, deployment) - `--type <type>` — override auto-detection: code-pr, code-release, deployment, content, docs, package, config - `--dry-run` — validate everything but don't ship - `--auto` — auto-approve if no errors found - `--force` — skip non-critical items (blockers still enforced) - `--rollback` — undo last ship action - `--monitor N` — post-ship monitoring for N minutes - `--checklist-only` — only generate checklist, don't execute - `--chain`, `--<subcommand>` Remaining text = description of what to ship. ## Setup (if Target or Type unclear) 1. Auto-detect ship type from context: - Has uncommitted changes or PR → code-pr - Has version bump / changelog → code-release - Has Dockerfile / deploy config → deployment - Has markdown / content files → content - Has package.json version change → package 2. If still unclear → request_user_input (single batch): Q1 (What): "What are you shipping?" — code PR, release, deployment, content, docs, package Q2 (Target): "Specific target?" — current branch, specific PR, specific path Q3 (Mode): "How to ship?" — full workflow, dry-run only, checklist only If all clear → skip. ## Phase 1: Identify - Determine ship type (auto-detected or --type override) - Identify target artifact(s) - Map to domain-specific checklist ## Phase 2: Inventory Gather everything that will be shipped: - Files changed (git diff) - Dependencies affected - Config changes - Migration files - Breaking changes ## Phase 3: Checklist Generate domain-specific checklist: **Code PR:** tests pass, types check, lint clean, no secrets, PR description, reviewers assigned **Release:** version bumped, changelog updated, migration tested, rollback plan **Deployment:** env vars set, health checks configured, rollback ready, monitoring active **Content:** links valid, images optimized, SEO metadata, spell check **Package:** version bumped, README updated, breaking changes documented, CI green If `--checklist-only` → output checklist and stop. ## Phase 4: Prepare Execute pre-ship tasks: - Run test suite - Run type checker - Run linter - Check for secrets in diff - Validate configs - Flag blockers (must-fix) vs warnings (can-ship-with) If blockers found → STOP, report blockers, ask user to fix. ## Phase 5: Dry-Run If `--dry-run` or always before actual ship: - Simulate the ship action without executing - Report what WOULD happen - If `--dry-run` → stop here ## Phase 6: Ship **REQUIRES EXPLICIT USER APPROVAL** (unless --auto with zero errors). Execute the ship action: - Code PR: create/update PR, request reviewers - Release: tag, build, publish - Deployment: deploy to target environment - Content: publish to CMS/platform ## Phase 7: Verify Post-ship verification: - Confirm artifact is live/accessible - Run smoke tests if available - Check monitoring for errors - If `--monitor N` → watch for N minutes ## Phase 8: Log Create output directory: `autoresearch/ship-{YYMMDD}-{HHMM}/` Write: - `checklist.md` — completed checklist with pass/fail per item - `summary.md` — what was shipped, verification results - `ship-log.tsv` — phase-by-phase log ## Rollback If `--rollback`: - Identify last ship action from most recent ship log - Reverse it (revert PR, unpublish, rollback deployment) - Verify rollback succeeded ## Chain Handoff Write handoff.json: version "2.1.0", source "ship", timestamp, status (COMPLETE|DRY_RUN|ROLLBACK|ERROR), findings = blockers/warnings found during prep. Invoke next target in --chain order. -
SKILL.md 7.8 KB
--- name: autoresearch description: "Autonomous iteration loop: modify, verify, keep/discard against any metric" metadata: version: "2.2.2" --- # Autoresearch — Autonomous Goal-directed Iteration ## Safety Invariants (all subcommands) - Never push, publish, or deploy without explicit user approval. - Bounded by default. Override with `Iterations: unlimited`. - All results logged to `autoresearch/{subcommand}-{YYMMDD}-{HHMM}/` directory. - Chain handoff via `handoff.json`. Evals reads `*-results.tsv`. ## Dispatch (bare `$autoresearch`) Parse the invocation in this order: | Condition | Mode | |---|---| | `Metric:` or `Verify:` present | **Classic** — existing metric loop, unchanged | | Free-form natural-language goal, no metric/verify | **Orchestrator** — see Orchestrator section | | Nothing | **Setup wizard** — interactive config builder | | `--classic` flag | Force Classic regardless of goal text | | `--auto` flag | Force Orchestrator regardless of goal text | Print a banner on every invocation: `[autoresearch] mode: classic | orchestrator | wizard`. ## Subcommands | Command | Does | Default Iterations | |---|---|---| | `$autoresearch` | Iterate against a metric: modify → verify → keep/discard | 25 | | `$autoresearch plan` | Convert a goal into validated Scope, Metric, Verify config | N/A | | `$autoresearch debug` | Hunt bugs: hypothesize → test → falsify → repeat | 15 | | `$autoresearch fix` | Crush errors one-by-one until zero remain | 20 | | `$autoresearch security` | STRIDE + OWASP audit with red-team personas | 15 | | `$autoresearch ship` | Ship through 8 phases: checklist → dry-run → deploy → verify | N/A | | `$autoresearch scenario` | Generate edge cases across 12 dimensions | 20 | | `$autoresearch predict` | 5 expert personas debate before implementation | N/A | | `$autoresearch learn` | Scout codebase → generate docs or wiki → validate → fix loop | 10 | | `$autoresearch reason` | Adversarial debate with blind judges until convergence | 8 | | `$autoresearch probe` | 8 personas interrogate requirements until saturation | 15 | | `$autoresearch improve` | Research ICP challenges, discover improvements, generate PRDs | 15 | | `$autoresearch evals` | Analyze iteration results: trends, plateaus, regressions | N/A | | `$autoresearch regression` | Regression stability gate: baseline vs candidate, verdict STABLE/UNSTABLE | N/A | ## Universal Flags | Flag | Applies To | Purpose | |---|---|---| | `Iterations: N` | All looping | Set iteration count | | `Iterations: unlimited` | All looping | Opt-in unbounded | | `--evals` | All looping | Mid-loop checkpoints + final summary | | `--evals-interval N` | All looping | Override checkpoint frequency | | `--chain <targets>` | All | Sequential handoff after completion | | `--<subcommand>` | All | Shorthand for `--chain <subcommand>` | | `--dry-run` | Orchestrator | Print derived config + planned pipeline; no execution | | `--max-cycles N` | Orchestrator | Hard ceiling on orchestration cycles (default 50) | | `--classic` | Bare `$autoresearch` | Force Classic metric-loop mode | | `--auto` | Bare `$autoresearch` | Force Orchestrator mode | ## Orchestrator Activated when a plain-language goal is given without `Metric:`/`Verify:`. Classifies the goal into a **Goal archetype** — see `references/orchestrator-routing.md` for the archetype table and router decision table. Resolve every `scripts/...` path below relative to this installed skill directory, never relative to the caller's working directory. **Two modes based on archetype:** - **Orchestration loop** — predicate-bearing archetypes (ship-ready, optimize-metric, fix-broken, harden, build-feature, explore). Goal has a mechanical Success predicate; the loop runs until that predicate is met. - **Single-pass dispatch** — subjective/terminal archetypes (document, what-to-build, decide-design). Routes once to the fitting subcommand (learn / improve / reason), lets it self-terminate, then reports. No loop, no Plateau, no ship gate. ### Orchestration Loop Steps Backed by `scripts/orchestrate.sh` (deterministic seam — all routing logic lives there). Subcommands exposed: `classify`, `next-hop`, `units`, `plateau`, `screen-cmd`, `verdict`, `validate-state`, `screen-state-predicate`. 1. **Classify** — `scripts/orchestrate.sh classify "<goal>"` → archetype label + mode. 2. **Derive predicate** — reuse `plan` logic to produce a concrete Success predicate: exact shell command + expected output. For `optimize-metric`, run the full plan/wizard derivation internally. 3. **Confirm** — ONE `request_user_input` showing: archetype, mode, concrete predicate (command + expected output), terminal choice (stop-at-verified vs proceed-to-ship). Misclassifications are caught here, not mid-run. 4. **Round-0 dry-run** — prove the predicate command runs and returns a value; safety-screen every derived command via `screen-cmd`; print projected cycle budget. Stop here if `--dry-run`. 5. **Loop** until predicate satisfied: a. Assess state via cheap signals (last `handoff.json`, regression verdict, error count) + affected-test verify. b. `scripts/orchestrate.sh next-hop orchestrator-state.json` → next subcommand. c. Run subcommand (its own bounded inner loop). d. Record per-hop outcome ∈ {progressed, no-op, failed, blocked}. e. Fold hop's `handoff.json` into `orchestrator-state.json`. f. `scripts/orchestrate.sh units` → recompute **Units remaining**. 6. **Stop conditions** (checked after each hop): - Predicate met → ship gate (only if ship is in the pipeline) else `CONVERGED`. - `scripts/orchestrate.sh plateau orchestrator-state.json` → true → stop + report `PLATEAU`. - Cycles > ceiling (default 50, override `--max-cycles N`) → stop + report `CEILING`. - Hop outcome `blocked`/`failed` with no alternative route → checkpoint + stop + report `BLOCKED`. ### Orchestrator State `orchestrator-state.json` — orchestrator-owned, additive. Tracks: goal, archetype, predicate, terminal-choice, `units_remaining` history, cycle count, per-hop pipeline log with outcomes, current incumbent. Each hop's `handoff.json` is unchanged (single-hop bridge); the orchestrator reads it and folds it in. Two clearly-owned state objects, no overlap. ### Orchestrator Safety Invariants - **Never auto-approve ship/deploy/push.** The orchestrator never passes `--auto` to `ship`; deploy always requires explicit user approval. - **Data-migration behind anchored DB-URL allowlist.** Reuses regression's allowlist — host must be `localhost`/`127.0.0.1`/container hostname, or database name carries `_test`/`_ci` suffix. Bare substring match does not qualify. Anything else refused. - **screen-cmd on every derived command** — run before the loop starts AND on every command read from a persisted state file on resume. Persisted commands are never trusted; resume re-screens the pinned predicate via `screen-state-predicate` and refuses on `refuse`. - **No un-screened commands mid-loop.** The autonomous loop cannot introduce new shell commands that bypass `screen-cmd`. - **Predicate pinned, not re-derived.** Round-0 writes the derived Success predicate verbatim into `orchestrator-state.json`; every cycle and every resume reuses that exact string so "done" is reproducible across runs. - **Validate the ledger before routing.** `validate-state` gates `orchestrator-state.json` (required fields + coarse types); a malformed ledger is not trusted to route from. - **Independent verify before convergence.** High-impact changes accepted on the working signal set `pending_verify`; `next-hop` routes to a `verify` hop (held-out / adversarial check) before `DONE` or ship. The verify hop never auto-approves ship. - **Unknown-units cycles excluded from Plateau counter.** A cycle where `units` returns `unknown` (e.g. runner crash) is not counted as zero-progress; repeated `unknown` routes to `BLOCKED`.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.