ax-audit
Audits agentic products for tool parity, authority, approval payloads, recovery, and trust using 27 rules and a ship verdict. Use when asked for an "AX audit", to review an agent approval flow, or whether an agent can operate the product. For human-facing API ergonomics use dx-au
Install
npx skills add https://github.com/mblode/agent-skills/tree/main/skills/ax-audit
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install mblode-agent-skills@llmmart
git clone https://github.com/mblode/agent-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole mblode/agent-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
AX Audit
Feature-level reviewer for apps where an agent acts for the user. One question: does it earn trust, and where does it break?
- IS: rules-based audit of agentic surfaces (chat, tool execution, config, dashboards) across architecture (
rules-arch/) and trust (rules-ax/), ending in a ship-readiness verdict plus an AX Relationship Summary. - IS NOT: traditional frontend UX (use
ui-designAudit mode); developer-facing API, CLI, or type ergonomics (usedx-audit); public site or docs agent scores (useagent-ready); agent instruction files (useagents-md); what the product should do before it exists (useproduct-design).
No agentic features in scope? Stop. AX rules against forms and lists are noise.
Contents
- Audit workflow
- Two rule layers
- Tiers and verdict
- AX Relationship Summary
- Reference files
- Gotchas
- Audit self-check
- Related skills
Audit workflow
AX Audit progress:
- [ ] Step 1: Scope, via the diff against the PR base merge-base (PR mode) or explicit path (full sweep)
- [ ] Step 2: Detect agentic features per references/feature-playbooks.md
- [ ] Step 3: Run each detected feature's playbook in order, plus the diff-wide checks (PR mode only)
- [ ] Step 4: For each check, load the rule file and follow its detection recipe
- [ ] Step 5: Tier each finding per references/ship-readiness.md (rule override table wins)
- [ ] Step 6: Render verdict + findings + AX Relationship Summary per references/output-format.md
- [ ] Step 7: Run the audit self-check and report its evidence counts
PR-mode scope is the diff plus the tool definitions and orchestrator it touches. Findings in untouched files belong in a full sweep, not this verdict. Playbook annotations are a scan copy; the rule file is authoritative. parity-orphan-ui-action runs on every PR-mode audit and never in a full sweep, where there is no diff for it to read.
Rule greps name the most common identifiers, not every framework's spelling. When a grep misses in code that plainly does the thing (a gate, a stream, a tool result), check references/framework-signals.md for the stack's name for it before recording unknown.
Two rule layers
| Layer | Folder | Rules | Load when a playbook names |
|---|---|---|---|
| 1: Architecture | rules-arch/ |
12 | rules-arch/<category>-<slug>.md |
| 2: Experience | rules-ax/ |
15 | rules-ax/<category>-<slug>.md |
Categories: arch = parity, granularity, context, comm; ax = trust, control, context, comm. Shared prefixes are different rules: rules-arch/comm-no-approval-gate.md (no gate on the execution path) is not rules-ax/control-no-approval-gate.md (gate exists, stakes are wrong).
Run Layer 1 comm/parity and Layer 2 control/trust first. They hold the blockers. Category map: rules-arch/_sections.md, rules-ax/_sections.md.
| Priority | Layer | Category | Prefix | Rules |
|---|---|---|---|---|
| 1 | arch | Communication | comm- |
3 |
| 2 | arch | Parity | parity- |
4 |
| 3 | ax | Control | control- |
4 |
| 4 | ax | Trust | trust- |
4 |
| 5 | arch | Context | context- |
3 |
| 6 | ax | Communication | comm- |
4 |
| 7 | ax | Context | context- |
3 |
| 8 | arch | Granularity | granularity- |
2 |
Tiers and verdict
Three tiers. Full trigger lists and the generic surface bump live in references/ship-readiness.md.
Precedence: the rule's own surface-override table > the generic bump > defaultTier. Apply at most one adjustment.
Verdict: ✅ READY (0 blockers, ≤3 sprint) · ⚠️ READY WITH FOLLOW-UP (0 blockers, ≥4 sprint) · ❌ NOT READY (≥1 blocker) · 🚫 INCOMPLETE (self-check failed).
Blockers outrank an incomplete audit. With ≥1 release-blocker and a failed self-check, report ❌ NOT READY and note the self-check failure beneath it: the blockers are established findings and stay actionable, while 🚫 reads as "nothing was learned" and sends the reader away. Reserve 🚫 for an audit with no blockers whose coverage you cannot vouch for.
AX Relationship Summary
Render after findings when any agentic feature was detected. Findings serve engineers; this serves designers and PMs. Four fields: evolution stage (behavior, not a label), trust signal (high/moderate/low plus one-line reason), key gap (one actionable sentence), trust question (one question only research can answer).
Reference files
| File | Read when |
|---|---|
references/feature-playbooks.md |
Steps 2-3: detection heuristics, per-feature ordered checks, diff-wide checks |
references/framework-signals.md |
Step 4, when the code uses AI SDK, MCP, the Claude Agent SDK, or AG-UI: where the gate, the stream, the completion signal, and the structured result live in each, with the spec defaults the rules lean on |
references/ship-readiness.md |
Step 5: tier triggers, precedence, verdict logic |
references/output-format.md |
Step 6: findings JSON schema, summary schema, terminal rendering |
references/agent-native-principles.md |
A Layer 1 finding needs grounding the rule file does not carry |
references/ax-evolution-curve.md |
Writing the AX Relationship Summary: stage, action depth, costume vs intelligence |
references/invisible-interface.md |
Grounding for structured tool output, approval payload, access scope, unprompted action; also the arguments that stay in keyGap |
references/evaluation-scenarios.md |
When changing this skill. Never loads during a user audit |
rules-arch/_sections.md |
Layer 1 categories, default tiers, co-firing pairs |
rules-ax/_sections.md |
Layer 2 categories, default tiers, co-firing pairs |
Gotchas
- Scope before rules. Running all 27 rules repo-wide on a 3-file PR buries a new release-blocker under pre-existing backlog noise; the verdict stops meaning "can this PR merge."
- The rule's override table is authoritative.
comm-no-intent-handshakedefaults tofix-this-sprintbut its table saysrelease-blockeron tool execution. Stacking the generic "+1 tier on tool execution" bump on an explicit override double-upgrades backlog findings into blockers. - A stop button not wired to
AbortController.abort()is a false affordance.control-no-escape-hatchstill fails: verify theabort()call, not the button label, or the audit passes a UI that lies to users. - A client
stop()that only closes the stream leaves the executor running.useChat().stop()aborts the fetch. Unless the route passesreq.signalintostreamText({ abortSignal })and toolexecutereads it, the server finishes every remaining tool call after the user pressed Stop. Trace the signal to the loop, not to the button. - Tool annotations are hints, not stakes. MCP tells clients to treat
annotationsfrom untrusted servers as untrusted; a gate that auto-approves on a third-party server'sreadOnlyHint: truehas handed the gate to that server.comm-no-approval-gatefails it. The spec defaults (destructiveHint: true,readOnlyHint: false) are the fail-closed baseline. - A framework approval flag is the gate's input, not the gate. AI SDK
toolApproval: "user-approval"emits atool-approval-requestpart and waits. A UI that never rendersstate === "approval-requested", or answers it withaddToolApprovalResponse({ approved: true })on arrival, has a gate in the type system and none for the user. Check the renderer and the response call, not the option. - Absence checks need a recorded file list. "Find components lacking X" greps return nothing both when everything passes and when nothing was scanned. List candidate files first (
rg -l <feature-pattern>), check each for the counter-pattern, and cite the file list as evidence. detection: observationalrules cannot fail on grep evidence alone.granularity-static-api-mapping,trust-no-uncertainty-markers,control-over-conversational, andcomm-no-generative-momentumneed interaction-flow judgment; on static evidence alone, returnunknownwith a reason, notfail.- Gates fail in three separate places. Absent from the path (
comm-no-approval-gate), present but mismatched to the stakes (control-no-approval-gate), or correct and unreadable (control-thin-approval-payload). Report the first that holds and fix in that order. - Interactive gates do not cover unattended runs. Cron, webhook, and queue entry points reach the same executor with nobody to prompt.
comm-unrequested-action-no-consentaudits that path; evidence names the entry point, not the executor. ax-audit-ignore:<slug>comments count assuppressed, notpass. Report the count in the verdict block; a suppression with no reason is itself awarn.- Don't inflate tiers.
comm-no-generative-momentumandgranularity-static-api-mappingdefault tobacklog. One finding promoted torelease-blockerflips the whole PR to ❌ NOT READY, so promoting cosmetic ones trains the team to ignore the verdict entirely. - Don't duplicate
ui-designAudit mode findings. "Missing loading state" and "form clears on error" are its territory; duplicating them trains engineers to dismiss the whole AX report. - A Personally Intelligent agent that only ever suggests has plateaued. Memory stage is not trust. Name the highest action rung in
evolutionStage.behavioror the summary flatters a polite chatbot.
Audit self-check
Flag the audit INCOMPLETE if any of these hold, and include the counts as evidence (planned vs. run rules per playbook, unknown rate, suppressed count):
- Fewer rules ran than the playbooks planned
- More than 30% of rules returned
unknown. Count onlyunknownhere, neverout-of-scope: a rule whose layer is absent from the scope you were given was answered correctly, and a narrow diff is the scope Step 1 asks for. Marking a correctly scoped audit INCOMPLETE buries its real blockers under a verdict that reads as "we learned nothing". - Any
fail/warnfinding lacksfile:lineevidence or a fix snippet - Every finding landed in the same tier (suspect blanket assignment)
- AX Relationship Summary is missing despite detected agentic features
Related skills
ui-designAudit mode: traditional frontend UX around agentic surfaces; run both on agentic feature PRsdx-audit: same files, different reader. This skill asks whether an agent can operate and recover;dx-auditasks whether a human adopting the API, CLI, or types finds it ergonomicagent-ready: whether public docs and HTTP APIs are discoverable to coding agents; this skill audits in-product agent UXproduct-design: what the agentic feature should do, before this auditagents-md: CLAUDE.md / AGENTS.md instruction files
Maintenance only: evals/evals.json contains regression scenarios for changes to this skill; it does not load during a user task.
Files (agent-skills)
-
evals
-
evals.json 1.4 KB
{ "skill_name": "ax-audit", "evals": [ { "id": 1, "prompt": "Audit an agent tool with this execution path: the UI displays a confirmation, but the API directly invokes deleteWorkspace(args). A direct API caller can bypass the UI.", "expected_output": "Find the missing execution-path gate with the applicable architecture rule.", "files": [], "assertions": [ "Cites comm-no-approval-gate", "Explains the direct API bypass", "Does not accept the UI dialog as enforcement" ] }, { "id": 2, "prompt": "Run an AX audit on a static contact form with no model, agent, or tool execution.", "expected_output": "Report that the agentic rule set does not apply.", "files": [], "assertions": [ "Does not invent an agentic feature", "Does not apply approval-flow rules to the form", "Routes ordinary UI review to ui-design" ] } ], "routing": { "should_trigger": [ "Audit an agent tool with this execution path: the UI displays a confirmation, but the API directly invokes deleteWorkspace(args). A direct API caller can bypass the UI.", "Run an AX audit on a static contact form with no model, agent, or tool execution." ], "near_miss": [ { "prompt": "Audit the flags and exit codes of our CLI.", "expected": "dx-audit" } ] } }
-
-
references
-
agent-native-principles.md 7.1 KB
# Agent-Native Principles (Condensed) Condensed from the Every agent-native guide (<https://every.to/guides/agent-native>), with Anthropic's tool-writing and context-engineering guidance folded in where it sharpens a rule. <!-- TOC --> - [Core Principles](#core-principles) - [Tool Design](#tool-design) - [Context Patterns](#context-patterns) - [Agent-UI Communication](#agent-ui-communication) <!-- /TOC --> ## Core Principles - **Parity:** every UI capability has a tool; if not, add one. - **Granularity:** atomic primitives, one action per tool; decision logic lives in prompts, so behavior changes are prompt edits, not refactors. - **Composability:** atomic tools plus parity make new features new prompts. - **Emergent capability:** ship atomic tools, watch requests, add domain tools for common patterns. - **Improvement over time:** context files plus refined prompts improve the app without code; self-modification needs audit logs and rollback. ## Tool Design **Atomic primitives first.** One action per tool, scoped to a domain noun: `read_note`, `update_note`, `list_projects`. Prove the architecture on these before bundling them into workflow tools. **Atomic is not raw.** A domain primitive has a blast radius you can state in a sentence. Raw substrate access over production data (arbitrary code eval, a shell on the live host, a SQL or GraphQL passthrough, an untyped SDK call) has the blast radius of the whole system and makes every tool boundary above it advisory. Ship the first. Some long-tail requests then stop being servable; that is a refusal, not a reason to add the substrate tool back. **CRUD completeness.** Verify every entity has create, read, update, delete. Common failure: `create_note` + `read_notes` exist but `update_note` and `delete_note` are missing. **Domain tools.** Add deliberately for vocabulary anchoring, guardrails (validation not left to judgment), or bundling a multi-step operation. **Consolidate mechanics, not judgment.** Anthropic's guidance for agent tools runs the other way from "one tool per endpoint": a few tools aimed at whole workflows beat many thin wrappers, and bloated, overlapping tool sets are the first failure mode it names. The two agree once the axis is clear. Chaining mechanical steps into one user action (`schedule_event` finds availability and books) is consolidation the agent loses nothing to; folding a decision into code (which role, what counts as stale) moves judgment out of the prompt and is what `granularity-workflow-shaped-tool` catches. **Tool results are a contract.** MCP gives the shape a name: `structuredContent` under an `outputSchema` on success, `isError: true` on failure (assumed false when unset), and `annotations` that describe stakes (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) which clients treat as hints, not authority. `parity-unstructured-tool-output` and `comm-no-approval-gate` audit against that shape whatever the stack. **Dynamic capability discovery.** For evolving type systems (CMS, CRM custom objects), expose `list_available_types()` + `read_data(type)` over one wrapper per type. A product agent whose tools *are* the product nouns (`create_issue`, `update_document`) does not need to discover them at runtime. MCP standardizes discover and access for external services. Prefer those servers over hand-coded wrappers. **Graduation.** Hot paths can move to optimized code, but the agent still triggers them and falls back to domain primitives for edge cases. ## Context Patterns **Entity-scoped directories.** `{entity_type}/{entity_id}/`; separate ephemeral (`AgentCheckpoints/`, `AgentLogs/`) from durable (`Research/`). **The `context.md` pattern.** Read at session start, updated as state changes (agent identity, user knowledge, what exists, recent activity, current state). Portable working memory, no code changes. **Context injection.** System prompts include three sections, scoped to this run, not the whole product: 1. **Available resources**: what data exists, where 2. **Capabilities**: what the agent can do 3. **Recent activity**: what happened since last session Context is a budget with two ends. Too little and the agent asks redundant questions. Too much (every tool, every procedure, every run) and it picks the wrong one from a longer list. **Durable state outside the transcript.** A session that outgrows its window silently drops the earliest decisions, so anything the agent must still honour at step 40 belongs in a file it re-reads, not in message history (see `rules-arch/context-no-checkpoint-resume`). Audit the store, not the window-management technique. ## Agent-UI Communication **Completion signals.** Every model API returns a terminal reason for a turn (`stop_reason` in the Messages API, `finishReason` in AI SDK). The audit question is whether the orchestrator reads that field, or infers "done" from idle time, an empty delta, or a timeout. Reading it is not enough on its own: `pause_turn` means continue and `max_tokens` means truncated, so a loop that treats every non-tool reason as completion has smuggled the heuristic back in. Orchestrators that add their own states (`pause`, `escalate`, `retry`) must emit them as explicitly as completion, or the UI guesses again. **Partial completion tracking.** Per-task status (pending, in_progress, completed, failed, skipped); show `3/5 tasks complete (60%)` with error notes. **Agent event types.** Emit typed events (`thinking`, `toolCall`, `toolResult`, `textResponse`, `statusChange`); an `ephemeralToolCalls` flag hides noisy internals. **Shared workspace.** Agents and users share one data space, each building on the other's work. Sandbox only when security or data integrity requires it. **Stable names on the pixel surface.** When an agent must operate the same UI a human sees, expose durable names in the DOM (`data-agent-id`, aria names that match the tool nouns) rather than making the agent guess from CSS and copy. Screenshot-only targeting drifts when layout shifts; a named control is a tool. This is the product-side half of annotating a UI for agents, not a visual-polish concern. **Approval gates.** Match approval to stakes and reversibility: | Stakes | Reversibility | Pattern | |--------|--------------|---------| | Low | Easy | Auto-apply | | Low | Hard | Quick confirm | | High | Easy | Suggest + apply (show diff) | | High | Hard | Explicit approval | An explicit user request is already approval. Self-modification always requires explicit approval + audit log + rollback. **Provenance is the third axis.** Stakes and reversibility classify by tool name, so a gate built from them treats every delete the same. Two questions adjust the row: - **Did the agent create this in the current conversation?** Deleting a draft it just made is not the same act as deleting a record that existed before the session. - **Does the target reach outside the workspace?** Posting to an internal thread and posting to one synced with a public repo differ in blast radius, not in the tool name. A gate that cannot read those two facts can only be tuned by making it stricter, and a gate strict enough to catch the pre-existing delete will also interrupt the draft cleanup. Pass provenance into the checkpoint alongside stakes. -
ax-evolution-curve.md 5.3 KB
# AX Evolution Curve A 4-stage model for how deep the user-agent relationship is in a design. Calibrates audit expectations: a Conversational agent missing memory is fine; a Personally Intelligent one missing memory visibility is a finding. ## The Four Stages ### 1. Conversational Starts from scratch every time: no memory across sessions, the user re-explains everything. Behavior: a chatbot that forgets on refresh; "like I mentioned earlier" means nothing to it. ### 2. Task-Aware Watches and adjusts in the moment: tracks current task state and multi-step progress, reacts to now. Forgets between sessions. Behavior: sees your current document and suggests, but does not know your preferences or recall past decisions. ### 3. Personally Intelligent Remembers preferences and history across sessions: accumulates context, adapts to patterns, gets better with use. Behavior: knows you prefer concise answers, remembers your conventions, recalls decisions from weeks ago. ### 4. Socially Embedded Understands role, team, and cultural context: speaks for the user to others, manages cross-team comms, navigates org dynamics. Behavior: drafts messages to your team in your voice, knowing who needs what context and how to frame requests for each audience. ## The Defensibility Line Sits between Task-Aware and Personally Intelligent. Below it, features are commoditized (anyone can build a stateless chatbot or task tracker). Above it, accumulated context is a moat: switching costs rise because the agent knows the user, so the longer it is used the harder it is to leave. Unused product starves that context: no contact, stale help, then replacement. When auditing, note where the product sits. Below the line: differentiate through execution quality. Above it: make accumulated context visible and portable, or risk trust erosion when users feel locked in. ## Action depth The four stages are memory. They cannot see an agent that remembers everything and only ever suggests. When the agent takes actions, name the highest rung it actually uses. A Personally Intelligent agent stuck at Tip has plateaued. 1. **Tip:** the user reads a suggestion. 2. **Preview:** evidence, then ask. 3. **Receipt:** it acted, said what changed, undo is one tap. 4. **Maintain:** it keeps the job in order and reports without being asked. 5. **Partner:** it proposes the next job before the user asked. ## Mapping to Rules Which ax-audit rules matter most at each stage: | Stage | Key rules | |---|---| | Conversational | `control-over-conversational`, `comm-no-progress-signal` | | Task-Aware | `comm-no-intent-handshake`, `control-no-escape-hatch`, `control-no-approval-gate`, `control-thin-approval-payload`, `trust-no-escalation-path` | | Personally Intelligent | `context-memory-not-visible`, `context-under-contextual`, `trust-no-confidence-cues`, `trust-no-uncertainty-markers`, `trust-undisclosed-access-scope` | | Socially Embedded | `context-no-adaptive-canvas`, `comm-no-generative-momentum`, `comm-unrequested-action-no-consent` | Earlier-stage rules still apply at later stages. A Socially Embedded agent lacking an escape hatch is still a finding. Maintain and Partner rungs without standing consent are `comm-unrequested-action-no-consent`. ## Costume vs intelligence Feeling intelligent and feeling like AI are different axes. Users already love the top-left. They reject the right edge when chat is bolted onto a tool they already had. | | Does not wear the costume | Wears the costume | |---|---|---| | **Feels intelligent** | Native intelligence. Maps ETA, Discover Weekly, For You. Nobody calls it AI. | Destination AI. ChatGPT, Claude. Fine when chat is the product. | | **Does not feel intelligent** | Static tool. | Sparkle graveyard. Bolted-on "Ask AI". | When writing the AX Relationship Summary: - If chat is the product, destination chrome is fine. - If chat is bolted onto an existing tool, sparkle, "Ask AI", "How can I help you", or a named persona as the UI is the finding. Put it in `keyGap` or `trustQuestion` when it is the most important gap. - Thinking dots and token streaming become costume when they are the product, not a way to show work in progress. - Thumbs up/down as the only feedback is costume, not a trust mechanism. Strip the interface and the costume has nothing to hang on. What a user recognises across a chat thread, a phone, a voice, and a notification is character. No single finding can carry that. Put it in `keyGap` when the service behaves like a different product in each place, and in `trustQuestion` when only research can tell you whether it does. ## Assessment To place a design: - **What persists between sessions?** Nothing = Conversational; task state only = Task-Aware; preferences + history = Personally Intelligent; relationships + org context = Socially Embedded. - **Does it adapt to individual users?** If two users get identical responses in identical situations, it is at most Task-Aware. - **Does it act on others' behalf?** If yes, check whether it grasps enough social context to avoid harm. - **If it takes actions, which rung is the highest it actually uses?** Write that into `evolutionStage.behavior` with the memory stage. Do not add a second field. Describe behaviors in output, not labels: write "remembers preferences across sessions, acts at Preview," not "Stage 3 product." The framework is for reasoning about depth, not vocabulary. -
evaluation-scenarios.md 6.8 KB
# Evaluation scenarios Rubric for changing this skill. Never loaded during a user audit. No runner: treat each `expected_behavior` as a pass/fail checklist and run it by hand (or a thin harness) against a fixture repo. Ablate one rule at a time. Keep a rule only if a scenario below regresses without it. House opinions (evolution curve, costume vs intelligence, verdict thresholds) are not ablatable this way. ## Contents - Scenarios 1-3: thin approval payload, unattended cron, prose tool result - Scenarios 4-6: no agentic surface, weak detection signals, generic `Action` toolbar - Scenarios 7-8: executor code outside the detection table, out-of-scope layers - Scenarios 9-11: MCP `isError`, gate only in `canUseTool`, client-only Stop ## Scenario 1: thin approval on an execution surface **Query:** "AX review this PR. The agent can send email." **Fixture:** an `ApprovalDialog` that receives `call.args` and renders only `call.name`, on a tool-execution panel. **Expected behavior:** - Detects an agent tool-execution surface - Runs `control-thin-approval-payload` and returns `fail` with `file:line` - Assigns `release-blocker` from the rule's override table, not from stacking the generic bump - Does not also file `control-no-approval-gate` on the same dialog (the gate exists) ## Scenario 2: unattended cron, interactive gates pass **Query:** "Audit this for AX. We have approval dialogs on every tool." **Fixture:** a `cron.schedule` job that calls `runAgent` with no standing policy and no user notification. Interactive tool calls go through a populated approval dialog. **Expected behavior:** - Interactive path passes `control-no-approval-gate` and `control-thin-approval-payload` - `comm-unrequested-action-no-consent` fails, evidence names the cron entry point, not `runAgent` - A path with a standing policy and no notice still fails (either half missing is enough) - Assigned tier on the execution surface is `release-blocker` - Does not report the cron path as `control-no-approval-gate` ## Scenario 3: prose tool result with HTTP 200 **Query:** "Can an agent use our product? Review the tool handlers." **Fixture:** `sendInvoice` catches, logs, and returns `"Something went wrong."` with status 200. A sibling `draft_reply` tool returns a string on purpose. **Expected behavior:** - `parity-unstructured-tool-output` fails on `sendInvoice` with `file:line` - `draft_reply` is `pass` or `suppressed` (the draft is the payload) - Routes to this skill, not `dx-audit` (the question is whether an agent can recover, not whether a human likes the error string) - Verdict is ❌ NOT READY ## Scenario 4: traditional form PR, no agentic surface **Query:** "AX review this PR." **Fixture:** a settings form with a loading-state bug and no chat, tools, or agent routes. **Expected behavior:** - Feature detection finds nothing - Stops. Does not run the 27 rules - Routes to `ui-design` Audit mode - Does not file AX findings about the missing spinner ## Scenario 5: non-agentic code that trips a weak detection signal **Query:** "AX review this PR." **Fixture:** an onboarding widget computing `const completion = done / total`, and an upload component with an `isStreaming` flag for video. No chat, tools, agent routes, or model calls anywhere. **Expected behavior:** - Feature detection finds nothing; neither token counts as a signal on its own - Stops. Does not run the chat playbook's rules - Routes to `ui-design` Audit mode - Regression guard: this scenario failed before the weak-signal rule was added to `feature-playbooks.md`, when bare `completion` and `isStreaming` were listed as chat signals ## Scenario 6: a generic toolbar named Action **Query:** "AX review this PR." **Fixture:** `function Action({ icon, onClick })`, a generic icon-button in a toolbar, beside a plain to-do list. No tool call, no executor, no agent context. **Expected behavior:** - Does not detect a tool-execution surface; `<Action>` alone is a weak signal - Stops and routes to `ui-design` Audit mode - Regression guard: two models independently invented this guard themselves when `<Action>` was listed as a strong signal ## Scenario 7: real executor code that matches no listed string **Query:** "Can an agent use our product?" **Fixture:** `server.tool("send_invoice", ...)` handlers, and a `cron.schedule` calling `runAgent(...)`. Neither matches `<ToolCall>`, `tool_use`, or `executeAction`. **Expected behavior:** - Detects a tool-execution surface anyway; the detection table is illustrative, not a checklist - Does not report "no agentic features detected" over obvious executor code ## Scenario 8: correctly scoped audit with layers out of scope **Query:** "AX review this PR." (a UI-only diff: an approval component, no orchestrator, no connectors) **Expected behavior:** - Rules whose layer is absent return `out-of-scope`, not `unknown` - The self-check's 30% threshold counts only `unknown`, so the audit is not flagged INCOMPLETE - With a release-blocker present, the verdict is ❌ NOT READY, not 🚫 INCOMPLETE - Regression guard: two models both returned 🚫 INCOMPLETE here while holding two to six real blockers ## Scenario 9: MCP handler that reports failure without `isError` **Query:** "Audit our MCP server for AX." **Fixture:** `server.tool("send_invoice", ...)` whose `catch` returns `{ content: [{ type: "text", text: "Could not send the invoice." }] }` with no `isError`. A sibling tool returns `isError: true` with the same prose. **Expected behavior:** - `parity-unstructured-tool-output` fails on `send_invoice` with `file:line`, citing the missing `isError` - The sibling is `pass`: the flag is the machine-readable half - Does not report the prose text itself as the defect ## Scenario 10: gate that lives only in `canUseTool` **Query:** "AX review this PR. Every tool goes through our approval callback." **Fixture:** a Claude Agent SDK query with `canUseTool` prompting the user, and `allowedTools: ["Bash", "Write"]` bare in the same options. **Expected behavior:** - `comm-no-approval-gate` fails: allow rules resolve those tools before the callback runs, so the gate is off the path for them - Fix names a `PreToolUse` hook or scoped rules, not a change to the callback body - Does not also file `control-no-approval-gate` on the callback, which is well-shaped for the tools that do reach it ## Scenario 11: Stop button that stops the stream, not the run **Query:** "Critique this AI feature." **Fixture:** a chat panel whose Stop button calls `useChat().stop()`, and a route handler that calls `streamText` without `abortSignal` and whose tool `execute` functions ignore the signal. **Expected behavior:** - `control-no-escape-hatch` fails, evidence at the route handler and a tool `execute`, not at the button - Tier is `release-blocker` on the chat surface per the rule's table - A fixture where `req.signal` is threaded through both passes -
feature-playbooks.md 9.6 KB
# Feature Playbooks Detect each agentic feature from element + filename + route signals, then run its checks in order. Every check names a rule file in `rules-ax/` (Layer 2, agentic experience) or `rules-arch/` (Layer 1, architecture, marked explicitly). The tier in parentheses is a scan copy of the rule's override for that surface. The rule file's override table wins if they disagree. ## Table of contents - [Feature detection](#feature-detection) - [Diff-wide checks](#diff-wide-checks) - [Agent Chat / Copilot](#agent-chat--copilot) - [Agent Tool Execution / Action Panel](#agent-tool-execution--action-panel) - [Agent Configuration / System Prompt Editor](#agent-configuration--system-prompt-editor) - [Agent Dashboard / Status](#agent-dashboard--status) - [Coverage](#coverage) ## Feature detection | Feature | Detect by | |---|---| | agent chat / copilot | `<Chat>`, `<Assistant>`, `<Copilot>`, `role="assistant"`, `aiResponse`, `useChat`, `useCompletion`, `chatCompletion`, route `/chat`, `/assistant`, `/copilot` | | agent tool execution / action panel | `<ToolCall>`, `tool_use`, `function_call`, `executeAction`, `agentAction`, `server.tool(`, `registerTool(`, `runAgent(`, `toolApproval`, `needsApproval`, `addToolApprovalResponse`, `canUseTool`, `PreToolUse`, `ToolLoopAgent`, `destructiveHint`, `ui/resourceUri`, component `*ToolPanel*`, `*ActionLog*` | | agent configuration / system prompt editor | `<SystemPrompt>`, `<AgentConfig>`, `<PromptEditor>`, route `/agent/settings`, `/configure`, `systemPrompt`, `permissionMode`, `allowedTools` | | agent dashboard / status | `<AgentStatus>`, `<TaskList>`, `<RunHistory>`, `<RunLog>`, `RUN_FINISHED`, component `*AgentDashboard*`, route `/agent`, `/runs` | **The table is illustrative, not exhaustive.** It lists the signals seen most often, not every form agent code takes. A handler registered as `server.tool("send_invoice", ...)`, or an executor reached only as `runAgent(...)` from a scheduler, is an agentic surface whether or not it matches a listed string. Detect on what the code does; the table is a starting sweep, not a checklist that licenses "no agentic features detected" over obvious executor code. **Weak signals, never on their own.** `completion`, `isStreaming`, and `<Action>` match ordinary non-agentic code: a progress percentage (`const completion = done / total`), a video upload flag, and a generic icon-button component. Count any of them only alongside a strong signal from the table. A chat surface detected from a bare `completion`, or a tool-execution surface from a toolbar's `<Action>`, runs a full playbook against a form and produces exactly the noise the stop condition below exists to prevent. No agentic features detected → stop; this skill does not apply. Route to `ui-design` Audit mode. ## Diff-wide checks Run on every PR-mode audit, regardless of detected features: 1. **`parity-orphan-ui-action`** (rules-arch, fix-this-sprint): the diff adds a UI capability (button, form action, route handler) with no matching tool in the same PR; each orphan widens the user/agent capability gap. ## Agent Chat / Copilot User need: get help from the agent, trust its output, control what it does. Checks in order: 1. **`comm-no-progress-signal`** (release-blocker): streaming/thinking indicator visible during the response; never a frozen UI. 2. **`control-no-escape-hatch`** (release-blocker): chat-triggered actions are interruptible mid-execution and reversible after completion. 3. **`context-no-injection`** (rules-arch, release-blocker): sessions initialize with dynamic context (preferences, recent activity, project state), not a bare static prompt. 4. **`trust-no-confidence-cues`** (fix-this-sprint): output includes rationale or sources so the user can verify correctness. 5. **`trust-no-uncertainty-markers`** (fix-this-sprint): agent hedges when uncertain rather than presenting guesses as fact. 6. **`comm-no-intent-handshake`** (fix-this-sprint): non-trivial or destructive actions confirmed before execution. 7. **`control-thin-approval-payload`** (fix-this-sprint): chat-triggered approvals show the call's arguments, not just the tool's name; the user approves an act, not a category. 8. **`control-over-conversational`** (fix-this-sprint): parallel direct-manipulation controls exist for common actions; users not forced into chat. 9. **`context-memory-not-visible`** (fix-this-sprint): the user can see and edit what the agent remembers across sessions. 10. **`comm-no-generative-momentum`** (backlog): blank-canvas entry points offer an agent-generated draft when the agent has the context. ## Agent Tool Execution / Action Panel User need: understand what the agent is doing, stop it if wrong, trust the outcome. Checks in order: 1. **`trust-no-escalation-path`** (release-blocker): high-stakes actions (deletes, payments, external calls) can hand off to a human first. 2. **`control-no-approval-gate`** (release-blocker): the approval UI matches the stakes and reversibility of the action. 3. **`comm-no-approval-gate`** (rules-arch, release-blocker): a gate exists on the orchestrator's execution path and every tool reaches it, not just the ones with a confirmation dialog at the call site. 4. **`control-thin-approval-payload`** (release-blocker on this surface): the gate renders the call's arguments, target, and blast radius; a prompt naming only the tool is a click-through. Runs after check 2, which establishes that a gate exists at all. 5. **`comm-unrequested-action-no-consent`** (release-blocker on this surface): scheduled, webhook, and queue entry points that reach the executor carry a standing consent boundary and emit a notice the user can act on. Checks 2 to 4 only cover the path a user started. 6. **`control-no-escape-hatch`** (release-blocker): every completed action has undo or revise; the user is never locked into an agent decision. 7. **`comm-no-progress-visibility`** (rules-arch, release-blocker): the server emits step-level events during a multi-step run; `comm-no-progress-signal` covers whether the UI shows them. 8. **`comm-no-completion-signal`** (rules-arch, release-blocker): completion is explicitly signalled (`stop_reason`, completion tool), never inferred from idle time. 9. **`parity-unstructured-tool-output`** (rules-arch, release-blocker on this surface): tool handlers return a typed result whose failure branch is machine-readable, never prose with a success status. 10. **`comm-no-intent-handshake`** (release-blocker on this surface): ambiguous or multi-interpretation requests get a playback/confirmation before the agent acts. 11. **`trust-undisclosed-access-scope`** (release-blocker on this surface): the accounts and scopes the agent acts through are visible and individually revocable; undisclosed reach on an autonomous surface is the case the user cannot discover by watching. 12. **`context-under-contextual`** (fix-this-sprint on this surface): the agent uses available context (current page, selection, recent actions) instead of asking redundant questions. 13. **`granularity-static-api-mapping`** (rules-arch, backlog): evolving APIs use discover + access tools, not one hard-coded tool per endpoint. ## Agent Configuration / System Prompt Editor User need: customize agent behavior without breaking it, understand what changed. Checks in order: 1. **`parity-no-tool-parity`** (rules-arch, release-blocker): every UI config option has an agent-accessible equivalent; no GUI-only settings. 2. **`granularity-workflow-shaped-tool`** (rules-arch, fix-this-sprint): config tools are atomic primitives, not bundled workflows that hide individual options. 3. **`context-starvation`** (rules-arch, fix-this-sprint): the system prompt injects available resources, tools, and constraints so the agent knows what it can do. 4. **`trust-undisclosed-access-scope`** (fix-this-sprint): connected accounts, their scopes, and what is retained are all nameable in the user's terms, with a revoke per connector rather than one global disconnect. 5. **`context-memory-not-visible`** (fix-this-sprint): the user can see the full context the agent receives, including injected system prompts. 6. **`context-no-adaptive-canvas`** (backlog): the UI surfaces downstream effects when config changes alter agent behavior. ## Agent Dashboard / Status User need: see what the agent has done, what it's doing now, and what went wrong. Checks in order: 1. **`comm-no-completion-signal`** (rules-arch, release-blocker): completed tasks are explicitly marked done, not left ambiguous. 2. **`parity-crud-incomplete`** (rules-arch, release-blocker): tasks have full CRUD: create, view, cancel, retry, and delete. 3. **`comm-no-progress-visibility`** (rules-arch, fix-this-sprint on this surface): the run loop emits step events the dashboard can subscribe to or poll, not just a start and end row. 4. **`trust-no-confidence-cues`** (fix-this-sprint): completed results include reasoning or a summary of what was done and why. 5. **`comm-unrequested-action-no-consent`** (fix-this-sprint on this surface): unattended runs appear here with what they touched and a way back; the dashboard is where a scheduled action becomes visible at all, so a missing notice is the defect rather than a report of one. 6. **`context-memory-not-visible`** (backlog on this surface): the agent's accumulated context is viewable and editable. 7. **`context-no-checkpoint-resume`** (rules-arch, backlog): interrupted or failed tasks resume from the last checkpoint, not from scratch. ## Coverage All 27 rules are reachable: 10 via chat, 13 via tool execution, 6 via config, 7 via dashboard, 1 diff-wide (rules repeat across playbooks; unique total = 27). To add a rule, copy `rules-<layer>/_template.md` as the starting structure, then add the rule to at least one playbook or it will never run. -
framework-signals.md 8.3 KB
# Framework Signals Where the things the rules audit live in the four stacks agentic code is usually written in. Rule greps carry the common spellings; this file carries the rest, plus the spec defaults the rules lean on. When a grep misses in code that plainly does the thing, look the concept up here before recording `unknown`. Identifiers are current for AI SDK 7, MCP 2025-11-25, and the Claude Agent SDK as of the sources listed at the end. ## Contents - [Approval gate](#approval-gate) - [Progress and completion](#progress-and-completion) - [Tool results](#tool-results) - [Escape hatch](#escape-hatch) - [Handshake and clarification](#handshake-and-clarification) - [Agent-rendered UI](#agent-rendered-ui) - [Sources](#sources) ## Approval gate | Stack | Where the gate lives | What the rules read off it | |---|---|---| | AI SDK 7 | `toolApproval` on `streamText` or `ToolLoopAgent`: per tool one of `'not-applicable'`, `'approved'`, `'denied'`, `'user-approval'`, or a function of `({ toolCall, tools, messages })`. `'user-approval'` emits a `tool-approval-request` part and pauses; the client answers with `useChat().addToolApprovalResponse({ id, approved, reason })`. `needsApproval` on `tool()` is the deprecated v6 spelling and still works | A catch-all returning `'approved'` is `control-no-approval-gate` Scenario A; `'user-approval'` on read-only tools is Scenario B. A renderer that branches on `part.state === 'approval-requested'` and prints the tool name without `part.input` is `control-thin-approval-payload`. `experimental_toolApprovalSecret` HMAC-binds an approval to its call id and input, so a replayed approval cannot authorize different arguments | | MCP 2025-11-25 | The host. The spec says there SHOULD always be a human in the loop able to deny tool invocations, and clients SHOULD show tool inputs to the user before calling the server. Servers describe stakes through `annotations`: `readOnlyHint` (default false), `destructiveHint` (default true), `idempotentHint` (default false), `openWorldHint` (default true). Clients MUST treat annotations as untrusted unless the server is trusted | The defaults already fail closed; a host that relaxes them from an untrusted server's hints is `comm-no-approval-gate`. In Claude Code hosts a server can force a prompt with `_meta["anthropic/requiresUserInteraction"]` | | Claude Agent SDK | Six steps in order: `PreToolUse` hooks, deny rules, ask rules, `permissionMode`, allow rules, then `canUseTool(toolName, input, { suggestions, signal })` returning `{ behavior: 'allow', updatedInput }` or `{ behavior: 'deny', message }`. Modes: `default`, `dontAsk`, `acceptEdits`, `bypassPermissions`, `plan`, `auto` | A bare `allowedTools` entry, `acceptEdits`, or `bypassPermissions` resolves a call before `canUseTool` runs, so a check that exists only in the callback is off the path for every pre-approved tool (`comm-no-approval-gate`); a `PreToolUse` hook runs before every other step and is on it. `dontAsk` plus an explicit allowlist is a standing boundary for `comm-unrequested-action-no-consent`; the notice half still has to exist | ## Progress and completion | Stack | Step-level events | Terminal signal | |---|---|---| | AI SDK 7 | Server: `createUIMessageStream` with `writer.write({ type: 'data-<name>', ... })`, `createUIMessageStreamResponse`, `result.toUIMessageStream()`. Persistent parts reach the client in `message.parts`; parts written with `transient: true` reach it only through the `onData` callback. Client: `status` is `'submitted'`, `'streaming'`, `'ready'`, or `'error'`; `isLoading` no longer exists | `finishReason` on the result; the loop stops by `stopWhen` (`isStepCount(n)` in v7, `stepCountIs` before). A default `ToolLoopAgent` stops at 20 steps | | Anthropic Messages API | Streaming events per content block | `stop_reason`: `end_turn`, `tool_use`, `max_tokens`, `stop_sequence`, `pause_turn`, `refusal`, `model_context_window_exceeded`. `pause_turn` means resend and continue, `max_tokens` means truncated; a loop that treats either as done fails `comm-no-completion-signal` | | AG-UI | `RUN_STARTED`, `STEP_STARTED` and `STEP_FINISHED`, `TEXT_MESSAGE_START/CONTENT/END`, `TOOL_CALL_START/ARGS/END/RESULT`, `STATE_SNAPSHOT` and `STATE_DELTA` (JSON Patch) | `RUN_FINISHED` with an outcome, or `RUN_ERROR` | | MCP | `notifications/progress` keyed by the request's `progressToken`; long calls can negotiate tasks through `execution.taskSupport` | The `tools/call` result | ## Tool results | Stack | Success shape | Failure shape | |---|---|---| | MCP | `content[]` plus `structuredContent` conforming to the tool's `outputSchema`; servers MUST conform, clients SHOULD validate | `isError: true` on the result, with actionable text the model can retry on. The schema says: if not set, this is assumed to be false (the call was successful). Error prose in `content` with no `isError` is `parity-unstructured-tool-output` | | AI SDK 7 | `tool({ inputSchema, outputSchema, execute })`; `toModelOutput` shapes what the model sees versus what the UI keeps | Throw, or a discriminated union in the output. UI part states: `output-available`, `output-error`, `output-denied` | | Anthropic tool-writing guidance | Return meaningful, human-readable identifiers and offer a `response_format` of `concise` or `detailed` | Errors steer: say what to do next, not just what failed | ## Escape hatch | Stack | Where it lives | |---|---| | AI SDK 7 | `useChat().stop()` aborts the client fetch only. The route passes `req.signal` as `abortSignal` to `streamText`, and each tool's `execute` reads `abortSignal` from its options, or the server finishes the loop after Stop. `resumeStream` reconnects only when the server kept the run | | Claude Agent SDK | `query.interrupt()`; the `signal` handed to `canUseTool` is an `AbortSignal`. A `PreToolUse` hook returning `defer` persists the session so an approval can wait past process exit | | AG-UI | `RUN_FINISHED` carrying an interrupt outcome | ## Handshake and clarification | Stack | Primitive | |---|---| | Claude Agent SDK | `AskUserQuestion` (1 to 4 questions, 2 to 4 options each) routed through `canUseTool`; `plan` mode sends every write to the callback | | MCP | `elicitation/create` in `form` mode (flat primitives, exposed to the client) or `url` mode (credentials and payments, never through the client). Answered `accept`, `decline`, or `cancel`; only `accept` means proceed, and treating `cancel` as `accept` fails `comm-no-intent-handshake` | | AI SDK 7 | `addToolApprovalResponse({ id, approved: false, reason })` carries the user's correction back into the loop | ## Agent-rendered UI | Stack | Shape | |---|---| | MCP Apps | Servers publish `ui://` resources with `mimeType: text/html+mcp` and link them from a tool's `_meta["ui/resourceUri"]`; hosts render them in a sandboxed iframe and talk over JSON-RPC on `postMessage`, with consent for UI-initiated tool calls. Rendered controls count as direct manipulation for `control-over-conversational` and as phase-specific UI for `context-no-adaptive-canvas` | | AI SDK 7 | `data-<name>` parts rendered by a component chosen on `part.type`; a chat that renders only `text` parts has the tool and step states in hand and drops them | ## Sources - AI SDK 7 tool calling and approvals: <https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling>, <https://ai-sdk.dev/docs/agents/tool-approvals>, <https://ai-sdk.dev/docs/ai-sdk-ui/chatbot-tool-usage>, <https://ai-sdk.dev/docs/ai-sdk-ui/streaming-data>, <https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat>, <https://ai-sdk.dev/docs/migration-guides/migration-guide-7-0> - MCP tools, elicitation, apps: <https://modelcontextprotocol.io/specification/2025-11-25/server/tools>, <https://modelcontextprotocol.io/specification/2025-11-25/client/elicitation>, <https://blog.modelcontextprotocol.io/posts/2025-11-21-mcp-apps/> - Claude Agent SDK permissions and approvals: <https://code.claude.com/docs/en/agent-sdk/permissions>, <https://code.claude.com/docs/en/agent-sdk/user-input> - Anthropic stop reasons: <https://platform.claude.com/docs/en/api/handling-stop-reasons> - Anthropic tool and context guidance: <https://www.anthropic.com/engineering/writing-tools-for-agents>, <https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents>, <https://www.anthropic.com/engineering/building-effective-agents> - AG-UI events: <https://docs.ag-ui.com/concepts/events> -
invisible-interface.md 3 KB
# The Invisible Interface The user reaches an agent connected to their mail, calendar, files, and accounts. The apps underneath do the work without being looked at. Four rules in this skill exist because of that. Three further arguments carry no rule: they are real, and nothing about them can change a ship verdict, so they belong in `keyGap` or `trustQuestion`. Source: <https://designplusai.com/p/invisible-interfaces>. ## Contents - [What the four rules catch](#what-the-four-rules-catch) - [What stays in the AX summary](#what-stays-in-the-ax-summary) - [Where the arguments land](#where-the-arguments-land) ## What the four rules catch The service layer is closer to a protocol than a screen: legible, structured, operable, and honest about its state. A tool that answers failure with a sentence and a 200 is not honest, and every capability above it is guessing (`parity-unstructured-tool-output`). The approval moment is one of the few surfaces left. It has to carry enough of the act for a confident yes or no. A gate that names the tool and hides its arguments is a click-through (`control-thin-approval-payload`). Legitimacy is what the agent can reach, what it keeps, and what it does when nobody is watching. The checkable form: the user can see the grants in their own terms and take one back without disconnecting everything (`trust-undisclosed-access-scope`). An action nobody requested cannot borrow permission from a request. Scheduled and triggered runs need a boundary agreed in advance and a notice afterwards the user can act on (`comm-unrequested-action-no-consent`). That is the safety cost of the Maintain and Partner rungs. ## What stays in the AX summary **Connectability.** The best interface still loses if the layer where the user lives cannot reach it. That is a strategy gap, not a user-harm gap. A rule for it would sit at `backlog` on every surface and never change a verdict. Raise it in `keyGap`. **Considered transparency.** `comm-no-progress-signal` catches showing too little. Showing too much (a raw token and tool-call dump with no summary) needs the rendered flow, so a rule for it could only return `unknown` on static evidence. Say so in `keyGap`. **Character across surfaces.** Tone, restraint, and whether the thing admits what it does not know. No rule. `keyGap` when it behaves like a different product in each place; `trustQuestion` when only research can tell. ## Where the arguments land | Argument | Lands in | |---|---| | The tool surface is a protocol, honest about its state | `rules-arch/parity-unstructured-tool-output` | | The approval moment carries the decision | `rules-ax/control-thin-approval-payload` | | Legitimacy: what can it reach | `rules-ax/trust-undisclosed-access-scope` | | Proactive action needs standing consent | `rules-ax/comm-unrequested-action-no-consent` | | Connectability is existential | No rule; `keyGap` | | Considered transparency, not a log wall | No rule; `keyGap` | | Character across surfaces | No rule; `keyGap` and `trustQuestion` | -
output-format.md 5.2 KB
# Output Format Output structure for ax-audit results: a findings table, then the AX relationship summary. ## Table of contents - [Findings table](#findings-table) - [Field reference](#field-reference) - [AX relationship summary](#ax-relationship-summary) - [AX relationship summary, field descriptions](#ax-relationship-summary-field-descriptions) - [Terminal rendering](#terminal-rendering) ## Findings table Each finding is a JSON object (schema is compatible with `ui-design` Audit mode findings, so the two reports merge): ```json { "rule": "trust-no-confidence-cues", "layer": "ax", "category": "trust", "feature": "agent-chat", "surface": "ChatPanel", "file": "src/chat/ChatPanel.tsx", "line": 42, "result": "fail", "defaultTier": "fix-this-sprint", "assignedTier": "fix-this-sprint", "tierReason": "Default tier; agent chat surface.", "observed": "Agent output rendered in <AssistantMessage> with no citation, source, or reasoning child components.", "evidence": ["src/chat/ChatPanel.tsx:42, <AssistantMessage content={message.content} /> with no children"], "fix": "Add a <Sources> or <Reasoning> component inside agent message rendering.", "suppressed": false } ``` ## Field reference | Field | Values / notes | |---|---| | `rule` | Rule slug, matches the rule filename without `.md` | | `layer` | `arch` (Layer 1) or `ax` (Layer 2) | | `category` | arch: `parity \| granularity \| context \| comm`. ax: `trust \| control \| context \| comm` | | `feature` | One of 4 playbooks: `agent-chat`, `agent-tool-execution`, `agent-config`, `agent-dashboard` | | `surface` | Component or page name the finding sits on (groups the report) | | `file`, `line` | Evidence location; required on every `fail`/`warn` | | `result` | `pass \| warn \| fail \| unknown \| out-of-scope`. `unknown`: the evidence was reachable and no judgment could be reached, which counts against the self-check. `out-of-scope`: the layer the rule audits is not in this scope at all (no orchestrator in a UI-only diff, no connector code in a tool-handler slice), which does not. Both require a reason in `observed` | | `defaultTier`, `assignedTier`, `tierReason` | Tier from the rule file, tier after surface override, and one-sentence justification. Tier is the only ship-impact signal; no separate severity field | | `observed` | What the code actually does, in one sentence | | `evidence` | Array of `file:line: excerpt` strings backing the finding | | `fix` | Concrete change; a snippet or one-sentence instruction | | `suppressed` | `true` when an `ax-audit-ignore:<slug>` comment covers the match, report suppressed counts, never silently drop | ## AX relationship summary Produced after findings, only when agentic features are detected. Four fields naming the user-agent relationship in behavioral terms: ```json { "axSummary": { "evolutionStage": { "stage": 2, "label": "Task-Aware", "behavior": "Agent tracks current task state and adjusts in the moment, but starts fresh each session with no memory of user preferences or history." }, "trustSignal": { "level": "moderate", "reasoning": "Escape hatches present for all agent actions. Confidence cues missing: agent output has no rationale or source attribution." }, "keyGap": "Agent accumulates no session context; every interaction starts cold. Users re-explain preferences and constraints each time.", "trustQuestion": "Will users accept inline rationale (sources, reasoning steps) on every agent response, or will it feel like noise?" } } ``` ## AX relationship summary: field descriptions | Field | Description | |---|---| | `evolutionStage` | Which of the 4 stages (see `ax-evolution-curve.md`). Describe the behavior, not the label (label for JSON, behavior for the reader). | | `trustSignal` | `high \| moderate \| low` with one-sentence reasoning, from the trust-critical rules that ran: escalation, escape hatch, approval gates and what they showed, access scope, unprompted action, confidence cues. | | `keyGap` | Single most important architectural or trust gap. One sentence, specific enough to act on. | | `trustQuestion` | One question for the designer/developer to answer before the next round; only prototyping or research can resolve it. | ## Terminal rendering Terminal (not JSON) format: ``` ═══════════════════════════════════════════════════════════ AX VERDICT: ⚠️ READY WITH FOLLOW-UP (0 blockers, 4 fix-this-sprint) Surfaces: 2 (ChatPanel, ToolExecutionPanel) Findings: 6 Release blockers: 0 Fix this sprint: 4 ⚠️ Backlog: 2 📋 AX Relationship: Stage: Task-Aware (2 of 4) Trust: Moderate: escape hatches present, confidence cues missing Key gap: No session context; every interaction starts cold Question: Will users accept inline rationale on every response? Cross-reference: Run ui-design Audit mode for traditional UX findings ═══════════════════════════════════════════════════════════ ``` -
ship-readiness.md 5.6 KB
# Ship Readiness: Three-Tier Verdict for Agentic Surfaces Every finding gets one of three tiers, deciding whether the PR ships, waits, or merges with follow-up. ## Table of contents - [The three tiers](#the-three-tiers) - [Tier assignment rules](#tier-assignment-rules) - [Verdict logic](#verdict-logic) ## The three tiers ### ⛔ release-blocker: fix before merge Cause user harm, unsafe autonomous behavior, or unrecoverable agent actions in production. Triggers: - **No escape hatch**: agent takes actions the user cannot interrupt, undo, or override; locked into an autonomous workflow with no way out. - **No approval gate on high-stakes actions**: agent autonomously runs destructive, financial, or external actions (deleting records, sending emails, charging cards) without confirmation. - **No escalation path**: high-stakes decisions with no human handoff; failures cascade without intervention. - **Silent execution**: multi-step task with no progress indication; user cannot tell if it is working, stalled, or failed. - **Heuristic completion**: completion detected by idle time, not an explicit signal; downstream steps race (fire too early or too late). - **Broken tool parity**: user can do something the agent cannot, or vice versa; breaks the mental model of what the agent can do. - **Missing CRUD**: entity has create but no delete, or read but no update; agent gets stuck mid-workflow with no way to correct or clean up. The next four are blockers on the agent tool execution surface and one tier lower elsewhere, because each is a property of code that acts rather than code that reports: - **Unconsented proactive execution**: a scheduled, webhook, or queued run reaches the executor missing a standing boundary, a notice the user can act on, or both; the interactive gates do not cover it, because nobody was there to prompt. - **Undisclosed reach**: the agent acts through accounts and scopes the user cannot see or individually revoke; the one thing watching the agent work will never reveal. - **Approval with nothing to decide on**: a gate that fires correctly and renders only the tool's name, so the user approves a category rather than an act. - **Tool output that misreports its own state**: failures returned as prose with a success status; every capability above the tool is guessing, silently. ### ⚠️ fix-this-sprint: merge but log issue Degrade the agentic experience but don't block shipping. Need a tracking issue before merge, resolved within the current sprint. Triggers: - Agent output with no confidence cues or reasoning (functional but trust-eroding) - No intent handshake before non-trivial actions (agent acts without confirming it understood the request) - Chat-only interface for button-worthy actions (common tasks buried in free-text input) - Agent uses context but user cannot see or edit what is remembered (opaque but not dangerous) - System prompt missing resource injection (agent under-informed for the task) - Config tools bundled instead of atomic (inflexible; user cannot grant fine-grained permissions) ### 📋 backlog: track, ship Real but low-stakes. Ship the PR, log a backlog issue, prioritize by frequency or impact later. Triggers: - Interface does not reshape with agent task progression (static but functional) - Agent does not leverage all available context (underperforms but does not break) - No generative momentum on blank-canvas surfaces (missed proactive-suggestion opportunity) - Static API mapping instead of dynamic discovery (less flexible when tools change) - No checkpoint/resume for long-running tasks (risky on interruption but rare in practice) ## Tier assignment rules Precedence, highest first; apply exactly one: 1. **The rule's own surface-override table** (in the rule file). Most carry one; it is authoritative. 2. **The generic surface adjustment below**: only for rules with no override row for the surface. 3. **The rule's `defaultTier`.** Never stack adjustments: a rule whose table already says `release-blocker` on tool execution is not bumped again. | Surface context | Generic adjustment | |---|---| | Agent tool execution / action panel | Bump 1 tier (sprint → blocker; backlog → sprint): autonomous actions demand higher safety | | Agent chat / copilot | No adjustment: conversational surfaces tolerate slightly more friction | | Agent config / system prompt editor | No adjustment | | Agent dashboard / status | Down 1 tier (blocker → sprint; sprint → backlog): monitoring is less critical than action surfaces | ## Verdict logic Aggregate the per-finding tiers into a top-level verdict (shown in the summary block at the top of every report): | Verdict | Condition | |---|---| | ✅ READY | 0 release-blockers AND ≤3 fix-this-sprint | | ⚠️ READY WITH FOLLOW-UP | 0 release-blockers AND ≥4 fix-this-sprint | | ❌ NOT READY | ≥1 release-blocker | | 🚫 INCOMPLETE | Audit-self-check failed; re-run | Justify every assigned tier in `tierReason` ("release-blocker because agent action panel"). Bare tiers without that sentence are incomplete. Tier per finding, not per rule: a rule's `defaultTier` is where the assignment starts, and the surface decides where it lands. Two ways to get this wrong, both of which cost the verdict its meaning: - **Inflation.** Everything becomes `release-blocker`. One inflated finding flips the whole PR to ❌ NOT READY, so a report that does this twice teaches the team to read the verdict as noise and merge anyway. - **Deflation.** Everything slides to `backlog` for a greener verdict. That reads well once and catches up at the next production incident. The test for either: if a finding could not honestly block a merge, it is not a blocker; if it would cause user harm, it is not backlog.
-
-
rules-arch
-
comm-no-approval-gate.md 5.9 KB
--- title: Orchestrator executes high-stakes tools with no gate on the code path slug: comm-no-approval-gate category: comm defaultTier: release-blocker surfaces: agent-tool-execution agent-native-principle: Parity (agent-UI communication) detection: hybrid related: comm-no-progress-visibility, control-no-approval-gate, control-thin-approval-payload, comm-unrequested-action-no-consent --- ## Orchestrator executes high-stakes tools with no gate on the code path The tool-use loop calls `tool.execute()` directly, so nothing in the orchestrator can interpose a confirmation. Whatever the UI renders, a destructive tool registered tomorrow ships ungated by default. Violates Parity: oversight belongs on the execution path both the user and the agent go through. Scope: this rule asks whether a gate exists in the orchestrator at all, and whether tool definitions carry the risk metadata a gate needs. Which approval treatment each risk level earns (auto-apply, quick confirm, diff, modal) is `rules-ax/control-no-approval-gate`. ## What goes wrong A chat surface ships a confirmation modal for the three delete tools it knows about, wired at the call site. Someone adds `transfer_funds` to the registry. The loop calls it the same way it calls `list_files`, and no reviewer notices, because the gate was never on the path, only on three call sites. ## Detection **Surfaces:** agent-tool-execution **Static signals:** 1. Find the tool-use loop and every call site that reaches `.execute(`. More than one path to execution means there is no single gate. 2. Identify destructive/financial/external operations: send, delete, publish, deploy, charge, transfer. 3. Check whether an approval handler sits between dispatch and execution on that path, and whether it is reached for every tool or only for named ones. 4. Check what happens to a tool with no stakes metadata: defaulting to auto-approve is a fail even when today's registry is fully labelled. 5. Check what the checkpoint receives. Tool name and arguments alone cannot express provenance (did the agent create this object this conversation, does the target leave the workspace), so a gate given only those can be tuned by strictness and nothing else. 6. Check where the stakes come from. MCP `annotations` (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) are hints the spec tells clients to treat as untrusted from untrusted servers; a checkpoint that auto-approves on a third-party server's `readOnlyHint: true` has delegated itself. The spec defaults (`destructiveHint: true`, `readOnlyHint: false`) are the fail-closed baseline. 7. In Claude Agent SDK code, find which step holds the check. Allow rules, `acceptEdits`, and `bypassPermissions` resolve a call before `canUseTool` runs, so a check that exists only in that callback is off the path for every pre-approved tool. A `PreToolUse` hook runs before every other step and is on it. **Runtime signals:** a destructive tool added to the registry with no other change executes without prompting. **Concrete commands:** ```bash rg '(name|toolName).*["'"'"'](send|delete|remove|publish|deploy|charge|transfer)' --type=ts src/ rg '(executeTool|callTool|invokeTool)' --type=ts -A 10 src/ | rg -v '(confirm|approve|requireApproval)' rg '(requireApproval|confirmBefore|approvalGate|stakesLevel)' --type=ts src/ rg '(toolApproval|needsApproval|canUseTool|PreToolUse|permissionMode|bypassPermissions|readOnlyHint|destructiveHint)' --type=ts src/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:comm-no-approval-gate`. - Skip read-only operations (get, list, search, fetch). - Skip operations marked safe/reversible in tool metadata. - Skip test files and fixtures. ## Fix Every tool call passes through one checkpoint, and tools declare their own risk so the checkpoint can classify them without a hardcoded name list. Fail closed: a tool with no declared stakes is treated as high. ```tsx // before: no interposition point, gating is per-call-site or absent async function executeTool(tc: ToolCall) { return tools[tc.name].execute(tc.args); } // after: single choke point, risk declared on the tool async function executeTool(tc: ToolCall, onApproval: ApprovalHandler) { const tool = tools[tc.name]; const risk = tool.stakes ?? "high"; // unlabelled tools are not auto-approved const decision = await onApproval({ tool: tc.name, args: tc.args, risk, reversibility: tool.reversibility }); if (decision !== "approved") return { status: "cancelled", reason: decision }; return tool.execute(tc.args); } ``` The `onApproval` handler owns the treatment per risk level, so it is audited by `rules-ax/control-no-approval-gate`, not here. This rule passes once the checkpoint exists, covers every tool, and cannot be bypassed by registering a new one. The choke point has a name in each stack: `toolApproval` on the agent in AI SDK 7, a `PreToolUse` hook in the Claude Agent SDK, the `tools/call` dispatcher in an MCP host. ## Default tier and overrides **Defaults to:** `release-blocker` The gap is in shared orchestrator code, so it reaches every surface that can trigger a mutating tool. The rows do not taper the way a presentation rule's do. | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | release-blocker | ## Examples **Anti-pattern (fails):** the tool-use loop calls `tools[tc.name].execute(tc.args)` with no handler parameter threaded in, and `grep` for `requireApproval|approvalGate` in the orchestrator returns nothing. Confirmation modals in the chat component do not count; they cannot see a tool call the loop makes on the server. **Applied (passes):** one `executeTool` wrapper, every registration path goes through it, and tool definitions carry `stakes` / `reversibility` fields the checkpoint reads. ## Suppression ```tsx // ax-audit-ignore:comm-no-approval-gate, internal cleanup, operates only on temp files const cleanupTool = { execute: (args) => fs.rm(args.tempDir) }; ``` -
comm-no-completion-signal.md 3.6 KB
--- title: Agent completion detected by heuristic instead of explicit signal slug: comm-no-completion-signal category: comm defaultTier: release-blocker surfaces: agent-tool-execution, agent-dashboard agent-native-principle: Parity (agent-UI communication) detection: code-auditable related: comm-no-progress-visibility, parity-unstructured-tool-output --- ## Agent completion detected by heuristic instead of explicit signal Orchestrator detects "done" by counting idle iterations, checking output files, or waiting for a timeout. A thinking pause looks like completion; slow API calls trigger premature termination. Violates Parity: the UI must receive an explicit signal, not guess. ## What goes wrong Agent researches a complex question. Makes 3 tool calls, then pauses 8 seconds composing a response. Orchestrator counts 2 idle iterations, hits `maxIdleIterations: 2`, terminates. User sees a truncated answer. ## Detection **Surfaces:** agent-tool-execution, agent-dashboard **Static signals:** 1. Find the orchestrator control loop that decides continue/stop. 2. Check for idle-counting, timeout-based completion, or file-existence as termination. 3. Check which terminal reasons the loop handles. `end_turn` alone is not enough: `pause_turn` means resend and continue, `max_tokens` means truncated, so a loop that returns on "anything but `tool_use`" presents a cut-off answer as complete. The full list is in `references/framework-signals.md`. 4. Flag any heuristic used as the primary completion signal. **Concrete commands:** ```bash rg '(consecutiveIdle|noToolCall|idleCount|maxIdle)' --type=ts src/ rg '(setTimeout|setInterval)' --type=ts -A 5 src/ | rg '(done|complete|finish|terminate)' rg '(stop_reason|stopReason|finishReason|end_turn|pause_turn|RUN_FINISHED|shouldContinue)' --type=ts src/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:comm-no-completion-signal`. - Skip timeout logic alongside an explicit signal (both `stop_reason` AND `setTimeout`). - Skip test files and fixtures. ## Fix ```tsx // before: heuristic completion let idle = 0; while (idle < 3) { const res = await llm.chat(messages); if (!res.toolCalls.length) { idle++; continue; } idle = 0; await executeTools(res.toolCalls, messages); } // after: every terminal reason handled explicitly, none inferred while (true) { const res = await llm.chat(messages); switch (res.stopReason) { case "tool_use": for (const tc of res.toolCalls) { if (tc.name === "task_complete") return { status: "complete", summary: tc.args.summary }; messages.push({ role: "tool", content: await executeTool(tc) }); } continue; case "pause_turn": continue; // server tools mid-run: resend, not done case "end_turn": return { status: "complete", content: res.content }; case "max_tokens": return { status: "truncated", content: res.content }; default: return { status: "failed", reason: res.stopReason }; // refusal, context window } } ``` ## Default tier and overrides **Defaults to:** `release-blocker` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent dashboard | release-blocker | | Agent chat | fix-this-sprint | | Agent config | backlog | ## Examples **Anti-pattern (fails):** `while (noToolCalls < 2)`: thinking pause triggers premature termination. **Applied (passes):** `if (res.stopReason === "end_turn") return res.content`: explicit model signal. ## Suppression ```tsx // ax-audit-ignore:comm-no-completion-signal, timeout is safety net, primary signal is stop_reason const SAFETY_TIMEOUT = 120_000; ``` -
comm-no-progress-visibility.md 4.3 KB
--- title: Orchestrator emits no progress events during execution slug: comm-no-progress-visibility category: comm defaultTier: release-blocker surfaces: agent-tool-execution, agent-dashboard agent-native-principle: Parity (agent-UI communication) detection: code-auditable related: comm-no-completion-signal, comm-no-approval-gate, comm-no-progress-signal --- ## Orchestrator emits no progress events during execution The run loop awaits every tool call and returns one payload at the end, so there is no event for any client to subscribe to. Violates Parity: the agent knows which step it is on and the transport throws that away. Scope: this rule asks whether the server emits step-level events at all. Whether the client renders them as a step list, a status line, or streamed text is `rules-ax/comm-no-progress-signal`. A streaming route with a UI that ignores the events fails that rule, not this one. ## What goes wrong The agent makes 12 tool calls over 45s analyzing a codebase. The route handler is `const result = await agent.run(msg); return Response.json(result)`, so the only thing a client can show is a spinner. No UI change can fix it: the information never left the server. ## Detection **Surfaces:** agent-tool-execution, agent-dashboard **Static signals:** 1. Find the server-side run loop: the route handler or worker that iterates tool calls. 2. Check its return type. A resolved object or `Response.json(...)` after the loop means events were never emitted; a `ReadableStream`, SSE response, or async generator means they were. 3. Inside the loop, check whether each tool call and each reasoning step yields or publishes an event, or whether the loop only accumulates into a local variable. 4. Flag loops whose only observable output is the final return value. **Concrete commands:** ```bash rg -l '(agent|orchestrator)\.(run|invoke|execute)\(' --type=ts src/app/api/ src/server/ rg -n 'return (NextResponse|Response)\.json' --type=ts -B 15 src/app/api/ | rg '(toolCall|tool_use|for await|while)' rg -n '(ReadableStream|text/event-stream|createUIMessageStream|toUIMessageStream|RUN_STARTED|STEP_STARTED|TOOL_CALL_START|async function\*|yield |emit\()' --type=ts src/app/api/ src/server/ ``` **Judgment signals:** - A route that streams text deltas while tool calls run silently between them is a partial pass: the user sees typing, then an unexplained pause. Report `warn` and name the missing step events. **False-positive guards:** - Skip files with `// ax-audit-ignore:comm-no-progress-visibility`. - Skip sub-second operations. - Skip loops whose consumer is another server process with no user waiting on it. - Skip test files and fixtures. ## Fix ```ts // before: the loop accumulates, the route returns once export async function POST(req: Request) { const result = await agent.run(await req.text()); // 30s, nothing observable return Response.json(result); } // after: the loop yields typed events, the route streams them async function* run(msg: string) { for (const step of await plan(msg)) { yield { type: "toolCall", toolName: step.tool }; const out = await tools[step.tool].execute(step.args); yield { type: "toolResult", toolName: step.tool, summary: summarize(out) }; } yield { type: "done" }; } export async function POST(req: Request) { return new Response(toSSE(run(await req.text())), { headers: { "content-type": "text/event-stream" }, }); } ``` Event names are a contract: `rules-ax/comm-no-progress-signal` audits the client against the same set, so renaming them mid-stack breaks the surface that displays them. In AI SDK 7 the equivalent is `createUIMessageStream` with `writer.write({ type: "data-step", ... })` per tool call; in AG-UI it is `STEP_STARTED` and `TOOL_CALL_START`. ## Default tier and overrides **Defaults to:** `release-blocker` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | release-blocker | | Agent dashboard | fix-this-sprint | ## Examples **Anti-pattern (fails):** the route awaits the whole run and returns `Response.json(result)`. The client has nothing to subscribe to. **Applied (passes):** the run loop is an async generator yielding `toolCall` / `toolResult` / `done`, and the route returns a stream of them. ## Suppression ```tsx {/* ax-audit-ignore:comm-no-progress-visibility, instant lookup, <500ms */} <QuickLookupAgent /> ``` -
context-no-checkpoint-resume.md 3 KB
--- title: Long-running agent with no checkpoint/resume slug: context-no-checkpoint-resume category: context defaultTier: backlog surfaces: agent-dashboard agent-native-principle: Improvement Over Time detection: hybrid related: comm-no-completion-signal --- ## Long-running agent with no checkpoint/resume Agent runs a multi-step task with no durability. Browser closes, network drops, session times out, all progress lost. User starts from scratch. Violates Improvement Over Time: completed work should survive interruption. ## What goes wrong User asks the agent to refactor 15 files. Agent completes 12 over 4 minutes. Laptop sleeps. On reconnect the session is gone, with no record of what was done. Agent redoes all 15, possibly making different choices. ## Detection **Surfaces:** agent-dashboard **Static signals:** 1. Find execution loops or multi-step task handlers. 2. Check whether they persist state between iterations. 3. Check whether resume/recovery logic exists. 4. Flag multi-step agents with no checkpoint writes. **Runtime signals:** Agent runs >60s with no state persistence. Reconnect restarts from scratch. **Judgment signals:** - A client `resumeStream` (AI SDK 7) resumes only if the server kept the run; a Claude Agent SDK `PreToolUse` hook returning `defer` persists the session across an approval wait. Either is a pass only when the server side is there; a reconnect helper over a run that died with the request is not. **Concrete commands:** ```bash rg '(for\s*\(|while\s*\(|for await)' --type=ts -A 5 src/ | rg -B 1 '(toolCall|executeStep|runTool)' rg '(checkpoint|saveState|persistSession|saveProgress)' --type=ts src/ rg '(resume|recover|restoreSession|loadCheckpoint|resumeStream|defer)' --type=ts src/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:context-no-checkpoint-resume`. - Skip single-step agents (no loop, single tool call). - Skip agents reliably under 10 seconds. - Skip test files and fixtures. ## Fix ```tsx // before async function refactorFiles(files: string[]) { for (const file of files) await agent.refactor(file); } // after: checkpoint after each step, resume on reconnect async function refactorFiles(sessionId: string, files: string[]) { const cp = await loadCheckpoint(sessionId); const done = new Set(cp?.completed ?? []); for (const file of files) { if (done.has(file)) continue; await agent.refactor(file); done.add(file); await saveCheckpoint(sessionId, { completed: [...done], updatedAt: Date.now() }); } } ``` ## Default tier and overrides **Defaults to:** `backlog` | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent chat | backlog | | Agent config | backlog | | Agent dashboard | backlog | ## Examples **Anti-pattern (fails):** `for (const t of tasks) await agent.execute(t)`: tab closes at task 8, all lost. **Applied (passes):** Loop resumes from `loadCheckpoint(id)` index, calls `saveCheckpoint` after each step. ## Suppression ```tsx // ax-audit-ignore:context-no-checkpoint-resume, sub-second operation await agent.formatSingleFile(filePath); ``` -
context-no-injection.md 2.6 KB
--- title: Agent session starts without knowing what data exists slug: context-no-injection category: context defaultTier: fix-this-sprint surfaces: agent-chat agent-native-principle: Improvement Over Time detection: code-auditable related: context-starvation, context-no-checkpoint-resume --- ## Agent session starts without knowing what data exists Static system prompt, no dynamic context. Every session starts ignorant of projects, preferences, and prior work that already exists. Violates Improvement Over Time: each session should build on the last. ## What goes wrong User opens a design review agent for the third time today. It has no memory of earlier sessions, the 5 files reviewed, or the user's accessibility-first preference. It asks "What would you like me to review?" again. ## Detection **Surfaces:** agent-chat **Static signals:** 1. Find session initialization: agent constructors, chat init, session start handlers. 2. Check whether initialization loads dynamic context (context files, preferences, recent activity). 3. Flag sessions that use only static/hardcoded prompt content. **Concrete commands:** ```bash rg '(new Agent|createAgent|initSession|startChat)' --type=ts -A 15 src/ rg 'messages\s*[:=]\s*\[' --type=ts -A 5 src/ | rg 'role.*system' | rg -v 'await|fetch|load|get' rg '(context\.md|loadContext|getContext|sessionContext)' --type=ts src/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:context-no-injection`. - Skip test files and fixtures. - Skip constructors where context is injected by a parent orchestrator. ## Fix ```tsx // before: static initialization function createSession(userId: string) { return { messages: [{ role: "system", content: STATIC_PROMPT }] }; } // after: read context.md at session start async function createSession(userId: string) { const ctx = await readContextFile(userId); const prefs = await getUserPreferences(userId); return { messages: [{ role: "system", content: `${STATIC_PROMPT}\n\n${ctx}\n\n${prefs.summary}` }], }; } ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent chat | release-blocker | | Agent tool execution | fix-this-sprint | | Agent config | backlog | | Agent dashboard | backlog | ## Examples **Anti-pattern (fails):** `private messages = [{ role: "system", content: "You review code." }]` **Applied (passes):** `static async create(uid) { const ctx = await loadProjectContext(uid); ... }` ## Suppression ```tsx // ax-audit-ignore:context-no-injection, stateless utility agent, no user context needed const agent = new StatelessAgent(STATIC_PROMPT); ``` -
context-starvation.md 2.9 KB
--- title: System prompt missing resource injection slug: context-starvation category: context defaultTier: fix-this-sprint surfaces: agent-config agent-native-principle: Improvement Over Time detection: code-auditable related: context-no-injection --- ## System prompt missing resource injection System prompt says "You are a helpful assistant" with zero dynamic context. Agent asks "What files do you have?" instead of using them. Violates Improvement Over Time: agents should accumulate context, not start blind. ## What goes wrong User opens a project management agent. System prompt has role instructions but nothing about 3 active projects or 12 unread notifications. First message: "What would you like to work on today?" ## Detection **Surfaces:** agent-config **Static signals:** 1. Find system prompt assembly: string templates, prompt builders, message arrays. 2. Check whether the prompt injects: (a) available resources, (b) capabilities, (c) recent activity. 3. Flag prompts missing any of the three. **Concrete commands:** ```bash rg 'role:\s*["\x27]system["\x27]' --type=ts -A 10 src/ | rg -v '\$\{|concat|join|append' rg '(availableResources|recentActivity|capabilities|context\.md)' --type=ts src/ ``` **Judgment signals:** - Missing any of the three sections is the usual fail (too little). - A prompt that injects every tool and every procedure on every run also fails: the run should carry what this task needs, not the whole product. A small agent with a short resident toolset is not this. **False-positive guards:** - Skip files with `// ax-audit-ignore:context-starvation`. - Skip test files and fixtures. - Skip prompts that delegate context loading to a separate init step. - Just-in-time retrieval counts. A prompt that names what exists and hands the agent a `read_context` or `list_*` tool to fetch the rest passes the resources section; the fail is data that is neither present nor discoverable. ## Fix ```tsx // before const messages = [{ role: "system", content: "You are a helpful assistant." }, ...userMessages]; // after: inject Available Data, What You Can Do, Recent Context const ctx = await loadProjectContext(session.userId); const messages = [ { role: "system", content: `You are an assistant.\n\n## Available Data\n${ctx.resources}\n\n## Capabilities\n${ctx.capabilities}\n\n## Recent Context\n${ctx.recent}` }, ...userMessages, ]; ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent config | fix-this-sprint | ## Examples **Anti-pattern (fails):** ```tsx const messages = [{ role: "system", content: "You are a helpful assistant." }]; ``` **Applied (passes):** ```tsx const ctx = await loadProjectContext(userId); const messages = [{ role: "system", content: `You assist with code.\n\n${ctx.format()}` }]; ``` ## Suppression ```tsx // ax-audit-ignore:context-starvation, bootstrapping prompt, context injected by middleware const basePrompt = "You are a helpful assistant."; ``` -
granularity-static-api-mapping.md 3.9 KB
--- title: One tool per API endpoint instead of dynamic discovery slug: granularity-static-api-mapping category: granularity defaultTier: backlog surfaces: agent-tool-execution agent-native-principle: Granularity detection: observational related: granularity-workflow-shaped-tool --- ## One tool per API endpoint instead of dynamic discovery 50 tools for 50 API endpoints. Adding an endpoint requires a code change and redeploy. The agent can only access what was anticipated at build time. For evolving APIs, a discover-and-access pattern keeps capabilities in sync automatically. ## What goes wrong CMS has 30 content types, 90 tools total. An editor adds "Press Release" in the CMS admin. The agent can't access it: no `read_press_release` tool exists yet. ## Detection **Surfaces:** agent-tool-execution **Static signals:** 1. Count tool definitions. High counts (>20) with the *same* parameter shape (read_X, read_Y, read_Z over one API) suggest static mapping. 2. Check whether the data source is an evolving type system (CMS, CRM custom objects) that supports dynamic type discovery. **Concrete commands:** ```bash rg 'name:\s*["\x27]' --type=ts src/tools/ -c | awk -F: '{sum+=$2} END {print "Total tools:", sum}' rg 'name:\s*["\x27](read|get|list|create|update|delete)_' --type=ts -o --no-filename src/tools/ | awk -F'_' '{print $1}' | sort | uniq -c | sort -rn ``` **Judgment signals:** - A product agent whose tools are the product nouns (`create_issue`, `update_document`, `list_requests`) passes even above 20 tools: the params differ, and the nouns are the product. - Fail when the tools are mechanical wrappers over one shape and a new type in the source system would need a new tool and a redeploy. - Overlap is a `warn` even under the threshold: two wrappers a human could not choose between are a coin flip for the agent too, and bloated, ambiguous tool sets are the failure Anthropic's context-engineering guidance names first. - An MCP client that hardcodes its tool list and ignores `notifications/tools/list_changed` is static mapping over a source that already offers discovery. **False-positive guards:** - Skip tools with genuinely different params, small stable APIs (<10 types), and `// ax-audit-ignore:granularity-static-api-mapping`. ## Fix Replace static tools with discover + access. ```ts // before: read_blog_post, read_landing_page ... 30 identical tools // after: two tools cover the entire surface export const listContentTypes = tool({ name: "list_content_types", execute: async () => api.get("/content/types"), }); export const readContent = tool({ name: "read_content", parameters: { type: { type: "string" }, id: { type: "string" } }, execute: async ({ type, id }) => api.get(`/content/${type}/${id}`), }); ``` ## Default tier and overrides **Defaults to:** `backlog`: scaling problem, not correctness. Works fine for small, stable APIs. | Surface | Tier | |---|---| | Agent tool execution | backlog | The row exists to block the generic tool-execution bump. Static mapping is a flexibility ceiling, not an unsafe action, so it stays `backlog` on the surface where every other rule rises a tier. ## Examples **Anti-pattern (fails):** ```ts export const readContact = tool({ name: "read_contact", execute: ({ id }) => api.get(`/crm/contact/${id}`) }); export const readDeal = tool({ name: "read_deal", execute: ({ id }) => api.get(`/crm/deal/${id}`) }); // ... 48 more: new custom "Partner" object added in CRM, agent can't access it ``` **Applied (passes):** ```ts // Two tools: discover + access. New "Partner" type works immediately. export const listObjectTypes = tool({ name: "list_crm_object_types", execute: () => api.get("/crm/objects") }); export const readObject = tool({ name: "read_crm_object", execute: ({ objectType, id }) => api.get(`/crm/${objectType}/${id}`) }); ``` ## Suppression ```ts // ax-audit-ignore:granularity-static-api-mapping, stable API with <10 types export const readUser = tool({ name: "read_user", /* ... */ }); ``` -
granularity-workflow-shaped-tool.md 3.6 KB
--- title: Tool bundles decision logic instead of being atomic slug: granularity-workflow-shaped-tool category: granularity defaultTier: fix-this-sprint surfaces: agent-config agent-native-principle: Granularity detection: hybrid related: granularity-static-api-mapping --- ## Tool bundles decision logic instead of being atomic A tool like `analyze_and_organize(folder)` bundles judgment into code. Changing what "organize" means needs a code refactor, not a prompt edit, and the agent can't apply judgment to intermediate steps. ## What goes wrong `analyze_and_organize_inbox` scans emails, decides importance, files them. User asks "Why did you archive that?" The agent can't change the logic: it's hardcoded. With atomic primitives, the agent decides itself. ## Detection **Surfaces:** agent-config **Static signals:** 1. Grep tool definitions for compound names (`_and_`, `_then_`, `_with_`). 2. Check implementations for branching logic making domain decisions. 3. Count distinct API calls per tool; >1 suggests bundling. **Concrete commands:** ```bash rg 'name:\s*["\x27]\w+_(and|then|with)_\w+' --type=ts src/tools/ rg 'name:\s*["\x27](process|handle|manage|analyze|organize|auto)_' --type=ts src/tools/ rg -l 'tool\(|defineTool' --type=ts src/tools/ | while read f; do rg -c --with-filename 'if\s*\(|switch\s*\(' "$f" done | awk -F: '$2>3' ``` **Judgment signals:** - Consolidation that removes mechanical chaining passes. A `schedule_event` that finds availability and books is one user action with no step the user would want to veto, and fewer tools of that kind is the guidance Anthropic gives for tool sets. The fail is a bundled decision (which role to assign, what counts as stale): that is the step the user will disagree with and the agent cannot change. Test: is there an intermediate step a user would want to see or override? **False-positive guards:** - Skip atomic transactions (e.g., `transfer_funds`) and `// ax-audit-ignore:granularity-workflow-shaped-tool`. ## Fix Split into atomic primitives. Let the agent decide what to move and where. ```ts // before: analyze_and_organize_inbox: after: atomic primitives export const listEmails = tool({ name: "list_emails", /* ... */ }); export const readEmail = tool({ name: "read_email", /* ... */ }); export const moveEmail = tool({ name: "move_email", /* ... */ }); ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint`: works until the user disagrees with a bundled decision. | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent config | fix-this-sprint | No tool-execution bump: a bundled tool still does what it says, it just does too much of it. The blocker in that neighbourhood is an ungated action (`comm-no-approval-gate`), not a coarse one. ## Examples **Anti-pattern (fails):** ```ts export const processNewUser = tool({ name: "process_and_configure_new_user", execute: async ({ email, name }) => { const user = await api.post("/users", { email, name }); await api.post(`/users/${user.id}/roles`, { role: "member" }); // can't choose role await api.post("/emails/send", { to: email, template: "welcome" }); // can't skip }, }); ``` **Applied (passes):** ```ts export const createUser = tool({ name: "create_user", /* ... */ }); export const assignRole = tool({ name: "assign_role", /* ... */ }); export const sendEmail = tool({ name: "send_email", /* ... */ }); // Agent decides: skip welcome email, assign admin role ``` ## Suppression ```ts // ax-audit-ignore:granularity-workflow-shaped-tool, atomic transaction export const transferFunds = tool({ name: "transfer_funds", /* ... */ }); ``` -
parity-crud-incomplete.md 2.7 KB
--- title: Entity with incomplete CRUD tool coverage slug: parity-crud-incomplete category: parity defaultTier: release-blocker surfaces: agent-dashboard agent-native-principle: Parity detection: code-auditable related: parity-no-tool-parity --- ## Entity with incomplete CRUD tool coverage Entity has create and read tools but no update or delete: the agent creates a note but can't fix a typo, lists tasks but can't mark one complete. ## What goes wrong Agent creates a note. User spots a typo. No `update_note` tool exists. Agent says "I can't edit it." The agent generated work instead of completing it. ## Detection **Surfaces:** agent-dashboard **Static signals:** 1. List all entity types from tool definitions. 2. For each entity, verify create/read/update/delete tools exist. 3. Flag entities with <4 CRUD operations. **Concrete commands:** ```bash rg 'name:\s*["\x27](create|get|list|update|edit|delete|remove)_(\w+)' \ --type=ts -o --no-filename src/tools/ | awk -F'_' '{print $2}' | sort | uniq -c | sort -n rg 'name:\s*["\x27]create_' --type=ts -o --no-filename src/tools/ | \ sed 's/.*create_//' | sed 's/["\x27]//' | while read e; do rg -q "update_${e}|edit_${e}" src/tools/ || echo "MISSING update: $e"; done ``` **False-positive guards:** - Skip immutable entities (audit logs, event streams) and `// ax-audit-ignore:parity-crud-incomplete`. ## Fix Add the missing CRUD tools. Every entity needs all four. ```ts // before: [createNote, listNotes]: after: add update and delete export const updateNote = tool({ name: "update_note", execute: async ({ noteId, ...fields }) => api.patch(`/notes/${noteId}`, fields), }); export const deleteNote = tool({ name: "delete_note", execute: async ({ noteId }) => api.delete(`/notes/${noteId}`), }); ``` ## Default tier and overrides **Defaults to:** `release-blocker`: incomplete CRUD strands agents mid-workflow. | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent config | release-blocker | | Agent dashboard | release-blocker | Dashboard does not taper. A run the user can start and watch but not cancel or retry is the same dead end as a note the agent cannot edit. ## Examples **Anti-pattern (fails):** ```ts export const createTask = tool({ name: "create_task", /* ... */ }); export const listTasks = tool({ name: "list_tasks", /* ... */ }); // No update_task, no delete_task: agent can't mark tasks complete ``` **Applied (passes):** ```ts // All four CRUD operations present export const tools = [createTask, listTasks, getTask, updateTask, deleteTask]; ``` ## Suppression ```ts // ax-audit-ignore:parity-crud-incomplete, audit_log is intentionally immutable export const listAuditLogs = tool({ name: "list_audit_log", /* ... */ }); ``` -
parity-no-tool-parity.md 3.3 KB
--- title: UI action with no agent tool equivalent slug: parity-no-tool-parity category: parity defaultTier: release-blocker surfaces: agent-config agent-native-principle: Parity detection: code-auditable related: parity-crud-incomplete, parity-orphan-ui-action --- ## UI action with no agent tool equivalent A route or UI handler performs an operation no available tool exposes. User asks the agent to do it; it says "I can't do that." Parity means the agent can do everything the user can do. Scope: codebase-wide. This rule enumerates every mutating handler that ships today against the whole tool registry, so it fires on gaps nobody introduced in this PR. Run it in full-sweep mode or when auditing a config surface. For gaps the diff under review adds, use `parity-orphan-ui-action`, which is diff-scoped and tiered lower because new drift has an owner in the room. ## What goes wrong UI has an "Archive" button calling `POST /api/projects/:id/archive`, but no tool exposes the endpoint. The agent responds "I don't have the ability to archive projects." ## Detection **Surfaces:** agent-config **Static signals:** 1. Enumerate every mutating handler in the tree, not only changed files: `POST`/`PUT`/`PATCH`/`DELETE` route exports, server actions, form actions. 2. Enumerate the full tool registry. 3. Match one list against the other by operation, not by name; `archive_project` and `POST /projects/:id/archive` are the same capability under different spellings. 4. Report the unmatched handlers as a list. An empty grep is not a pass here: cite the handler count you compared. **Concrete commands:** ```bash rg -l 'export (async )?function (POST|PUT|PATCH|DELETE)' --type=ts src/app/api/ | sort # handler inventory rg -o 'name:\s*["\x27]\w+' --no-filename --type=ts src/tools/ | sed 's/.*["\x27]//' | sort # tool inventory rg -c 'export (async )?function (POST|PUT|PATCH|DELETE)' --type=ts src/app/api/ | wc -l # count to cite ``` **False-positive guards:** - Skip health-check endpoints (`/api/health`), webhook receivers, test files. - Skip files with `// ax-audit-ignore:parity-no-tool-parity`. ## Fix For every UI capability, ensure an equivalent tool exists. ```ts // before: route exists, no tool // POST /api/projects/[id]/archive exists; no archive_project tool // after: tool mirrors the UI action export const archiveProject = tool({ name: "archive_project", description: "Archive a project by ID.", parameters: { projectId: { type: "string", required: true } }, execute: async ({ projectId }) => api.post(`/projects/${projectId}/archive`), }); ``` ## Default tier and overrides **Defaults to:** `release-blocker`: a missing tool is a hard wall the agent cannot work around. | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent config | release-blocker | Config does not taper: a GUI-only setting is exactly the capability gap this rule exists to catch. ## Examples **Anti-pattern (fails):** ```ts // Route handler exists, tools array has no archive tool export const tools = [createProject, listProjects, getProject]; ``` **Applied (passes):** ```ts export const tools = [createProject, listProjects, getProject, archiveProject]; ``` ## Suppression ```ts // ax-audit-ignore:parity-no-tool-parity, internal admin endpoint export async function POST(req: Request) { ... } ``` -
parity-orphan-ui-action.md 3 KB
--- title: New UI capability without corresponding tool slug: parity-orphan-ui-action category: parity defaultTier: fix-this-sprint surfaces: agent-config, agent-tool-execution agent-native-principle: Parity detection: code-auditable related: parity-no-tool-parity, parity-crud-incomplete --- ## New UI capability without corresponding tool A PR adds a new UI feature (button, page, form action) but no new tool. Each PR without tool parity widens the gap between what users and agents can do. Scope: diff only. Every finding here names a capability introduced by the change under review, which is why it tiers lower than `parity-no-tool-parity`: the author is still in the room and the tool is a few lines away. A gap the diff did not introduce is not this rule's finding, even when the grep hits it. ## What goes wrong PR adds a "Duplicate project" button calling a new endpoint, but no tool. It merges. Months later a user asks the agent to duplicate a project. It can't. ## Detection **Surfaces:** agent-config, agent-tool-execution **Static signals:** 1. In the diff, find new onClick handlers, form actions, route handlers. 2. Cross-reference with new tool definitions in the same diff. 3. Flag new UI capabilities with no new tool. **Concrete commands:** ```bash git diff main --name-only -- '*.ts' '*.tsx' | while read f; do rg -l 'export (async )?function (POST|PUT|PATCH|DELETE)' "$f" done 2>/dev/null git diff main -U0 -- '*.tsx' | rg '^\+.*onClick' git diff main -U0 -- '*.ts' | rg '^\+.*(tool\(|defineTool|createTool)' ``` **False-positive guards:** - Skip cosmetic UI changes with no new backend call and `// ax-audit-ignore:parity-orphan-ui-action`. ## Fix When adding a UI capability, add the corresponding tool in the same PR. ```ts // before: POST /api/projects/[id]/duplicate added, no tool // after: tool ships in the same PR export const duplicateProject = tool({ name: "duplicate_project", execute: async ({ projectId }) => api.post(`/projects/${projectId}/duplicate`), }); ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint`: orphans are drift, not crisis. Cumulative effect degrades agent usefulness. | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent config | fix-this-sprint | No tool-execution bump: one new orphan is a gap the author can close next sprint, not a shipped hard wall. Promoting it to blocker on every PR is how teams learn to ignore ❌ verdicts. ## Examples **Anti-pattern (fails):** ```tsx <button onClick={() => fetch(`/api/reports/${id}/export`, { method: "POST" })}> Export CSV </button> // No export_report tool in this PR ``` **Applied (passes):** ```ts // Same PR adds the button AND the tool export const exportReport = tool({ name: "export_report", parameters: { reportId: { type: "string", required: true } }, execute: async ({ reportId }) => api.post(`/reports/${reportId}/export`), }); ``` ## Suppression ```tsx {/* ax-audit-ignore:parity-orphan-ui-action, cosmetic preview, no agent use case */} <button onClick={handlePreview}>Preview</button> ``` -
parity-unstructured-tool-output.md 6 KB
--- title: Tool returns prose instead of a structured, honest result slug: parity-unstructured-tool-output category: parity defaultTier: fix-this-sprint surfaces: agent-tool-execution agent-native-principle: The tool surface is a protocol, legible and honest about its state detection: code-auditable related: comm-no-completion-signal, parity-no-tool-parity, trust-no-uncertainty-markers --- ## Tool returns prose instead of a structured, honest result A tool handler returns a human sentence and swallows its failures into that same sentence. The agent reading it has to parse English to work out whether the call succeeded, and there is nothing to parse when the handler answers `"Something went wrong"` with a 200. Every capability above this tool is now guessing, and the guess is silent. When the product's primary caller is another agent, the return value is the interface. Prose is a rendering; a result object is a contract. ## What goes wrong `send_invoice` fails validation upstream. The handler catches, logs, and returns `"Could not send the invoice, please try again."` with HTTP 200. The orchestrator sees a successful tool result, marks the step done, and moves to `mark_invoice_paid`. The user is told the invoice went out. It did not, and no error ever surfaced, because the only signal that it failed was a sentence nobody typed a parser for. The same failure in the other direction: a handler returns a rendered HTML table of search results. The agent can quote it back but cannot filter, count, or select a row, so the next tool call is invented from a string. ## Detection **Surfaces:** agent-tool-execution **Static signals:** 1. List the tool handler files: `rg -l 'tool|handler|execute' --type=ts src/`, then narrow to files registering tools. 2. For each handler, read the return statements on the error path. Flag a bare string or template literal. 3. Flag `catch` blocks that return a value at all rather than rethrowing or returning a typed failure. 4. Flag responses that carry an error message with a 2xx status. 5. Compare the success return against a declared output type. No type or schema on the tool definition is itself the finding. 6. For MCP handlers, check the failure branch sets `isError: true`. The schema assumes false when unset, so a `catch` that returns `{ content: [{ type: "text", text: "Something went wrong" }] }` is a success to every client. On the success side, `structuredContent` under an `outputSchema` is the typed half; text-only `content` for a data-returning tool is the finding. **Concrete commands:** ```bash # error-path prose: a catch that returns a string literal or template literal rg -n -U 'catch\s*\([^)]*\)\s*\{[^}]*return\s+[`"'\'']' --type=ts src/ # a 2xx carrying an error payload rg -n 'status:\s*200' -A 3 --type=ts src/ | rg -i 'error|failed|could not' # tool definitions with no output schema alongside the input schema rg -n 'inputSchema|parameters:' --type=ts src/ -A 6 | rg -v 'outputSchema|returns' # MCP handlers: list the catch blocks that build a result, then check each for isError rg -n -U 'catch\s*\([^)]*\)\s*\{[^}]*content:\s*\[' --type=ts src/ rg -c 'isError:\s*true' --type=ts src/ ``` **False-positive guards:** - Skip handlers whose return type is a discriminated union (`{ ok: false, code, message }` counts as structured, message included). - A prose `message` field alongside a machine-readable `code` passes. The finding is prose *instead of*, not prose *in addition to*. An MCP result with `isError: true` and prose `content` passes the same way: the flag is the machine-readable half, and the text is meant to be feedback the model can retry on. - Skip files with `// ax-audit-ignore:parity-unstructured-tool-output` near the match. - Skip test fixtures, mocks, and Storybook files. - Skip tools whose whole job is text generation (a `draft_reply` returning a draft string is the payload, not a status). ## Fix **Concrete change:** return a discriminated result and let the status line up with it. ```ts // before: the caller cannot tell these two apart export async function sendInvoice(id: string) { try { await billing.send(id); return "Invoice sent."; } catch (e) { logger.error(e); return "Something went wrong."; } } // after: one shape, both branches machine-readable type ToolResult<T> = | { ok: true; data: T } | { ok: false; code: string; message: string; retryable: boolean }; export async function sendInvoice(id: string): Promise<ToolResult<{ sentAt: string }>> { try { const { sentAt } = await billing.send(id); return { ok: true, data: { sentAt } }; } catch (e) { logger.error(e); return { ok: false, code: e instanceof ValidationError ? "invoice_invalid" : "billing_unavailable", message: e.message, retryable: !(e instanceof ValidationError), }; } } ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | fix-this-sprint | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | A tool that reports failure as success is a correctness bug on the execution path, not an ergonomics one, which is why this is the rare parity rule that blocks only on the surface where the tool actually runs. ## Examples **Anti-pattern (fails):** ```ts server.tool("delete_record", async ({ id }) => { const res = await db.delete(id); return res.count > 0 ? `Deleted record ${id}.` : `No record matched ${id}.`; }); ``` The agent cannot distinguish "deleted" from "nothing matched" without string matching, so a retry loop either never fires or never stops. **Applied (passes):** ```ts server.tool("delete_record", async ({ id }) => { const res = await db.delete(id); return res.count > 0 ? { ok: true, data: { id, deleted: res.count } } : { ok: false, code: "not_found", message: `No record matched ${id}.`, retryable: false }; }); ``` ## Suppression ```ts // ax-audit-ignore:parity-unstructured-tool-output, the draft is the payload, not a status server.tool("draft_reply", async ({ threadId }) => generateDraft(threadId)); ``` -
_sections.md 5.2 KB
# Sections: Agent-Native Architecture (Layer 1) The 4 categories of agent-native architecture (Layer 1) audit rules. Each rule file uses one category prefix. --- ## 1. Parity (parity) **Default tier:** mostly release-blocker **Why critical:** If the agent can't do what the user can do, it's a second-class citizen. Gaps surface as "why can't the agent do X?" with no workaround. Missing CRUD operations strand agents mid-workflow. Parity also runs outward: a tool that answers failure with a sentence gives the caller above it nothing to act on, so the caller guesses. ## 2. Granularity (granularity) **Default tier:** mostly fix-this-sprint **Why critical:** Tools that bundle decision logic force the agent to accept or reject a whole workflow; atomic primitives let it apply judgment at each step. If behavior changes need code refactoring instead of prompt edits, granularity is too low. The axis is judgment, not step count: a tool that chains mechanical steps into one user action is fine, and fewer of those beats many thin wrappers. ## 3. Context (context) **Default tier:** mostly fix-this-sprint **Why critical:** An agent that doesn't know what exists or what the user has done asks redundant questions, misses relevant data, and feels unintelligent. Context starvation is the most common reason an agent underperforms despite capable tools. ## 4. Communication (comm) **Default tier:** release-blocker across all three (completion, progress, approval gates) **Why critical:** Silent agents feel broken. Heuristic completion detection creates race conditions. Missing progress indicators make users kill and restart tasks. An orchestrator with no gate on its execution path auto-executes whatever destructive tool is registered next, which is why the approval-gate rule blocks a release here rather than waiting a sprint. --- ## Rule index ``` parity-no-tool-parity parity-crud-incomplete parity-orphan-ui-action parity-unstructured-tool-output granularity-workflow-shaped-tool granularity-static-api-mapping context-starvation context-no-injection context-no-checkpoint-resume comm-no-completion-signal comm-no-progress-visibility comm-no-approval-gate ``` Total: 12 rules. --- ## Cross-rule interactions Within this layer, these pairs sit close enough to double-report. Pick one: - **no-tool-parity + orphan-ui-action**: same defect, different scope. `parity-no-tool-parity` sweeps the shipped codebase; `parity-orphan-ui-action` looks only at the diff. A capability the PR introduced is the orphan finding; anything older is the parity finding, never both. - **crud-incomplete + no-tool-parity**: `parity-crud-incomplete` is entity-level (this noun is missing an operation), `parity-no-tool-parity` is handler-level (this endpoint has no tool). A missing `update_note` matches both: report the CRUD finding, which names the operation. - **workflow-shaped-tool + static-api-mapping**: opposite ends of one axis. Too coarse (one tool making domain decisions) versus too many too-thin tools (one per endpoint, no discovery). They should not fire on the same tool; if they do, re-read the tool. - **context-starvation + context-no-injection**: one static prompt trips both greps. Report `context-starvation` for the missing what-exists block, and `context-no-injection` only when cross-session state is separately absent. - **no-completion-signal + no-progress-visibility**: both audit the orchestrator's event contract, at the end of the run and during it. A loop that emits nothing fails both; fix the emission once and re-run. - **unstructured-tool-output + no-completion-signal**: a tool that reports failure as prose with a success status defeats an honest terminal reason downstream. Fix the tool's return shape first; the orchestrator rule is unfixable while its inputs lie. Across layers, `rules-arch` owns the code path and `rules-ax` owns what the user sees on it: - **comm-no-approval-gate (arch) + control-no-approval-gate (ax) + control-thin-approval-payload (ax)**: three steps of one chain, not three reports of one defect. The gate exists on the execution path, the treatment matches the stakes, the prompt carries enough to decide. Fix in that order; without a checkpoint there is nowhere to put the ax fixes, and a gate the user cannot read is the last of the three to matter. - **comm-no-approval-gate (arch) + comm-unrequested-action-no-consent (ax)**: the arch rule audits the gate on the path a user started. Scheduled, webhook, and queue entry points reach the same executor with nobody to prompt, so a gate can pass the arch rule and still leave the unattended path open. File both when both hold, each naming its own entry point. - **parity-unstructured-tool-output (arch) + trust-no-uncertainty-markers (ax)**: honesty at two altitudes. The tool's result shape versus the agent's hedging in prose. An agent cannot hedge accurately about a tool that reports every outcome as success. - **comm-no-progress-visibility (arch) + comm-no-progress-signal (ax)**: emission versus presentation. If the server returns one final payload, only the arch rule fires and no component change can resolve it. If events stream and the UI ignores them, only the ax rule fires. -
_template.md 2.6 KB
--- title: <Rule title, short, descriptive> slug: <category>-<kebab-slug> category: parity | granularity | context | comm defaultTier: release-blocker | fix-this-sprint | backlog surfaces: <playbooks that name this rule: agent-chat, agent-tool-execution, agent-config, agent-dashboard> agent-native-principle: <which agent-native principle this enforces> detection: code-auditable | hybrid | observational related: <comma-separated other rule slugs (arch or ax); cross-layer pairs list each other in both files> --- ## <Rule title> One paragraph: the architectural failure mode in plain language, why it breaks agents, what principle it violates. ## What goes wrong A concrete, observable scenario: what the user or agent experiences, what the code does, why they diverge. ## Detection **Surfaces:** <which playbooks invoke this: agent-chat, agent-tool-execution, agent-config, agent-dashboard> **Static signals:** 1. Concrete grep / Read step. Use `rg` / `find` / file-extension filters. 2. Each step produces evidence: a file path, a line number, a presence/absence boolean, a count. 3. Last step compares evidence to a threshold. **Concrete commands:** ```bash # Inline grep recipes the agent can run. Note: ripgrep has no 'tsx' type: '--type=ts' covers *.ts and *.tsx. rg 'pattern' --type=ts src/ ``` **False-positive guards:** - Skip files that already have the expected pattern. - Skip files with `// ax-audit-ignore:<this-slug>` near the match. - Skip test and Storybook fixtures. ## Fix **Concrete change** with the architectural pattern: ```tsx // before: the anti-pattern // after: the corrected pattern ``` ## Default tier and overrides **Defaults to:** `<tier>` | Surface | Tier | |---|---| | Agent tool execution | <usually one tier higher> | | Agent chat | <same or one tier lower> | | Agent config | <same> | | Agent dashboard | <usually one tier lower> | Cover every surface listed in `surfaces:`. A missing row hands that surface to the generic bump in `references/ship-readiness.md`, so the table is required even when every row just repeats `defaultTier`: the repetition is what suppresses the bump. When a row does not follow the shape above, say why in one line under the table (shared orchestrator code does not taper by surface; a flexibility ceiling does not rise on tool execution). ## Examples **Anti-pattern (fails):** ```tsx // Real-world example showing the bug. ``` **Applied (passes):** ```tsx // Same component with the fix applied. ``` ## Suppression Ignore this rule on a specific component: ```tsx {/* ax-audit-ignore:<slug>, reason */} <Component /> ```
-
-
rules-ax
-
comm-no-generative-momentum.md 2.6 KB
--- title: Blank-canvas surface with no agent-generated starting content slug: comm-no-generative-momentum category: comm defaultTier: backlog surfaces: agent-chat ax-pattern: Generative Momentum detection: observational related: control-over-conversational, context-under-contextual --- ## Blank-canvas surface with no agent-generated starting content User opens a new document, email, or report: empty canvas, blinking cursor, agent available but silent. A half-written draft is easier to shape than an empty page, but the agent doesn't offer one. ## What goes wrong User clicks "New marketing email" in a tool that has their brand voice and audience data. Blank editor, agent idle. A contextual draft would have them editing in ten seconds. ## Detection **Surfaces:** agent-chat **Auditability:** observational **Judgment signals:** - Find creation surfaces (new/create routes, empty editors, blank composition areas). - Check whether agent-generated content or templates are offered on first load. - Flag blank-canvas surfaces with no generative starting point where the agent has enough context. **Concrete commands:** ```bash rg '(/new|/create|/compose|/draft)' --type=ts -l src/ rg '(EmptyState|BlankCanvas|emptyDocument|initialContent:\s*["'"'"']{2})' --type=ts -l src/ rg '(generateDraft|suggestDraft|aiDraft|startWithAI|TemplatePicker)' --type=ts -l src/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:comm-no-generative-momentum`. - Skip test/Storybook fixtures and code editors where blank is the expected state. ## Fix Offer an agent-generated draft on blank-canvas surfaces: "Start with AI draft" button, template suggestions, or outline. Always let the user dismiss and start from scratch. ## Default tier and overrides **Defaults to:** `backlog` | Surface | Tier | |---|---| | Agent chat | backlog | | Agent config | backlog | ## Examples **Anti-pattern (fails):** ```tsx export function NewReport() { // Agent has project data, metrics, goals: offers nothing return <RichTextEditor initialContent="" />; } ``` **Applied (passes):** ```tsx export function NewReport() { const project = useProject(); const { suggestion, dismiss } = useAgentSuggestion({ prompt: `Draft a report outline for ${project.name}`, }); return ( <div> {suggestion && ( <Banner onAccept={() => editor.setContent(suggestion)} onDismiss={dismiss}> Start with AI outline? </Banner>)} <RichTextEditor ref={editor} /> </div> ); } ``` ## Suppression ```tsx {/* ax-audit-ignore:comm-no-generative-momentum, code editor, blank canvas is intentional */} <CodeEditor /> ``` -
comm-no-intent-handshake.md 3.5 KB
--- title: Agent acts on non-trivial request without confirming intent slug: comm-no-intent-handshake category: comm defaultTier: fix-this-sprint surfaces: agent-chat, agent-tool-execution ax-pattern: Intent Handshake detection: hybrid related: control-no-escape-hatch, control-no-approval-gate --- ## Agent acts on non-trivial request without confirming intent User says "reorganize my files." The agent immediately moves files, but the user meant "suggest a new folder structure," not "execute a restructure right now." Intent Handshake requires agents to play back their interpretation before executing. The intent/interpretation gap stays invisible until the damage is done. ## What goes wrong User asks "clean up my project." The agent deletes unused files, renames directories, and updates imports in one shot. The user wanted a report. No playback, no scoping choices, no "here's what I'll do" first. Destructive, ambiguous requests get instant-execute treatment. ## Detection **Surfaces:** agent-chat, agent-tool-execution **Auditability:** hybrid **Static signals:** 1. Find agent action triggers for non-trivial operations (multi-step, destructive, ambiguous). 2. Check for a confirmation/playback step between request and execution. 3. Flag direct execution of complex requests with no preview. **Concrete commands:** ```bash rg '(executeTool|runAction|performAction|handleToolCall)' --type=ts -l src/ rg '(delete|remove|move|rename|reorganize|migrate|deploy|publish)' --type=ts src/tools/ src/actions/ rg '(confirm|approval|preview|playback|requireApproval)' --type=ts src/ rg '(autoExecute|skipConfirm|auto_approve)' --type=ts src/ rg '(AskUserQuestion|elicitation/create|permissionMode.*plan)' --type=ts src/ ``` **Judgment signals:** - Trivial, unambiguous requests ("what time is it?") don't need a handshake. - Targets multi-step, destructive, ambiguous, or high-stakes requests. - Framework primitives count when the agent can actually reach them: Claude Agent SDK `AskUserQuestion` or `plan` mode, MCP `elicitation/create`, an AI SDK approval carrying a `reason`. A system prompt that says "confirm before destructive actions" with no such primitive wired is a hope, not a handshake. **False-positive guards:** - Skip files with `// ax-audit-ignore:comm-no-intent-handshake`. - Skip test and Storybook fixtures. - Skip read-only operations (queries, lookups, status checks). ## Fix Before executing non-trivial actions, play back understanding: "I'll reorganize your files by moving X to Y. Proceed?" Options: text playback, structured plan preview, or scoping choices. ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | fix-this-sprint | ## Examples **Anti-pattern (fails):** ```tsx async function onToolCall(tool: string, args: Record<string, unknown>) { const result = await tools[tool].execute(args); // no confirmation, even for destructive ops return { role: "tool", content: result }; } ``` **Applied (passes):** ```tsx async function onToolCall(tool: string, args: Record<string, unknown>) { const meta = tools[tool].metadata; if (meta.destructive || meta.multiStep) return { type: "pending_approval", message: `I'll ${meta.describe(args)}. Proceed?`, onApprove: () => tools[tool].execute(args) }; return tools[tool].execute(args); } ``` ## Suppression ```tsx {/* ax-audit-ignore:comm-no-intent-handshake, read-only lookup, no side effects */} <QuickSearchAgent /> ``` -
comm-no-progress-signal.md 3.5 KB
--- title: Multi-step agent task shows no progress slug: comm-no-progress-signal category: comm defaultTier: release-blocker surfaces: agent-chat ax-pattern: Confidence Cues (progress dimension) detection: code-auditable related: comm-no-intent-handshake, context-no-adaptive-canvas, comm-no-progress-visibility --- ## Multi-step agent task shows no progress Agent runs a task that takes 30+ seconds. The UI shows nothing: no streaming, no step counter, no thinking indicator. User doesn't know if it's working, stuck, or crashed. Silent agents feel broken. Scope: this rule audits what the user sees. If the server never emitted progress events in the first place, the finding belongs to `rules-arch/comm-no-progress-visibility`, and fixing the component cannot resolve it. ## What goes wrong User asks the agent to analyze a dataset. Three tool calls, API waits, synthesis, 45 seconds. The user sees a spinner or nothing. At 15 seconds they wonder if it's broken. At 30 they refresh. ## Detection **Surfaces:** agent-chat **Auditability:** code-auditable **Static signals:** 1. Find the components that call the agent (chat submit handlers, action panel triggers). 2. Check whether they subscribe to progress (`onChunk`, `onToken`, `onProgress`, `onStatus`, `onData`, `useChat`) and render what arrives, not just the final value. 3. Flag components that receive events but only render on completion: a handler that sets state no JSX reads is the same silence to the user. In AI SDK 7, a component that reads `status` only to disable the send button and renders `message.parts` only once `status === "ready"` is this finding; so is one that reads `message.parts` while the server writes status as `transient` parts, which arrive only through `onData`. **Concrete commands:** ```bash rg '(useChat|useCompletion|agent\.chat|agent\.run|streamText|generateText)' --type=ts -l src/components/ src/app/ rg '(onChunk|onToken|onProgress|onStatus|stream:\s*true)' --type=ts src/components/ src/app/ rg -n "status === ['\"]streaming|isStreaming|isLoading|onData" --type=ts src/components/ src/app/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:comm-no-progress-signal`. - Skip test and Storybook fixtures. - Skip agent calls that reliably complete in under 2 seconds. ## Fix Render each event as it arrives instead of waiting for the final value. Name the current step in the user's words, not the tool's: "Searching for X..." then "Found 3 results, analyzing..." beats three identical spinners. A generic "Thinking..." held for 45 seconds is still a frozen UI. ## Default tier and overrides **Defaults to:** `release-blocker` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | release-blocker | | Agent dashboard | fix-this-sprint | ## Examples **Anti-pattern (fails):** ```tsx async function onAsk(query: string) { const data = await fetch("/api/agent/research", { method: "POST", body: JSON.stringify({ query }), }).then((r) => r.json()); // 30-60s silence, no feedback setResult(data); } ``` **Applied (passes):** ```tsx export function ResearchPanel() { const [steps, setSteps] = useState<string[]>([]); const { data, isStreaming } = useAgentStream("/api/agent/research", { onStatus: (s) => setSteps((prev) => [...prev, s]), }); return <> {isStreaming && <ProgressList steps={steps} current={steps.at(-1)} />} {data && <Results data={data} />} </>; } ``` ## Suppression ```tsx {/* ax-audit-ignore:comm-no-progress-signal, instant lookup, sub-second response */} <QuickLookup /> ``` -
comm-unrequested-action-no-consent.md 6.5 KB
--- title: Agent acts unprompted with no standing consent and no notice slug: comm-unrequested-action-no-consent category: comm defaultTier: fix-this-sprint surfaces: agent-tool-execution, agent-dashboard ax-pattern: Proactive action needs a different consent shape than requested action detection: code-auditable related: control-no-approval-gate, comm-no-approval-gate, trust-no-escalation-path, comm-no-intent-handshake --- ## Agent acts unprompted with no standing consent and no notice Every gate in the product sits on the path a user starts. The agent also runs on a schedule, a webhook, and a queue, and those paths reach the same executor without passing a gate, because a background job has nobody to prompt. So the agent acts on the user's accounts while the user is asleep, and the record of it is a log row. An agent worth having does not wait to be asked. That is the point of the layer. But an action nobody requested cannot borrow its permission from a request, so it needs the two things a prompt would have given it: a boundary agreed in advance, and a notice afterwards the user can act on. ## What goes wrong A weekly cleanup job is scheduled to "tidy stale drafts". The definition of stale changes when the underlying model is updated, and one run archives forty documents a user was still working on. The user finds out on Monday. There was no approval prompt, correctly, because nobody was at the keyboard. There was also no standing rule bounding what the job could touch, and no notification when it touched more than usual, so the first signal was the damage. ## Detection **Surfaces:** agent-tool-execution, agent-dashboard **Auditability:** code-auditable **Static signals:** 1. Find the unprompted entry points: cron and scheduler registrations, queue consumers, webhook handlers, event subscribers, retry workers. 2. Trace each to the tool executor or agent runner. Entry points that never reach it are out of scope. 3. On each path that does reach it, look for a standing consent check: a policy object, an allowlist of tools valid without a user present, a budget or blast-radius limit. 4. Look for a notification emitted on the same path, addressed to the user, carrying what was done. 5. Fail when a path reaches the executor missing either control. The boundary and the notice are not alternatives: a policy with no notice leaves the user unable to find out, and a notice with no policy tells them only after the blast radius was already unbounded. One present and one absent is still a finding, named for the missing half. **Concrete commands:** ```bash # unprompted entry points rg -n -i 'cron|schedule\(|CronJob|@Cron|queue\.(process|consume)|webhook|onEvent|subscribe\(' --type=ts src/ -l # do they reach the executor? rg -n -i 'runAgent|executeTool|orchestrator|invokeTools' --type=ts src/ -l # a boundary on the unattended path rg -n -i 'autonomousPolicy|allowUnattended|standingConsent|withoutUser|maxActions|budget|dontAsk|allowedTools' --type=ts src/ # a notice the user can act on rg -n -i 'notify|sendNotification|createActivity|digest' --type=ts src/ -A 2 ``` **Judgment signals:** - A notification that only records success is a partial pass. The user needs to be able to reverse what they read about, so check that the notice links to an undo or a review surface. - A boundary expressed only as "the prompt tells it not to" is not a boundary. Look for a check in code on the execution path. - A headless Claude Agent SDK run with `permissionMode: "dontAsk"` and an explicit `allowedTools` list is a standing boundary: anything unlisted is denied rather than prompted. That satisfies the boundary half only; the notice still has to exist. **False-positive guards:** - Skip read-only background work: a nightly index or summary that writes nothing the user owns. - A job whose entire effect is to draft something for later review passes: the draft is the notice, and the review is the gate. - Skip files with `// ax-audit-ignore:comm-unrequested-action-no-consent` near the match. - Skip test harnesses, local seed scripts, and CI jobs. - Do not double-report with `control-no-approval-gate`. That rule governs the path a user started; this one governs the path nobody started. If a single executor serves both and neither is gated, file the approval-gate finding for the interactive path and this one for the unattended path, each with its own entry point in evidence. ## Fix Give the unattended path its own policy and make it report. ```ts // before: the same executor, no user, no boundary cron.schedule("0 3 * * 1", async () => { await runAgent({ goal: "tidy stale drafts", userId }); }); // after: a standing boundary agreed in advance, and a notice with a way back cron.schedule("0 3 * * 1", async () => { const policy = await getStandingConsent(userId, "weekly-tidy"); if (!policy) return; const result = await runAgent({ goal: "tidy stale drafts", userId, allowedTools: policy.allowedTools, maxAffected: policy.maxAffected, onLimitExceeded: "pause", }); await notify(userId, { title: `Tidied ${result.affected.length} drafts`, items: result.affected, undo: result.undoToken, }); }); ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | fix-this-sprint | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | The dashboard row does not taper below `fix-this-sprint` the way monitoring findings usually do. A dashboard is where an unattended run becomes visible at all, so a missing notice is the defect itself rather than a report of one. ## Examples **Anti-pattern (fails):** ```ts webhooks.on("invoice.overdue", async ({ customerId }) => { await runAgent({ goal: "chase the overdue invoice", customerId }); }); ``` An external event causes the agent to email a customer. No policy bounds it, and the user learns about it if the customer replies. **Applied (passes):** ```ts webhooks.on("invoice.overdue", async ({ customerId }) => { const policy = await getStandingConsent(ownerOf(customerId), "invoice-chase"); if (!policy?.allowsOutbound) return queueForReview(customerId); const result = await runAgent({ goal: "chase the overdue invoice", customerId, policy }); await notify(ownerOf(customerId), { title: "Chased an overdue invoice", items: result.sent, undo: result.undoToken }); }); ``` ## Suppression ```ts // ax-audit-ignore:comm-unrequested-action-no-consent, read-only nightly summary, writes nothing cron.schedule("0 2 * * *", buildUsageDigest); ``` -
context-memory-not-visible.md 3 KB
--- title: Agent uses context the user can't see or edit slug: context-memory-not-visible category: context defaultTier: fix-this-sprint surfaces: agent-chat, agent-config, agent-dashboard ax-pattern: Memory in Motion detection: code-auditable related: context-under-contextual, context-no-adaptive-canvas, trust-undisclosed-access-scope --- ## Agent uses context the user can't see or edit Agent injects preferences, past interactions, or learned patterns into its prompt, but the user can't see what the agent "knows" about them. Opaque memory feels invasive. Memory in Motion requires every piece of stored context to have a user-facing view and edit path. ## What goes wrong Agent says "Based on your preference for concise answers..." and the user thinks "What preference? I never said that." The system built a profile from past interactions and injected it into the system prompt with zero user visibility: no settings page, no memory panel, no way to correct it. ## Detection **Surfaces:** agent-chat, agent-config, agent-dashboard **Auditability:** code-auditable **Static signals:** 1. Find context injection points (prompt builders, context loaders, preference injectors). 2. Search for UI that exposes this context (settings pages, memory panels). 3. Flag injected context with no user-facing view or edit path. **Concrete commands:** ```bash rg '(systemPrompt|buildPrompt|contextLoader|injectContext|userPreferences|userMemory)' --type=ts -l src/ rg '(MemoryPanel|PreferencesView|WhatIKnow|MemorySettings)' --type=ts -l src/ rg '(savePreference|updateMemory|storePattern|learnFrom)' --type=ts -l src/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:context-memory-not-visible`. - Skip test and Storybook fixtures. - Skip internal admin-only agent tools where the operator is the developer. ## Fix For every context item injected into the prompt, provide UI to view and edit it: a "Memory" or "What I know about you" panel with edit/delete per item. ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent chat | fix-this-sprint | | Agent config | fix-this-sprint | | Agent dashboard | backlog | ## Examples **Anti-pattern (fails):** ```tsx async function getAgentContext(userId: string) { const prefs = await redis.get(`user:${userId}:prefs`); const history = await redis.get(`user:${userId}:patterns`); return { preferences: prefs, patterns: history }; // never shown to user } ``` **Applied (passes):** ```tsx // Context store is shared: same data feeds the agent AND the settings UI async function getAgentContext(userId: string) { return await getVisibleMemory(userId); // MemorySettings reads the same store } function MemorySettings() { const memory = useMemory(); return memory.items.map((m) => ( <li key={m.id}>{m.summary} <button onClick={() => memory.delete(m.id)}>Delete</button></li> )); } ``` ## Suppression ```tsx {/* ax-audit-ignore:context-memory-not-visible, internal dev tool, operator is the developer */} <AgentPromptBuilder /> ``` -
context-no-adaptive-canvas.md 3.2 KB
--- title: Interface static during agent task progression slug: context-no-adaptive-canvas category: context defaultTier: backlog surfaces: agent-config ax-pattern: Adaptive Canvas detection: code-auditable related: context-memory-not-visible, comm-no-progress-signal --- ## Interface static during agent task progression Agent moves through phases (researching, drafting, reviewing, complete) but the UI looks identical in each. No phase indicator, no layout change, no context-appropriate tools surfaced. Adaptive Canvas requires the interface to reshape around the agent's current activity. ## What goes wrong Agent starts a research task. User sees "Searching..." then nothing changes for 45 seconds. The agent transitions through phases but the layout never shifts: no stepper, no phase-specific controls. The user can't tell where the agent is or how close to done. ## Detection **Surfaces:** agent-config **Auditability:** code-auditable **Static signals:** 1. Find agent workflow state: phase, status, or stage enums/state machines. 2. Check whether rendering differs across phases (conditional rendering, different components per phase). 3. Flag workflows where UI is identical regardless of agent phase. 4. In chat surfaces, look for per-phase rendering of stream parts: a component chosen on `part.type` for `data-*` and tool parts, or MCP Apps `ui://` resources per tool. A chat that renders only `text` parts while `part.type` carries tool and step states has the phase information and drops it. **Concrete commands:** ```bash rg '(phase|stage|status|workflow).*(enum|type|const)' --type=ts src/ rg '(stateMachine|createMachine|useReducer|switch.*phase)' --type=ts src/ rg '(Stepper|ProgressBar|PhaseIndicator|StageIndicator)' --type=ts -l src/ rg -n "part\.type|case ['\"]data-|ui://" --type=ts src/components/ ``` **False-positive guards:** - Skip files with `// ax-audit-ignore:context-no-adaptive-canvas`. - Skip test and Storybook fixtures. - Skip single-step agent interactions where no multi-phase workflow exists. ## Fix Show a phase indicator (stepper, progress bar). Surface phase-appropriate tools (research tools during research, editing tools during review). Reshape the layout to match the current activity. ## Default tier and overrides **Defaults to:** `backlog` | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent dashboard | backlog | | Agent config | backlog | ## Examples **Anti-pattern (fails):** ```tsx function ResearchAgent({ status }: { status: string }) { // status is "searching" | "analyzing" | "complete": UI never changes return <div className="flex"><ChatPanel /><Sidebar /></div>; } ``` **Applied (passes):** ```tsx function ResearchAgent({ status, data }: { status: AgentStatus; data: AgentData }) { return ( <div> <Stepper steps={["Searching", "Analyzing", "Complete"]} current={status} /> {status === "searching" && <SearchProgress queries={data.queries} />} {status === "analyzing" && <AnalysisView sources={data.sources} />} {status === "complete" && <ResultsView results={data.results} />} </div> ); } ``` ## Suppression ```tsx {/* ax-audit-ignore:context-no-adaptive-canvas, single-turn chat, no multi-phase workflow */} <AgentChat /> ``` -
context-under-contextual.md 2.9 KB
--- title: Agent ignores available context it should use slug: context-under-contextual category: context defaultTier: backlog surfaces: agent-tool-execution ax-pattern: Under-contextual (anti-pattern) detection: hybrid related: context-memory-not-visible, context-starvation --- ## Agent ignores available context it should use The system has the user's project history, preferences, recent activity, and team context, but the agent's prompt includes none of it. The agent asks questions it should already know the answer to, wasting time and making it feel stupid. ## What goes wrong User opens a project page and asks "help me write a status update." Agent responds: "What project are you working on?" The project name, recent commits, and open tickets are all in app state, but the prompt ignores them. Every needless question erodes confidence. ## Detection **Surfaces:** agent-tool-execution **Auditability:** hybrid **Static signals:** 1. Catalog available context sources (user profile, project state, recent activity, team info). 2. Find agent prompt/context assembly functions. 3. Check whether available sources are referenced in context injection. 4. Flag significant context sources never passed to the agent. **Concrete commands:** ```bash rg '(useUser|useProject|useTeam|useActivity|currentProject|activeWorkspace)' --type=ts -l src/ rg '(buildPrompt|systemPrompt|assembleContext|getAgentContext)' --type=ts -l src/ rg -A 15 '(buildPrompt|assembleContext|getAgentContext)' --type=ts src/ ``` **Judgment signals:** - Would a human assistant in this position already know the answer? - Is the missing context high-signal (project name, recent activity) or low-signal? **False-positive guards:** - Skip files with `// ax-audit-ignore:context-under-contextual`. - Skip test/Storybook fixtures and generic agent surfaces with no page-specific context. ## Fix Inject relevant context at session start using the context.md pattern: "What I Know About This User," "What Exists," "Recent Activity." Update dynamically during the session. ## Default tier and overrides **Defaults to:** `backlog` | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent chat | backlog | ## Examples **Anti-pattern (fails):** ```tsx // User is on /projects/acme-redesign but agent gets no project context export function ProjectAgent() { const { sendMessage } = useAgent({ system: "You are a helpful assistant." }); return <AgentChat onSend={sendMessage} />; } ``` **Applied (passes):** ```tsx export function ProjectAgent() { const project = useProject(); const activity = useRecentActivity(project.id); const { sendMessage } = useAgent({ system: `Assistant for ${project.name}. Recent: ${activity.map((a) => a.summary).join("; ")}`, }); return <AgentChat onSend={sendMessage} />; } ``` ## Suppression ```tsx {/* ax-audit-ignore:context-under-contextual, generic help chat, no page context needed */} <HelpAgent /> ``` -
control-no-approval-gate.md 4.2 KB
--- title: Autonomous agent action without stakes-appropriate approval slug: control-no-approval-gate category: control defaultTier: release-blocker surfaces: agent-tool-execution ax-pattern: Escape Hatch (pre-execution dimension) detection: hybrid related: control-no-escape-hatch, trust-no-escalation-path, comm-no-approval-gate, control-thin-approval-payload --- ## Autonomous agent action without stakes-appropriate approval Agent sends an email, posts to Slack, or deletes data without asking. Or it asks confirmation for every trivial action. Either extreme breaks trust. The approval model must match the stakes and reversibility of the action. ## What goes wrong Scenario A: User says "clean up my calendar." Agent deletes meetings including one with the VP. No confirmation. Scenario B: Agent asks "Move report.pdf? [Yes/No]" for 40 files. User gives up at file 12. Both are approval mismatches. ## Detection **Surfaces:** agent-tool-execution **Auditability:** hybrid **Static signals:** 1. Find agent-initiated side effects (send, delete, create, publish). 2. Classify by stakes and reversibility. Check whether approval precedes high-stakes actions. 3. Flag mismatches in both directions. 4. Where a framework holds the policy, read it. AI SDK 7 `toolApproval` maps each tool to `'not-applicable'`, `'approved'`, `'denied'`, or `'user-approval'`, or runs a function: a catch-all returning `'approved'` is Scenario A, `'user-approval'` on read-only tools is Scenario B. In Claude Agent SDK code, `permissionMode: "bypassPermissions"` in a user-facing product is Scenario A. 5. Where MCP elicitation carries the confirmation, check that three answers mean three outcomes: `decline` and `cancel` both mean do not proceed; only `accept` does. **Concrete commands:** ```bash rg -l 'sendEmail|sendMessage|deleteAccount|publishPost|processPayment' --type=ts src/ rg -B 10 'sendEmail|delete|publish' --type=ts src/ | rg 'confirm|approval|modal' rg -n "toolApproval|'user-approval'|needsApproval|permissionMode|elicitation/create" --type=ts src/ ``` **Judgment signals:** - "User-requested" vs. "agent-initiated" matters. "Clean up my inbox" per-email = user-requested. Agent proactively acting = agent-initiated. - A single "Are you sure?" for 50 actions is insufficient. - Stakes and reversibility alone treat every delete the same. If the gate cannot tell a draft the agent created this turn from a record that predates the session, or an internal target from one that leaves the workspace, it can only be tuned by getting stricter. That is a fail: pass provenance in with the stakes. **False-positive guards:** - Skip `// ax-audit-ignore:control-no-approval-gate`, test, and Storybook files. ## Fix Implement the stakes x reversibility matrix. Low/easy: auto-apply. Low/hard: quick confirm. High/easy: show diff. High/hard: explicit modal approval. ```ts // before: one policy for everything toolApproval: () => "approved", // after: the treatment follows stakes, reversibility, and provenance toolApproval: ({ toolCall }) => { const t = tools[toolCall.toolName]; if (t.readOnly) return "not-applicable"; if (t.reversible && !leavesWorkspace(toolCall.input)) return "approved"; // receipt with undo return "user-approval"; // diff or modal }, ``` ## Default tier and overrides **Defaults to:** `release-blocker` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | release-blocker | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | ## Examples **Anti-pattern (fails):** ```tsx async function handleSendEmail(draft: EmailDraft) { await emailClient.send(draft); return { status: "sent", message: `Email sent to ${draft.to}` }; } ``` **Applied (passes):** ```tsx async function handleSendEmail(draft: EmailDraft, ctx: AgentContext) { const approved = await ctx.modalApproval({ title: `Send email to ${draft.to}?`, preview: <EmailPreview draft={draft} />, actions: ["Send", "Edit", "Cancel"], }); if (!approved) return { status: "cancelled" }; await emailClient.send(draft); return { status: "sent" }; } ``` ## Suppression ```tsx {/* ax-audit-ignore:control-no-approval-gate, user opted into auto-apply mode */} <AutoApplyToggle enabled={userPreference.autoApply} /> ``` -
control-no-escape-hatch.md 3.3 KB
--- title: No way to interrupt, redirect, or undo agent action slug: control-no-escape-hatch category: control defaultTier: release-blocker surfaces: agent-chat, agent-tool-execution ax-pattern: Escape Hatch detection: hybrid related: control-no-approval-gate, trust-no-escalation-path, comm-no-intent-handshake, comm-unrequested-action-no-consent --- ## No way to interrupt, redirect, or undo agent action Agent starts a long response or multi-step workflow. User realizes it's wrong but there's no stop, undo, or "go back": they watch it do the wrong thing and can't intervene. Autonomy without exit is coercion. ## What goes wrong User asks the agent to refactor a module. It begins a 12-step migration; at step 3 the user sees it's wrong. No stop button: it runs to completion, leaving the codebase in an unwanted state. Manual revert takes longer than doing it themselves. ## Detection **Surfaces:** agent-chat, agent-tool-execution **Auditability:** hybrid **Static signals:** 1. Find agent execution UI (chat panels, action panels, tool execution views). 2. Check for cancel/stop during execution (`onCancel`, `AbortController`, `useChat().stop`, `query.interrupt`). 3. Trace the signal to the server. `stop()` aborts the client fetch; the route has to pass `req.signal` as `abortSignal` into `streamText` (or the loop), and each tool's `execute` has to read it, or the executor finishes every remaining call after Stop. 4. Check for undo/revert after completion. Flag flows with neither. **Concrete commands:** ```bash rg -l 'AbortController|onCancel|stopGenerat' --type=ts src/ rg -A 10 'isGenerating|isStreaming|isPending' --type=ts src/ | rg -v 'cancel|stop|abort' rg -n 'abortSignal|req\.signal|request\.signal' --type=ts src/app/api/ src/server/ ``` **Judgment signals:** - A cancel button not wired to `AbortController.abort()` is a false affordance, worse than nothing. - A `stop()` that closes the stream while the server loop keeps executing tools is the same false affordance one layer down: the UI goes quiet and the emails still go out. **False-positive guards:** - Skip `// ax-audit-ignore:control-no-escape-hatch`, test, and Storybook files. ## Fix During execution: stop button wired to `AbortController`, and the signal threaded through the route into the loop and each tool. After completion: undo/revert for reversible actions. For irreversible actions, the approval gate (`control-no-approval-gate`) is the pre-execution escape hatch. ## Examples **Anti-pattern (fails):** ```tsx <div> {messages.map((m) => <Message key={m.id} {...m} />)} {isGenerating && <Spinner />} {/* no stop button, no undo */} </div> ``` **Applied (passes):** ```tsx <div> {messages.map((m) => <Message key={m.id} {...m} />)} {isGenerating && ( <> <Spinner /> <Button onClick={onStop} aria-label="Stop generating">Stop</Button> </> )} {!isGenerating && <Button onClick={onUndo} variant="ghost">Undo</Button>} </div> ``` ## Default tier and overrides **Defaults to:** `release-blocker` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | release-blocker | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | ## Suppression ```tsx {/* ax-audit-ignore:control-no-escape-hatch, single status check, completes in <1s */} <StatusCheckResult result={result} /> ``` -
control-over-conversational.md 2.5 KB
--- title: Chat interface for actions that should be buttons slug: control-over-conversational category: control defaultTier: fix-this-sprint surfaces: agent-chat ax-pattern: Over-conversational (anti-pattern) detection: observational related: comm-no-intent-handshake, comm-no-generative-momentum --- ## Chat interface for actions that should be buttons User wants to toggle a setting or trigger a known action, but chat is the only interface. They type "turn on dark mode" and wait for a round-trip instead of flipping a switch. Chat is the ONLY path to deterministic actions. ## What goes wrong User types "enable dark mode" in chat; agent responds after 2 seconds. A toggle would take 50ms. Multiply across every simple action and chat becomes a bottleneck. ## Detection **Surfaces:** agent-chat **Auditability:** observational **Static signals:** 1. Find chat input surfaces. 2. Identify deterministic actions achievable through chat (toggles, selections, CRUD). 3. Flag cases where chat is the only path to a simple action. **Concrete commands:** ```bash rg -l 'ChatInput|MessageInput|PromptInput' --type=ts src/ rg -l 'Toggle|Switch|Select|Dropdown' --type=ts src/components/ ``` **Judgment signals:** - The anti-pattern is chat-only for deterministic actions. Some conversational interface is expected. - Controls rendered inside the chat stream count: an MCP Apps `ui://` resource or an AI SDK `data-*` part that renders a real toggle, table, or picker is direct manipulation. The fail is a text box as the only path. **False-positive guards:** - Skip `// ax-audit-ignore:control-over-conversational`, test, and Storybook files. ## Fix Add direct-manipulation controls alongside chat: quick-action buttons, command palette, context menus. Keep chat for ambiguous or multi-step requests. ## Examples **Anti-pattern (fails):** ```tsx <div> <DataTable data={data} /> <AgentChat onSend={handleAgentCommand} /> {/* no sort, filter, or action controls */} </div> ``` **Applied (passes):** ```tsx <div> <DataTable data={data} onSort={handleSort} sortable /> <QuickActions actions={[{ label: "Export CSV", handler: exportCsv }]} /> <AgentChat onSend={handleAgentCommand} /> </div> ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | backlog | | Agent chat | fix-this-sprint | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | ## Suppression ```tsx {/* ax-audit-ignore:control-over-conversational, chat-first product by design */} <AgentChat onSend={onSend} /> ``` -
control-thin-approval-payload.md 6 KB
--- title: Approval prompt names the action but not what it will do slug: control-thin-approval-payload category: control defaultTier: fix-this-sprint surfaces: agent-chat, agent-tool-execution ax-pattern: The Approval Moment detection: code-auditable related: control-no-approval-gate, comm-no-approval-gate, comm-no-intent-handshake --- ## Approval prompt names the action but not what it will do The gate is there. It fires on the right actions, it matches the stakes, and it asks "Allow the agent to send an email?" with a recipient, a subject, and a body the user never sees. There is nothing to weigh, so the answer is yes every time, which makes the gate a click-through rather than a decision. As the interface shrinks, the approval moment is one of the few surfaces left. It has to carry exactly enough of the action for a confident yes or no, and a tool name is not enough. ## What goes wrong The agent drafts a reply to a customer, gets it wrong, and asks for approval. The dialog reads "Send email? Allow / Deny". The user, six approvals into a session where the previous five were fine, allows it. The wrong reply goes to the customer. The gate did its job and prevented nothing, because the one fact that would have changed the answer, the body of the message, was the fact the dialog left out. The reverse also happens: the dialog is so thin the user denies everything, and the agent becomes unusable for exactly the tasks it exists to do. ## Detection **Surfaces:** agent-chat, agent-tool-execution **Auditability:** code-auditable **Static signals:** 1. Find approval and confirmation components: `ApprovalDialog`, `ConfirmAction`, `ToolApproval`, `PermissionPrompt`, anything rendering an allow and deny pair. 2. Read the props each one accepts, and the props the call site passes. 3. Flag components whose only action-describing input is a name or type (`toolName`, `action`, `actionType`, `title`) with no channel for the call arguments. 4. Flag call sites that have the arguments in scope and pass only the name. 5. For destructive actions, check that the target is named specifically (which record, how many rows), not by category. 6. In AI SDK 7 renderers, find the branch on `part.state === "approval-requested"`: `part.input` is in scope there, and a branch that prints the tool name without it is the finding. In Claude Agent SDK `canUseTool(toolName, input)` handlers, check that `input` reaches the prompt, not only `toolName`. MCP asks clients to show tool inputs before calling the server; a host that shows the tool title is not doing that. **Concrete commands:** ```bash # candidate approval surfaces: record this list, it is the evidence rg -l 'Approval|ConfirmAction|PermissionPrompt|requiresApproval' --type=ts src/ # per candidate, does the surface ever reference the call's arguments? # absence is the finding, so check each file rather than grepping for a negative for f in $(rg -l 'Approval|ConfirmAction|PermissionPrompt' --type=ts src/); do rg -q 'args|payload|preview|diff|target|params|describeIntent' "$f" \ || echo "thin approval surface: $f" done # the prop surface, to see what a call site is even able to pass rg -n -A 8 'interface .*(Approval|Confirm).*Props' --type=ts src/ # gates that stringify the tool and stop there rg -n 'confirm\(|window\.confirm' -A 2 --type=ts src/ # framework approval branches: read each for input/args rendering rg -n 'approval-requested|canUseTool' -A 8 --type=ts src/ ``` **Judgment signals:** - A payload rendered but truncated to one line for a multi-paragraph action is a partial pass; note it as `warn`, not `fail`. - A diff view for edits and a plain summary for sends both count, as long as what changes is legible before the answer. **False-positive guards:** - Skip gates for low-stakes reversible actions where a receipt with undo is the right pattern instead. That is `control-no-approval-gate` territory, and this rule should not push toward more friction than the stakes need. - A component that accepts a `children` or `preview` slot passes if the call sites populate it. Check the call sites, not just the type. - Approve-with-changes passes: a gate whose fields are editable and flow back as `updatedInput` (Agent SDK) or an approval `reason` (AI SDK) carries more than the payload this rule asks for. - Skip files with `// ax-audit-ignore:control-thin-approval-payload` near the match. - Skip test and Storybook fixtures. ## Fix Pass the call arguments through to the gate and render the ones that would change the answer. ```tsx // before: the user approves a category <ApprovalDialog toolName="send_email" onApprove={() => execute(call)} onDeny={reject} /> // after: the user approves this specific act <ApprovalDialog toolName="send_email" summary={`Reply to ${call.args.to} about "${call.args.subject}"`} onApprove={() => execute(call)} onDeny={reject} > <RecipientList to={call.args.to} cc={call.args.cc} /> <BodyPreview body={call.args.body} expandable /> </ApprovalDialog> ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | fix-this-sprint | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | ## Examples **Anti-pattern (fails):** ```tsx function ToolApproval({ call, onApprove, onDeny }: Props) { return ( <Prompt> <p>Allow the agent to run {call.name}?</p> <Button onClick={onApprove}>Allow</Button> <Button onClick={onDeny}>Deny</Button> </Prompt> ); } ``` `call.args` is in scope and never rendered. **Applied (passes):** ```tsx function ToolApproval({ call, onApprove, onDeny }: Props) { return ( <Prompt> <p>{describeIntent(call)}</p> <ArgumentPreview args={call.args} highlight={call.destructiveFields} /> <Button onClick={onApprove}>Allow</Button> <Button onClick={onDeny}>Deny</Button> </Prompt> ); } ``` ## Suppression ```tsx {/* ax-audit-ignore:control-thin-approval-payload, no arguments, the action is the whole payload */} <ApprovalDialog toolName="end_session" onApprove={stop} onDeny={reject} /> ``` -
trust-no-confidence-cues.md 3 KB
--- title: Agent output with no rationale or sources slug: trust-no-confidence-cues category: trust defaultTier: fix-this-sprint surfaces: agent-chat, agent-dashboard ax-pattern: Confidence Cues detection: hybrid related: trust-no-uncertainty-markers --- ## Agent output with no rationale or sources Agent says "You should refactor this function" with no explanation. User can't evaluate the advice: follows it blindly or ignores it. Neither builds trust. ## What goes wrong Agent responds with a confident directive and nothing else. User can't tell if it came from docs, past conversations, or hallucination. One wrong answer and they stop trusting all responses, having never had a way to tell good from bad. ## Detection **Surfaces:** agent-chat, agent-dashboard **Auditability:** hybrid **Static signals:** 1. Find agent output components (`role="assistant"`, `<AssistantMessage>`, `<AiResponse>`). 2. Check whether consequential claims expose supporting sources or a concise user-facing rationale, inline or through source components. 3. Flag missing support for a consequential claim, not the absence of a particular child component. Internal reasoning need not be displayed. **Concrete commands:** ```bash rg -l 'role.*assistant|AssistantMessage|AiResponse|completion' --type=ts src/ rg -A 15 'role.*assistant|<AssistantMessage|<AiResponse' --type=ts src/ | rg -v 'Citation|Source|Reasoning|Thinking' rg -n "type === ['\"](reasoning|source-url|source-document)|filter\(.*type === ['\"]text" --type=ts src/ ``` **Judgment signals:** - Even if `<Sources>` exists, check whether it's populated vs. always empty. - Rationale is needed where it helps assess a consequential recommendation; routine status or self-contained answers need no extra panel. - Dropping source parts can remove claim support. Omitting private reasoning is not itself a defect; inspect the user-facing explanation and sources. **False-positive guards:** - Skip `// ax-audit-ignore:trust-no-confidence-cues`, test, and Storybook files. - Skip status-only messages ("Done!" confirmations). ## Fix Expose relevant sources and a concise decision rationale. Do not require private chain-of-thought or a thinking panel. ## Examples **Anti-pattern (fails):** ```tsx <div className="agent-response" role="assistant"> <Markdown>{completion.text}</Markdown> </div> ``` **Applied (passes):** ```tsx <div className="agent-response" role="assistant"> <Markdown>{completion.text}</Markdown> {completion.explanation && <p>{completion.explanation}</p>} {completion.sources.length > 0 && <CitationList sources={completion.sources} />} </div> ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent chat | fix-this-sprint | | Agent config | backlog | | Agent dashboard | fix-this-sprint | ## Suppression ```tsx {/* ax-audit-ignore:trust-no-confidence-cues, status-only messages need no rationale */} <AgentMessage content={statusText} /> ``` -
trust-no-escalation-path.md 2.8 KB
--- title: High-stakes agent action with no human escalation slug: trust-no-escalation-path category: trust defaultTier: release-blocker surfaces: agent-tool-execution ax-pattern: Escape Hatch (escalation dimension) detection: code-auditable related: control-no-approval-gate, control-no-escape-hatch --- ## High-stakes agent action with no human escalation Agent handles a refund, medical question, or legal inquiry with no way to hand off to a human: it gives a dangerous answer or refuses entirely. An escalation path is the trust floor. ## What goes wrong User asks about a billing dispute. Agent applies a partial credit that doesn't match. No "talk to a person" button. It keeps trying, makes things worse, user files a chargeback. ## Detection **Surfaces:** agent-tool-execution **Auditability:** code-auditable **Static signals:** 1. Find action handlers for high-stakes operations (financial, medical, legal, account deletion). 2. Check for escalation/handoff logic. Flag high-stakes handlers with no escalation path. **Concrete commands:** ```bash rg -l 'refund|payment|delete.*account|send.*email|legal|medical' --type=ts src/ rg 'escalat|handoff|transfer.*human|transfer.*agent' --type=ts src/ ``` **Judgment signals:** - An escalation tool never referenced in the system prompt is effectively invisible. - Escalation is a handoff to a human. Refusal is declining a request the agent cannot safely complete. Improvising the nearest write to real state is not a success, and is not fixed by adding an escalate button. **False-positive guards:** - Skip `// ax-audit-ignore:trust-no-escalation-path`, test, and Storybook files. ## Fix Add `escalate_to_human(reason, context)` as an agent tool. Surface it in the UI as "Talk to a person." ## Examples **Anti-pattern (fails):** ```tsx const agentTools = { processRefund: async (amount: number) => { await api.refund(amount); return { success: true, message: "Refund processed." }; }, }; ``` **Applied (passes):** ```tsx const agentTools = { processRefund: async (amount: number) => { if (amount > ESCALATION_THRESHOLD) return { escalate: true, reason: "Exceeds limit" }; await api.refund(amount); return { success: true }; }, escalateToHuman: async (reason: string, ctx: AgentContext) => { await support.transfer({ reason, transcript: ctx.messages }); return { message: "Connecting you with a team member." }; }, }; ``` ## Default tier and overrides **Defaults to:** `release-blocker` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | release-blocker | | Agent config | backlog | | Agent dashboard | fix-this-sprint | ## Suppression ```tsx {/* ax-audit-ignore:trust-no-escalation-path, internal admin tool, operator is the human */} <AgentToolPanel tools={adminTools} /> ``` -
trust-no-uncertainty-markers.md 2.7 KB
--- title: Agent presents everything with equal certainty slug: trust-no-uncertainty-markers category: trust defaultTier: fix-this-sprint surfaces: agent-chat ax-pattern: Confidence Cues detection: observational related: trust-no-confidence-cues, trust-no-escalation-path, parity-unstructured-tool-output --- ## Agent presents everything with equal certainty Agent is 95% sure of one recommendation, 40% of another, but both render identically. When the 40% answer is wrong, the user distrusts not just it but everything. Confident wrong answers cause permanent trust damage. ## What goes wrong Two recommendations in one response: one well-supported, one a guess. Same font, weight, formatting. User treats both as equally reliable; the guess is wrong; now they second-guess every future response. Trust is binary when the interface gives no gradient. ## Detection **Surfaces:** agent-chat **Auditability:** observational **Static signals:** 1. Find agent output containers. 2. Check for confidence props (`confidence`, `certainty`, `score`) or uncertainty components. 3. Absence of all = flag. **Concrete commands:** ```bash rg 'confidence|certainty|ConfidenceBadge|UncertaintyIndicator' --type=ts src/ ``` **Judgment signals:** - Hedging in prompt instructions is weaker than structured indicators but better than nothing. - A badge always showing "high" is not meaningful: check for actual variation. **False-positive guards:** - Skip `// ax-audit-ignore:trust-no-uncertainty-markers`, test, and Storybook files. - Skip trivial outputs (confirmations, acknowledgments) where confidence is always 100%. ## Fix Add confidence indicators: numeric score, visual badge (high/medium/low), hedging language, or expandable reasoning that shows uncertainty. ## Examples **Anti-pattern (fails):** ```tsx <ul> {recommendations.map((rec) => ( <li key={rec.id}>{rec.text}</li> ))} </ul> ``` **Applied (passes):** ```tsx <ul> {recommendations.map((rec) => ( <li key={rec.id}> {rec.text} <ConfidenceBadge level={rec.confidence > 0.8 ? "high" : "low"} /> </li> ))} </ul> ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | fix-this-sprint | | Agent chat | fix-this-sprint | | Agent config | backlog | | Agent dashboard | fix-this-sprint | No tool-execution bump. Hedging in prose changes nothing about what a tool did, and an observational rule that can only return `unknown` on static evidence should not be the single finding that flips a verdict. The blockers on that surface are the gate, its payload, and the escape hatch. ## Suppression ```tsx {/* ax-audit-ignore:trust-no-uncertainty-markers, deterministic lookups, no uncertainty */} <AgentRecommendation text={result.text} /> ``` -
trust-undisclosed-access-scope.md 5.5 KB
--- title: What the agent can reach is never shown to the user slug: trust-undisclosed-access-scope category: trust defaultTier: fix-this-sprint surfaces: agent-config, agent-tool-execution ax-pattern: Legitimacy detection: code-auditable related: context-memory-not-visible, trust-no-escalation-path --- ## What the agent can reach is never shown to the user The agent holds tokens for the user's mail, calendar, files, and billing. Nowhere in the product can the user see that list, see which scopes each grant carries, or hand back one of them without disconnecting the whole thing. The product is asking for standing access to a person's life and answering the obvious question, what can you actually get to, with silence. Usability failures cost a task. This costs the relationship, and it is the loudest question asked of every product in this category: what can it access, what does it keep, and what does it do when nobody is watching. ## What goes wrong A user connects their mail so the agent can draft replies. The integration requests full mailbox read and send. Months later the user notices the agent citing a thread they thought was private, goes looking for what it can see, and finds a settings page with one switch labelled "Email: connected". The only available move is to disconnect everything and lose the feature. They disconnect, and they do not come back, because the product never gave them a smaller answer than all or nothing. ## Detection **Surfaces:** agent-config, agent-tool-execution **Auditability:** code-auditable **Static signals:** 1. Enumerate the grants the product requests: scope arrays, connector definitions, integration configs, service account roles. Record the list. 2. Find the settings or connections surface that a user actually sees. 3. Cross-check: every grant in step 1 should be nameable from step 2, in the user's terms rather than the provider's scope string. 4. Check for a per-connector revoke path, not just a global disconnect. 5. Check whether retention is stated anywhere the user can find: what the agent stores from that connection, and for how long. **Concrete commands:** ```bash # what the product asks for rg -n -i "scopes?\s*[:=]\s*\[|scope=|'https://www\.googleapis\.com/auth" --type=ts src/ rg -n -i 'connector|integration' --type=ts src/ -l # what the user can see: a surface that enumerates them rg -n -i 'connections|integrations|permissions|connected accounts' --type=ts src/app src/pages src/components # a per-connector revoke, not one global switch rg -n -i 'revoke|disconnect' --type=ts src/ -A 3 ``` **Judgment signals:** - A list of connected services with no scopes is a partial pass: the user knows what is attached but not what it can do. Report `warn`. - Scope strings rendered raw (`https://www.googleapis.com/auth/gmail.modify`) are a partial pass; legible only to developers. - Grants made through MCP URL-mode elicitation (`mode: "url"`) happen on the server's own page and never transit the client, so the product has no record of them unless the server exposes one. A connections list that names the MCP server but not the third-party accounts it now holds tokens for is a partial pass; report `warn`. **False-positive guards:** - Skip products whose agent touches only first-party data the user is already looking at. There is no external reach to disclose. - A provider-hosted consent screen counts for the initial grant but not for ongoing visibility: the question is whether the user can check later, not whether they clicked once. - Skip files with `// ax-audit-ignore:trust-undisclosed-access-scope` near the match. - Skip test fixtures and seed data. - Do not merge this with `context-memory-not-visible`. What the agent can reach and what it has kept are different disclosures; if both fail, file both, each with its own evidence. ## Fix Render the grant list the code already holds, in the user's terms, with a revoke per row. ```tsx // before: one switch, no scopes, all or nothing <Toggle label="Email" checked={mail.connected} onChange={disconnectAll} /> // after: what it can reach, and a way to take one thing back {connections.map((c) => ( <ConnectionRow key={c.id} name={c.displayName}> <ScopeList scopes={c.scopes.map(describeScope)} /> <RetentionNote keeps={c.retention} /> <Button onClick={() => revoke(c.id)}>Revoke</Button> </ConnectionRow> ))} ``` ## Default tier and overrides **Defaults to:** `fix-this-sprint` | Surface | Tier | |---|---| | Agent tool execution | release-blocker | | Agent chat | fix-this-sprint | | Agent config | fix-this-sprint | | Agent dashboard | fix-this-sprint | Undisclosed reach on a surface that acts autonomously is the case the user cannot discover by watching, which is why the execution surface blocks and the config surface, where the user is already looking for this, does not. ## Examples **Anti-pattern (fails):** ```ts // src/integrations/google.ts export const googleScopes = [ "https://www.googleapis.com/auth/gmail.modify", "https://www.googleapis.com/auth/calendar", "https://www.googleapis.com/auth/drive.readonly", ]; // no component anywhere renders googleScopes ``` **Applied (passes):** ```tsx // src/settings/Connections.tsx <ConnectionRow name="Google"> <ScopeList scopes={googleScopes.map(describeScope)} /> {/* "Read and send mail", "Read and edit calendar events", "Read files" */} <Button onClick={() => revoke("google")}>Revoke</Button> </ConnectionRow> ``` ## Suppression ```tsx {/* ax-audit-ignore:trust-undisclosed-access-scope, agent reads only the open document */} <AgentPanel document={doc} /> ``` -
_sections.md 4.3 KB
# Sections: Agentic Experience (Layer 2) This file defines the 4 categories of agentic experience audit rules. Each rule file uses one of these category prefixes. --- ## 1. Trust & Transparency (trust) **Default tier:** mostly fix-this-sprint; release-blocker for missing escalation paths **Why critical:** Users won't trust an agent, even when it's right, unless they can see why it decided what it did. Confident wrong answers without uncertainty markers or escalation paths cause permanent trust damage that future accuracy can't recover. Disclosure is part of trust, not just explanation: a user who cannot see what the agent can reach has no way to judge whether to keep it connected. ## 2. Control & Recovery (control) **Default tier:** release-blocker for missing escape hatches; fix-this-sprint for over-conversational **Why critical:** Autonomy without exit is coercion. Every agent action needs a visible path to undo, revise, or override. The approval model must match the stakes and reversibility of the action. Chat-only interfaces for button-worthy actions waste user time and patience. As the interface shrinks the approval moment carries more weight, so a gate that fires correctly and shows nothing to decide on is a click-through, not a control. ## 3. Context & Memory (context) **Default tier:** mostly fix-this-sprint to backlog **Why critical:** Agents that don't show what they remember feel opaque. Agents that don't use available context feel stupid. Interfaces that don't reshape with task progression feel static. All three erode the relationship depth that makes agent products defensible. ## 4. Agent Communication (comm) **Default tier:** release-blocker for silent execution; fix-this-sprint for missing handshake; backlog for missing drafts **Why critical:** Silent agents feel broken. The communication contract between agent and user (progress signals, intent confirmation, and generative momentum) is the difference between a tool that works and a black box. The contract also covers the turns the user did not start: an agent that acts on a schedule cannot borrow permission from a request that never happened, so unprompted action blocks on a standing boundary plus a notice the user can act on. --- ## Rule index ``` trust-no-confidence-cues trust-no-uncertainty-markers trust-no-escalation-path trust-undisclosed-access-scope control-no-escape-hatch control-no-approval-gate control-over-conversational control-thin-approval-payload context-memory-not-visible context-no-adaptive-canvas context-under-contextual comm-no-intent-handshake comm-no-progress-signal comm-no-generative-momentum comm-unrequested-action-no-consent ``` Total: 15 rules. --- ## Cross-rule interactions These pairings often co-fire on the same surface: - **no-confidence-cues + no-uncertainty-markers**: both address "why should I trust this." Different targets: rationale vs. hedging. - **no-escape-hatch + no-approval-gate**: for autonomous actions, both fire. Approval gate may partially satisfy escape hatch. - **no-progress-signal + no-intent-handshake**: long-running tasks that didn't confirm scope AND show no progress are doubly opaque. - **memory-not-visible + under-contextual**: complementary. One says the agent knows things the user can't see; the other says it doesn't know things it should. - **over-conversational + no-generative-momentum**: paradoxical pairing. Forcing chat where buttons would do, while failing to offer drafts where blanks would benefit. - **no-approval-gate + thin-approval-payload**: sequential, not simultaneous. The first asks whether the treatment matches the stakes, the second whether the prompt carries enough to decide. A surface with no gate at all is the first finding only; the second has nothing to inspect until a gate exists. - **undisclosed-access-scope + memory-not-visible**: what the agent can reach versus what it has kept. Different disclosures, so file both when both fail, each with its own evidence. Merging them hides whichever the team did not think of. - **unrequested-action-no-consent + no-escape-hatch**: an unattended run's notice is the only place an escape hatch can appear after the fact. If the notice is missing, report the consent finding; the escape-hatch fix has nowhere to attach until the user is told the action happened. -
_template.md 2.6 KB
--- title: <Rule title, short, descriptive> slug: <category>-<kebab-slug> category: trust | control | context | comm defaultTier: release-blocker | fix-this-sprint | backlog surfaces: <playbooks that name this rule: agent-chat, agent-tool-execution, agent-config, agent-dashboard> ax-pattern: <which AX pattern or anti-pattern this enforces> detection: code-auditable | hybrid | observational related: <comma-separated other rule slugs (arch or ax); cross-layer pairs list each other in both files> --- ## <Rule title> One paragraph explaining the trust or interaction failure mode in plain language. Why it erodes user trust. What AX pattern it violates. ## What goes wrong A concrete, observable scenario. What the user experiences, what the agent does, why trust breaks. ## Detection **Surfaces:** <which playbooks invoke this: agent-chat, agent-tool-execution, agent-config, agent-dashboard> **Auditability:** <code-auditable | hybrid | observational> **Static signals** (for code-auditable and hybrid rules): 1. Concrete grep / Read step. Use `rg` / `find` / file-extension filters. 2. Each step produces evidence: a file path, a line number, a presence/absence boolean, a count. 3. Last step compares evidence to a threshold. **Concrete commands:** ```bash # Inline grep recipes the agent can run. Note: ripgrep has no 'tsx' type: '--type=ts' covers *.ts and *.tsx. rg 'pattern' --type=ts src/ ``` **Judgment signals** (for hybrid and observational rules): - What to look for in the component tree or interaction flow. - What qualifies as present vs. missing vs. misapplied. **False-positive guards:** - Skip files that already have the expected pattern. - Skip files with `// ax-audit-ignore:<this-slug>` near the match. - Skip test and Storybook fixtures. ## Fix **Concrete change:** ```tsx // before: the anti-pattern // after: the corrected pattern ``` ## Default tier and overrides **Defaults to:** `<tier>` | Surface | Tier | |---|---| | Agent tool execution | <usually one tier higher> | | Agent chat | <same or one tier lower> | | Agent config | <same> | | Agent dashboard | <usually one tier lower> | Cover every surface listed in `surfaces:`. A missing row hands that surface to the generic bump in `references/ship-readiness.md`, so the table is required even when every row just repeats `defaultTier`: the repetition is what suppresses the bump. ## Examples **Anti-pattern (fails):** ```tsx // Real-world example showing the trust failure. ``` **Applied (passes):** ```tsx // Same component with trust pattern applied. ``` ## Suppression To ignore this rule on a specific component: ```tsx {/* ax-audit-ignore:<slug>, reason */} <Component /> ```
-
-
SKILL.md 11.1 KB
--- name: ax-audit description: Audits agentic products for tool parity, authority, approval payloads, recovery, and trust using 27 rules and a ship verdict. Use when asked for an "AX audit", to review an agent approval flow, or whether an agent can operate the product. For human-facing API ergonomics use dx-audit; for ordinary UI use ui-design. --- # AX Audit Feature-level reviewer for apps where an agent acts for the user. One question: **does it earn trust, and where does it break?** - **IS:** rules-based audit of agentic surfaces (chat, tool execution, config, dashboards) across architecture (`rules-arch/`) and trust (`rules-ax/`), ending in a ship-readiness verdict plus an AX Relationship Summary. - **IS NOT:** traditional frontend UX (use `ui-design` Audit mode); developer-facing API, CLI, or type ergonomics (use `dx-audit`); public site or docs agent scores (use `agent-ready`); agent instruction files (use `agents-md`); what the product should do before it exists (use `product-design`). No agentic features in scope? Stop. AX rules against forms and lists are noise. ## Contents - [Audit workflow](#audit-workflow) - [Two rule layers](#two-rule-layers) - [Tiers and verdict](#tiers-and-verdict) - [AX Relationship Summary](#ax-relationship-summary) - [Reference files](#reference-files) - [Gotchas](#gotchas) - [Audit self-check](#audit-self-check) - [Related skills](#related-skills) ## Audit workflow ```text AX Audit progress: - [ ] Step 1: Scope, via the diff against the PR base merge-base (PR mode) or explicit path (full sweep) - [ ] Step 2: Detect agentic features per references/feature-playbooks.md - [ ] Step 3: Run each detected feature's playbook in order, plus the diff-wide checks (PR mode only) - [ ] Step 4: For each check, load the rule file and follow its detection recipe - [ ] Step 5: Tier each finding per references/ship-readiness.md (rule override table wins) - [ ] Step 6: Render verdict + findings + AX Relationship Summary per references/output-format.md - [ ] Step 7: Run the audit self-check and report its evidence counts ``` PR-mode scope is the diff plus the tool definitions and orchestrator it touches. Findings in untouched files belong in a full sweep, not this verdict. Playbook annotations are a scan copy; the rule file is authoritative. `parity-orphan-ui-action` runs on every PR-mode audit and never in a full sweep, where there is no diff for it to read. Rule greps name the most common identifiers, not every framework's spelling. When a grep misses in code that plainly does the thing (a gate, a stream, a tool result), check `references/framework-signals.md` for the stack's name for it before recording `unknown`. ## Two rule layers | Layer | Folder | Rules | Load when a playbook names | |---|---|---|---| | 1: Architecture | `rules-arch/` | 12 | `rules-arch/<category>-<slug>.md` | | 2: Experience | `rules-ax/` | 15 | `rules-ax/<category>-<slug>.md` | Categories: arch = parity, granularity, context, comm; ax = trust, control, context, comm. Shared prefixes are different rules: `rules-arch/comm-no-approval-gate.md` (no gate on the execution path) is not `rules-ax/control-no-approval-gate.md` (gate exists, stakes are wrong). Run Layer 1 `comm`/`parity` and Layer 2 `control`/`trust` first. They hold the blockers. Category map: `rules-arch/_sections.md`, `rules-ax/_sections.md`. | Priority | Layer | Category | Prefix | Rules | |---|---|---|---|---| | 1 | arch | Communication | `comm-` | 3 | | 2 | arch | Parity | `parity-` | 4 | | 3 | ax | Control | `control-` | 4 | | 4 | ax | Trust | `trust-` | 4 | | 5 | arch | Context | `context-` | 3 | | 6 | ax | Communication | `comm-` | 4 | | 7 | ax | Context | `context-` | 3 | | 8 | arch | Granularity | `granularity-` | 2 | ## Tiers and verdict Three tiers. Full trigger lists and the generic surface bump live in `references/ship-readiness.md`. Precedence: the rule's own surface-override table > the generic bump > `defaultTier`. Apply at most one adjustment. Verdict: ✅ READY (0 blockers, ≤3 sprint) · ⚠️ READY WITH FOLLOW-UP (0 blockers, ≥4 sprint) · ❌ NOT READY (≥1 blocker) · 🚫 INCOMPLETE (self-check failed). Blockers outrank an incomplete audit. With ≥1 release-blocker and a failed self-check, report ❌ NOT READY and note the self-check failure beneath it: the blockers are established findings and stay actionable, while 🚫 reads as "nothing was learned" and sends the reader away. Reserve 🚫 for an audit with no blockers whose coverage you cannot vouch for. ## AX Relationship Summary Render after findings when any agentic feature was detected. Findings serve engineers; this serves designers and PMs. Four fields: evolution stage (behavior, not a label), trust signal (high/moderate/low plus one-line reason), key gap (one actionable sentence), trust question (one question only research can answer). ## Reference files | File | Read when | |---|---| | `references/feature-playbooks.md` | Steps 2-3: detection heuristics, per-feature ordered checks, diff-wide checks | | `references/framework-signals.md` | Step 4, when the code uses AI SDK, MCP, the Claude Agent SDK, or AG-UI: where the gate, the stream, the completion signal, and the structured result live in each, with the spec defaults the rules lean on | | `references/ship-readiness.md` | Step 5: tier triggers, precedence, verdict logic | | `references/output-format.md` | Step 6: findings JSON schema, summary schema, terminal rendering | | `references/agent-native-principles.md` | A Layer 1 finding needs grounding the rule file does not carry | | `references/ax-evolution-curve.md` | Writing the AX Relationship Summary: stage, action depth, costume vs intelligence | | `references/invisible-interface.md` | Grounding for structured tool output, approval payload, access scope, unprompted action; also the arguments that stay in `keyGap` | | `references/evaluation-scenarios.md` | When changing this skill. Never loads during a user audit | | `rules-arch/_sections.md` | Layer 1 categories, default tiers, co-firing pairs | | `rules-ax/_sections.md` | Layer 2 categories, default tiers, co-firing pairs | ## Gotchas - **Scope before rules.** Running all 27 rules repo-wide on a 3-file PR buries a new release-blocker under pre-existing backlog noise; the verdict stops meaning "can this PR merge." - **The rule's override table is authoritative.** `comm-no-intent-handshake` defaults to `fix-this-sprint` but its table says `release-blocker` on tool execution. Stacking the generic "+1 tier on tool execution" bump on an explicit override double-upgrades backlog findings into blockers. - **A stop button not wired to `AbortController.abort()` is a false affordance.** `control-no-escape-hatch` still fails: verify the `abort()` call, not the button label, or the audit passes a UI that lies to users. - **A client `stop()` that only closes the stream leaves the executor running.** `useChat().stop()` aborts the fetch. Unless the route passes `req.signal` into `streamText({ abortSignal })` and tool `execute` reads it, the server finishes every remaining tool call after the user pressed Stop. Trace the signal to the loop, not to the button. - **Tool annotations are hints, not stakes.** MCP tells clients to treat `annotations` from untrusted servers as untrusted; a gate that auto-approves on a third-party server's `readOnlyHint: true` has handed the gate to that server. `comm-no-approval-gate` fails it. The spec defaults (`destructiveHint: true`, `readOnlyHint: false`) are the fail-closed baseline. - **A framework approval flag is the gate's input, not the gate.** AI SDK `toolApproval: "user-approval"` emits a `tool-approval-request` part and waits. A UI that never renders `state === "approval-requested"`, or answers it with `addToolApprovalResponse({ approved: true })` on arrival, has a gate in the type system and none for the user. Check the renderer and the response call, not the option. - **Absence checks need a recorded file list.** "Find components lacking X" greps return nothing both when everything passes and when nothing was scanned. List candidate files first (`rg -l <feature-pattern>`), check each for the counter-pattern, and cite the file list as evidence. - **`detection: observational` rules cannot fail on grep evidence alone.** `granularity-static-api-mapping`, `trust-no-uncertainty-markers`, `control-over-conversational`, and `comm-no-generative-momentum` need interaction-flow judgment; on static evidence alone, return `unknown` with a reason, not `fail`. - **Gates fail in three separate places.** Absent from the path (`comm-no-approval-gate`), present but mismatched to the stakes (`control-no-approval-gate`), or correct and unreadable (`control-thin-approval-payload`). Report the first that holds and fix in that order. - **Interactive gates do not cover unattended runs.** Cron, webhook, and queue entry points reach the same executor with nobody to prompt. `comm-unrequested-action-no-consent` audits that path; evidence names the entry point, not the executor. - **`ax-audit-ignore:<slug>` comments count as `suppressed`, not `pass`.** Report the count in the verdict block; a suppression with no reason is itself a `warn`. - **Don't inflate tiers.** `comm-no-generative-momentum` and `granularity-static-api-mapping` default to `backlog`. One finding promoted to `release-blocker` flips the whole PR to ❌ NOT READY, so promoting cosmetic ones trains the team to ignore the verdict entirely. - **Don't duplicate `ui-design` Audit mode findings.** "Missing loading state" and "form clears on error" are its territory; duplicating them trains engineers to dismiss the whole AX report. - **A Personally Intelligent agent that only ever suggests has plateaued.** Memory stage is not trust. Name the highest action rung in `evolutionStage.behavior` or the summary flatters a polite chatbot. ## Audit self-check Flag the audit `INCOMPLETE` if any of these hold, and include the counts as evidence (planned vs. run rules per playbook, unknown rate, suppressed count): - Fewer rules ran than the playbooks planned - More than 30% of rules returned `unknown`. Count only `unknown` here, never `out-of-scope`: a rule whose layer is absent from the scope you were given was answered correctly, and a narrow diff is the scope Step 1 asks for. Marking a correctly scoped audit INCOMPLETE buries its real blockers under a verdict that reads as "we learned nothing". - Any `fail`/`warn` finding lacks `file:line` evidence or a fix snippet - Every finding landed in the same tier (suspect blanket assignment) - AX Relationship Summary is missing despite detected agentic features ## Related skills - `ui-design` Audit mode: traditional frontend UX around agentic surfaces; run both on agentic feature PRs - `dx-audit`: same files, different reader. This skill asks whether an agent can operate and recover; `dx-audit` asks whether a human adopting the API, CLI, or types finds it ergonomic - `agent-ready`: whether public docs and HTTP APIs are discoverable to coding agents; this skill audits in-product agent UX - `product-design`: what the agentic feature should do, before this audit - `agents-md`: CLAUDE.md / AGENTS.md instruction files Maintenance only: `evals/evals.json` contains regression scenarios for changes to this skill; it does not load during a user task.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.