ai-audit-docs
Audits docs against repo state for dead links, false claims, stale versions, and nonexistent commands. Input: paths or default docs/, README.md, AGENTS.md. Output: ranked findings only; no auto-fix. Trigger: "audit docs", "verify docs", or "documentation review".
Install
npx skills add https://github.com/arcasilesgroup/ai-engineering/tree/main/skills/ai-audit-docs
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install arcasilesgroup-ai-engineering@llmmart
git clone https://github.com/arcasilesgroup/ai-engineering.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole arcasilesgroup/ai-engineering collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Audit docs against repo reality
Lifecycle
Lane: light Writes: nothing — report-only, never modifies files Read by: humans (consumed as a findings table) Dies: on completion Next: ai-verify (to validate fixes if user acts on findings)
What it produces
A ranked markdown table of findings. Every finding names the doc file, the line, the severity, and the broken claim.
| Severity | Meaning |
|---|---|
| critical | Doc points users at something that does not exist — they will hit a wall |
| warning | Claim is outdated or inaccurate but the doc is still roughly usable |
| info | Cosmetic: stale version number, unused anchor, minor nit |
Steps
Determine scope. If the user gave a path, audit that. Otherwise default to
README.md,AGENTS.md, and anydocs/directory at the repo root. Glob for**/*.mdunder docs if the directory exists. Never audit vendored or generated files (node_modules/,dist/,*.generated.*).Read every target doc. Extract all of these per file:
- File links: markdown links pointing at local paths (e.g.
./path,../path,path/to/file). Skip anchors (#section) and bare URLs — those are checked in step 3. - External URLs: bare
http:///https://links. Collect for batch HEAD-check. - Command references: inline code or code blocks containing shell commands
(
npm run X,yarn X,make X,cargo X,python -m X,npx X). Strip arguments; extract just the command name. - Version strings:
v1.2.3,1.2.3,>=1.2.3, pinned deps in prose. - Feature claims: sentences that assert existence — "supports X", "includes X", "has X", "provides X", "X is built-in", "X is enabled by default".
- File links: markdown links pointing at local paths (e.g.
Cross-check file links. For each local path extracted from a markdown link:
- Resolve it relative to the doc file's directory.
findorreadto confirm the target exists. If not -> critical finding.
Cross-check command references. For each command name:
grepthe repo for a script/package-bin definition (package.jsonscripts,Makefiletargets,Cargo.tomlbin,setup.py/pyproject.tomlentry points, shell scripts inbin/orscripts/).- If no definition exists AND the command is not a well-known system binary
(
git,ls,cat,curl,jq, etc.) -> warning finding. - If a script IS defined, check that the arguments the doc uses still match the script's actual flags — flag mismatches -> info.
Cross-check feature claims. For each feature assertion:
findthe feature name (the noun, not the sentence) in the codebase: config keys, source files, type definitions, exported symbols.- If nothing matches -> warning (the feature may exist under a different name; the finding should note "searched for X, found nothing; may be named differently").
- If the feature exists but is behind a flag the doc doesn't mention -> info.
Check version strings. For each version reference found in prose:
- Compare against the repo's actual version:
package.jsonversion field,Cargo.toml,pyproject.toml,version.txt, etc. - If the doc version is older -> info (may be intentional; flag as potentially stale).
- If the doc references a version that is higher than what exists -> warning.
- Compare against the repo's actual version:
Batch-check external URLs. For each external URL:
- HEAD request (timeout 5s). 404/410/DNS failure -> critical (dead link).
- 403/429/connection refused -> info (may be rate-limited or geo-blocked; note it but do not flag as dead).
- 301/302 -> follow one hop; if the final target is the same domain, note the redirect as info.
Rank and output. Sort findings by severity (critical first), then by file path. Output as a markdown table:
| # | Severity | File | Line | Finding | |---|----------|------|------|---------| | 1 | critical | README.md | 42 | Link `./docs/setup.md` -> file does not exist | | 2 | warning | AGENTS.md | 15 | Command `yarn deploy` not defined in any script | | 3 | info | README.md | 8 | Version `v2.1.0` in doc vs `v2.3.0` in package.json |End with a one-line summary:
X critical, Y warnings, Z info — total N findings.
Anti-patterns
- Fixing anything. This skill is report-only. Never edit docs; the user decides what to fix.
- Auditing generated files.
dist/,build/,node_modules/,*.generated.*are out of scope — their content is derived, not authored. - Failing on external URLs that are merely slow. A 429 or timeout is not a dead link. Only 404/410/DNS failure are critical.
- Over-flagging version info. A doc pinned to an older version on purpose is info, not warning. The severity distinction matters.
- Assuming feature names are unique. When a feature claim doesn't match code, say "may be named differently" — don't assume absence.
- Checking anchors inside the same doc.
#sectionlinks to anchors the doc itself defines; checking those is useful but low value and fragile. Skip unless the user specifically asks.
Done when
- Every target doc has been read and all link/command/feature/version claims extracted.
- Each claim has been cross-checked against repo state or the network.
- The findings table is complete, ranked, and includes file:line for every row.
- Nothing in the repo was modified.
Not for
- Writing or fixing docs — use /ai-write for authoring, /ai-edit or manual edits for fixes.
- Reviewing prose quality or style — use /ai-review or /humanizer.
- Diagnosing a runtime failure that happens to mention docs — use /ai-debug.
- External research about a topic the docs cover — use /ai-research.
Files (ai-engineering)
-
SKILL.md 6 KB
--- name: ai-audit-docs description: >- Audits docs against repo state for dead links, false claims, stale versions, and nonexistent commands. Input: paths or default docs/, README.md, AGENTS.md. Output: ranked findings only; no auto-fix. Trigger: "audit docs", "verify docs", or "documentation review". license: MIT --- # Audit docs against repo reality ## Lifecycle Lane: light Writes: nothing — report-only, never modifies files Read by: humans (consumed as a findings table) Dies: on completion Next: ai-verify (to validate fixes if user acts on findings) ## What it produces A ranked markdown table of findings. Every finding names the doc file, the line, the severity, and the broken claim. | Severity | Meaning | |----------|---------| | critical | Doc points users at something that does not exist — they will hit a wall | | warning | Claim is outdated or inaccurate but the doc is still roughly usable | | info | Cosmetic: stale version number, unused anchor, minor nit | ## Steps 1. **Determine scope.** If the user gave a path, audit that. Otherwise default to `README.md`, `AGENTS.md`, and any `docs/` directory at the repo root. Glob for `**/*.md` under docs if the directory exists. Never audit vendored or generated files (`node_modules/`, `dist/`, `*.generated.*`). 2. **Read every target doc.** Extract all of these per file: - **File links:** markdown links pointing at local paths (e.g. `./path`, `../path`, `path/to/file`). Skip anchors (`#section`) and bare URLs — those are checked in step 3. - **External URLs:** bare `http://` / `https://` links. Collect for batch HEAD-check. - **Command references:** inline code or code blocks containing shell commands (`npm run X`, `yarn X`, `make X`, `cargo X`, `python -m X`, `npx X`). Strip arguments; extract just the command name. - **Version strings:** `v1.2.3`, `1.2.3`, `>=1.2.3`, pinned deps in prose. - **Feature claims:** sentences that assert existence — "supports X", "includes X", "has X", "provides X", "X is built-in", "X is enabled by default". 3. **Cross-check file links.** For each local path extracted from a markdown link: - Resolve it relative to the doc file's directory. - `find` or `read` to confirm the target exists. If not -> **critical** finding. 4. **Cross-check command references.** For each command name: - `grep` the repo for a script/package-bin definition (`package.json` scripts, `Makefile` targets, `Cargo.toml` bin, `setup.py`/`pyproject.toml` entry points, shell scripts in `bin/` or `scripts/`). - If no definition exists AND the command is not a well-known system binary (`git`, `ls`, `cat`, `curl`, `jq`, etc.) -> **warning** finding. - If a script IS defined, check that the arguments the doc uses still match the script's actual flags — flag mismatches -> **info**. 5. **Cross-check feature claims.** For each feature assertion: - `find` the feature name (the noun, not the sentence) in the codebase: config keys, source files, type definitions, exported symbols. - If nothing matches -> **warning** (the feature may exist under a different name; the finding should note "searched for X, found nothing; may be named differently"). - If the feature exists but is behind a flag the doc doesn't mention -> **info**. 6. **Check version strings.** For each version reference found in prose: - Compare against the repo's actual version: `package.json` version field, `Cargo.toml`, `pyproject.toml`, `version.txt`, etc. - If the doc version is older -> **info** (may be intentional; flag as potentially stale). - If the doc references a version that is *higher* than what exists -> **warning**. 7. **Batch-check external URLs.** For each external URL: - HEAD request (timeout 5s). 404/410/DNS failure -> **critical** (dead link). - 403/429/connection refused -> **info** (may be rate-limited or geo-blocked; note it but do not flag as dead). - 301/302 -> follow one hop; if the final target is the same domain, note the redirect as **info**. 8. **Rank and output.** Sort findings by severity (critical first), then by file path. Output as a markdown table: ``` | # | Severity | File | Line | Finding | |---|----------|------|------|---------| | 1 | critical | README.md | 42 | Link `./docs/setup.md` -> file does not exist | | 2 | warning | AGENTS.md | 15 | Command `yarn deploy` not defined in any script | | 3 | info | README.md | 8 | Version `v2.1.0` in doc vs `v2.3.0` in package.json | ``` End with a one-line summary: `X critical, Y warnings, Z info — total N findings`. ## Anti-patterns - **Fixing anything.** This skill is report-only. Never edit docs; the user decides what to fix. - **Auditing generated files.** `dist/`, `build/`, `node_modules/`, `*.generated.*` are out of scope — their content is derived, not authored. - **Failing on external URLs that are merely slow.** A 429 or timeout is not a dead link. Only 404/410/DNS failure are critical. - **Over-flagging version info.** A doc pinned to an older version on purpose is info, not warning. The severity distinction matters. - **Assuming feature names are unique.** When a feature claim doesn't match code, say "may be named differently" — don't assume absence. - **Checking anchors inside the same doc.** `#section` links to anchors the doc itself defines; checking those is useful but low value and fragile. Skip unless the user specifically asks. ## Done when - Every target doc has been read and all link/command/feature/version claims extracted. - Each claim has been cross-checked against repo state or the network. - The findings table is complete, ranked, and includes file:line for every row. - Nothing in the repo was modified. ## Not for - Writing or fixing docs — use /ai-write for authoring, /ai-edit or manual edits for fixes. - Reviewing prose quality or style — use /ai-review or /humanizer. - Diagnosing a runtime failure that happens to mention docs — use /ai-debug. - External research about a topic the docs cover — use /ai-research.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.