skill-graph-audit
Audit Skill() refs; detect hubs, isolates, and dangling targets. Use when auditing skills.
Install
npx skills add https://github.com/athola/claude-night-market/tree/master/plugins/abstract/skills/skill-graph-audit
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install athola-claude-night-market@llmmart
git clone https://github.com/athola/claude-night-market.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole athola/claude-night-market collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Skill Graph Audit
Overview
Build a directed graph of Skill(plugin:name) invocations across the
marketplace and surface composition patterns: which skills are heavily
referenced (hubs), which orchestrate many others (orchestrators), which
have no incoming or outgoing references (isolates), and which point at
non-existent skills (dangling references).
The federation graph is now derivable from source rather than hand-curated.
When To Use
- Before a documentation pass on skill composition
- After a renaming or retirement to catch broken
Skill()references - During quarterly audits to spot orphaned skills
- When evaluating consolidation candidates (hubs are higher-risk to merge)
- When a new skill's outbound references should be sanity-checked
When NOT To Use
- For per-skill quality scoring, use
Skill(abstract:skills-eval)instead - For frontmatter/structure validation, use
Skill(abstract:plugin-review) - For hook-specific audits, use
Skill(abstract:hooks-eval)
Quick Start
python3 plugins/abstract/scripts/skill_graph.py \
--plugins-root plugins --top-n 10
For machine-readable output:
python3 plugins/abstract/scripts/skill_graph.py \
--plugins-root plugins --format json --output reports/skill-graph.json
See modules/usage.md for full CLI reference and example workflows.
Core Outputs
| Output | Meaning | Action when high |
|---|---|---|
| Hubs | Most-referenced skills | Treat as core API; retire with extreme care |
| Orchestrators | Skills that call many others | Verify each ref still resolves |
| Isolates | Zero in / zero out | Check role: library? entrypoint? typo? |
| Dangling: bugs | Missing internal target | Fix immediately (typo or retired skill) |
| Dangling: external | Reference to external plugin | Document plugin dependency |
| Dangling: placeholders | Template text like -NAME |
Verify intentional |
See modules/interpretation.md for false-positive guidance and
isolation taxonomy.
Dogfood Evidence
This skill itself was scaffolded TDD-first; on first run against
plugins/, it caught two genuine dangling refs that the manual
audit (2026-04-25) had missed:
attune:makefile-generation -> abstract:makefile-dogfooder(script name confused with skill name)imbue:karpathy-principles -> spec-kit:speckit-clarify(command referenced as skill)
Both were converted to correct command-style references in the same session.
Verification
Two ways to validate the audit output is trustworthy:
- Test-suite correctness check: Run
pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.pyto confirm extraction, graph construction, ranking, isolate detection, and dangling-ref classification all pass on the current code. The-o addopts=flag bypasses the package-wide coverage gate, which would otherwise fail on a single-file run. - Round-trip smoke check: Note the dangling-ref count from a baseline run, fix one or more flagged references, then rerun and verify the count drops by at least the number fixed. If the count does not move, the report is stale or the regex missed a syntax variant.
Exit Criteria
- The graph builds:
skill_graph.pyruns againstplugins/without error and emits a node/edge count. - Dangling references are classified into bugs, external, and
placeholders (the three
Core Outputsrows resolve). - Every
Dangling: bugsentry is either fixed in the same session or filed as a tracked issue. -
pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.pypasses. - The round-trip smoke check shows the dangling-ref count drops by at least the number of references fixed.
Related Skills
Skill(abstract:skills-eval): per-skill quality scoringSkill(abstract:plugin-review): plugin manifest and structureSkill(abstract:hooks-eval): hook-specific validationSkill(abstract:rules-eval): rules directory validation
References
- Implementation:
plugins/abstract/scripts/skill_graph.py - Tests:
plugins/abstract/tests/scripts/test_skill_graph.py - Composition documentation:
docs/quality-gates.md#skill-level-quality-gate-composition - Skill role taxonomy:
docs/skill-integration-guide.md#skill-role-taxonomy
Files (claude-night-market)
-
modules
-
interpretation.md 3.7 KB
--- name: skill-graph-audit-interpretation description: How to interpret graph metrics, including isolate taxonomy and false-positive guidance. --- # Interpreting Graph Metrics ## Isolate Taxonomy A skill flagged as "isolate" (zero inbound, zero outbound) is not necessarily broken. Per `docs/skill-integration-guide.md#skill-role-taxonomy`, three legitimate roles produce zero edges: ### 1. Library skills Skills consumed via `dependencies:` frontmatter from other skills or via Python imports rather than `Skill()` calls. Example: `abstract:skills-eval`, which `rules-eval` and `plugin-review` list under `dependencies:`. **Action**: confirm `dependencies:` field in callers. ### 2. Entrypoint skills Skills invoked directly by users via slash commands or by an external orchestrator (e.g. `egregore:summon`). Example: `abstract:plugin-review`. **Action**: confirm a corresponding command file exists in `plugins/<plugin>/commands/`. ### 3. Hook-target skills Skills that hooks redirect to. Example: `imbue:proof-of-work`. **Action**: confirm a `PreToolUse`/`PostToolUse` hook in `plugins/<plugin>/hooks.json` references the skill. A skill that fits none of the three is a true orphan and a candidate for retirement. ## Hub Sensitivity Skills with high inbound count are load-bearing. Before retiring or splitting one: - Run `rg "Skill\\(<plugin>:<name>\\)" plugins/` to enumerate callers - Open a deprecation issue with at least 30-day notice - Provide a migration target in the deprecation note The current top-5 hubs (as of 2026-04-25) are: 1. `scribe:slop-detector` 2. `attune:project-brainstorming` 3. `sanctum:git-workspace-review` 4. `attune:project-planning` 5. `attune:project-specification` ## Dangling Reference Triage | Class | Default action | |-------|----------------| | bugs | Fix in the same PR; do not merge with bugs > 0 | | external | Confirm external plugin is documented in plugin.json | | placeholders | Annotate with `<!-- template -->` to suppress | ## Cross-Plugin Coupling A high count of cross-plugin edges (src plugin != dst plugin) is healthy ecosystem behaviour, not a problem. A high count of intra-plugin edges (src plugin == dst plugin) suggests a plugin-internal federation worth documenting in the plugin's README. ## Neither Reference Count Is Trustworthy Alone An August 2026 triage of 209 skills ran the count two ways and got two different answers. Both are wrong, and the way each fails is what a reader of this skill's output needs: | Measure | Result | How it fails | |---------|-------:|--------------| | Qualified `plugin:skill` outside its own directory | 21 skills at zero | Undercounts. A router naming its targets by bare name scores all of them zero while they are demonstrably reachable | | Bare directory name as a whole word outside its own directory | 0 skills at zero | Overcounts, badly. Short names collide with ordinary English | The overcount is not marginal. `tome:papers` scored 10,571 "references", `tome:research` 5,315, `gauntlet:extract` 4,818. Those are occurrences of the words *papers*, *research* and *extract*. Read a zero as a prompt to look, never as a verdict. Before acting on one, check whether a router or a command names the skill by bare name, and whether the name is a common word. A skill whose name is a common English word cannot be measured this way at all, and the audit should say so instead of reporting a number. ## Common False Positives - Skill names in code blocks demonstrating example usage are still parsed. If documenting a hypothetical skill, use `<plugin>:<name>` without backticks or surround with `<!-- example -->`. - Skill names mentioned in `docs/decisions/` outside SKILL.md files are not parsed (only SKILL.md is the source of truth). -
usage.md 2 KB
--- name: skill-graph-audit-usage description: CLI reference and example workflows for the skill graph audit tool. --- # Usage Reference ## CLI Flags ```text python3 plugins/abstract/scripts/skill_graph.py [OPTIONS] --plugins-root PATH Root containing <plugin>/skills/<name>/ tree (default: plugins) --top-n INT Top N hubs/orchestrators to show (default: 10) --format {text,json} Output format (default: text) --output PATH Write to file instead of stdout ``` ## Common Workflows ### Pre-release dangling-ref check ```bash python3 plugins/abstract/scripts/skill_graph.py \ --plugins-root plugins --format json --output /tmp/graph.json python3 -c " import json report = json.load(open('/tmp/graph.json')) bugs = report['dangling_refs']['bugs'] if bugs: print(f'BLOCKING: {len(bugs)} dangling refs') for b in bugs: print(f' {b[\"source\"]} -> {b[\"target\"]}') raise SystemExit(1) print('OK: 0 internal dangling references') " ``` ### Find consolidation candidates Hubs with >5 inbound references are core API; orchestrators with >5 outbound references are coordination points. The intersection (hub AND orchestrator) is the federation backbone. ```bash python3 plugins/abstract/scripts/skill_graph.py --top-n 20 \ | tee /tmp/graph.txt ``` ### Update composition documentation Generate the federation table for `docs/quality-gates.md` from report JSON instead of curating manually. ## Updating External Plugin Allowlist If a new external plugin is referenced (one not yet in `KNOWN_EXTERNAL_PLUGINS`), update the constant in `plugins/abstract/scripts/skill_graph.py` so refs to it are classified as `external` rather than `bugs`. ## Limitations - Detects only `Skill(plugin:name)` invocations. Free-text mentions in prose are not parsed. - Self-references (a skill referencing itself) are skipped to avoid cycles in counts. - Module-level `dependencies:` and `modules:` frontmatter are not yet treated as edges; see backlog item for planned extension.
-
-
SKILL.md 4.5 KB
--- name: skill-graph-audit role: library description: 'Audit Skill() refs; detect hubs, isolates, and dangling targets. Use when auditing skills.' category: skill-management alwaysApply: false tools: [] modules: - modules/usage.md - modules/interpretation.md --- # Skill Graph Audit ## Overview Build a directed graph of `Skill(plugin:name)` invocations across the marketplace and surface composition patterns: which skills are heavily referenced (hubs), which orchestrate many others (orchestrators), which have no incoming or outgoing references (isolates), and which point at non-existent skills (dangling references). The federation graph is now derivable from source rather than hand-curated. ## When To Use - Before a documentation pass on skill composition - After a renaming or retirement to catch broken `Skill()` references - During quarterly audits to spot orphaned skills - When evaluating consolidation candidates (hubs are higher-risk to merge) - When a new skill's outbound references should be sanity-checked ## When NOT To Use - For per-skill quality scoring, use `Skill(abstract:skills-eval)` instead - For frontmatter/structure validation, use `Skill(abstract:plugin-review)` - For hook-specific audits, use `Skill(abstract:hooks-eval)` ## Quick Start ```bash python3 plugins/abstract/scripts/skill_graph.py \ --plugins-root plugins --top-n 10 ``` For machine-readable output: ```bash python3 plugins/abstract/scripts/skill_graph.py \ --plugins-root plugins --format json --output reports/skill-graph.json ``` See `modules/usage.md` for full CLI reference and example workflows. ## Core Outputs | Output | Meaning | Action when high | |--------|---------|------------------| | Hubs | Most-referenced skills | Treat as core API; retire with extreme care | | Orchestrators | Skills that call many others | Verify each ref still resolves | | Isolates | Zero in / zero out | Check role: library? entrypoint? typo? | | Dangling: bugs | Missing internal target | Fix immediately (typo or retired skill) | | Dangling: external | Reference to external plugin | Document plugin dependency | | Dangling: placeholders | Template text like `-NAME` | Verify intentional | See `modules/interpretation.md` for false-positive guidance and isolation taxonomy. ## Dogfood Evidence This skill itself was scaffolded TDD-first; on first run against `plugins/`, it caught two genuine dangling refs that the manual audit (2026-04-25) had missed: - `attune:makefile-generation -> abstract:makefile-dogfooder` (script name confused with skill name) - `imbue:karpathy-principles -> spec-kit:speckit-clarify` (command referenced as skill) Both were converted to correct command-style references in the same session. ## Verification Two ways to validate the audit output is trustworthy: 1. **Test-suite correctness check**: Run `pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py` to confirm extraction, graph construction, ranking, isolate detection, and dangling-ref classification all pass on the current code. The `-o addopts=` flag bypasses the package-wide coverage gate, which would otherwise fail on a single-file run. 2. **Round-trip smoke check**: Note the dangling-ref count from a baseline run, fix one or more flagged references, then rerun and verify the count drops by at least the number fixed. If the count does not move, the report is stale or the regex missed a syntax variant. ## Exit Criteria - [ ] The graph builds: `skill_graph.py` runs against `plugins/` without error and emits a node/edge count. - [ ] Dangling references are classified into bugs, external, and placeholders (the three `Core Outputs` rows resolve). - [ ] Every `Dangling: bugs` entry is either fixed in the same session or filed as a tracked issue. - [ ] `pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py` passes. - [ ] The round-trip smoke check shows the dangling-ref count drops by at least the number of references fixed. ## Related Skills - `Skill(abstract:skills-eval)`: per-skill quality scoring - `Skill(abstract:plugin-review)`: plugin manifest and structure - `Skill(abstract:hooks-eval)`: hook-specific validation - `Skill(abstract:rules-eval)`: rules directory validation ## References - Implementation: `plugins/abstract/scripts/skill_graph.py` - Tests: `plugins/abstract/tests/scripts/test_skill_graph.py` - Composition documentation: `docs/quality-gates.md#skill-level-quality-gate-composition` - Skill role taxonomy: `docs/skill-integration-guide.md#skill-role-taxonomy`
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.