Claude Skill

skill-graph-audit

Audit Skill() refs; detect hubs, isolates, and dangling targets. Use when auditing skills.

LLM Mart · 0 points · 6 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download athola-claude-night-market-plugins_abstract_skills_skill-graph-audit-9045831.zip · 5 KB
Part of athola/claude-night-market — 46 skills

Install

skills CLI npx skills add https://github.com/athola/claude-night-market/tree/master/plugins/abstract/skills/skill-graph-audit
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install athola-claude-night-market@llmmart
Git git clone https://github.com/athola/claude-night-market.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole athola/claude-night-market collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Skill Graph Audit

Overview

Build a directed graph of Skill(plugin:name) invocations across the marketplace and surface composition patterns: which skills are heavily referenced (hubs), which orchestrate many others (orchestrators), which have no incoming or outgoing references (isolates), and which point at non-existent skills (dangling references).

The federation graph is now derivable from source rather than hand-curated.

When To Use

  • Before a documentation pass on skill composition
  • After a renaming or retirement to catch broken Skill() references
  • During quarterly audits to spot orphaned skills
  • When evaluating consolidation candidates (hubs are higher-risk to merge)
  • When a new skill's outbound references should be sanity-checked

When NOT To Use

  • For per-skill quality scoring, use Skill(abstract:skills-eval) instead
  • For frontmatter/structure validation, use Skill(abstract:plugin-review)
  • For hook-specific audits, use Skill(abstract:hooks-eval)

Quick Start

python3 plugins/abstract/scripts/skill_graph.py \
  --plugins-root plugins --top-n 10

For machine-readable output:

python3 plugins/abstract/scripts/skill_graph.py \
  --plugins-root plugins --format json --output reports/skill-graph.json

See modules/usage.md for full CLI reference and example workflows.

Core Outputs

Output Meaning Action when high
Hubs Most-referenced skills Treat as core API; retire with extreme care
Orchestrators Skills that call many others Verify each ref still resolves
Isolates Zero in / zero out Check role: library? entrypoint? typo?
Dangling: bugs Missing internal target Fix immediately (typo or retired skill)
Dangling: external Reference to external plugin Document plugin dependency
Dangling: placeholders Template text like -NAME Verify intentional

See modules/interpretation.md for false-positive guidance and isolation taxonomy.

Dogfood Evidence

This skill itself was scaffolded TDD-first; on first run against plugins/, it caught two genuine dangling refs that the manual audit (2026-04-25) had missed:

  • attune:makefile-generation -> abstract:makefile-dogfooder (script name confused with skill name)
  • imbue:karpathy-principles -> spec-kit:speckit-clarify (command referenced as skill)

Both were converted to correct command-style references in the same session.

Verification

Two ways to validate the audit output is trustworthy:

  1. Test-suite correctness check: Run pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py to confirm extraction, graph construction, ranking, isolate detection, and dangling-ref classification all pass on the current code. The -o addopts= flag bypasses the package-wide coverage gate, which would otherwise fail on a single-file run.
  2. Round-trip smoke check: Note the dangling-ref count from a baseline run, fix one or more flagged references, then rerun and verify the count drops by at least the number fixed. If the count does not move, the report is stale or the regex missed a syntax variant.

Exit Criteria

  • The graph builds: skill_graph.py runs against plugins/ without error and emits a node/edge count.
  • Dangling references are classified into bugs, external, and placeholders (the three Core Outputs rows resolve).
  • Every Dangling: bugs entry is either fixed in the same session or filed as a tracked issue.
  • pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py passes.
  • The round-trip smoke check shows the dangling-ref count drops by at least the number of references fixed.

Related Skills

  • Skill(abstract:skills-eval): per-skill quality scoring
  • Skill(abstract:plugin-review): plugin manifest and structure
  • Skill(abstract:hooks-eval): hook-specific validation
  • Skill(abstract:rules-eval): rules directory validation

References

  • Implementation: plugins/abstract/scripts/skill_graph.py
  • Tests: plugins/abstract/tests/scripts/test_skill_graph.py
  • Composition documentation: docs/quality-gates.md#skill-level-quality-gate-composition
  • Skill role taxonomy: docs/skill-integration-guide.md#skill-role-taxonomy
Files (claude-night-market)
  • modules
    • interpretation.md 3.7 KB
      ---
      name: skill-graph-audit-interpretation
      description: How to interpret graph metrics, including isolate taxonomy and false-positive guidance.
      ---
      
      # Interpreting Graph Metrics
      
      ## Isolate Taxonomy
      
      A skill flagged as "isolate" (zero inbound, zero outbound) is not
      necessarily broken. Per `docs/skill-integration-guide.md#skill-role-taxonomy`, three legitimate
      roles produce zero edges:
      
      ### 1. Library skills
      
      Skills consumed via `dependencies:` frontmatter from other skills
      or via Python imports rather than `Skill()` calls. Example:
      `abstract:skills-eval`, which `rules-eval` and `plugin-review` list
      under `dependencies:`. **Action**: confirm `dependencies:` field
      in callers.
      
      ### 2. Entrypoint skills
      
      Skills invoked directly by users via slash commands or by an
      external orchestrator (e.g. `egregore:summon`). Example:
      `abstract:plugin-review`. **Action**: confirm a corresponding
      command file exists in `plugins/<plugin>/commands/`.
      
      ### 3. Hook-target skills
      
      Skills that hooks redirect to. Example: `imbue:proof-of-work`.
      **Action**: confirm a `PreToolUse`/`PostToolUse` hook in
      `plugins/<plugin>/hooks.json` references the skill.
      
      A skill that fits none of the three is a true orphan and a
      candidate for retirement.
      
      ## Hub Sensitivity
      
      Skills with high inbound count are load-bearing. Before retiring
      or splitting one:
      
      - Run `rg "Skill\\(<plugin>:<name>\\)" plugins/` to enumerate callers
      - Open a deprecation issue with at least 30-day notice
      - Provide a migration target in the deprecation note
      
      The current top-5 hubs (as of 2026-04-25) are:
      
      1. `scribe:slop-detector`
      2. `attune:project-brainstorming`
      3. `sanctum:git-workspace-review`
      4. `attune:project-planning`
      5. `attune:project-specification`
      
      ## Dangling Reference Triage
      
      | Class | Default action |
      |-------|----------------|
      | bugs | Fix in the same PR; do not merge with bugs > 0 |
      | external | Confirm external plugin is documented in plugin.json |
      | placeholders | Annotate with `<!-- template -->` to suppress |
      
      ## Cross-Plugin Coupling
      
      A high count of cross-plugin edges (src plugin != dst plugin) is
      healthy ecosystem behaviour, not a problem. A high count of
      intra-plugin edges (src plugin == dst plugin) suggests a
      plugin-internal federation worth documenting in the plugin's
      README.
      
      ## Neither Reference Count Is Trustworthy Alone
      
      An August 2026 triage of 209 skills ran the count two ways and got two
      different answers. Both are wrong, and the way each fails is what a
      reader of this skill's output needs:
      
      | Measure | Result | How it fails |
      |---------|-------:|--------------|
      | Qualified `plugin:skill` outside its own directory | 21 skills at zero | Undercounts. A router naming its targets by bare name scores all of them zero while they are demonstrably reachable |
      | Bare directory name as a whole word outside its own directory | 0 skills at zero | Overcounts, badly. Short names collide with ordinary English |
      
      The overcount is not marginal. `tome:papers` scored 10,571
      "references", `tome:research` 5,315, `gauntlet:extract` 4,818. Those
      are occurrences of the words *papers*, *research* and *extract*.
      
      Read a zero as a prompt to look, never as a verdict. Before acting on
      one, check whether a router or a command names the skill by bare name,
      and whether the name is a common word. A skill whose name is a common
      English word cannot be measured this way at all, and the audit should
      say so instead of reporting a number.
      
      ## Common False Positives
      
      - Skill names in code blocks demonstrating example usage are still
        parsed. If documenting a hypothetical skill, use `<plugin>:<name>`
        without backticks or surround with `<!-- example -->`.
      - Skill names mentioned in `docs/decisions/` outside SKILL.md files
        are not parsed (only SKILL.md is the source of truth).
      
    • usage.md 2 KB
      ---
      name: skill-graph-audit-usage
      description: CLI reference and example workflows for the skill graph audit tool.
      ---
      
      # Usage Reference
      
      ## CLI Flags
      
      ```text
      python3 plugins/abstract/scripts/skill_graph.py [OPTIONS]
      
        --plugins-root PATH    Root containing <plugin>/skills/<name>/ tree
                               (default: plugins)
        --top-n INT            Top N hubs/orchestrators to show (default: 10)
        --format {text,json}   Output format (default: text)
        --output PATH          Write to file instead of stdout
      ```
      
      ## Common Workflows
      
      ### Pre-release dangling-ref check
      
      ```bash
      python3 plugins/abstract/scripts/skill_graph.py \
        --plugins-root plugins --format json --output /tmp/graph.json
      
      python3 -c "
      import json
      report = json.load(open('/tmp/graph.json'))
      bugs = report['dangling_refs']['bugs']
      if bugs:
          print(f'BLOCKING: {len(bugs)} dangling refs')
          for b in bugs:
              print(f'  {b[\"source\"]} -> {b[\"target\"]}')
          raise SystemExit(1)
      print('OK: 0 internal dangling references')
      "
      ```
      
      ### Find consolidation candidates
      
      Hubs with >5 inbound references are core API; orchestrators with
      >5 outbound references are coordination points. The intersection
      (hub AND orchestrator) is the federation backbone.
      
      ```bash
      python3 plugins/abstract/scripts/skill_graph.py --top-n 20 \
        | tee /tmp/graph.txt
      ```
      
      ### Update composition documentation
      
      Generate the federation table for `docs/quality-gates.md` from
      report JSON instead of curating manually.
      
      ## Updating External Plugin Allowlist
      
      If a new external plugin is referenced (one not yet in
      `KNOWN_EXTERNAL_PLUGINS`), update the constant in
      `plugins/abstract/scripts/skill_graph.py` so refs to it are
      classified as `external` rather than `bugs`.
      
      ## Limitations
      
      - Detects only `Skill(plugin:name)` invocations. Free-text mentions
        in prose are not parsed.
      - Self-references (a skill referencing itself) are skipped to avoid
        cycles in counts.
      - Module-level `dependencies:` and `modules:` frontmatter are not
        yet treated as edges; see backlog item for planned extension.
      
  • SKILL.md 4.5 KB
    ---
    name: skill-graph-audit
    role: library
    description: 'Audit Skill() refs; detect hubs, isolates, and dangling targets. Use when auditing skills.'
    category: skill-management
    alwaysApply: false
    tools: []
    modules:
    - modules/usage.md
    - modules/interpretation.md
    ---
    
    # Skill Graph Audit
    
    ## Overview
    
    Build a directed graph of `Skill(plugin:name)` invocations across the
    marketplace and surface composition patterns: which skills are heavily
    referenced (hubs), which orchestrate many others (orchestrators), which
    have no incoming or outgoing references (isolates), and which point at
    non-existent skills (dangling references).
    
    The federation graph is now derivable from source rather than
    hand-curated.
    
    ## When To Use
    
    - Before a documentation pass on skill composition
    - After a renaming or retirement to catch broken `Skill()` references
    - During quarterly audits to spot orphaned skills
    - When evaluating consolidation candidates (hubs are higher-risk to merge)
    - When a new skill's outbound references should be sanity-checked
    
    ## When NOT To Use
    
    - For per-skill quality scoring, use `Skill(abstract:skills-eval)` instead
    - For frontmatter/structure validation, use `Skill(abstract:plugin-review)`
    - For hook-specific audits, use `Skill(abstract:hooks-eval)`
    
    ## Quick Start
    
    ```bash
    python3 plugins/abstract/scripts/skill_graph.py \
      --plugins-root plugins --top-n 10
    ```
    
    For machine-readable output:
    
    ```bash
    python3 plugins/abstract/scripts/skill_graph.py \
      --plugins-root plugins --format json --output reports/skill-graph.json
    ```
    
    See `modules/usage.md` for full CLI reference and example workflows.
    
    ## Core Outputs
    
    | Output | Meaning | Action when high |
    |--------|---------|------------------|
    | Hubs | Most-referenced skills | Treat as core API; retire with extreme care |
    | Orchestrators | Skills that call many others | Verify each ref still resolves |
    | Isolates | Zero in / zero out | Check role: library? entrypoint? typo? |
    | Dangling: bugs | Missing internal target | Fix immediately (typo or retired skill) |
    | Dangling: external | Reference to external plugin | Document plugin dependency |
    | Dangling: placeholders | Template text like `-NAME` | Verify intentional |
    
    See `modules/interpretation.md` for false-positive guidance and
    isolation taxonomy.
    
    ## Dogfood Evidence
    
    This skill itself was scaffolded TDD-first; on first run against
    `plugins/`, it caught two genuine dangling refs that the manual
    audit (2026-04-25) had missed:
    
    - `attune:makefile-generation -> abstract:makefile-dogfooder`
      (script name confused with skill name)
    - `imbue:karpathy-principles -> spec-kit:speckit-clarify`
      (command referenced as skill)
    
    Both were converted to correct command-style references in the
    same session.
    
    ## Verification
    
    Two ways to validate the audit output is trustworthy:
    
    1. **Test-suite correctness check**: Run `pytest -o addopts=
       plugins/abstract/tests/scripts/test_skill_graph.py` to confirm
       extraction, graph construction, ranking, isolate detection, and
       dangling-ref classification all pass on the current code. The
       `-o addopts=` flag bypasses the package-wide coverage gate, which
       would otherwise fail on a single-file run.
    2. **Round-trip smoke check**: Note the dangling-ref count from a
       baseline run, fix one or more flagged references, then rerun and
       verify the count drops by at least the number fixed. If the count
       does not move, the report is stale or the regex missed a syntax
       variant.
    
    ## Exit Criteria
    
    - [ ] The graph builds: `skill_graph.py` runs against `plugins/`
          without error and emits a node/edge count.
    - [ ] Dangling references are classified into bugs, external, and
          placeholders (the three `Core Outputs` rows resolve).
    - [ ] Every `Dangling: bugs` entry is either fixed in the same
          session or filed as a tracked issue.
    - [ ] `pytest -o addopts= plugins/abstract/tests/scripts/test_skill_graph.py`
          passes.
    - [ ] The round-trip smoke check shows the dangling-ref count drops
          by at least the number of references fixed.
    
    ## Related Skills
    
    - `Skill(abstract:skills-eval)`: per-skill quality scoring
    - `Skill(abstract:plugin-review)`: plugin manifest and structure
    - `Skill(abstract:hooks-eval)`: hook-specific validation
    - `Skill(abstract:rules-eval)`: rules directory validation
    
    ## References
    
    - Implementation: `plugins/abstract/scripts/skill_graph.py`
    - Tests: `plugins/abstract/tests/scripts/test_skill_graph.py`
    - Composition documentation:
      `docs/quality-gates.md#skill-level-quality-gate-composition`
    - Skill role taxonomy: `docs/skill-integration-guide.md#skill-role-taxonomy`
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related