Claude Skill

diff-skill

Compare two versions of a skill's instructions to detect degradations and complexity increases. TRIGGER when: user asks to diff, compare, or review changes to a skill's markdown instructions. Asymmetric check — only regressions count. Additions, clarifications, and strengthenings

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download serpro69-claude-toolbox-kodex-plugin_skills_diff-skill-4b66a41.zip · 12 KB
Part of serpro69/claude-toolbox — 15 skills

Install

skills CLI npx skills add https://github.com/serpro69/claude-toolbox/tree/master/kodex-plugin/skills/diff-skill
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install serpro69-claude-toolbox@llmmart
Git git clone https://github.com/serpro69/claude-toolbox.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole serpro69/claude-toolbox collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Diff Skill Instructions

Goal: Detect whether an edit to a skill's markdown instructions introduced degradations or made the skill harder for an LLM to follow.

Two judgment axes:

  1. Degradation (asymmetric) — load-bearing instruction dropped, constraint weakened, verification step removed, scope narrowed, required output dropped, or reference broken. Content relocated but preserved is neutral.
  2. Complexity — split into regressions (introduced or worsened by the edit, affects verdict) and pre-existing advisories (exist regardless of the edit, surfaced as improvement opportunities, does NOT affect verdict).

Conventions

Required Outputs

  • Report written to docs/reviews/diff-skill/<slug>-<sha-a>-<sha-b>.md
  • Inline summary presented (under 10 lines: verdict, finding counts, report path)
  • Capy indexed under kk:review-findings if any degradation or complexity regression findings exist (skip on clean results and pre-existing advisories)

Invocation

$kk:diff-skill <skill-name>

Convenience shortcut: resolves klaude-plugin/skills/<skill-name>/SKILL.md. If not found, asks the user for the skill directory path.

Defaults: HEAD (before) → working tree (after). User can specify different refs in the prompt.

Workflow

Mandatory order — load instructions before acting on subject matter. The phases below are strictly sequential. Do not read skill content, build file sets, or produce judgments until you have loaded and internalized the full process file. The only early contact with the target skill is its name and SKILL.md path — enough to validate, not enough to judge.

See diff-process.md for the detailed step-by-step process.

Files (claude-toolbox)
  • evals
    • clean-refactor
      • test-files
        • after
          • review-process.md 758 B
            ### Validation Rules
            
            You MUST check every configuration file for:
            - Required fields present (`name`, `version`, `entrypoint`)
            - No deprecated fields (`legacy_mode`, `compat_shim`)
            - Version string matches semver format
            
            ### Error Handling
            
            When a validation error is found:
            1. Record the file path, field name, and violation type
            2. Continue checking remaining files — do not stop on first error
            3. Group errors by severity: blocking vs. warning
            
            ### Phases
            
            1. **Load schemas** — read all configuration schemas
            2. **Scan files** — validate each target file against the rules above
            3. **Classify errors** — group by severity (blocking vs. warning)
            4. **Write report** — output to `docs/reports/`
            5. **Present summary** — inline pass/fail counts
            
          • SKILL.md 735 B
            ---
            name: review-tool
            description: Reviews tool configurations for correctness.
            ---
            <!-- codex: tool-name mapping applied. See .codex/scripts/session-start.sh -->
            
            # Review Tool
            
            **Goal: Validate tool configuration files against the project schema.**
            
            ## Conventions
            
            Follow standard review conventions.
            
            ## Required Outputs
            
            - [ ] Validation report written to `docs/reports/`
            - [ ] Inline summary with pass/fail counts
            - [ ] Blocking errors indexed under `kk:review-findings`
            
            ## Workflow
            
            **Mandatory order — load rules before validating.** Read all configuration schemas before examining any target files.
            
            See [review-process.md](review-process.md) for validation rules, error handling, and the detailed phase-by-phase workflow.
            
        • before
          • SKILL.md 1.1 KB
            ---
            name: review-tool
            description: Reviews tool configurations for correctness.
            ---
            <!-- codex: tool-name mapping applied. See .codex/scripts/session-start.sh -->
            
            # Review Tool
            
            **Goal: Validate tool configuration files against the project schema.**
            
            ## Conventions
            
            Follow standard review conventions.
            
            ## Validation Rules
            
            You MUST check every configuration file for:
            - Required fields present (`name`, `version`, `entrypoint`)
            - No deprecated fields (`legacy_mode`, `compat_shim`)
            - Version string matches semver format
            
            ## Error Handling
            
            When a validation error is found:
            1. Record the file path, field name, and violation type
            2. Continue checking remaining files — do not stop on first error
            3. Group errors by severity: blocking vs. warning
            
            ## Required Outputs
            
            - [ ] Validation report written to `docs/reports/`
            - [ ] Inline summary with pass/fail counts
            - [ ] Blocking errors indexed under `kk:review-findings`
            
            ## Workflow
            
            **Mandatory order — load rules before validating.** Read all configuration schemas before examining any target files.
            
            Phases: load schemas → scan files → classify errors → write report → present summary.
            
      • eval.json 1.5 KB
        {
          "id": 2,
          "name": "clean-refactor",
          "description": "Two sections (Validation Rules, Error Handling) extracted from SKILL.md into a new review-process.md with links updated. No substance lost — should report no degradations.",
          "skills": ["diff-skill"],
          "prompt": "Run $kk:diff-skill on the review-tool skill, comparing the before/ and after/ directories in test-files/.",
          "trap": "Model flags the extraction as a degradation because the sections were 'removed' from SKILL.md, instead of recognizing that the substance was relocated to review-process.md and is still reachable via the updated link.",
          "files": [
            "test-files/before/SKILL.md",
            "test-files/after/SKILL.md",
            "test-files/after/review-process.md"
          ],
          "assertions": [
            { "id": "2.1", "text": "No degradations reported: the report does NOT flag the extraction of Validation Rules and Error Handling as degradations — their substance survived in review-process.md." },
            { "id": "2.2", "text": "Relocation recognized: the report or neutral-changes section acknowledges that content was moved from SKILL.md to review-process.md." },
            { "id": "2.3", "text": "Verdict is 'No issues' or complexity-only — NOT 'Degraded'. The extraction did not lose substance." },
            { "id": "2.4", "text": "Pre-existing complexity advisory is acceptable: if the report flags structural issues in the after-state (e.g., the Phases list being a flat enumeration), that is fine — but these must appear in the Pre-existing Complexity section, not as regressions." }
          ]
        }
        
    • known-degradation
      • test-files
        • after
          • example-process.md 555 B
            ### Workflow
            
            **Phase 1: Load input**
            
            Read the target files. You SHOULD validate that each file exists before processing.
            
            **Verify:** All target files confirmed present.
            
            **Phase 2: Analyze**
            
            Compare each file against the conformance rules. Flag violations with file path and line number.
            
            **Verify:** Every violation has a file path and line reference.
            
            **Phase 3: Write report**
            
            Write findings to `docs/reports/<name>-<date>.md`. Include a summary section and a detailed findings section.
            
            **Verify:** Report file exists and contains both sections.
            
          • shared-conventions.md 160 B
            # Shared Conventions
            
            Follow the project's standard naming and formatting rules when producing output files.
            
            See [style-guide.md](style-guide.md) for details.
            
          • SKILL.md 754 B
            ---
            name: example-skill
            description: An example skill for testing diff-skill evaluation.
            ---
            <!-- codex: tool-name mapping applied. See .codex/scripts/session-start.sh -->
            
            # Example Skill
            
            **Goal: Validate input data and produce a conformance report.**
            
            ## Conventions
            
            Read shared conventions at [shared-conventions.md](shared-conventions.md).
            
            ## Required Outputs
            
            - [ ] Conformance report written to `docs/reports/`
            - [ ] Inline summary presented to the user
            
            ## Workflow
            
            **Mandatory order — load instructions before acting.** The phases below are strictly sequential. You MUST read all referenced process files before examining any input data. Do not skip verification steps.
            
            See [example-process.md](example-process.md) for the detailed steps.
            
        • before
          • example-process.md 553 B
            ### Workflow
            
            **Phase 1: Load input**
            
            Read the target files. You MUST validate that each file exists before processing.
            
            **Verify:** All target files confirmed present.
            
            **Phase 2: Analyze**
            
            Compare each file against the conformance rules. Flag violations with file path and line number.
            
            **Verify:** Every violation has a file path and line reference.
            
            **Phase 3: Write report**
            
            Write findings to `docs/reports/<name>-<date>.md`. Include a summary section and a detailed findings section.
            
            **Verify:** Report file exists and contains both sections.
            
          • shared-conventions.md 109 B
            # Shared Conventions
            
            Follow the project's standard naming and formatting rules when producing output files.
            
          • SKILL.md 804 B
            ---
            name: example-skill
            description: An example skill for testing diff-skill evaluation.
            ---
            <!-- codex: tool-name mapping applied. See .codex/scripts/session-start.sh -->
            
            # Example Skill
            
            **Goal: Validate input data and produce a conformance report.**
            
            ## Conventions
            
            Read shared conventions at [shared-conventions.md](shared-conventions.md).
            
            ## Required Outputs
            
            - [ ] Conformance report written to `docs/reports/`
            - [ ] Inline summary presented to the user
            - [ ] Failures indexed under `kk:review-findings`
            
            ## Workflow
            
            **Mandatory order — load instructions before acting.** The phases below are strictly sequential. You MUST read all referenced process files before examining any input data. Do not skip verification steps.
            
            See [example-process.md](example-process.md) for the detailed steps.
            
      • eval.json 1.9 KB
        {
          "id": 1,
          "name": "known-degradation",
          "description": "Three deliberate degradations: a MUST weakened to SHOULD, a required-output bullet removed, and a newly added link to a non-existent file (style-guide.md). All three should be flagged.",
          "skills": ["diff-skill"],
          "prompt": "Run $kk:diff-skill on the example-skill, comparing the before/ and after/ directories in test-files/.",
          "trap": "Model treats the MUST→SHOULD weakening as a minor style edit rather than a constraint degradation, or misses the removed required-output bullet because the surrounding list still looks complete, or ignores the broken link to style-guide.md because it was added (not removed) — but it references a file that does not exist.",
          "files": [
            "test-files/before/SKILL.md",
            "test-files/before/example-process.md",
            "test-files/before/shared-conventions.md",
            "test-files/after/SKILL.md",
            "test-files/after/example-process.md",
            "test-files/after/shared-conventions.md"
          ],
          "assertions": [
            { "id": "1.1", "text": "Degradation — constraint weakened: the report flags that 'MUST validate' was weakened to 'SHOULD validate' in example-process.md Phase 1 as a degradation." },
            { "id": "1.2", "text": "Degradation — required output dropped: the report flags that the 'Failures indexed under kk:review-findings' bullet was removed from SKILL.md Required Outputs." },
            { "id": "1.3", "text": "Degradation — broken reference: the report flags that shared-conventions.md now links to style-guide.md which does not exist in the after-state." },
            { "id": "1.4", "text": "Verdict includes 'Degraded' — the summary verdict names degradation (not 'No issues' or complexity-only)." },
            { "id": "1.5", "text": "No false positives on neutral content: the report does NOT flag the unchanged workflow structure, phase headings, or verify steps as degradations." }
          ]
        }
        
  • diff-process.md 8.3 KB
    ### Workflow
    
    Copy this checklist and check off items as you complete them:
    
    ```
    Task Progress:
    - [ ] Phase 1: Parse invocation
    - [ ] Phase 2: Validate
    - [ ] Phase 3: Build reachable file sets
    - [ ] Phase 4: Judgment
    - [ ] Phase 5: Write report
    - [ ] Phase 6: Present inline summary
    - [ ] Phase 7: Index to capy
    ```
    
    **Input:** A skill name or skill directory path, optionally with comparison refs.
    
    **Output:** A report file at `docs/reviews/diff-skill/<slug>-<sha-a>-<sha-b>.md` and an inline summary.
    
    ---
    
    **Phase 1: Parse invocation**
    
    Extract `<skill-name>` from the user's prompt.
    
    Locate SKILL.md:
    - Try `klaude-plugin/skills/<skill-name>/SKILL.md` as a convenience shortcut
    - If not found, ask the user for the path to the skill directory
    - Normalize any user-provided path to repo-relative: strip the leading absolute prefix up to the repo root. Reject paths outside the repo — `git show` and `git cat-file` require repo-relative paths
    
    Determine the **report slug**: read SKILL.md's YAML frontmatter `name` field if present, otherwise use the parent directory basename of the SKILL.md path.
    
    Determine **comparison refs**: default is `HEAD` (before) → working tree (after). "Before" is the baseline; "after" is what's being judged. If the user specifies different refs, use those.
    
    **Verify:** SKILL.md path is resolved and both refs are determined.
    
    ---
    
    **Phase 2: Validate**
    
    For each git ref (not the working tree):
    - Run `git rev-parse <ref>` to confirm the ref resolves
    
    Confirm SKILL.md exists at both sides:
    - For git refs: `git cat-file -e <ref>:<repo-relative-path>`
    - For the working tree: check the file exists on disk
    
    If either side is missing, stop with a clear error message naming which ref and path failed.
    
    **Verify:** Both refs resolve and SKILL.md exists at both.
    
    ---
    
    **Phase 3: Build reachable file sets**
    
    For each ref, build the set of all files transitively reachable via markdown links from SKILL.md.
    
    **Algorithm:**
    
    1. Initialize `frontier = [SKILL.md path]`, `visited = {}`, `missing_links = []`
    2. Pop a file from frontier, add to visited
    3. Retrieve content:
       - For git refs: check `git ls-tree <ref> <path>` first
         - Mode `120000` (symlink): read the target path with `git cat-file -p <ref>:<path>`, resolve it relative to the symlink's directory, then retrieve the resolved target at the same ref (recurse if the target is also a symlink)
         - Otherwise: `git show <ref>:<path>` returns content directly
       - For working tree: use the `Read` tool (follows symlinks transparently)
    4. Extract markdown links from the content:
       - Match `[text](relative/path.md)` and `[text](relative/path.md#anchor)` patterns
       - Strip fragment identifiers (`#anchor`) before resolution — `[text](file.md#section)` resolves to `file.md`
       - Skip external URLs (`http://`, `https://`)
       - Skip anchor-only refs (`#section`)
       - Skip links inside fenced code blocks (triple-backtick or triple-tilde, with or without info strings) — use best-effort fence detection; don't build an elaborate parser
    5. Resolve each extracted path relative to the containing file's directory
    6. For each resolved path:
       - If it exists at the ref and is not in visited → add to frontier
       - If it does NOT exist at the ref → add to `missing_links` as `{source_file, raw_href, resolved_path}`
    7. Repeat until frontier is empty
    
    After building both sets, estimate total combined content size (both sides together). If it exceeds ~100KB, warn the user with the size and file count, and offer to proceed or narrow scope (e.g., compare only changed files). This fails loud about context-window risk without refusing to run.
    
    **Verify:** Two file sets produced, one per ref. Each set maps `{path → content}`. A `missing_links` list per ref tracks `{source_file, raw_href, resolved_path}` for broken references.
    
    ---
    
    **Phase 4: Judgment**
    
    Present yourself with the full content from both sides and judge across three axes. The judgment is **asymmetric** — additions, clarifications, and strengthenings are NOT degradations.
    
    **Input to judgment:**
    1. Full content of all reachable files at ref-a ("before" state)
    2. Full content of all reachable files at ref-b ("after" state)
    3. Files present on only one side
    4. Missing links per ref (broken references are a core degradation signal)
    
    **Degradation axis** (affects verdict):
    
    Only these count as degradations:
    - Load-bearing instruction dropped
    - Constraint weakened (`MUST` → `SHOULD`, `NEVER` → `AVOID`)
    - Verification step removed
    - Scope narrowed (fewer cases covered)
    - Required output dropped
    - Reference broken — a link whose target does not exist in the after-state. This covers two cases: a previously-resolving link whose target was removed, and a newly-added link whose target never existed. Missing links present in both states are pre-existing advisories, not degradations
    
    Content that was **relocated** (moved between files, inlined elsewhere, extracted to a new file) is neutral — check whether the substance survived. If yes, neutral. If substance was lost in the move, degradation.
    
    **Complexity regression axis** (affects verdict):
    
    Compare both states. Flag complexity that was *introduced or worsened* by the edit:
    - Instruction density — too many rules crammed into one section
    - Cross-turn state burden — gates and sub-phase state the LLM must track across multiple messages
    - Inline multi-path logic — branching workflows that could be extracted into dedicated files
    - Contradictory or tension-creating instructions — rules pulling in opposite directions without clear precedence
    - Deep reference chains — files linking to files linking to files, inflating the working set
    - Ambiguous conditional logic — gates without clear evaluation criteria
    
    **Pre-existing complexity advisory** (does NOT affect verdict):
    
    Separately, flag structural issues in the after-state that exist regardless of whether the edit caused them. These are improvement opportunities — surfaced for the author's benefit but kept in a distinct report section.
    
    For each finding: state what was found, where (file and section), and why it matters. Classify each as degradation, complexity regression, or pre-existing advisory.
    
    **Verdict** (combining both axes):
    - **No issues** — no degradation, no complexity regressions
    - **Degraded** — content was lost or weakened
    - **Complexity regressed** — no content lost, but the edit made the skill harder to follow
    - **Both** — degradation and complexity regression
    
    Pre-existing complexity advisories appear in the report regardless of the verdict but do not influence it.
    
    **Verify:** Each finding has a clear what/where/why description and is classified. A verdict is determined.
    
    ---
    
    **Phase 5: Write report**
    
    Create the report directory if absent:
    ```bash
    mkdir -p docs/reviews/diff-skill
    ```
    
    Compute the filename: `<report-slug>-<short-sha-a>-<short-sha-b>.md`
    - Short SHA = 7 characters from `git rev-parse --short=7 <ref>`
    - Working-tree side → `WORKTREE`
    
    Write the report using the `Write` tool, following this structure:
    
    ```markdown
    # diff-skill: <name> (<ref-a> → <ref-b>)
    
    ## Summary
    <one-paragraph verdict combining both axes>
    
    ## Degradations
    <each finding: what was lost/weakened, where, and why it matters>
    
    ## Complexity Regressions
    <each finding: what became harder to follow due to this edit, and how it could be simplified>
    
    ## Pre-existing Complexity
    <improvement opportunities that exist in the after-state regardless of this edit>
    
    ## Neutral Changes
    <brief list of changes that are neither degradations nor complexity issues>
    ```
    
    Omit sections that have no findings (except Summary, which is always present).
    
    **Verify:** Report file exists at the expected path and follows the template structure.
    
    ---
    
    **Phase 6: Present inline summary**
    
    Print a short block in the conversation — under 10 lines. Include:
    - The verdict
    - Finding counts per axis (e.g., "2 degradations, 1 complexity regression, 3 pre-existing advisories")
    - The path to the full report file
    
    **Verify:** Summary is under 10 lines and contains the report file path.
    
    ---
    
    **Phase 7: Index to capy**
    
    If any degradation or complexity regression findings exist, index a concise summary under source label `kk:review-findings`. Follow the protocol in [shared-capy-knowledge-protocol.md](shared-capy-knowledge-protocol.md).
    
    Skip indexing on clean results — nothing durable to record.
    
    **Verify:** Capy index call made if findings exist, skipped if clean.
    
  • shared-capy-knowledge-protocol.md 1.8 KB
    # Capy Knowledge Base Protocol
    
    If `capy` MCP tools are not available in this session, skip all search and index steps below and proceed normally.
    
    ## Source Label Taxonomy
    
    All plugin-managed labels use the `kk:` namespace prefix.
    
    | Label                    | Contents                                                              |
    | ------------------------ | --------------------------------------------------------------------- |
    | `kk:arch-decisions`      | Architecture decisions, design rationale, trade-offs                  |
    | `kk:review-findings`     | Code review patterns, recurring issues, anti-patterns                 |
    | `kk:lang-idioms`         | Language best practices, idiomatic patterns from external sources     |
    | `kk:project-conventions` | Discovered project patterns, naming conventions, structural decisions |
    | `kk:test-patterns`       | Testing approaches, edge cases, test infrastructure decisions         |
    | `kk:debug-context`       | Root causes, tricky bugs and their fixes, environment gotchas         |
    
    ## Search Conventions
    
    - Use 2-4 specific terms per query — not vague keywords
    - Always scope with `source` filter to relevant `kk:*` labels
    - Use `source: "kk:"` only for broad cross-domain searches (e.g., CoVe verification)
    - Default `limit: 3` per query unless more context is needed
    - **Cold-start fallback:** If no results, proceed with standard guidelines — empty results are normal for new projects
    
    ## Index Conventions
    
    - Only index non-obvious learnings not derivable from reading the code or git history
    - Keep content concise — summarize the insight, don't dump raw output
    - Always use a `kk:` prefixed label from the taxonomy above
    - One concept per `capy_index` call — don't bundle unrelated learnings
    - Skip indexing if the insight is already captured in design docs or CLAUDE.md
    
  • SKILL.md 2.4 KB
    ---
    name: diff-skill
    description: |
      Compare two versions of a skill's instructions to detect degradations and complexity increases.
      TRIGGER when: user asks to diff, compare, or review changes to a skill's markdown instructions.
      Asymmetric check — only regressions count. Additions, clarifications, and strengthenings are not degradations.
      Compares HEAD → working tree by default. Walks the full reachable file set from SKILL.md via markdown links.
      Produces a report file under docs/reviews/diff-skill/ and an inline summary.
    ---
    <!-- codex: tool-name mapping applied. See .codex/scripts/session-start.sh -->
    
    # Diff Skill Instructions
    
    **Goal: Detect whether an edit to a skill's markdown instructions introduced degradations or made the skill harder for an LLM to follow.**
    
    Two judgment axes:
    
    1. **Degradation (asymmetric)** — load-bearing instruction dropped, constraint weakened, verification step removed, scope narrowed, required output dropped, or reference broken. Content relocated but preserved is neutral.
    2. **Complexity** — split into _regressions_ (introduced or worsened by the edit, affects verdict) and _pre-existing advisories_ (exist regardless of the edit, surfaced as improvement opportunities, does NOT affect verdict).
    
    ## Conventions
    
    - **Read capy knowledge base conventions** at [shared-capy-knowledge-protocol.md](shared-capy-knowledge-protocol.md).
    
    ## Required Outputs
    
    - [ ] Report written to `docs/reviews/diff-skill/<slug>-<sha-a>-<sha-b>.md`
    - [ ] Inline summary presented (under 10 lines: verdict, finding counts, report path)
    - [ ] Capy indexed under `kk:review-findings` if any degradation or complexity regression findings exist (skip on clean results and pre-existing advisories)
    
    ## Invocation
    
    ```
    $kk:diff-skill <skill-name>
    ```
    
    Convenience shortcut: resolves `klaude-plugin/skills/<skill-name>/SKILL.md`. If not found, asks the user for the skill directory path.
    
    Defaults: `HEAD` (before) → working tree (after). User can specify different refs in the prompt.
    
    ## Workflow
    
    **Mandatory order — load instructions before acting on subject matter.** The phases below are strictly sequential. Do not read skill content, build file sets, or produce judgments until you have loaded and internalized the full process file. The only early contact with the target skill is its name and SKILL.md path — enough to validate, not enough to judge.
    
    See [diff-process.md](diff-process.md) for the detailed step-by-step process.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related