Claude Skill

graph-evolution

Compares Trailmark code graphs at two source code snapshots (git commits, tags, or directories) to surface security-relevant structural changes. Detects new attack paths, complexity shifts, blast radius growth, taint propagation changes, and privilege boundary modifications that

LLM Mart · 0 points · 6 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download trailofbits-skills-plugins_trailmark_skills_graph-evolution-123037e.zip · 13 KB
trailofbits/skills 7234 616 forks CC-BY-SA-4.0 Updated 8h ago
Part of trailofbits/skills — 100 skills

Install

skills CLI npx skills add https://github.com/trailofbits/skills/tree/main/plugins/trailmark/skills/graph-evolution
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install trailofbits-skills@llmmart
Git git clone https://github.com/trailofbits/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole trailofbits/skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Graph Evolution

Builds Trailmark code graphs at two source snapshots and computes a structural diff. Surfaces security-relevant changes that text-level diffs miss: new attack paths, complexity shifts, blast radius growth, taint propagation changes, and privilege boundary modifications.

When to Use

  • Comparing two git refs to understand what structurally changed
  • Auditing a range of commits for security-relevant evolution
  • Detecting new attack paths created by code changes
  • Finding functions whose blast radius or complexity grew silently
  • Identifying taint propagation changes across refactors
  • Pre-release structural comparison (tag-to-tag or branch-to-branch)

When NOT to Use

  • Line-level code review (use differential-review for text-diff analysis)
  • Single-snapshot analysis (use the trailmark skill directly)
  • Diagram generation from a single snapshot (use the diagramming-code skill)
  • Mutation testing triage (use the genotoxic skill)

Rationalizations to Reject

Rationalization Why It's Wrong Required Action
"We just need the structural diff, skip pre-analysis" Without pre-analysis, you miss taint changes, blast radius growth, and privilege boundary shifts Run engine.preanalysis() on both snapshots
"Text diff covers what changed" Text diffs miss new attack paths, transitive complexity shifts, and subgraph membership changes Use structural diff to complement text diff
"Only added nodes matter" Removed security functions and shifted privilege boundaries are equally dangerous Review removals and modifications, not just additions
"Low-severity structural changes can be ignored" INFO-level changes (dead code removal) can mask removed security checks Classify every change, review removals for replaced functionality
"One snapshot's graph is enough for comparison" Single-snapshot analysis can't detect evolution — you need both before and after Always build and export both graphs
"Tool isn't installed, I'll compare manually" Manual comparison misses what graph analysis catches Install trailmark first
"The diff came back empty, so nothing changed structurally" trailmark diff defaults --language to python and exits 0 with empty arrays on any other target, so an empty diff reads identically whether the code is unchanged or the language was wrong Pass --language explicitly and re-run before concluding no change

Prerequisites

trailmark must be installed. If uv run trailmark fails, run:

uv tool install trailmark
# Python snippets: uv run --with trailmark python -   (a tool env is not importable)

DO NOT fall back to "manual comparison" or reading source files as a substitute for running trailmark. The tool must be installed and used programmatically. If installation fails, report the error.


Quick Start

# Compare two git refs (e.g., tags, branches, commits)
# 1. Build graphs at each snapshot
# 2. Run pre-analysis on both
# 3. Compute structural diff
# 4. Generate report

# Step-by-step: see Workflow below

Decision Tree

├─ Need to understand what each metric means?
│  └─ Read: references/evolution-metrics.md
│
├─ Need the report output format?
│  └─ Read: references/report-format.md
│
├─ Already have two graph JSON exports?
│  └─ Jump to Phase 3 (run native diff + graph_diff.py)
│
└─ Starting from two git refs?
   └─ Start at Phase 1

Workflow

Graph Evolution Progress:
- [ ] Phase 1: Create snapshots (git worktrees)
- [ ] Phase 2: Build graphs + pre-analysis on both snapshots
- [ ] Phase 3: Compute structural diff
- [ ] Phase 4: Interpret diff and generate report
- [ ] Phase 5: Clean up worktrees

Phase 1: Create Snapshots

Use git worktrees to get clean copies of each ref without disturbing the working tree.

# Create temp directories for worktrees
BEFORE_DIR=$(mktemp -d)
AFTER_DIR=$(mktemp -d)

# Create worktrees (run from repo root)
git worktree add "$BEFORE_DIR" {before_ref}
git worktree add "$AFTER_DIR" {after_ref}

If comparing two directories instead of git refs, skip this phase and use the directory paths directly in Phase 2.

Phase 2: Build Graphs and Run Pre-Analysis

Build Trailmark graphs for both snapshots and run pre-analysis on each. Pre-analysis computes blast radius, taint propagation, privilege boundaries, and entrypoint enumeration.

from trailmark.query.api import QueryEngine

def build_and_export(target_dir, output_path, language="auto"):
    """Build graph, run pre-analysis, export JSON."""
    engine = QueryEngine.from_directory(target_dir, language=language)
    engine.preanalysis()
    json_str = engine.to_json()
    with open(output_path, "w") as f:
        f.write(json_str)
    return engine.summary()

import tempfile, os
work_dir = tempfile.mkdtemp(prefix="trailmark_evolution_")
before_json = os.path.join(work_dir, "before_graph.json")
after_json = os.path.join(work_dir, "after_graph.json")

before_summary = build_and_export(
    "{before_dir}", before_json
)
after_summary = build_and_export(
    "{after_dir}", after_json
)

Verify both graphs built successfully by checking the summary output. If either fails, rerun with an explicit language or comma-separated list instead of auto.

Phase 3: Compute Structural Diff

Run both:

  1. Trailmark's native structural diff for nodes, edges, and entrypoints
  2. The plugin's graph_diff.py helper for subgraph membership changes

Use the same work_dir from Phase 2, and pass the same --language value Phase 2 built with. trailmark diff defaults that flag to python, so on any other target the default exits 0 and writes empty arrays rather than reporting a mismatch.

trailmark diff --json --language auto "{before_dir}" "{after_dir}" > "{work_dir}/trailmark_diff.json" || \
  uv run trailmark diff --json --language auto "{before_dir}" "{after_dir}" > "{work_dir}/trailmark_diff.json"

uv run {baseDir}/scripts/graph_diff.py \
    --before "{before_json}" \
    --after "{after_json}" > "{work_dir}/subgraph_diff.json"

If Phase 2 needed an explicit language or a comma-separated list instead of auto, use that same value here.

If either diff command fails or writes an empty JSON file, stop and report the error instead of continuing to Phase 4.

A trailmark_diff.json whose nodes, edges, and entrypoints arrays are all empty means either nothing changed structurally or both snapshots parsed to (near-)empty graphs. Decide which using Phase 2's graph summaries: if either snapshot's node count is zero or implausibly small for the target, the parse missed the code — name the language set explicitly (rust, solidity, python,rust) and re-run. Healthy node counts on both snapshots plus an empty diff is genuine structural stability.

The native Trailmark diff contains:

Key Contents
summary_delta Changes in node/edge/entrypoint counts
nodes.added New functions, classes, methods
nodes.removed Deleted functions, classes, methods
nodes.modified Functions with changed CC, params, line span
edges.added New call/inheritance/import relationships
edges.removed Deleted relationships
entrypoints Added, removed, and modified entrypoints

The subgraph diff contains:

Key Contents
subgraphs Per-subgraph membership changes (tainted, high_blast_radius, etc.)

Phase 4: Interpret Diff and Generate Report

Read both diff JSON files and generate a security-focused markdown report. See references/report-format.md for the full template.

Interpretation priorities (highest to lowest):

  1. New tainted paths — nodes entering the tainted subgraph, especially if they also appear in added edges targeting sensitive functions
  2. Privilege boundary changes — new or removed trust transitions from the native entrypoint/edge diff plus the subgraph diff
  3. Attack surface growth — new entrypoints, especially untrusted_external, from trailmark_diff.json
  4. Blast radius increases — nodes entering high_blast_radius
  5. Complexity spikes — CC increases > 3 on tainted or entrypoint-reachable nodes
  6. Structural additions — new nodes and edges (review needed)
  7. Structural removals — verify removed security functions were replaced

Cross-reference structural changes with git diff {before_ref}..{after_ref} to add source-level context to findings.

Severity classification:

Severity Structural Signal
CRITICAL New tainted path to sensitive function, removed auth boundary
HIGH New entrypoint + high blast radius, large CC increase on tainted node
MEDIUM New trust-boundary-crossing edges, moderate CC increase
LOW Added nodes without entrypoint reachability
INFO Dead code removal, complexity reductions

For detailed metric definitions, see references/evolution-metrics.md.

Phase 5: Clean Up

Remove git worktrees after the report is written:

git worktree remove "{before_dir}"
git worktree remove "{after_dir}"

Diff Reference

trailmark diff --json --language auto BEFORE AFTER
uv run {baseDir}/scripts/graph_diff.py [OPTIONS]

trailmark diff --language defaults to python. On a target in any other language that default still exits 0, emitting well-formed JSON with empty nodes, edges, and entrypoints arrays, so always pass the flag: auto detects and merges every supported language found under the target, and a single name (rust, solidity) or comma-separated list (python,rust) pins an explicit set. auto fails loudly with No supported languages detected under <path> when a snapshot holds nothing it can parse, which is the outcome you want. Confirm the language first; only then can an empty diff count as evidence that nothing changed.

Use trailmark diff for:

  • Node/edge changes
  • Added/removed/modified entrypoints
  • Human-readable structural diff reports

Use graph_diff.py for:

  • Subgraph membership changes derived from engine.preanalysis()
  • tainted, high_blast_radius, privilege_boundary, and related sets
Argument Default Description
--before required Path to the "before" graph JSON
--after required Path to the "after" graph JSON
--indent 2 JSON output indentation

graph_diff.py input format: Trailmark JSON exports from engine.to_json(). graph_diff.py output: JSON structural diff for nodes, edges, and subgraphs.


Quality Checklist

Before delivering the report:

  • Both graphs built successfully (check summaries)
  • Pre-analysis ran on both snapshots
  • Native Trailmark diff computed (trailmark_diff.json); if it is empty, both snapshots' Phase 2 node counts were non-zero, so empty means stable
  • Subgraph diff computed and non-empty (subgraph_diff.json)
  • All subgraph changes interpreted (tainted, blast radius, etc.)
  • Critical findings include evidence (node IDs, edge diffs)
  • Severity levels assigned to all findings
  • Source-level context added via git diff cross-reference
  • Worktrees cleaned up (or temp dirs removed)
  • Report written to GRAPH_EVOLUTION_*.md

Integration

trailmark skill: Phase 2 uses the trailmark API for graph building and pre-analysis. All trailmark query patterns work on either snapshot's engine.

differential-review skill: Use graph-evolution for structural analysis, differential-review for line-level code review. The two are complementary — graph-evolution finds attack paths that text diffs miss, while differential-review provides git blame context and micro-adversarial analysis.

trailmark-review-gate skill: Use trailmark-review-gate after graph-evolution when a branch, pull request, fix commit, or release diff needs a PASS/WARN/FAIL/UNKNOWN structural review packet. The gate applies deterministic review rules to graph-evolution output; it does not replace human review.

genotoxic skill: If graph-evolution reveals new high-CC tainted nodes, feed them to genotoxic for mutation testing triage.

diagramming-code skill: Generate before/after diagrams to visualize structural changes. Use call-graph or data-flow diagrams focused on changed nodes.


Supporting Documentation

Files (skills)
  • agents
    • openai.yaml 246 B
      interface:
        display_name: "Graph Evolution"
        short_description: "Compare code graphs for security-relevant structural changes"
        icon_small: "assets/trail-of-bits-mark.svg"
        icon_large: "assets/trail-of-bits-mark.svg"
        brand_color: "#D83A34"
      
  • assets
    • trail-of-bits-mark.svg 3 KB · in bundle
  • references
    • evolution-metrics.md 5.1 KB
      # Evolution Metrics Reference
      
      This document explains each structural metric the graph-evolution skill
      tracks and why it matters for security analysis.
      
      ## Contents
      
      - Node changes (added, removed, modified)
      - Edge changes (added, removed)
      - Complexity evolution
      - Attack surface changes
      - Blast radius shifts
      - Taint propagation changes
      - Privilege boundary changes
      
      ---
      
      ## Node Changes
      
      ### Added Nodes
      
      New functions, methods, classes, or modules introduced between snapshots.
      
      **Security relevance:**
      - New code has no review history and may lack test coverage
      - New public functions expand the attack surface
      - New classes may introduce state management complexity
      
      **Triage:** Cross-reference added nodes against the `after` graph's
      `entrypoints` subgraph. Added nodes that are entrypoint-reachable get
      highest review priority.
      
      ### Removed Nodes
      
      Functions, methods, or classes deleted between snapshots.
      
      **Security relevance:**
      - Removed validation functions may indicate weakened security controls
      - Removed error handlers can expose unhandled edge cases
      - Dead code removal is generally positive (reduces attack surface)
      
      **Triage:** Check if removed nodes had `privilege_boundary` or
      `taint_propagation` annotations. If so, verify the security function
      was replaced, not just deleted.
      
      ### Modified Nodes
      
      Nodes present in both snapshots whose properties changed. Tracked
      properties:
      
      | Property | What Changed | Security Concern |
      |----------|-------------|-----------------|
      | `cyclomatic_complexity` | Control flow complexity | Higher CC = more paths to test |
      | `parameters` | Function signature | New params may accept untrusted input |
      | `return_type` | Return type annotation | Type changes can break callers |
      | `line_span` | Lines of code | Significant growth may indicate added logic |
      
      ---
      
      ## Edge Changes
      
      ### Added Edges
      
      New call relationships between functions.
      
      **Security relevance:**
      - New calls from untrusted entrypoints to sensitive functions create
        attack paths that did not previously exist
      - New `inherits` or `implements` edges can change polymorphic dispatch
      - Cross-module calls may violate existing trust boundaries
      
      **Triage:** For each added `calls` edge, check if `source` is in the
      `tainted` subgraph and `target` handles sensitive operations.
      
      ### Removed Edges
      
      Call relationships that no longer exist.
      
      **Security relevance:**
      - Removed validation calls may mean input is no longer checked
      - Removed authorization calls can create privilege escalation
      - Usually benign during refactoring, but verify removed edges
        between security-relevant nodes
      
      ---
      
      ## Complexity Evolution
      
      Tracks per-node cyclomatic complexity changes between snapshots.
      
      **Thresholds:**
      
      | CC Delta | Significance |
      |----------|-------------|
      | +1 to +3 | Minor — likely a new branch or error check |
      | +4 to +9 | Moderate — new logic paths need test coverage |
      | +10 or more | Major — function is becoming difficult to reason about |
      | Negative | Positive — simplification usually reduces bug surface |
      
      **Aggregate signals:**
      - Mean CC increase across all modified nodes indicates codebase is
        growing more complex
      - Functions that crossed the CC > 10 threshold are new fuzzing
        candidates (per the genotoxic skill's criteria)
      
      ---
      
      ## Attack Surface Changes
      
      Derived from the `entrypoints` subgraph.
      
      **New entrypoints:** Nodes that appear in the `after` entrypoints but
      not `before`. Each new entrypoint is a new way external input reaches
      the system.
      
      **Removed entrypoints:** Nodes that were entrypoints in `before` but
      not `after`. Usually positive (reduced surface), but verify the
      functionality wasn't just moved.
      
      **Trust level changes:** Compare entrypoint trust levels between
      snapshots. A function changing from `trusted_internal` to
      `untrusted_external` is a significant security event.
      
      ---
      
      ## Blast Radius Shifts
      
      Derived from the `high_blast_radius` subgraph (nodes with 10+
      downstream dependents).
      
      **New high-blast nodes:** Nodes that entered `high_blast_radius` in
      `after`. These now affect many downstream functions — bugs here have
      wide impact.
      
      **Reduced blast radius:** Nodes that left `high_blast_radius`. Usually
      positive (decoupling), but verify the downstream functions weren't
      orphaned.
      
      ---
      
      ## Taint Propagation Changes
      
      Derived from the `tainted` subgraph (nodes reachable from untrusted
      entrypoints).
      
      **Newly tainted:** Nodes that entered `tainted` in `after`. These can
      now be reached by untrusted input and must validate their inputs.
      
      **De-tainted:** Nodes that left `tainted`. Usually means a trust
      boundary was added or an entrypoint was removed.
      
      **Critical combination:** Nodes that are both newly tainted AND had
      their CC increase. These are the highest-priority review targets.
      
      ---
      
      ## Privilege Boundary Changes
      
      Derived from the `privilege_boundary` subgraph (edges where trust
      levels change).
      
      **New boundary crossings:** Functions that appeared on a privilege
      boundary. These are points where trust transitions happen — common
      vulnerability locations.
      
      **Removed boundaries:** Privilege boundaries that disappeared. Could
      mean trust was flattened (potentially unsafe) or that the boundary
      moved (needs verification).
      
    • report-format.md 4.2 KB
      # Report Format Reference
      
      Output format for graph-evolution reports. The report is a markdown file
      summarizing structural changes between two code graph snapshots.
      
      ## Contents
      
      - Report filename convention
      - Section-by-section template
      - Severity classification
      - Example snippets
      
      ---
      
      ## Filename Convention
      
      ```
      GRAPH_EVOLUTION_<project>_<before-ref>_<after-ref>.md
      ```
      
      Example: `GRAPH_EVOLUTION_myapp_v1.2.0_v1.3.0.md`
      
      ---
      
      ## Report Template
      
      ```markdown
      # Graph Evolution Report
      
      **Project:** {project_name}
      **Before:** {before_ref} ({before_date})
      **After:** {after_ref} ({after_date})
      **Language:** {language}
      
      ## Summary
      
      | Metric | Before | After | Delta |
      |--------|--------|-------|-------|
      | Total nodes | N | N | +/-N |
      | Functions | N | N | +/-N |
      | Classes | N | N | +/-N |
      | Call edges | N | N | +/-N |
      | Entrypoints | N | N | +/-N |
      
      ## Critical Structural Changes
      
      Changes with direct security implications. Each finding includes
      the structural evidence and affected nodes.
      
      ### [SEVERITY] Finding title
      
      **What changed:** Description of the structural change
      **Evidence:** Node IDs, edge diffs, subgraph membership
      **Security impact:** Why this matters
      **Recommendation:** What to review or test
      
      ## Attack Surface Evolution
      
      ### New Entrypoints
      | Node | Kind | Trust Level | File |
      |------|------|------------|------|
      
      ### Removed Entrypoints
      | Node | Kind | Trust Level | File |
      |------|------|------------|------|
      
      ## Complexity Evolution
      
      ### Increased Complexity (CC delta > 0)
      | Node | Before CC | After CC | Delta | File |
      |------|-----------|----------|-------|------|
      
      ### Decreased Complexity (CC delta < 0)
      | Node | Before CC | After CC | Delta | File |
      |------|-----------|----------|-------|------|
      
      ## Taint Propagation Changes
      
      ### Newly Tainted Nodes
      | Node | Kind | Tainted Via | File |
      |------|------|-------------|------|
      
      ### De-Tainted Nodes
      | Node | Kind | File |
      |------|------|------|
      
      ## Blast Radius Shifts
      
      ### Nodes Entering high_blast_radius
      | Node | Kind | Downstream Count | File |
      |------|------|-----------------|------|
      
      ### Nodes Leaving high_blast_radius
      | Node | Kind | File |
      |------|------|------|
      
      ## Privilege Boundary Changes
      
      ### New Boundary Crossings
      | Node | Trust Transition | File |
      |------|-----------------|------|
      
      ### Removed Boundary Crossings
      | Node | Trust Transition | File |
      |------|-----------------|------|
      
      ## New Code (Added Nodes)
      | Node | Kind | CC | File |
      |------|------|----|------|
      
      ## Removed Code (Deleted Nodes)
      | Node | Kind | CC | File |
      |------|------|----|------|
      
      ## New Call Relationships (Added Edges)
      | Source | Target | Kind |
      |--------|--------|------|
      
      ## Removed Call Relationships (Deleted Edges)
      | Source | Target | Kind |
      |--------|--------|------|
      
      ## Methodology
      
      - **Tool:** Trailmark graph-evolution
      - **Before snapshot:** {before_ref}
      - **After snapshot:** {after_ref}
      - **Pre-analysis:** blast radius, taint, privilege boundaries,
        entrypoints
      - **Limitations:** {honest scope disclosure}
      ```
      
      ---
      
      ## Severity Classification
      
      Classify structural findings by security impact:
      
      | Severity | Criteria |
      |----------|----------|
      | **CRITICAL** | New tainted path to sensitive function, removed auth boundary |
      | **HIGH** | New entrypoint + high blast radius, CC increase > 10 on tainted node |
      | **MEDIUM** | New call edges crossing trust boundaries, moderate CC increase |
      | **LOW** | Added nodes without entrypoint reachability, cosmetic changes |
      | **INFO** | Dead code removal, complexity reductions, positive changes |
      
      ---
      
      ## Example: Critical Finding
      
      ```markdown
      ### [CRITICAL] New untrusted path to database query
      
      **What changed:** Function `parse_user_input` (added) calls
      `execute_query` (existing, tainted). This edge did not exist in the
      before snapshot.
      
      **Evidence:**
      - Added edge: `parse_user_input` → `execute_query` (calls, certain)
      - `execute_query` is in `high_blast_radius` (47 downstream nodes)
      - `parse_user_input` is in `tainted` subgraph
      - `parse_user_input` CC = 12 (above fuzzing threshold)
      
      **Security impact:** Untrusted external input can now reach database
      query execution through a complex, high-blast-radius path.
      
      **Recommendation:**
      1. Verify input validation on `parse_user_input`
      2. Add parameterized query usage in `execute_query`
      3. Write fuzz harness targeting `parse_user_input`
      ```
      
  • scripts
    • graph_diff.py 7.7 KB
      # /// script
      # requires-python = ">=3.12"
      # ///
      """Compute structural diff between two Trailmark graph JSON exports.
      
      Compares nodes, edges, complexity, subgraph membership, and
      pre-analysis results to surface security-relevant structural changes.
      """
      
      from __future__ import annotations
      
      import argparse
      import json
      import sys
      from pathlib import Path
      from typing import Any
      
      
      def load_graph(path: str) -> dict[str, Any]:
          """Load and validate a Trailmark JSON export."""
          data = json.loads(Path(path).read_text())
          for key in ("nodes", "edges"):
              if key not in data:
                  print(f"ERROR: Missing '{key}' in {path}", file=sys.stderr)
                  sys.exit(1)
          return data
      
      
      def diff_nodes(
          before: dict[str, Any],
          after: dict[str, Any],
      ) -> dict[str, Any]:
          """Compute added, removed, and modified nodes."""
          before_ids = set(before.keys())
          after_ids = set(after.keys())
      
          added = _summarize_nodes(after, after_ids - before_ids)
          removed = _summarize_nodes(before, before_ids - after_ids)
          modified = _find_modified(before, after, before_ids & after_ids)
      
          return {"added": added, "removed": removed, "modified": modified}
      
      
      def _summarize_nodes(
          nodes: dict[str, Any],
          ids: set[str],
      ) -> list[dict[str, Any]]:
          """Extract summary dicts for a set of node IDs."""
          result = []
          for nid in sorted(ids):
              node = nodes[nid]
              result.append(
                  {
                      "id": nid,
                      "name": node.get("name", ""),
                      "kind": node.get("kind", ""),
                      "file": _node_file(node),
                      "cyclomatic_complexity": node.get("cyclomatic_complexity"),
                  }
              )
          return result
      
      
      def _node_file(node: dict[str, Any]) -> str:
          """Extract file path from a node's location."""
          loc = node.get("location", {})
          if isinstance(loc, dict):
              return loc.get("file_path", "")
          return ""
      
      
      def _find_modified(
          before: dict[str, Any],
          after: dict[str, Any],
          shared_ids: set[str],
      ) -> list[dict[str, Any]]:
          """Find nodes present in both with changed properties."""
          modified = []
          for nid in sorted(shared_ids):
              b, a = before[nid], after[nid]
              changes = _compare_node_properties(b, a)
              if changes:
                  modified.append({"id": nid, "changes": changes})
          return modified
      
      
      def _compare_node_properties(
          before: dict[str, Any],
          after: dict[str, Any],
      ) -> dict[str, Any]:
          """Compare security-relevant properties of two node versions."""
          changes: dict[str, Any] = {}
          cc_b = before.get("cyclomatic_complexity")
          cc_a = after.get("cyclomatic_complexity")
          if cc_b != cc_a:
              changes["cyclomatic_complexity"] = {
                  "before": cc_b,
                  "after": cc_a,
              }
      
          params_b = _param_signature(before)
          params_a = _param_signature(after)
          if params_b != params_a:
              changes["parameters"] = {
                  "before": params_b,
                  "after": params_a,
              }
      
          ret_b = _return_type_str(before)
          ret_a = _return_type_str(after)
          if ret_b != ret_a:
              changes["return_type"] = {"before": ret_b, "after": ret_a}
      
          span_b = _line_span(before)
          span_a = _line_span(after)
          if span_b != span_a:
              changes["line_span"] = {"before": span_b, "after": span_a}
      
          return changes
      
      
      def _param_signature(node: dict[str, Any]) -> list[str]:
          """Extract parameter names from a node."""
          params = node.get("parameters", ())
          if isinstance(params, (list, tuple)):
              return [p.get("name", "") if isinstance(p, dict) else str(p) for p in params]
          return []
      
      
      def _return_type_str(node: dict[str, Any]) -> str | None:
          """Extract return type string from a node."""
          rt = node.get("return_type")
          if isinstance(rt, dict):
              return rt.get("name")
          return rt
      
      
      def _line_span(node: dict[str, Any]) -> int:
          """Compute line count from a node's location."""
          loc = node.get("location", {})
          if isinstance(loc, dict):
              start = loc.get("start_line", 0)
              end = loc.get("end_line", 0)
              return max(0, end - start + 1)
          return 0
      
      
      def diff_edges(
          before: list[dict[str, Any]],
          after: list[dict[str, Any]],
      ) -> dict[str, Any]:
          """Compute added and removed edges."""
          before_set = {_edge_key(e) for e in before}
          after_set = {_edge_key(e) for e in after}
      
          added = sorted(after_set - before_set)
          removed = sorted(before_set - after_set)
      
          return {
              "added": [_parse_edge_key(k) for k in added],
              "removed": [_parse_edge_key(k) for k in removed],
          }
      
      
      def _edge_key(edge: dict[str, Any]) -> str:
          """Create a hashable key for an edge."""
          src = edge.get("source", edge.get("source_id", ""))
          tgt = edge.get("target", edge.get("target_id", ""))
          kind = edge.get("kind", "")
          return f"{src}|{tgt}|{kind}"
      
      
      def _parse_edge_key(key: str) -> dict[str, str]:
          """Convert an edge key back to a dict."""
          source, target, kind = key.split("|", 2)
          return {"source": source, "target": target, "kind": kind}
      
      
      def diff_subgraphs(
          before: dict[str, list[str]],
          after: dict[str, list[str]],
      ) -> dict[str, Any]:
          """Compute per-subgraph membership changes."""
          all_names = sorted(set(before.keys()) | set(after.keys()))
          changes: dict[str, Any] = {}
      
          for name in all_names:
              b_ids = set(before.get(name, []))
              a_ids = set(after.get(name, []))
              added = sorted(a_ids - b_ids)
              removed = sorted(b_ids - a_ids)
              if added or removed:
                  changes[name] = {"added": added, "removed": removed}
      
          return changes
      
      
      def compute_summary_delta(
          before: dict[str, Any],
          after: dict[str, Any],
      ) -> dict[str, Any]:
          """Compute deltas for summary statistics."""
          b_sum = before.get("summary", {})
          a_sum = after.get("summary", {})
          delta: dict[str, Any] = {}
      
          for key in ("total_nodes", "functions", "classes", "call_edges", "entrypoints"):
              b_val = b_sum.get(key, 0)
              a_val = a_sum.get(key, 0)
              if b_val != a_val:
                  delta[key] = {
                      "before": b_val,
                      "after": a_val,
                      "delta": a_val - b_val,
                  }
          return delta
      
      
      def compute_diff(
          before: dict[str, Any],
          after: dict[str, Any],
      ) -> dict[str, Any]:
          """Compute the full structural diff between two graphs."""
          return {
              "summary_delta": compute_summary_delta(before, after),
              "nodes": diff_nodes(
                  before.get("nodes", {}),
                  after.get("nodes", {}),
              ),
              "edges": diff_edges(
                  before.get("edges", []),
                  after.get("edges", []),
              ),
              "subgraphs": diff_subgraphs(
                  before.get("subgraphs", {}),
                  after.get("subgraphs", {}),
              ),
          }
      
      
      def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
          """Parse command-line arguments."""
          parser = argparse.ArgumentParser(
              description="Structural diff between Trailmark graphs",
          )
          parser.add_argument(
              "--before",
              required=True,
              help="Path to the 'before' graph JSON export",
          )
          parser.add_argument(
              "--after",
              required=True,
              help="Path to the 'after' graph JSON export",
          )
          parser.add_argument(
              "--indent",
              type=int,
              default=2,
              help="JSON output indentation (default: 2)",
          )
          return parser.parse_args(argv)
      
      
      def main(argv: list[str] | None = None) -> None:
          """Entry point: load graphs, compute diff, print JSON."""
          args = parse_args(argv)
          before = load_graph(args.before)
          after = load_graph(args.after)
          diff = compute_diff(before, after)
          print(json.dumps(diff, indent=args.indent))
      
      
      if __name__ == "__main__":
          main()
      
  • SKILL.md 13.2 KB
    ---
    name: graph-evolution
    description: >
      Compares Trailmark code graphs at two source code snapshots (git commits,
      tags, or directories) to surface security-relevant structural changes.
      Detects new attack paths, complexity shifts, blast radius growth, taint
      propagation changes, and privilege boundary modifications that text diffs
      miss. Use when comparing code between commits or tags, analyzing structural
      evolution, detecting attack surface growth, reviewing what changed between
      audit snapshots, or finding security-relevant changes that text diffs miss.
    ---
    
    # Graph Evolution
    
    Builds Trailmark code graphs at two source snapshots and computes a
    structural diff. Surfaces security-relevant changes that text-level
    diffs miss: new attack paths, complexity shifts, blast radius growth,
    taint propagation changes, and privilege boundary modifications.
    
    ## When to Use
    
    - Comparing two git refs to understand what structurally changed
    - Auditing a range of commits for security-relevant evolution
    - Detecting new attack paths created by code changes
    - Finding functions whose blast radius or complexity grew silently
    - Identifying taint propagation changes across refactors
    - Pre-release structural comparison (tag-to-tag or branch-to-branch)
    
    ## When NOT to Use
    
    - Line-level code review (use `differential-review` for text-diff analysis)
    - Single-snapshot analysis (use the `trailmark` skill directly)
    - Diagram generation from a single snapshot (use the `diagramming-code` skill)
    - Mutation testing triage (use the `genotoxic` skill)
    
    ## Rationalizations to Reject
    
    | Rationalization | Why It's Wrong | Required Action |
    |-----------------|----------------|-----------------|
    | "We just need the structural diff, skip pre-analysis" | Without pre-analysis, you miss taint changes, blast radius growth, and privilege boundary shifts | Run `engine.preanalysis()` on both snapshots |
    | "Text diff covers what changed" | Text diffs miss new attack paths, transitive complexity shifts, and subgraph membership changes | Use structural diff to complement text diff |
    | "Only added nodes matter" | Removed security functions and shifted privilege boundaries are equally dangerous | Review removals and modifications, not just additions |
    | "Low-severity structural changes can be ignored" | INFO-level changes (dead code removal) can mask removed security checks | Classify every change, review removals for replaced functionality |
    | "One snapshot's graph is enough for comparison" | Single-snapshot analysis can't detect evolution — you need both before and after | Always build and export both graphs |
    | "Tool isn't installed, I'll compare manually" | Manual comparison misses what graph analysis catches | Install trailmark first |
    | "The diff came back empty, so nothing changed structurally" | `trailmark diff` defaults `--language` to `python` and exits 0 with empty arrays on any other target, so an empty diff reads identically whether the code is unchanged or the language was wrong | Pass `--language` explicitly and re-run before concluding no change |
    
    ---
    
    ## Prerequisites
    
    **trailmark** must be installed. If `uv run trailmark` fails, run:
    
    ```bash
    uv tool install trailmark
    # Python snippets: uv run --with trailmark python -   (a tool env is not importable)
    ```
    
    **DO NOT** fall back to "manual comparison" or reading source files as a
    substitute for running trailmark. The tool must be installed and used
    programmatically. If installation fails, report the error.
    
    ---
    
    ## Quick Start
    
    ```bash
    # Compare two git refs (e.g., tags, branches, commits)
    # 1. Build graphs at each snapshot
    # 2. Run pre-analysis on both
    # 3. Compute structural diff
    # 4. Generate report
    
    # Step-by-step: see Workflow below
    ```
    
    ---
    
    ## Decision Tree
    
    ```
    ├─ Need to understand what each metric means?
    │  └─ Read: references/evolution-metrics.md
    │
    ├─ Need the report output format?
    │  └─ Read: references/report-format.md
    │
    ├─ Already have two graph JSON exports?
    │  └─ Jump to Phase 3 (run native diff + graph_diff.py)
    │
    └─ Starting from two git refs?
       └─ Start at Phase 1
    ```
    
    ---
    
    ## Workflow
    
    ```
    Graph Evolution Progress:
    - [ ] Phase 1: Create snapshots (git worktrees)
    - [ ] Phase 2: Build graphs + pre-analysis on both snapshots
    - [ ] Phase 3: Compute structural diff
    - [ ] Phase 4: Interpret diff and generate report
    - [ ] Phase 5: Clean up worktrees
    ```
    
    ### Phase 1: Create Snapshots
    
    Use git worktrees to get clean copies of each ref without disturbing
    the working tree.
    
    ```bash
    # Create temp directories for worktrees
    BEFORE_DIR=$(mktemp -d)
    AFTER_DIR=$(mktemp -d)
    
    # Create worktrees (run from repo root)
    git worktree add "$BEFORE_DIR" {before_ref}
    git worktree add "$AFTER_DIR" {after_ref}
    ```
    
    If comparing two directories instead of git refs, skip this phase and
    use the directory paths directly in Phase 2.
    
    ### Phase 2: Build Graphs and Run Pre-Analysis
    
    Build Trailmark graphs for both snapshots and run pre-analysis on each.
    Pre-analysis computes blast radius, taint propagation, privilege
    boundaries, and entrypoint enumeration.
    
    ```python
    from trailmark.query.api import QueryEngine
    
    def build_and_export(target_dir, output_path, language="auto"):
        """Build graph, run pre-analysis, export JSON."""
        engine = QueryEngine.from_directory(target_dir, language=language)
        engine.preanalysis()
        json_str = engine.to_json()
        with open(output_path, "w") as f:
            f.write(json_str)
        return engine.summary()
    
    import tempfile, os
    work_dir = tempfile.mkdtemp(prefix="trailmark_evolution_")
    before_json = os.path.join(work_dir, "before_graph.json")
    after_json = os.path.join(work_dir, "after_graph.json")
    
    before_summary = build_and_export(
        "{before_dir}", before_json
    )
    after_summary = build_and_export(
        "{after_dir}", after_json
    )
    ```
    
    Verify both graphs built successfully by checking the summary output.
    If either fails, rerun with an explicit language or comma-separated list
    instead of `auto`.
    
    ### Phase 3: Compute Structural Diff
    
    Run **both**:
    
    1. Trailmark's native structural diff for nodes, edges, and entrypoints
    2. The plugin's `graph_diff.py` helper for subgraph membership changes
    
    Use the same `work_dir` from Phase 2, and pass the same `--language` value Phase 2
    built with. `trailmark diff` defaults that flag to `python`, so on any other
    target the default exits 0 and writes empty arrays rather than reporting a
    mismatch.
    
    ```bash
    trailmark diff --json --language auto "{before_dir}" "{after_dir}" > "{work_dir}/trailmark_diff.json" || \
      uv run trailmark diff --json --language auto "{before_dir}" "{after_dir}" > "{work_dir}/trailmark_diff.json"
    
    uv run {baseDir}/scripts/graph_diff.py \
        --before "{before_json}" \
        --after "{after_json}" > "{work_dir}/subgraph_diff.json"
    ```
    
    If Phase 2 needed an explicit language or a comma-separated list instead of
    `auto`, use that same value here.
    
    If either diff command fails or writes an empty JSON file, stop and report the
    error instead of continuing to Phase 4.
    
    A `trailmark_diff.json` whose `nodes`, `edges`, and `entrypoints` arrays are all
    empty means either nothing changed structurally or both snapshots parsed to
    (near-)empty graphs. Decide which using Phase 2's graph summaries: if either
    snapshot's node count is zero or implausibly small for the target, the parse
    missed the code — name the language set explicitly (`rust`, `solidity`,
    `python,rust`) and re-run. Healthy node counts on both snapshots plus an empty
    diff is genuine structural stability.
    
    The native Trailmark diff contains:
    
    | Key | Contents |
    |-----|----------|
    | `summary_delta` | Changes in node/edge/entrypoint counts |
    | `nodes.added` | New functions, classes, methods |
    | `nodes.removed` | Deleted functions, classes, methods |
    | `nodes.modified` | Functions with changed CC, params, line span |
    | `edges.added` | New call/inheritance/import relationships |
    | `edges.removed` | Deleted relationships |
    | `entrypoints` | Added, removed, and modified entrypoints |
    
    The subgraph diff contains:
    
    | Key | Contents |
    |-----|----------|
    | `subgraphs` | Per-subgraph membership changes (tainted, high_blast_radius, etc.) |
    
    ### Phase 4: Interpret Diff and Generate Report
    
    Read **both** diff JSON files and generate a security-focused markdown
    report.
    See [references/report-format.md](references/report-format.md) for
    the full template.
    
    **Interpretation priorities (highest to lowest):**
    
    1. **New tainted paths** — nodes entering the `tainted` subgraph,
       especially if they also appear in added edges targeting sensitive
       functions
    2. **Privilege boundary changes** — new or removed trust transitions
       from the native entrypoint/edge diff plus the subgraph diff
    3. **Attack surface growth** — new entrypoints, especially
       `untrusted_external`, from `trailmark_diff.json`
    4. **Blast radius increases** — nodes entering `high_blast_radius`
    5. **Complexity spikes** — CC increases > 3 on tainted or
       entrypoint-reachable nodes
    6. **Structural additions** — new nodes and edges (review needed)
    7. **Structural removals** — verify removed security functions were
       replaced
    
    Cross-reference structural changes with `git diff {before_ref}..{after_ref}`
    to add source-level context to findings.
    
    **Severity classification:**
    
    | Severity | Structural Signal |
    |----------|------------------|
    | CRITICAL | New tainted path to sensitive function, removed auth boundary |
    | HIGH | New entrypoint + high blast radius, large CC increase on tainted node |
    | MEDIUM | New trust-boundary-crossing edges, moderate CC increase |
    | LOW | Added nodes without entrypoint reachability |
    | INFO | Dead code removal, complexity reductions |
    
    For detailed metric definitions, see
    [references/evolution-metrics.md](references/evolution-metrics.md).
    
    ### Phase 5: Clean Up
    
    Remove git worktrees after the report is written:
    
    ```bash
    git worktree remove "{before_dir}"
    git worktree remove "{after_dir}"
    ```
    
    ---
    
    ## Diff Reference
    
    ```
    trailmark diff --json --language auto BEFORE AFTER
    uv run {baseDir}/scripts/graph_diff.py [OPTIONS]
    ```
    
    `trailmark diff --language` defaults to `python`. On a target in any other
    language that default still exits 0, emitting well-formed JSON with empty
    `nodes`, `edges`, and `entrypoints` arrays, so always pass the flag: `auto`
    detects and merges every supported language found under the target, and a single
    name (`rust`, `solidity`) or comma-separated list (`python,rust`) pins an
    explicit set. `auto` fails loudly with `No supported languages detected under
    <path>` when a snapshot holds nothing it can parse, which is the outcome you
    want. Confirm the language first; only then can an empty diff count as evidence
    that nothing changed.
    
    Use `trailmark diff` for:
    - Node/edge changes
    - Added/removed/modified entrypoints
    - Human-readable structural diff reports
    
    Use `graph_diff.py` for:
    - Subgraph membership changes derived from `engine.preanalysis()`
    - `tainted`, `high_blast_radius`, `privilege_boundary`, and related sets
    
    | Argument | Default | Description |
    |----------|---------|-------------|
    | `--before` | required | Path to the "before" graph JSON |
    | `--after` | required | Path to the "after" graph JSON |
    | `--indent` | `2` | JSON output indentation |
    
    `graph_diff.py` input format: Trailmark JSON exports from `engine.to_json()`.
    `graph_diff.py` output: JSON structural diff for nodes, edges, and subgraphs.
    
    ---
    
    ## Quality Checklist
    
    Before delivering the report:
    
    - [ ] Both graphs built successfully (check summaries)
    - [ ] Pre-analysis ran on both snapshots
    - [ ] Native Trailmark diff computed (`trailmark_diff.json`); if it is empty,
          both snapshots' Phase 2 node counts were non-zero, so empty means stable
    - [ ] Subgraph diff computed and non-empty (`subgraph_diff.json`)
    - [ ] All subgraph changes interpreted (tainted, blast radius, etc.)
    - [ ] Critical findings include evidence (node IDs, edge diffs)
    - [ ] Severity levels assigned to all findings
    - [ ] Source-level context added via git diff cross-reference
    - [ ] Worktrees cleaned up (or temp dirs removed)
    - [ ] Report written to `GRAPH_EVOLUTION_*.md`
    
    ---
    
    ## Integration
    
    **trailmark skill:**
    Phase 2 uses the trailmark API for graph building and pre-analysis.
    All trailmark query patterns work on either snapshot's engine.
    
    **differential-review skill:**
    Use graph-evolution for structural analysis, differential-review for
    line-level code review. The two are complementary — graph-evolution
    finds attack paths that text diffs miss, while differential-review
    provides git blame context and micro-adversarial analysis.
    
    **trailmark-review-gate skill:**
    Use trailmark-review-gate after graph-evolution when a branch, pull request,
    fix commit, or release diff needs a PASS/WARN/FAIL/UNKNOWN structural review
    packet. The gate applies deterministic review rules to graph-evolution output;
    it does not replace human review.
    
    **genotoxic skill:**
    If graph-evolution reveals new high-CC tainted nodes, feed them to
    genotoxic for mutation testing triage.
    
    **diagramming-code skill:**
    Generate before/after diagrams to visualize structural changes.
    Use `call-graph` or `data-flow` diagrams focused on changed nodes.
    
    ---
    
    ## Supporting Documentation
    
    - **[references/evolution-metrics.md](references/evolution-metrics.md)** —
      What each structural metric means and why it matters for security
    - **[references/report-format.md](references/report-format.md)** —
      Report template, severity classification, and example findings
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related