hallucination-guard
Verify AI-generated imports, API calls, and packages actually exist. Use when finishing AI-generated code that introduces new imports or external API calls — auto-fires after fix/cook to catch phantom functions, non-existent packages, and slopsquatting. Checks the code itself; ag
Install
npx skills add https://github.com/Rune-kit/rune/tree/master/skills/hallucination-guard
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install rune-kit-rune@llmmart
git clone https://github.com/Rune-kit/rune.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole rune-kit/rune collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
hallucination-guard
Purpose
Post-generation validation that verifies AI-generated code references actually exist. Catches the 42% of AI code that contains hallucinated imports, non-existent packages, phantom functions, and incorrect API signatures. Also defends against "slopsquatting" — where attackers register package names that AI commonly hallucinates.
Triggers
- Called by
cookafter code generation, before commit - Called by
fixafter applying fixes - Called by
preflightas import verification sub-check - Called by
reviewduring code review - Auto-trigger: when new import statements are added to codebase
Calls (outbound)
Exception: L3→L3 coordination
research(L3): verify package existence on npm/pypi
Called By (inbound)
cook(L1): after code generation, before commitfix(L2): after applying fixespreflight(L2): import verification sub-checkreview(L2): during code reviewdb(L2): verify SQL syntax and ORM method calls are realreview-intake(L2): verify imports in code submitted for reviewskill-forge(L2): verify imports in newly generated skill codeadversary(L2): verify APIs/packages in plan actually existlogic-guardian(L2): verify referenced functions and imports after guarded logic edits
Execution
Step 1 — Extract imports
Use Grep to find all import/require/use statements in changed files:
Grep pattern: ^(import|require|use|from)\s
Files: changed files passed as input
Output mode: content
Collect every imported module name and file path. Separate into:
- Internal imports (start with
./,../,@/,~/) - External packages (bare module names)
Step 2 — Verify internal imports
For each internal import path, use Glob to confirm the file exists in the codebase.
Glob pattern: <resolved import path>.* (try .ts, .tsx, .js, .jsx, .py, .rs etc.)
If Glob returns no results → mark as BLOCK (file does not exist).
Also use Grep to verify that the specific exported name (function/class/const) exists in the resolved file:
Grep pattern: export (function|class|const|default) <name>
File: resolved file path
If export not found → mark as WARN (symbol may not be exported).
Step 3 — Verify external packages (Dependency Check Before Import)
Before importing ANY third-party library, check the dependency manifest.
Use Read on the project's dependency manifest to confirm each external package is listed:
- JavaScript/TypeScript:
package.json→ checkdependenciesanddevDependencies - Python:
requirements.txtorpyproject.toml→[project.dependencies]and[project.optional-dependencies] - Rust:
Cargo.toml→[dependencies]and[dev-dependencies]
Pre-import gate (BEFORE writing import statements, not just after):
- If the agent is ABOUT to import a package → check manifest FIRST
- If package is NOT in manifest → output install command before writing the import:
⚠ Package '<name>' not in dependencies. Install first: npm install <name> # JS/TS pip install <name> # Python cargo add <name> # Rust - If package IS in manifest → proceed with import
Post-import verification (after code is written):
- If package is not listed in the manifest → mark as BLOCK (phantom dependency)
- If package is listed but not installed (no lockfile entry) → mark as WARN (not yet installed)
Also check for typosquatting: if package name has edit distance ≤ 2 from a known popular package (axios/axois, lodash/lodahs, react/recat), mark as SUSPICIOUS.
Step 3.5 — Slopsquatting Registry Verification
For each NEW external package (present in manifest but absent from lockfile):
3.5a. Registry existence check:
JavaScript: Bash: npm view <package-name> version 2>/dev/null
Python: Bash: pip index versions <package-name> 2>/dev/null
Rust: Bash: cargo search <package-name> --limit 1 2>/dev/null
If command returns empty/error → BLOCK (package does not exist on registry — likely hallucinated name).
3.5b. Popularity check (slopsquatting defense):
JavaScript: Bash: npm view <package-name> 'dist-tags.latest' 'time.modified' 2>/dev/null
→ If last modified > 2 years ago AND weekly downloads < 100: SUSPICIOUS
Python: Use rune:research to check PyPI page for download stats
Low-popularity packages with names similar to popular ones = SUSPICIOUS (potential slopsquatting attack).
3.5c. Known slopsquatting patterns:
Popular Package → Common AI Hallucination
axios → axois, axio, axioss
lodash → lodahs, loadash, lo-dash
express → expresss, express-js
react-router → react-routes, react-routing
python-dotenv → dotenv (wrong package in Python context)
Flag any match with edit distance ≤ 2 from these known pairs.
Step 4 — Verify API calls
For any API endpoint or SDK method call found in the diff, use rune:docs-seeker (Context7) to confirm:
- The method/function exists in the library's documented API
- The parameter signature matches usage in code
Mark unverifiable API calls as WARN (cannot confirm without docs).
Step 5 — Report
Emit the report in the Output Format below. If any BLOCK items exist, return status BLOCK to the calling skill to halt commit/deploy.
Check Types
INTERNAL — file exists, function/class exists, signature matches
EXTERNAL — package exists on registry, version is valid
API — endpoint pattern valid, method correct
TYPE — assertion matches actual type
SUSPICIOUS — package name similar to popular package (slopsquatting)
Output Format
## Hallucination Guard Report
- **Status**: PASS | WARN | BLOCK
- **References Checked**: [count]
- **Verified**: [count] | **Unverified**: [count] | **Suspicious**: [count]
### BLOCK (hallucination detected)
- `import { formatDate } from 'date-utils'` — Package 'date-utils' not found on npm. Did you mean 'date-fns'?
- `import { useAuth } from '@/hooks/useAuth'` — File '@/hooks/useAuth' does not exist
### WARN (verify manually)
- `import { newFunction } from 'popular-lib'` — Function 'newFunction' not found in popular-lib@3.2.0 exports
### SUSPICIOUS (potential slopsquatting)
- `import axios from 'axois'` — Typo? Similar to popular package 'axios'
### Verified
- 12/15 references verified successfully
Constraints
- MUST verify every import against actual installed packages — not just check if name looks reasonable
- MUST verify API signatures against docs — not assume from function name
- MUST report BLOCK verdict with specific evidence — never "looks suspicious"
- MUST NOT say "no hallucinations found" without listing what was checked
Sharp Edges
Known failure modes for this skill. Check these before declaring done.
| Failure Mode | Severity | Mitigation |
|---|---|---|
| Declaring "no hallucinations found" without listing what was checked | CRITICAL | Constraint 4 blocks this — always list verified count vs total |
| Marking phantom package (not in manifest) as WARN instead of BLOCK | HIGH | Unlisted package in manifest = BLOCK — not installed = won't run |
| Missing typosquatting check on external packages | MEDIUM | Edit distance ≤2 check is mandatory — check every external package name |
| Only checking package name, not the specific exported symbol | MEDIUM | Step 2: verify the specific function/class is exported, not just the file exists |
| Skipping registry verification for new packages | CRITICAL | Step 3.5 HARD-GATE: new packages MUST be verified against actual registry |
| AI-hallucinated package name passes because it "sounds right" | HIGH | Slopsquatting defense: check registry existence, not name plausibility |
| Low-popularity package with similar name to popular one not flagged | HIGH | Popularity check catches slopsquatting attacks on newly registered packages |
Done When
- All imports extracted from changed files (internal + external separated)
- Internal imports: file existence AND symbol export verified
- External packages: manifest presence checked for every package
- Suspicious package names flagged (edit distance ≤2 from popular packages)
- API signatures checked via docs-seeker for new SDK/library calls
- Hallucination Guard Report emitted with PASS/WARN/BLOCK and verified count
Cost Profile
~500-1500 tokens input, ~200-500 tokens output. Haiku for speed — this runs frequently as a sub-check.
Files (rune)
-
SKILL.md 9.2 KB
--- name: hallucination-guard description: "Verify AI-generated imports, API calls, and packages actually exist. Use when finishing AI-generated code that introduces new imports or external API calls — auto-fires after fix/cook to catch phantom functions, non-existent packages, and slopsquatting. Checks the code itself; agent claims are completion-gate, HARD-GATE compliance is constraint-check." metadata: author: runedev version: "0.3.0" layer: L3 model: sonnet group: validation tools: "Read, Bash, Glob, Grep" --- # hallucination-guard ## Purpose Post-generation validation that verifies AI-generated code references actually exist. Catches the 42% of AI code that contains hallucinated imports, non-existent packages, phantom functions, and incorrect API signatures. Also defends against "slopsquatting" — where attackers register package names that AI commonly hallucinates. ## Triggers - Called by `cook` after code generation, before commit - Called by `fix` after applying fixes - Called by `preflight` as import verification sub-check - Called by `review` during code review - Auto-trigger: when new import statements are added to codebase ## Calls (outbound) # Exception: L3→L3 coordination - `research` (L3): verify package existence on npm/pypi ## Called By (inbound) - `cook` (L1): after code generation, before commit - `fix` (L2): after applying fixes - `preflight` (L2): import verification sub-check - `review` (L2): during code review - `db` (L2): verify SQL syntax and ORM method calls are real - `review-intake` (L2): verify imports in code submitted for review - `skill-forge` (L2): verify imports in newly generated skill code - `adversary` (L2): verify APIs/packages in plan actually exist - `logic-guardian` (L2): verify referenced functions and imports after guarded logic edits ## Execution ### Step 1 — Extract imports Use `Grep` to find all import/require/use statements in changed files: ``` Grep pattern: ^(import|require|use|from)\s Files: changed files passed as input Output mode: content ``` Collect every imported module name and file path. Separate into: - Internal imports (start with `./`, `../`, `@/`, `~/`) - External packages (bare module names) ### Step 2 — Verify internal imports For each internal import path, use `Glob` to confirm the file exists in the codebase. ``` Glob pattern: <resolved import path>.* (try .ts, .tsx, .js, .jsx, .py, .rs etc.) ``` If `Glob` returns no results → mark as **BLOCK** (file does not exist). Also use `Grep` to verify that the specific exported name (function/class/const) exists in the resolved file: ``` Grep pattern: export (function|class|const|default) <name> File: resolved file path ``` If export not found → mark as **WARN** (symbol may not be exported). ### Step 3 — Verify external packages (Dependency Check Before Import) Before importing ANY third-party library, check the dependency manifest. Use `Read` on the project's dependency manifest to confirm each external package is listed: - JavaScript/TypeScript: `package.json` → check `dependencies` and `devDependencies` - Python: `requirements.txt` or `pyproject.toml` → `[project.dependencies]` and `[project.optional-dependencies]` - Rust: `Cargo.toml` → `[dependencies]` and `[dev-dependencies]` **Pre-import gate** (BEFORE writing import statements, not just after): 1. If the agent is ABOUT to import a package → check manifest FIRST 2. If package is NOT in manifest → output install command before writing the import: ``` ⚠ Package '<name>' not in dependencies. Install first: npm install <name> # JS/TS pip install <name> # Python cargo add <name> # Rust ``` 3. If package IS in manifest → proceed with import **Post-import verification** (after code is written): - If package is **not listed** in the manifest → mark as **BLOCK** (phantom dependency) - If package is listed but not installed (no lockfile entry) → mark as **WARN** (not yet installed) Also check for typosquatting: if package name has edit distance ≤ 2 from a known popular package (axios/axois, lodash/lodahs, react/recat), mark as **SUSPICIOUS**. ### Step 3.5 — Slopsquatting Registry Verification <HARD-GATE> Any NEW package added to the manifest (not previously in the lockfile) MUST be verified against the actual registry. AI agents hallucinate package names at high rates. A package that doesn't exist on npm/PyPI/crates.io = supply chain risk. </HARD-GATE> For each NEW external package (present in manifest but absent from lockfile): **3.5a. Registry existence check:** ``` JavaScript: Bash: npm view <package-name> version 2>/dev/null Python: Bash: pip index versions <package-name> 2>/dev/null Rust: Bash: cargo search <package-name> --limit 1 2>/dev/null ``` If command returns empty/error → **BLOCK** (package does not exist on registry — likely hallucinated name). **3.5b. Popularity check (slopsquatting defense):** ``` JavaScript: Bash: npm view <package-name> 'dist-tags.latest' 'time.modified' 2>/dev/null → If last modified > 2 years ago AND weekly downloads < 100: SUSPICIOUS Python: Use rune:research to check PyPI page for download stats ``` Low-popularity packages with names similar to popular ones = **SUSPICIOUS** (potential slopsquatting attack). **3.5c. Known slopsquatting patterns:** ``` Popular Package → Common AI Hallucination axios → axois, axio, axioss lodash → lodahs, loadash, lo-dash express → expresss, express-js react-router → react-routes, react-routing python-dotenv → dotenv (wrong package in Python context) ``` Flag any match with edit distance ≤ 2 from these known pairs. ### Step 4 — Verify API calls For any API endpoint or SDK method call found in the diff, use `rune:docs-seeker` (Context7) to confirm: - The method/function exists in the library's documented API - The parameter signature matches usage in code Mark unverifiable API calls as **WARN** (cannot confirm without docs). ### Step 5 — Report Emit the report in the Output Format below. If any **BLOCK** items exist, return status `BLOCK` to the calling skill to halt commit/deploy. ## Check Types ``` INTERNAL — file exists, function/class exists, signature matches EXTERNAL — package exists on registry, version is valid API — endpoint pattern valid, method correct TYPE — assertion matches actual type SUSPICIOUS — package name similar to popular package (slopsquatting) ``` ## Output Format ``` ## Hallucination Guard Report - **Status**: PASS | WARN | BLOCK - **References Checked**: [count] - **Verified**: [count] | **Unverified**: [count] | **Suspicious**: [count] ### BLOCK (hallucination detected) - `import { formatDate } from 'date-utils'` — Package 'date-utils' not found on npm. Did you mean 'date-fns'? - `import { useAuth } from '@/hooks/useAuth'` — File '@/hooks/useAuth' does not exist ### WARN (verify manually) - `import { newFunction } from 'popular-lib'` — Function 'newFunction' not found in popular-lib@3.2.0 exports ### SUSPICIOUS (potential slopsquatting) - `import axios from 'axois'` — Typo? Similar to popular package 'axios' ### Verified - 12/15 references verified successfully ``` ## Constraints 1. MUST verify every import against actual installed packages — not just check if name looks reasonable 2. MUST verify API signatures against docs — not assume from function name 3. MUST report BLOCK verdict with specific evidence — never "looks suspicious" 4. MUST NOT say "no hallucinations found" without listing what was checked ## Sharp Edges Known failure modes for this skill. Check these before declaring done. | Failure Mode | Severity | Mitigation | |---|---|---| | Declaring "no hallucinations found" without listing what was checked | CRITICAL | Constraint 4 blocks this — always list verified count vs total | | Marking phantom package (not in manifest) as WARN instead of BLOCK | HIGH | Unlisted package in manifest = BLOCK — not installed = won't run | | Missing typosquatting check on external packages | MEDIUM | Edit distance ≤2 check is mandatory — check every external package name | | Only checking package name, not the specific exported symbol | MEDIUM | Step 2: verify the specific function/class is exported, not just the file exists | | Skipping registry verification for new packages | CRITICAL | Step 3.5 HARD-GATE: new packages MUST be verified against actual registry | | AI-hallucinated package name passes because it "sounds right" | HIGH | Slopsquatting defense: check registry existence, not name plausibility | | Low-popularity package with similar name to popular one not flagged | HIGH | Popularity check catches slopsquatting attacks on newly registered packages | ## Done When - All imports extracted from changed files (internal + external separated) - Internal imports: file existence AND symbol export verified - External packages: manifest presence checked for every package - Suspicious package names flagged (edit distance ≤2 from popular packages) - API signatures checked via docs-seeker for new SDK/library calls - Hallucination Guard Report emitted with PASS/WARN/BLOCK and verified count ## Cost Profile ~500-1500 tokens input, ~200-500 tokens output. Haiku for speed — this runs frequently as a sub-check.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.