Claude Cursor GitHub Copilot Skill

reality-verification

This skill should be used when the user asks to "verify a fix", "reproduce failure", "diagnose issue", "check BEFORE/AFTER state", "VF task", "reality check", "check test quality", "mock-only tests", or needs guidance on verifying fixes by reproducing failures before and after im

LLM Mart · 0 points · 14 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download tzachbon-smart-ralph-plugins_ralph-specum_skills_reality-verification-4890dd3.zip · 4 KB
Part of tzachbon/smart-ralph — 44 skills

Install

skills CLI npx skills add https://github.com/tzachbon/smart-ralph/tree/main/plugins/ralph-specum/skills/reality-verification
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tzachbon-smart-ralph@llmmart
Git git clone https://github.com/tzachbon/smart-ralph.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tzachbon/smart-ralph collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Reality Verification

For fix goals: reproduce the failure BEFORE work, verify resolution AFTER.

Goal Detection

Classify user goals to determine if diagnosis is needed. See references/goal-detection-patterns.md for detailed patterns.

Quick reference:

  • Fix indicators: fix, repair, resolve, debug, patch, broken, failing, error, bug
  • Add indicators: add, create, build, implement, new
  • Conflict resolution: If both present, treat as Fix

Command Mapping

Goal Keywords Reproduction Command
CI, pipeline gh run view --log-failed
test, tests project test command
type, typescript pnpm check-types or tsc --noEmit
lint pnpm lint
build pnpm build
E2E, UI Playwright MCP browser tools
API, endpoint WebFetch tool

For E2E/deployment verification, use MCP tools (Playwright MCP browser tools for UI, WebFetch tool for APIs).

BEFORE/AFTER Documentation

BEFORE State (Diagnosis)

Document in .progress.md under ## Reality Check (BEFORE):

## Reality Check (BEFORE)

**Goal type**: Fix
**Reproduction command**: `pnpm test`
**Failure observed**: Yes
**Output**:

FAIL src/auth.test.ts Expected: 200 Received: 401

**Timestamp**: 2026-01-16T10:30:00Z

AFTER State (Verification)

Document in .progress.md under ## Reality Check (AFTER):

## Reality Check (AFTER)

**Command**: `pnpm test`
**Result**: PASS
**Output**:

PASS src/auth.test.ts All tests passed

**Comparison**: BEFORE failed with 401, AFTER passes
**Verified**: Issue resolved

VF Task Format

Add as task 4.3 (after PR creation) for fix-type specs:

- [ ] 4.3 VF: Verify original issue resolved
  - **Do**:
    1. Read BEFORE state from .progress.md
    2. Re-run reproduction command: `<command>`
    3. Compare output with BEFORE state
    4. Document AFTER state in .progress.md
  - **Verify**: `grep -q "Verified: Issue resolved" ./specs/<name>/.progress.md`
  - **Done when**: AFTER shows issue resolved, documented in .progress.md
  - **Commit**: `chore(<name>): verify fix resolves original issue`

Test Quality Checks

When verifying test-related fixes, check for mock-only test anti-patterns. See references/mock-quality-checks.md for detailed patterns.

Quick reference red flags:

  • Mock declarations > 3x real assertions
  • Missing import of actual module under test
  • All assertions are mock interaction checks (toHaveBeenCalled)
  • No integration tests
  • Missing mock cleanup (afterEach)

Why This Matters

Without With
"Fix CI" spec completes but CI still red CI verified green before merge
Tests "fixed" but original failure unknown Before/after comparison proves fix
Silent regressions Explicit failure reproduction
Manual verification required Automated verification in workflow
Tests pass but only test mocks Tests verify real behavior, not mock behavior
False sense of security from green tests Confidence that tests catch real bugs
Files (smart-ralph)
  • references
    • goal-detection-patterns.md 2.3 KB
      # Goal Detection Patterns
      
      Regex patterns and heuristics for classifying user goals as Fix or Add type.
      
      ## Detection Heuristics
      
      | Pattern | Type | Regex |
      |---------|------|-------|
      | fix, repair, resolve, debug, patch | Fix | `\b(fix\|repair\|resolve\|debug\|patch)\b` |
      | broken, failing, error, bug, issue | Fix | `\b(broken\|failing\|error\|bug\|issue)\b` |
      | "not working", "doesn't work" | Fix | `not\s+working\|doesn't\s+work` |
      | crash, crashing | Fix | `\b(crash\|crashing)\b` |
      | add, create, build, implement, new | Add | `\b(add\|create\|build\|implement\|new)\b` |
      | enable, introduce, support | Add | `\b(enable\|introduce\|support)\b` |
      
      ## Classification Algorithm
      
      ```text
      1. Scan goal text for Fix indicators (higher priority)
      2. If Fix indicators found -> classify as Fix
      3. Else scan for Add indicators
      4. If Add indicators found -> classify as Add
      5. If both present -> classify as Fix (fixing enables the feature)
      6. If neither -> default to Add
      ```
      
      ## Examples
      
      | Goal | Classification | Reason |
      |------|---------------|--------|
      | "Fix login bug" | Fix | Contains "fix" and "bug" |
      | "Add dark mode" | Add | Contains "add" |
      | "Create auth endpoint" | Add | Contains "create" |
      | "Resolve failing tests" | Fix | Contains "resolve" and "failing" |
      | "Build new API and fix errors" | Fix | Both present, Fix takes priority |
      | "Implement user registration" | Add | Contains "implement" |
      | "Debug performance issue" | Fix | Contains "debug" and "issue" |
      
      ## Command Mapping
      
      Map goal keywords to reproduction commands:
      
      | Goal Keywords | Reproduction Command |
      |---------------|---------------------|
      | CI, pipeline, actions | `gh run view --log-failed` |
      | test, tests, spec | project test command (package.json scripts.test) |
      | type, types, typescript | `pnpm check-types` or `tsc --noEmit` |
      | lint, linting | `pnpm lint` or `eslint .` |
      | build, compile | `pnpm build` or `npm run build` |
      | deploy, deployment | `gh api` or MCP fetch to check status |
      | E2E, UI, browser, visual | MCP playwright to screenshot or run E2E suite |
      | endpoint, API, response | MCP fetch with expected status/response validation |
      | site, page, live | MCP fetch/playwright to verify live behavior |
      
      ## Fallback
      
      If no keyword match:
      1. Ask user for reproduction steps
      2. Or skip diagnosis if goal is clearly Add type
      
    • mock-quality-checks.md 3.4 KB
      # Mock Quality Checks
      
      Detect mock-only test anti-patterns where tests pass but don't actually test implementation.
      
      ## Red Flags
      
      ### 1. Mockery Pattern
      
      Test file has more mock setup than real assertions.
      
      **Detection:**
      ```bash
      grep -c "mock\|stub\|spy" test-file
      grep -c "expect(" test-file
      ```
      
      **Red flag:** Mock lines > 3x assertion lines
      
      ### 2. Missing Real Imports
      
      Test only imports test libraries, not actual module.
      
      **Detection:** Check if test imports the real implementation:
      ```javascript
      // Good: imports real module
      import { authenticate } from '../auth'
      
      // Bad: only imports test libraries
      import { describe, it, expect, vi } from 'vitest'
      ```
      
      **Red flag:** Only imports from jest/vitest/testing-library
      
      ### 3. Behavioral-Only Testing
      
      All assertions are mock interactions, no state/value checks.
      
      **Detection:**
      ```bash
      grep "toHaveBeenCalled\|spy.calledWith" test-file
      ```
      
      **Red flag:** No `toBe`, `toEqual`, `toMatch` assertions on real values
      
      ### 4. No Integration Coverage
      
      Every dependency is mocked, no real integration tested.
      
      **Detection:** Check if any tests run without mocks
      
      **Red flag:** 100% mock coverage = 0% real integration tested
      
      ### 5. Partial Mocking Issues
      
      Excessive use of `vi.spyOn`/`jest.spyOn`.
      
      **Red flag:** Mixing real and mocked behavior creates unpredictability
      
      ### 6. No Mock Cleanup
      
      Mocks persist across tests.
      
      **Detection:**
      ```bash
      grep "afterEach\|mockClear\|mockReset" test-file
      ```
      
      **Red flag:** Missing cleanup = flaky tests
      
      ## Quality Check Report Format
      
      ### When Issues Detected
      
      ```text
      Mock Quality Issues Detected
      
      File: src/auth.test.ts
      - Mock declarations: 15
      - Real assertions: 3
      - Mock ratio: 5.0x (threshold: 3x)
      - Real module import: MISSING
      - Integration tests: 0
      
      Issues:
      1. Missing import of actual auth module
      2. All assertions verify mock interactions, none check real behavior
      3. No integration test coverage
      
      Suggested fixes:
      - Import actual auth module: import { authenticate } from '../auth'
      - Add state-based assertions: expect(result).toEqual({...})
      - Create integration test with real dependencies
      - Reduce mocking to only external services (network, DB)
      
      Status: VERIFICATION_FAIL (test quality issues)
      ```
      
      ### When Tests Are Healthy
      
      ```text
      Mock Quality Check: PASS
      
      File: src/auth.test.ts
      - Mock declarations: 2 (external services only)
      - Real assertions: 12
      - Real module import: YES
      - Integration tests: 3
      - Mock cleanup: afterEach present
      
      Tests verify real behavior, not mock behavior.
      ```
      
      ## Reality Check for Test Fixes
      
      For "fix tests" specs, verify BOTH conditions:
      1. Tests pass
      2. Tests actually test real behavior (not just mocks)
      
      ### Before State
      
      ```markdown
      ## Reality Check (BEFORE)
      
      **Tests**: FAILING
      **Mock ratio**: N/A (tests failing)
      ```
      
      ### After State (Good)
      
      ```markdown
      ## Reality Check (AFTER)
      
      **Tests**: PASSING
      **Mock quality check**:
      - Mock declarations: 2
      - Real assertions: 15
      - Real module import: YES
      - Integration tests: 3
      - Mock cleanup: afterEach present
      
      **Verdict**: Tests pass AND test real behavior
      ```
      
      ### After State (Bad)
      
      ```markdown
      ## Reality Check (AFTER)
      
      **Tests**: PASSING (warning)
      **Mock quality issues**:
      - Mock declarations: 20
      - Real assertions: 4
      - Mock ratio: 5.0x (exceeds 3x threshold)
      - Real module import: MISSING
      - Integration tests: 0
      
      **Verdict**: Tests pass but only test mocks, not implementation
      **Required**: Fix test quality before marking VERIFICATION_PASS
      ```
      
  • SKILL.md 3.4 KB
    ---
    name: reality-verification
    description: This skill should be used when the user asks to "verify a fix", "reproduce failure", "diagnose issue", "check BEFORE/AFTER state", "VF task", "reality check", "check test quality", "mock-only tests", or needs guidance on verifying fixes by reproducing failures before and after implementation, or detecting mock-heavy test anti-patterns.
    version: 0.2.0
    user-invocable: false
    ---
    
    # Reality Verification
    
    For fix goals: reproduce the failure BEFORE work, verify resolution AFTER.
    
    ## Goal Detection
    
    Classify user goals to determine if diagnosis is needed. See `references/goal-detection-patterns.md` for detailed patterns.
    
    **Quick reference:**
    - Fix indicators: fix, repair, resolve, debug, patch, broken, failing, error, bug
    - Add indicators: add, create, build, implement, new
    - Conflict resolution: If both present, treat as Fix
    
    ## Command Mapping
    
    | Goal Keywords | Reproduction Command |
    |---------------|---------------------|
    | CI, pipeline | `gh run view --log-failed` |
    | test, tests | project test command |
    | type, typescript | `pnpm check-types` or `tsc --noEmit` |
    | lint | `pnpm lint` |
    | build | `pnpm build` |
    | E2E, UI | Playwright MCP browser tools |
    | API, endpoint | WebFetch tool |
    
    For E2E/deployment verification, use MCP tools (Playwright MCP browser tools for UI, WebFetch tool for APIs).
    
    ## BEFORE/AFTER Documentation
    
    ### BEFORE State (Diagnosis)
    
    Document in `.progress.md` under `## Reality Check (BEFORE)`:
    
    ```markdown
    ## Reality Check (BEFORE)
    
    **Goal type**: Fix
    **Reproduction command**: `pnpm test`
    **Failure observed**: Yes
    **Output**:
    ```
    FAIL src/auth.test.ts
      Expected: 200
      Received: 401
    ```
    **Timestamp**: 2026-01-16T10:30:00Z
    ```
    
    ### AFTER State (Verification)
    
    Document in `.progress.md` under `## Reality Check (AFTER)`:
    
    ```markdown
    ## Reality Check (AFTER)
    
    **Command**: `pnpm test`
    **Result**: PASS
    **Output**:
    ```
    PASS src/auth.test.ts
    All tests passed
    ```
    **Comparison**: BEFORE failed with 401, AFTER passes
    **Verified**: Issue resolved
    ```
    
    ## VF Task Format
    
    Add as task 4.3 (after PR creation) for fix-type specs:
    
    ```markdown
    - [ ] 4.3 VF: Verify original issue resolved
      - **Do**:
        1. Read BEFORE state from .progress.md
        2. Re-run reproduction command: `<command>`
        3. Compare output with BEFORE state
        4. Document AFTER state in .progress.md
      - **Verify**: `grep -q "Verified: Issue resolved" ./specs/<name>/.progress.md`
      - **Done when**: AFTER shows issue resolved, documented in .progress.md
      - **Commit**: `chore(<name>): verify fix resolves original issue`
    ```
    
    ## Test Quality Checks
    
    When verifying test-related fixes, check for mock-only test anti-patterns. See `references/mock-quality-checks.md` for detailed patterns.
    
    **Quick reference red flags:**
    - Mock declarations > 3x real assertions
    - Missing import of actual module under test
    - All assertions are mock interaction checks (toHaveBeenCalled)
    - No integration tests
    - Missing mock cleanup (afterEach)
    
    ## Why This Matters
    
    | Without | With |
    |---------|------|
    | "Fix CI" spec completes but CI still red | CI verified green before merge |
    | Tests "fixed" but original failure unknown | Before/after comparison proves fix |
    | Silent regressions | Explicit failure reproduction |
    | Manual verification required | Automated verification in workflow |
    | Tests pass but only test mocks | Tests verify real behavior, not mock behavior |
    | False sense of security from green tests | Confidence that tests catch real bugs |
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related