reality-verification
This skill should be used when the user asks to "verify a fix", "reproduce failure", "diagnose issue", "check BEFORE/AFTER state", "VF task", "reality check", "check test quality", "mock-only tests", or needs guidance on verifying fixes by reproducing failures before and after im
Install
npx skills add https://github.com/tzachbon/smart-ralph/tree/main/plugins/ralph-specum/skills/reality-verification
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tzachbon-smart-ralph@llmmart
git clone https://github.com/tzachbon/smart-ralph.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tzachbon/smart-ralph collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Reality Verification
For fix goals: reproduce the failure BEFORE work, verify resolution AFTER.
Goal Detection
Classify user goals to determine if diagnosis is needed. See references/goal-detection-patterns.md for detailed patterns.
Quick reference:
- Fix indicators: fix, repair, resolve, debug, patch, broken, failing, error, bug
- Add indicators: add, create, build, implement, new
- Conflict resolution: If both present, treat as Fix
Command Mapping
| Goal Keywords | Reproduction Command |
|---|---|
| CI, pipeline | gh run view --log-failed |
| test, tests | project test command |
| type, typescript | pnpm check-types or tsc --noEmit |
| lint | pnpm lint |
| build | pnpm build |
| E2E, UI | Playwright MCP browser tools |
| API, endpoint | WebFetch tool |
For E2E/deployment verification, use MCP tools (Playwright MCP browser tools for UI, WebFetch tool for APIs).
BEFORE/AFTER Documentation
BEFORE State (Diagnosis)
Document in .progress.md under ## Reality Check (BEFORE):
## Reality Check (BEFORE)
**Goal type**: Fix
**Reproduction command**: `pnpm test`
**Failure observed**: Yes
**Output**:
FAIL src/auth.test.ts Expected: 200 Received: 401
**Timestamp**: 2026-01-16T10:30:00Z
AFTER State (Verification)
Document in .progress.md under ## Reality Check (AFTER):
## Reality Check (AFTER)
**Command**: `pnpm test`
**Result**: PASS
**Output**:
PASS src/auth.test.ts All tests passed
**Comparison**: BEFORE failed with 401, AFTER passes
**Verified**: Issue resolved
VF Task Format
Add as task 4.3 (after PR creation) for fix-type specs:
- [ ] 4.3 VF: Verify original issue resolved
- **Do**:
1. Read BEFORE state from .progress.md
2. Re-run reproduction command: `<command>`
3. Compare output with BEFORE state
4. Document AFTER state in .progress.md
- **Verify**: `grep -q "Verified: Issue resolved" ./specs/<name>/.progress.md`
- **Done when**: AFTER shows issue resolved, documented in .progress.md
- **Commit**: `chore(<name>): verify fix resolves original issue`
Test Quality Checks
When verifying test-related fixes, check for mock-only test anti-patterns. See references/mock-quality-checks.md for detailed patterns.
Quick reference red flags:
- Mock declarations > 3x real assertions
- Missing import of actual module under test
- All assertions are mock interaction checks (toHaveBeenCalled)
- No integration tests
- Missing mock cleanup (afterEach)
Why This Matters
| Without | With |
|---|---|
| "Fix CI" spec completes but CI still red | CI verified green before merge |
| Tests "fixed" but original failure unknown | Before/after comparison proves fix |
| Silent regressions | Explicit failure reproduction |
| Manual verification required | Automated verification in workflow |
| Tests pass but only test mocks | Tests verify real behavior, not mock behavior |
| False sense of security from green tests | Confidence that tests catch real bugs |
Files (smart-ralph)
-
references
-
goal-detection-patterns.md 2.3 KB
# Goal Detection Patterns Regex patterns and heuristics for classifying user goals as Fix or Add type. ## Detection Heuristics | Pattern | Type | Regex | |---------|------|-------| | fix, repair, resolve, debug, patch | Fix | `\b(fix\|repair\|resolve\|debug\|patch)\b` | | broken, failing, error, bug, issue | Fix | `\b(broken\|failing\|error\|bug\|issue)\b` | | "not working", "doesn't work" | Fix | `not\s+working\|doesn't\s+work` | | crash, crashing | Fix | `\b(crash\|crashing)\b` | | add, create, build, implement, new | Add | `\b(add\|create\|build\|implement\|new)\b` | | enable, introduce, support | Add | `\b(enable\|introduce\|support)\b` | ## Classification Algorithm ```text 1. Scan goal text for Fix indicators (higher priority) 2. If Fix indicators found -> classify as Fix 3. Else scan for Add indicators 4. If Add indicators found -> classify as Add 5. If both present -> classify as Fix (fixing enables the feature) 6. If neither -> default to Add ``` ## Examples | Goal | Classification | Reason | |------|---------------|--------| | "Fix login bug" | Fix | Contains "fix" and "bug" | | "Add dark mode" | Add | Contains "add" | | "Create auth endpoint" | Add | Contains "create" | | "Resolve failing tests" | Fix | Contains "resolve" and "failing" | | "Build new API and fix errors" | Fix | Both present, Fix takes priority | | "Implement user registration" | Add | Contains "implement" | | "Debug performance issue" | Fix | Contains "debug" and "issue" | ## Command Mapping Map goal keywords to reproduction commands: | Goal Keywords | Reproduction Command | |---------------|---------------------| | CI, pipeline, actions | `gh run view --log-failed` | | test, tests, spec | project test command (package.json scripts.test) | | type, types, typescript | `pnpm check-types` or `tsc --noEmit` | | lint, linting | `pnpm lint` or `eslint .` | | build, compile | `pnpm build` or `npm run build` | | deploy, deployment | `gh api` or MCP fetch to check status | | E2E, UI, browser, visual | MCP playwright to screenshot or run E2E suite | | endpoint, API, response | MCP fetch with expected status/response validation | | site, page, live | MCP fetch/playwright to verify live behavior | ## Fallback If no keyword match: 1. Ask user for reproduction steps 2. Or skip diagnosis if goal is clearly Add type -
mock-quality-checks.md 3.4 KB
# Mock Quality Checks Detect mock-only test anti-patterns where tests pass but don't actually test implementation. ## Red Flags ### 1. Mockery Pattern Test file has more mock setup than real assertions. **Detection:** ```bash grep -c "mock\|stub\|spy" test-file grep -c "expect(" test-file ``` **Red flag:** Mock lines > 3x assertion lines ### 2. Missing Real Imports Test only imports test libraries, not actual module. **Detection:** Check if test imports the real implementation: ```javascript // Good: imports real module import { authenticate } from '../auth' // Bad: only imports test libraries import { describe, it, expect, vi } from 'vitest' ``` **Red flag:** Only imports from jest/vitest/testing-library ### 3. Behavioral-Only Testing All assertions are mock interactions, no state/value checks. **Detection:** ```bash grep "toHaveBeenCalled\|spy.calledWith" test-file ``` **Red flag:** No `toBe`, `toEqual`, `toMatch` assertions on real values ### 4. No Integration Coverage Every dependency is mocked, no real integration tested. **Detection:** Check if any tests run without mocks **Red flag:** 100% mock coverage = 0% real integration tested ### 5. Partial Mocking Issues Excessive use of `vi.spyOn`/`jest.spyOn`. **Red flag:** Mixing real and mocked behavior creates unpredictability ### 6. No Mock Cleanup Mocks persist across tests. **Detection:** ```bash grep "afterEach\|mockClear\|mockReset" test-file ``` **Red flag:** Missing cleanup = flaky tests ## Quality Check Report Format ### When Issues Detected ```text Mock Quality Issues Detected File: src/auth.test.ts - Mock declarations: 15 - Real assertions: 3 - Mock ratio: 5.0x (threshold: 3x) - Real module import: MISSING - Integration tests: 0 Issues: 1. Missing import of actual auth module 2. All assertions verify mock interactions, none check real behavior 3. No integration test coverage Suggested fixes: - Import actual auth module: import { authenticate } from '../auth' - Add state-based assertions: expect(result).toEqual({...}) - Create integration test with real dependencies - Reduce mocking to only external services (network, DB) Status: VERIFICATION_FAIL (test quality issues) ``` ### When Tests Are Healthy ```text Mock Quality Check: PASS File: src/auth.test.ts - Mock declarations: 2 (external services only) - Real assertions: 12 - Real module import: YES - Integration tests: 3 - Mock cleanup: afterEach present Tests verify real behavior, not mock behavior. ``` ## Reality Check for Test Fixes For "fix tests" specs, verify BOTH conditions: 1. Tests pass 2. Tests actually test real behavior (not just mocks) ### Before State ```markdown ## Reality Check (BEFORE) **Tests**: FAILING **Mock ratio**: N/A (tests failing) ``` ### After State (Good) ```markdown ## Reality Check (AFTER) **Tests**: PASSING **Mock quality check**: - Mock declarations: 2 - Real assertions: 15 - Real module import: YES - Integration tests: 3 - Mock cleanup: afterEach present **Verdict**: Tests pass AND test real behavior ``` ### After State (Bad) ```markdown ## Reality Check (AFTER) **Tests**: PASSING (warning) **Mock quality issues**: - Mock declarations: 20 - Real assertions: 4 - Mock ratio: 5.0x (exceeds 3x threshold) - Real module import: MISSING - Integration tests: 0 **Verdict**: Tests pass but only test mocks, not implementation **Required**: Fix test quality before marking VERIFICATION_PASS ```
-
-
SKILL.md 3.4 KB
--- name: reality-verification description: This skill should be used when the user asks to "verify a fix", "reproduce failure", "diagnose issue", "check BEFORE/AFTER state", "VF task", "reality check", "check test quality", "mock-only tests", or needs guidance on verifying fixes by reproducing failures before and after implementation, or detecting mock-heavy test anti-patterns. version: 0.2.0 user-invocable: false --- # Reality Verification For fix goals: reproduce the failure BEFORE work, verify resolution AFTER. ## Goal Detection Classify user goals to determine if diagnosis is needed. See `references/goal-detection-patterns.md` for detailed patterns. **Quick reference:** - Fix indicators: fix, repair, resolve, debug, patch, broken, failing, error, bug - Add indicators: add, create, build, implement, new - Conflict resolution: If both present, treat as Fix ## Command Mapping | Goal Keywords | Reproduction Command | |---------------|---------------------| | CI, pipeline | `gh run view --log-failed` | | test, tests | project test command | | type, typescript | `pnpm check-types` or `tsc --noEmit` | | lint | `pnpm lint` | | build | `pnpm build` | | E2E, UI | Playwright MCP browser tools | | API, endpoint | WebFetch tool | For E2E/deployment verification, use MCP tools (Playwright MCP browser tools for UI, WebFetch tool for APIs). ## BEFORE/AFTER Documentation ### BEFORE State (Diagnosis) Document in `.progress.md` under `## Reality Check (BEFORE)`: ```markdown ## Reality Check (BEFORE) **Goal type**: Fix **Reproduction command**: `pnpm test` **Failure observed**: Yes **Output**: ``` FAIL src/auth.test.ts Expected: 200 Received: 401 ``` **Timestamp**: 2026-01-16T10:30:00Z ``` ### AFTER State (Verification) Document in `.progress.md` under `## Reality Check (AFTER)`: ```markdown ## Reality Check (AFTER) **Command**: `pnpm test` **Result**: PASS **Output**: ``` PASS src/auth.test.ts All tests passed ``` **Comparison**: BEFORE failed with 401, AFTER passes **Verified**: Issue resolved ``` ## VF Task Format Add as task 4.3 (after PR creation) for fix-type specs: ```markdown - [ ] 4.3 VF: Verify original issue resolved - **Do**: 1. Read BEFORE state from .progress.md 2. Re-run reproduction command: `<command>` 3. Compare output with BEFORE state 4. Document AFTER state in .progress.md - **Verify**: `grep -q "Verified: Issue resolved" ./specs/<name>/.progress.md` - **Done when**: AFTER shows issue resolved, documented in .progress.md - **Commit**: `chore(<name>): verify fix resolves original issue` ``` ## Test Quality Checks When verifying test-related fixes, check for mock-only test anti-patterns. See `references/mock-quality-checks.md` for detailed patterns. **Quick reference red flags:** - Mock declarations > 3x real assertions - Missing import of actual module under test - All assertions are mock interaction checks (toHaveBeenCalled) - No integration tests - Missing mock cleanup (afterEach) ## Why This Matters | Without | With | |---------|------| | "Fix CI" spec completes but CI still red | CI verified green before merge | | Tests "fixed" but original failure unknown | Before/after comparison proves fix | | Silent regressions | Explicit failure reproduction | | Manual verification required | Automated verification in workflow | | Tests pass but only test mocks | Tests verify real behavior, not mock behavior | | False sense of security from green tests | Confidence that tests catch real bugs |
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.