testing
Testing: TDD, E2E, preferred patterns, verification, agent testing.
Install
npx skills add https://github.com/notque/vexjoy-agent/tree/main/skills/process/testing
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install notque-vexjoy-agent@llmmart
git clone https://github.com/notque/vexjoy-agent.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole notque/vexjoy-agent collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Testing
Seven modes. Match the request to one mode and follow its section. Read repository CLAUDE.md first -- project conventions override defaults here.
Mode Selection
| Request matches | Go to |
|---|---|
| Write tests first, TDD, red-green-refactor | TDD |
| Flaky, brittle, test smell, over-mocking, slow tests | Pattern Quality |
| Audit or prune low-value, duplicate, or implementation-coupled tests | Test Audit |
| Test an agent, subagent testing, validate agent | Agent Testing |
| Run vitest, JavaScript/TypeScript tests | Vitest Runner |
| Playwright, E2E, end-to-end, browser test | E2E (Playwright) |
| Verify completion, final check, defense in depth | Verification |
TDD
RED-GREEN-REFACTOR cycle with strict phase gates. Each feature gets its own cycle. Do not batch multiple features into one cycle.
Phase 1: RED -- Write a Failing Test
Write a test describing desired behavior before implementation exists. Use Arrange-Act-Assert, descriptive names, one concept per test. Run the test and show full output.
Every new or changed test, in any mode, must first pass the authoring gate in
references/audit-test-value.md.
Gate -- proceed only when all true:
- Test file created and saved
- Test executed
- Output shows FAILURE (not syntax/import error)
- Failure indicates missing implementation
If test passes before implementation: assertions are too weak, or the feature already exists. If test fails for wrong reason (syntax, import, setup): fix those first, then re-run until it fails for the right reason.
Phase 2: GREEN -- Minimum Implementation
Write ONLY enough code to make the failing test pass. No extra features. Hardcoded values are acceptable initially. Run the test and the full suite; show complete output.
Gate -- proceed only when all true:
- New test passes
- Full suite executed
- No other tests broken
Phase 3: REFACTOR
Improve code quality without changing behavior. Establish a green baseline, refactor incrementally, run tests after every step. Test behavior, not internals.
Gate -- proceed only when all true:
- Full suite passes
- Code quality evaluated
Phase 4: Commit
Run the full suite first. Commit only when the user asked for commits: then commit test and implementation together as one unit, with a one-sentence lowercase message in the AGENTS.md writing style (what changed and why). Otherwise leave the change uncommitted and report it.
TDD Error Recovery
| Symptom | Cause | Fix |
|---|---|---|
| Test passes in RED phase | Weak assertions or feature exists | Strengthen assertions; check for existing implementation |
| Wrong failure reason | Setup incomplete, missing deps | Fix syntax/imports first, re-run |
| Tests green but feature broken | Tests miss actual usage | Add integration tests; test with real data |
| Refactoring breaks tests | Tests coupled to internals | Test behavior not implementation; refactor in smaller steps |
Pattern Quality
Identify and fix testing mistakes across unit, integration, and E2E suites. Test behavior, be reliable, run fast, fail for the right reasons.
Phase 1: SCAN
Locate test files (*_test.go, test_*.py, *.test.ts, *.spec.js). Scan
for these 10 failure modes:
| # | Pattern | Detection Signal |
|---|---|---|
| 1 | Testing implementation details | Asserts on private fields, spy on private methods |
| 2 | Over-mocking / brittle selectors | Mock setup > 50% of test code, CSS nth-child |
| 3 | Order-dependent tests | Shared mutable state, numbered test names |
| 4 | Incomplete assertions | != nil, > 0, toBeTruthy(), no value checks |
| 5 | Over-specification | Exact timestamps, hardcoded IDs, asserting defaults |
| 6 | Ignored failures | @skip, .skip, xit, empty catch, _ = err |
| 7 | Poor naming | testFunc2, it('works'), it('handles case') |
| 8 | Missing edge cases | Only happy path, no empty/null/boundary/error tests |
| 9 | Slow test suites | Full DB reset per test, no parallelization |
| 10 | Flaky tests | sleep(), time.Sleep(), unsynchronized goroutines |
Document each finding with file:line, severity, issue, and impact.
Gate: At least one quality issue identified with file:line reference.
Phase 2: PRIORITIZE
- HIGH -- Flaky, order-dependent, ignored failures (erode trust)
- MEDIUM -- Over-mocking, incomplete assertions, missing edges (false confidence)
- LOW -- Poor naming, over-specification, slow suites (maintenance burden)
Fix one pattern at a time. Preserve test intent. Prevent over-engineering.
Gate: Findings ranked. User agrees on fix scope.
Phase 3: FIX
For each issue (highest priority first): show current code, show fixed code, apply fix, run tests. Guide toward behavior testing:
- Asserts on private fields -> test the public behavior those fields enable
- Spies on
_getUser()-> test what happens when a user exists or not - Checks exact regex -> test that validation succeeds/fails for representative inputs
Run the specific fixed test first, then the full file or package. If a fix breaks a previously-passing test, investigate before proceeding.
Gate: Each fix verified. Tests pass after each change.
Phase 4: VERIFY
Run full suite. Verify flaky tests are now deterministic (run 3x). Confirm no tests were accidentally removed or disabled. Report: bad patterns fixed, files modified, tests affected, suite status.
Gate: Full suite passes. Summary delivered.
Pattern Error Recovery
| Problem | Fix |
|---|---|
| Cannot determine if pattern is a quality issue | Check comments, consider test layer, flag MEDIUM with trade-offs |
| Fix changes test behavior | Identify original intent, write correct assertion, note as separate finding |
| Suite has hundreds of quality issues | Fix HIGH severity first, recommend TDD going forward, suggest fix-on-touch |
Test Audit
Find and remove tests that do not earn their maintenance cost, and the
test-only production seams they keep alive. Load
references/audit-test-value.md and follow it; it holds the authoring gate,
junk patterns, retention bar, evidence fields, validation, and handoff.
Use Pattern Quality to fix a test that guards real behavior badly. Use Test Audit to decide whether a test should exist at all.
| Scope | Sub-mode |
|---|---|
| Writing or changing a test | Authoring gate |
| Focused sweep of a few candidates | Audit |
| Every test one subsystem owns | Campaign |
Gate: Every deletion has all candidate evidence fields recorded. Owner and full suites pass. Production and test LOC reported separately.
To find hook or validator tests that miss real bugs, run
python3 scripts/mutation-check.py --hook hooks/<name>.py. It plants small AST
bugs, runs the tests, and lists surviving mutants. Use --all for every hook
(about 30 minutes) and --json for machine output. Run it on demand, not in CI.
Mutation testing: jev_limits.py, jev_router_common.py
Real mutation testing (cosmic-ray, not the homegrown AST checker above, which
is scoped to hooks/ and scripts/validate-*.py) for two load-bearing,
deterministic scripts outside that checker's scope: scripts/jev_limits.py
(split/budget math) and scripts/jev_router_common.py (retry/backoff logic).
Local-only, on request, same as the AST checker above -- not wired into CI.
Mirrors the switch project's own tradeoff of keeping its Stryker (JS mutation
testing) run local for speed rather than gating CI on it.
Tool choice: cosmic-ray over mutmut. mutmut 3.x copies the mutated
source into a sandboxed mutants/ tree and imports it from there; that
breaks on this repo's convention of scripts/tests/conftest.py inserting
scripts/ onto sys.path so tests can import jev_limits as a flat
top-level module (mutmut's own code warns this exact pattern causes a
silent "No Tests" result for every mutant). cosmic-ray mutates the real
file on disk, runs the configured test command, and restores the file --
no sandbox, no import-path mismatch, no extra also_copy plumbing needed
for hooks/lib/, scripts/jev_images.py, etc.
Run (installs cosmic-ray into a disposable venv; never committed):
python3 -m venv .venv && ./.venv/bin/python -m pip install cosmic-ray
./.venv/bin/cosmic-ray init scripts/cosmic-ray-jev-limits.toml /tmp/jev-limits-mutation.sqlite
./.venv/bin/cr-filter-operators /tmp/jev-limits-mutation.sqlite scripts/cosmic-ray-jev-limits.toml
./.venv/bin/cosmic-ray exec scripts/cosmic-ray-jev-limits.toml /tmp/jev-limits-mutation.sqlite
./.venv/bin/cr-report /tmp/jev-limits-mutation.sqlite
./.venv/bin/cosmic-ray init scripts/cosmic-ray-jev-router-common.toml /tmp/jev-router-common-mutation.sqlite
./.venv/bin/cr-filter-operators /tmp/jev-router-common-mutation.sqlite scripts/cosmic-ray-jev-router-common.toml
./.venv/bin/cosmic-ray exec scripts/cosmic-ray-jev-router-common.toml /tmp/jev-router-common-mutation.sqlite
./.venv/bin/cr-report /tmp/jev-router-common-mutation.sqlite
Each config's [cosmic-ray.filters.operators-filter] excludes cosmic-ray's
full pairwise operator-swap matrix (every arithmetic/bitwise operator
replaced by every other, every comparison operator by every other) down to
one classic mutant per real operator family. Most of that matrix is noise
here: both files use X | None PEP 604 type unions, which Python parses as
a BitOr node, and from __future__ import annotations stores every
annotation as a string, so a mutant that swaps that BitOr is never
evaluated and can never be killed. Without the filter, cosmic-ray init
reports 714 candidates for jev_limits.py and 1200 for
jev_router_common.py; with it, 203 and 435 actually run.
cr-report's own percentage divides survivors by the unfiltered total
job count (714 / 1200), which understates survivors once filtered-out
mutants are mixed into that denominator. Read the score as killed /
(killed + survived) over the mutants that actually ran instead.
Baseline (run 2026-10-30, cosmic-ray 8.7.0):
| File | Ran | Killed | Survived | Other | Score |
|---|---|---|---|---|---|
jev_limits.py |
203 | 90 | 113 | -- | 44% |
jev_router_common.py |
435 | 188 | 245 | 2 incompetent | 43% |
The 2 incompetent jev_router_common.py mutants are a cosmic-ray 8.7.0
ExceptionReplacer bug (AttributeError: 'PythonNode' object has no attribute 'value') on an except (A, B, ...) tuple; it is a tool
limitation, not a test gap, and is excluded from the score above.
Most survivors are NumberReplacer hits on policy constants the tests
check relationally (over/under a limit) rather than pin to an exact value
(REQUEST_TOKEN_LIMIT, SAFE_FRACTION, cache TTLs), plus check_run's
>/>= boundary comparisons and call_jev's untested telemetry/logging
branches. _retry_max's default-retry-count constant also survives --
worth a boundary test (JEV_RETRY_MAX=0 and the documented default) if this
module changes again.
Treat a re-run that drops either score below 35% as a regression signal worth investigating before merging whatever change caused it; this baseline is not a CI gate, just a documented trip wire for local, on-request runs.
Agent Testing
TDD methodology applied to agent development. Test what the agent DOES, not what the prompt SAYS. Each test runs in a fresh subagent to avoid context pollution.
Minimum Test Counts
| Agent Type | Min Tests | Coverage |
|---|---|---|
| Reviewer | 6 | 2 real issues, 2 clean, 1 edge, 1 ambiguous |
| Implementation | 5 | 2 typical, 1 complex, 1 minimal, 1 error |
| Analysis | 4 | 2 standard, 1 edge, 1 malformed |
| Routing/orchestration | 4 | 2 correct route, 1 ambiguous, 1 invalid |
No agent is simple enough to skip testing.
Phase 1: RED -- Observe Current Behavior
Read the agent file and referenced skills. Extract testable claims (inputs, output structure, routing triggers, error conditions). Write a test plan to a file. Dispatch subagent via Task tool with test inputs. Capture results verbatim. Identify failure patterns.
Gate: All cases executed. Outputs captured. Failures documented.
Phase 2: GREEN -- Fix Agent Definition
Prioritize failures by severity. Make one fix at a time. Re-run ALL test cases after each fix. If a fix causes regression, revert and try a different approach.
Gate: All cases pass. No regressions.
Phase 3: REFACTOR -- Edge Cases and Robustness
Add edge case tests (empty, large, unusual, ambiguous inputs). Run consistency tests (same input 3x; outputs should have same structure and key findings). Run full regression suite.
Gate: Edge cases handled. Consistency verified. Full suite green.
Vitest Runner
Run existing Vitest tests and report results. A check-only request does not authorize changing tests, assertions, dependencies, or configuration.
Check package.json, vitest.config.*, and vite.config.* to confirm Vitest.
Use the installed project version; avoid implicit npx downloads. If Vitest is
unavailable, report setup needed rather than installing it. Always use run;
bare vitest enters watch mode.
| Scope | Command |
|---|---|
| Full suite | npx --no vitest run --reporter=verbose 2>&1 |
| File or directory | npx --no vitest run path/to/test.ts 2>&1 |
| Test-name pattern | npx --no vitest run -t "pattern" 2>&1 |
| Coverage | npx --no vitest run --coverage 2>&1 |
--no makes npx fail instead of downloading Vitest when the project has none.
Capture exit code and full output. Report pass/fail, scope, counts, duration. For failures: retain file, test name, assertion diff, relevant stack. Nonzero exit is failure; partial output is not a passing run.
Vitest Recovery
| Problem | Fix |
|---|---|
| Vitest missing / no node_modules | Report setup needed. npm install (declared deps) or npm install -D vitest (edits package.json) only when the user asks |
| No test files found | Check naming (*.test.ts, *.spec.ts) and include/exclude globs |
| Missing DOM environment | Check for jsdom/happy-dom in config; suggest devDependency |
| Out of memory | Batch by directory, use --pool=forks or --shard=1/N |
| Failing assertions | Report mismatch; if fixing authorized, determine whether implementation or test is wrong |
E2E (Playwright)
Playwright-based E2E testing: Scaffold, Build, Run, Validate. Each phase produces an artifact and must pass its gate.
Phase 1: SCAFFOLD
- Verify
@playwright/testinstalled:npx playwright --version. If missing:npm install -D @playwright/test && npx playwright install. - Create directory structure:
tests/e2e/{auth,features,api}/,pages/,artifacts/{screenshots,traces,videos}/. - Write
playwright.config.ts. Bake in failure diagnostics:screenshot: 'only-on-failure',trace: 'on-first-retry',video: 'retain-on-failure'. CI retries:retries: process.env.CI ? 2 : 0. - Verify:
npx tsc --noEmit.
Gate: playwright.config.ts exists AND tests/e2e/ exists.
Phase 2: BUILD
Write POM classes in pages/ for each feature area. All locators use
data-testid via page.getByTestId(). No inline locators in spec files.
Write spec files in tests/e2e/<area>/. Verify: npx tsc --noEmit.
Gate: At least one .spec.ts under tests/e2e/ AND npx tsc --noEmit
exits 0.
Phase 3: RUN
- Ensure app is running (or document
BASE_URL). - Run:
npx playwright test. - If failures, isolate with
--repeat-each=5to distinguish flaky from broken. - Quarantine confirmed flaky tests with
test.fixme()and a tracking TODO. Never delete a failing test. Usetest.skip()only for environment guards.
Gate: playwright-results.json exists and parses as valid JSON.
Phase 4: VALIDATE
- Deterministic checks first: parse JSON, extract counts, identify
unexpectedandflakyentries. - LLM triage: classify each failure as (a) broken assertion, (b) selector mismatch, (c) timing/async, or (d) application bug.
- Write
e2e-report.md.
Gate: e2e-report.md exists.
E2E Error Recovery
| Symptom | Fix |
|---|---|
npx tsc --noEmit fails |
Check @playwright/test in devDeps, verify tsconfig includes test dir |
| Pass locally, fail CI | npx playwright install --with-deps in CI; verify BASE_URL |
| Results JSON missing | Check JSON reporter in config; check for OOM/process kill |
| Locator timeout on existing element | await expect(locator).toBeVisible() before interaction; check overlays |
fill() appends |
locator.clear() then locator.fill() |
| Flaky (4/5 pass) | Quarantine with test.fixme(), reproduce with --repeat-each=10, check missing waitFor |
Confirm flaky vs. broken: --repeat-each=5 --retries=0. If fails at least once
in 5, it is flaky. Fix if root cause is clear; quarantine otherwise. Verify fix
with --repeat-each=10 --retries=0 (must pass 10/10).
Verification
Defense-in-depth verification before declaring any task complete. Match checks to affected behavior and repository requirements.
Steps
- Inspect changes.
git status --shortandgit diff. Read changed code; check imports, error handling, compatibility, unintended edits. - Run required checks. Tests, build, lint, format per repository config. Start with relevant tests; run full affected suite when shared behavior changed. Do not substitute syntax checks for behavior tests.
- Verify artifacts. Check generated artifacts at expected paths. For integrations, verify four levels: EXISTS on disk, SUBSTANTIVE implementation, WIRED into callers, real DATA FLOWS through it. An unused file or hardcoded empty result is not a working feature.
- Inspect diff for problems. Debug code, secrets, placeholders, unfinished
work. Review in context: an intentional
passis not automatically a stub. - Fix and rerun. Fix failures within authorized scope; rerun affected checks. A failed required build or test blocks a success claim.
- Report. Commands, observed status, counts, limitations. Retain full logs; show actionable excerpts. Distinguish automated, manual, and unrun checks.
Default Commands
| Language | Tests | Build/syntax | Lint |
|---|---|---|---|
| Python | pytest -v |
python -m py_compile {files} |
ruff check {files} |
| Go | go test ./... -v -race |
go build ./... |
golangci-lint run ./... |
| JavaScript | npm test |
npm run build |
npm run lint |
| TypeScript | npm test |
npx tsc --noEmit |
npm run lint |
| Rust | cargo test |
cargo build |
cargo clippy |
Evidence Reuse
Reuse a passing result when it covers the current task, checked files, dependencies, and environment. Keep its command, scope, state, and log path. After edits, rerun affected checks. Do not claim inherited results as your own. Required CI checks still apply to the delivered commit.
Verification Recovery
| Problem | Fix |
|---|---|
| No tests | Manual checks; state coverage gap; add regression test if warranted |
| Missing dependencies | Use repo environment; report missing tool; unrun checks are not passes |
| Build/test failure | Retain failing command and diagnostic; identify cause, fix, rerun |
| Missing wiring or data flow | Name where integration stops; repair it |
Anti-Rationalization
| Rationalization | Required Action |
|---|---|
| "I loaded the patterns, that's enough" | Loading is not applying. Check against patterns at each gate. |
| "This task is simple, full rigor is overkill" | Apply proportionate rigor, never zero. |
| "The gate basically passes" | Either it passes with evidence or it does not. |
Completion self-check: Did I verify or assume? Did I run tests or just read code? Did I complete everything or just the "important" parts? Can I show evidence?
Deep References
Load when the signal applies.
| Signal | Load | Content |
|---|---|---|
| TDD phase steps, language commands | references/tdd-phase-guidance.md |
RED-GREEN-REFACTOR steps per language |
| TDD walkthroughs | references/tdd-examples.md |
Go, Python, JavaScript worked examples |
| BAD/GOOD code per failure mode | references/patterns-preferred-pattern-catalog.md |
Code examples per pattern per language |
| Failure mode classification | references/patterns-quality-catalog.md |
10 failure mode descriptions |
| Language-specific fix strategies | references/patterns-fix-strategies.md |
Fix patterns and tooling per language |
| Test blind spots | references/patterns-blind-spot-taxonomy.md |
6-category gap taxonomy |
| Load test scenarios | references/patterns-load-test-scenarios.md |
Smoke, stress, spike, soak configs |
| New-test gate, test audits, pruning campaigns | references/audit-test-value.md |
Authoring gate, junk patterns, retention bar, evidence, validation |
| Agent dispatch patterns | references/agents-testing-patterns.md |
Dispatch, negative, A/B |
| Agent testing examples | references/agents-examples-and-errors.md |
Worked examples and error cases |
| E2E async patterns | references/e2e-async.md |
Promise.all, race conditions, teardown |
| E2E auth testing | references/e2e-auth.md |
Login, storageState, OAuth, SSO, JWT |
| E2E config templates | references/e2e-templates.md |
playwright.config.ts, POM, CI/CD |
| E2E POM and waiting | references/e2e-playwright-patterns.md |
POM examples, multi-browser |
| E2E Web3 wallet | references/e2e-wallet-testing.md |
MetaMask testing patterns |
| E2E financial flows | references/e2e-financial-flows.md |
Payment flow testing |
| Stub detection | references/verify-adversarial-methodology.md |
Four-level checks, goal-backward verification |
| Domain checklists | references/verify-checklist.md |
Schema change, compatibility checks |
| Verification examples | references/verify-verification-examples.md |
Bug fix, refactor, migration walkthroughs |
Quick Reference: Red Flags
@skip,@ignore,xit,.skipwithout expiration datetime.sleep(),setTimeout()in test code- Test names with sequential numbers (
test1,test2) - Global mutable state accessed by multiple tests
- Mock setup spanning 20+ lines
- Empty catch blocks in tests
- Assertions like
!= nil,> 0,toBeTruthy()without value checks
Strict TDD prevents most quality issues: RED catches incomplete assertions, GREEN minimum prevents over-specification, watching failure confirms you test behavior not mocks, incremental cycles prevent interdependence, refactor phase reveals implementation coupling.
Files (vexjoy-agent)
-
references
-
agents-examples-and-errors.md 5.4 KB
Loading… -
agents-testing-patterns.md 4 KB
Loading… -
audit-test-value.md 8.3 KB
Loading… -
e2e-async.md 8.9 KB
Loading… -
e2e-auth.md 11.6 KB
Loading… -
e2e-financial-flows.md 1.7 KB
Loading… -
e2e-playwright-patterns.md 8.4 KB
Loading… -
e2e-templates.md 6.9 KB
Loading… -
e2e-wallet-testing.md 2.2 KB
Loading… -
patterns-blind-spot-taxonomy.md 4.5 KB
Loading… -
patterns-fix-strategies.md 5 KB
Loading… -
patterns-load-test-scenarios.md 4.8 KB
Loading… -
patterns-preferred-pattern-catalog.md 14.3 KB
Loading… -
patterns-quality-catalog.md 7.9 KB
Loading… -
tdd-examples.md 19.4 KB
Loading… -
tdd-phase-guidance.md 10.1 KB
Loading… -
verify-adversarial-methodology.md 9.5 KB
Loading… -
verify-checklist.md 15.4 KB
Loading… -
verify-verification-examples.md 14.6 KB
Loading…
-
-
SKILL.md 23.1 KB
--- name: testing description: "Testing: TDD, E2E, preferred patterns, test-value audits, verification, agent testing." user-invocable: false allowed-tools: - Read - Write - Bash - Grep - Glob - Edit - Task - Skill - Agent agent: testing-automation-engineer routing: not_for: "code review (use review), linting (use code-quality)" triggers: - "TDD" - "test first" - "red green refactor" - "write tests first" - "test-driven" - "tests before code" - "flaky test" - "brittle test" - "test smell" - "test quality issue" - "slow tests" - "over-mocking" - "test audit" - "audit tests" - "low-value tests" - "prune tests" - "test bloat" - "delete useless tests" - "test agents" - "agent testing" - "subagent testing" - "run vitest" - "JavaScript tests" - "TypeScript tests" - "playwright" - "E2E test" - "end-to-end" - "browser test" - "verify completion" - "run tests" - "final verification" category: testing pairs_with: - review - code-quality - workflow --- # Testing Seven modes. Match the request to one mode and follow its section. Read repository CLAUDE.md first -- project conventions override defaults here. ## Mode Selection | Request matches | Go to | |---|---| | Write tests first, TDD, red-green-refactor | **TDD** | | Flaky, brittle, test smell, over-mocking, slow tests | **Pattern Quality** | | Audit or prune low-value, duplicate, or implementation-coupled tests | **Test Audit** | | Test an agent, subagent testing, validate agent | **Agent Testing** | | Run vitest, JavaScript/TypeScript tests | **Vitest Runner** | | Playwright, E2E, end-to-end, browser test | **E2E (Playwright)** | | Verify completion, final check, defense in depth | **Verification** | --- ## TDD RED-GREEN-REFACTOR cycle with strict phase gates. Each feature gets its own cycle. Do not batch multiple features into one cycle. ### Phase 1: RED -- Write a Failing Test Write a test describing desired behavior before implementation exists. Use Arrange-Act-Assert, descriptive names, one concept per test. Run the test and show full output. Every new or changed test, in any mode, must first pass the authoring gate in `references/audit-test-value.md`. **Gate** -- proceed only when all true: - Test file created and saved - Test executed - Output shows FAILURE (not syntax/import error) - Failure indicates missing implementation If test passes before implementation: assertions are too weak, or the feature already exists. If test fails for wrong reason (syntax, import, setup): fix those first, then re-run until it fails for the right reason. ### Phase 2: GREEN -- Minimum Implementation Write ONLY enough code to make the failing test pass. No extra features. Hardcoded values are acceptable initially. Run the test and the full suite; show complete output. **Gate** -- proceed only when all true: - New test passes - Full suite executed - No other tests broken ### Phase 3: REFACTOR Improve code quality without changing behavior. Establish a green baseline, refactor incrementally, run tests after every step. Test behavior, not internals. **Gate** -- proceed only when all true: - Full suite passes - Code quality evaluated ### Phase 4: Commit Run the full suite first. Commit only when the user asked for commits: then commit test and implementation together as one unit, with a one-sentence lowercase message in the AGENTS.md writing style (what changed and why). Otherwise leave the change uncommitted and report it. ### TDD Error Recovery | Symptom | Cause | Fix | |---|---|---| | Test passes in RED phase | Weak assertions or feature exists | Strengthen assertions; check for existing implementation | | Wrong failure reason | Setup incomplete, missing deps | Fix syntax/imports first, re-run | | Tests green but feature broken | Tests miss actual usage | Add integration tests; test with real data | | Refactoring breaks tests | Tests coupled to internals | Test behavior not implementation; refactor in smaller steps | --- ## Pattern Quality Identify and fix testing mistakes across unit, integration, and E2E suites. Test behavior, be reliable, run fast, fail for the right reasons. ### Phase 1: SCAN Locate test files (`*_test.go`, `test_*.py`, `*.test.ts`, `*.spec.js`). Scan for these 10 failure modes: | # | Pattern | Detection Signal | |---|---|---| | 1 | Testing implementation details | Asserts on private fields, spy on private methods | | 2 | Over-mocking / brittle selectors | Mock setup > 50% of test code, CSS nth-child | | 3 | Order-dependent tests | Shared mutable state, numbered test names | | 4 | Incomplete assertions | `!= nil`, `> 0`, `toBeTruthy()`, no value checks | | 5 | Over-specification | Exact timestamps, hardcoded IDs, asserting defaults | | 6 | Ignored failures | `@skip`, `.skip`, `xit`, empty catch, `_ = err` | | 7 | Poor naming | `testFunc2`, `it('works')`, `it('handles case')` | | 8 | Missing edge cases | Only happy path, no empty/null/boundary/error tests | | 9 | Slow test suites | Full DB reset per test, no parallelization | | 10 | Flaky tests | `sleep()`, `time.Sleep()`, unsynchronized goroutines | Document each finding with file:line, severity, issue, and impact. **Gate**: At least one quality issue identified with file:line reference. ### Phase 2: PRIORITIZE 1. **HIGH** -- Flaky, order-dependent, ignored failures (erode trust) 2. **MEDIUM** -- Over-mocking, incomplete assertions, missing edges (false confidence) 3. **LOW** -- Poor naming, over-specification, slow suites (maintenance burden) Fix one pattern at a time. Preserve test intent. Prevent over-engineering. **Gate**: Findings ranked. User agrees on fix scope. ### Phase 3: FIX For each issue (highest priority first): show current code, show fixed code, apply fix, run tests. Guide toward behavior testing: - Asserts on private fields -> test the public behavior those fields enable - Spies on `_getUser()` -> test what happens when a user exists or not - Checks exact regex -> test that validation succeeds/fails for representative inputs Run the specific fixed test first, then the full file or package. If a fix breaks a previously-passing test, investigate before proceeding. **Gate**: Each fix verified. Tests pass after each change. ### Phase 4: VERIFY Run full suite. Verify flaky tests are now deterministic (run 3x). Confirm no tests were accidentally removed or disabled. Report: bad patterns fixed, files modified, tests affected, suite status. **Gate**: Full suite passes. Summary delivered. ### Pattern Error Recovery | Problem | Fix | |---|---| | Cannot determine if pattern is a quality issue | Check comments, consider test layer, flag MEDIUM with trade-offs | | Fix changes test behavior | Identify original intent, write correct assertion, note as separate finding | | Suite has hundreds of quality issues | Fix HIGH severity first, recommend TDD going forward, suggest fix-on-touch | --- ## Test Audit Find and remove tests that do not earn their maintenance cost, and the test-only production seams they keep alive. Load `references/audit-test-value.md` and follow it; it holds the authoring gate, junk patterns, retention bar, evidence fields, validation, and handoff. Use Pattern Quality to fix a test that guards real behavior badly. Use Test Audit to decide whether a test should exist at all. | Scope | Sub-mode | |---|---| | Writing or changing a test | Authoring gate | | Focused sweep of a few candidates | Audit | | Every test one subsystem owns | Campaign | **Gate**: Every deletion has all candidate evidence fields recorded. Owner and full suites pass. Production and test LOC reported separately. To find hook or validator tests that miss real bugs, run `python3 scripts/mutation-check.py --hook hooks/<name>.py`. It plants small AST bugs, runs the tests, and lists surviving mutants. Use `--all` for every hook (about 30 minutes) and `--json` for machine output. Run it on demand, not in CI. ### Mutation testing: jev_limits.py, jev_router_common.py Real mutation testing (cosmic-ray, not the homegrown AST checker above, which is scoped to `hooks/` and `scripts/validate-*.py`) for two load-bearing, deterministic scripts outside that checker's scope: `scripts/jev_limits.py` (split/budget math) and `scripts/jev_router_common.py` (retry/backoff logic). Local-only, on request, same as the AST checker above -- not wired into CI. Mirrors the switch project's own tradeoff of keeping its Stryker (JS mutation testing) run local for speed rather than gating CI on it. **Tool choice: cosmic-ray over mutmut.** mutmut 3.x copies the mutated source into a sandboxed `mutants/` tree and imports it from there; that breaks on this repo's convention of `scripts/tests/conftest.py` inserting `scripts/` onto `sys.path` so tests can `import jev_limits` as a flat top-level module (mutmut's own code warns this exact pattern causes a silent "No Tests" result for every mutant). cosmic-ray mutates the real file on disk, runs the configured test command, and restores the file -- no sandbox, no import-path mismatch, no extra `also_copy` plumbing needed for `hooks/lib/`, `scripts/jev_images.py`, etc. **Run** (installs cosmic-ray into a disposable venv; never committed): ```bash python3 -m venv .venv && ./.venv/bin/python -m pip install cosmic-ray ./.venv/bin/cosmic-ray init scripts/cosmic-ray-jev-limits.toml /tmp/jev-limits-mutation.sqlite ./.venv/bin/cr-filter-operators /tmp/jev-limits-mutation.sqlite scripts/cosmic-ray-jev-limits.toml ./.venv/bin/cosmic-ray exec scripts/cosmic-ray-jev-limits.toml /tmp/jev-limits-mutation.sqlite ./.venv/bin/cr-report /tmp/jev-limits-mutation.sqlite ./.venv/bin/cosmic-ray init scripts/cosmic-ray-jev-router-common.toml /tmp/jev-router-common-mutation.sqlite ./.venv/bin/cr-filter-operators /tmp/jev-router-common-mutation.sqlite scripts/cosmic-ray-jev-router-common.toml ./.venv/bin/cosmic-ray exec scripts/cosmic-ray-jev-router-common.toml /tmp/jev-router-common-mutation.sqlite ./.venv/bin/cr-report /tmp/jev-router-common-mutation.sqlite ``` Each config's `[cosmic-ray.filters.operators-filter]` excludes cosmic-ray's full pairwise operator-swap matrix (every arithmetic/bitwise operator replaced by every other, every comparison operator by every other) down to one classic mutant per real operator family. Most of that matrix is noise here: both files use `X | None` PEP 604 type unions, which Python parses as a `BitOr` node, and `from __future__ import annotations` stores every annotation as a string, so a mutant that swaps that `BitOr` is never evaluated and can never be killed. Without the filter, `cosmic-ray init` reports 714 candidates for `jev_limits.py` and 1200 for `jev_router_common.py`; with it, 203 and 435 actually run. `cr-report`'s own percentage divides survivors by the *unfiltered* total job count (714 / 1200), which understates survivors once filtered-out mutants are mixed into that denominator. Read the score as killed / (killed + survived) over the mutants that actually ran instead. **Baseline** (run 2026-10-30, cosmic-ray 8.7.0): | File | Ran | Killed | Survived | Other | Score | |---|---|---|---|---|---| | `jev_limits.py` | 203 | 90 | 113 | -- | 44% | | `jev_router_common.py` | 435 | 188 | 245 | 2 incompetent | 43% | The 2 incompetent `jev_router_common.py` mutants are a cosmic-ray 8.7.0 `ExceptionReplacer` bug (`AttributeError: 'PythonNode' object has no attribute 'value'`) on an `except (A, B, ...)` tuple; it is a tool limitation, not a test gap, and is excluded from the score above. Most survivors are `NumberReplacer` hits on policy constants the tests check relationally (over/under a limit) rather than pin to an exact value (`REQUEST_TOKEN_LIMIT`, `SAFE_FRACTION`, cache TTLs), plus `check_run`'s `>`/`>=` boundary comparisons and `call_jev`'s untested telemetry/logging branches. `_retry_max`'s default-retry-count constant also survives -- worth a boundary test (`JEV_RETRY_MAX=0` and the documented default) if this module changes again. Treat a re-run that drops either score below 35% as a regression signal worth investigating before merging whatever change caused it; this baseline is not a CI gate, just a documented trip wire for local, on-request runs. --- ## Agent Testing TDD methodology applied to agent development. Test what the agent DOES, not what the prompt SAYS. Each test runs in a fresh subagent to avoid context pollution. ### Minimum Test Counts | Agent Type | Min Tests | Coverage | |---|---|---| | Reviewer | 6 | 2 real issues, 2 clean, 1 edge, 1 ambiguous | | Implementation | 5 | 2 typical, 1 complex, 1 minimal, 1 error | | Analysis | 4 | 2 standard, 1 edge, 1 malformed | | Routing/orchestration | 4 | 2 correct route, 1 ambiguous, 1 invalid | No agent is simple enough to skip testing. ### Phase 1: RED -- Observe Current Behavior Read the agent file and referenced skills. Extract testable claims (inputs, output structure, routing triggers, error conditions). Write a test plan to a file. Dispatch subagent via Task tool with test inputs. Capture results verbatim. Identify failure patterns. **Gate**: All cases executed. Outputs captured. Failures documented. ### Phase 2: GREEN -- Fix Agent Definition Prioritize failures by severity. Make one fix at a time. Re-run ALL test cases after each fix. If a fix causes regression, revert and try a different approach. **Gate**: All cases pass. No regressions. ### Phase 3: REFACTOR -- Edge Cases and Robustness Add edge case tests (empty, large, unusual, ambiguous inputs). Run consistency tests (same input 3x; outputs should have same structure and key findings). Run full regression suite. **Gate**: Edge cases handled. Consistency verified. Full suite green. --- ## Vitest Runner Run existing Vitest tests and report results. A check-only request does not authorize changing tests, assertions, dependencies, or configuration. Check `package.json`, `vitest.config.*`, and `vite.config.*` to confirm Vitest. Use the installed project version; avoid implicit npx downloads. If Vitest is unavailable, report setup needed rather than installing it. Always use `run`; bare `vitest` enters watch mode. | Scope | Command | |---|---| | Full suite | `npx --no vitest run --reporter=verbose 2>&1` | | File or directory | `npx --no vitest run path/to/test.ts 2>&1` | | Test-name pattern | `npx --no vitest run -t "pattern" 2>&1` | | Coverage | `npx --no vitest run --coverage 2>&1` | `--no` makes npx fail instead of downloading Vitest when the project has none. Capture exit code and full output. Report pass/fail, scope, counts, duration. For failures: retain file, test name, assertion diff, relevant stack. Nonzero exit is failure; partial output is not a passing run. ### Vitest Recovery | Problem | Fix | |---|---| | Vitest missing / no node_modules | Report setup needed. `npm install` (declared deps) or `npm install -D vitest` (edits `package.json`) only when the user asks | | No test files found | Check naming (`*.test.ts`, `*.spec.ts`) and include/exclude globs | | Missing DOM environment | Check for `jsdom`/`happy-dom` in config; suggest devDependency | | Out of memory | Batch by directory, use `--pool=forks` or `--shard=1/N` | | Failing assertions | Report mismatch; if fixing authorized, determine whether implementation or test is wrong | --- ## E2E (Playwright) Playwright-based E2E testing: Scaffold, Build, Run, Validate. Each phase produces an artifact and must pass its gate. ### Phase 1: SCAFFOLD 1. Verify `@playwright/test` installed: `npx playwright --version`. If missing: `npm install -D @playwright/test && npx playwright install`. 2. Create directory structure: `tests/e2e/{auth,features,api}/`, `pages/`, `artifacts/{screenshots,traces,videos}/`. 3. Write `playwright.config.ts`. Bake in failure diagnostics: `screenshot: 'only-on-failure'`, `trace: 'on-first-retry'`, `video: 'retain-on-failure'`. CI retries: `retries: process.env.CI ? 2 : 0`. 4. Verify: `npx tsc --noEmit`. **Gate**: `playwright.config.ts` exists AND `tests/e2e/` exists. ### Phase 2: BUILD Write POM classes in `pages/` for each feature area. All locators use `data-testid` via `page.getByTestId()`. No inline locators in spec files. Write spec files in `tests/e2e/<area>/`. Verify: `npx tsc --noEmit`. **Gate**: At least one `.spec.ts` under `tests/e2e/` AND `npx tsc --noEmit` exits 0. ### Phase 3: RUN 1. Ensure app is running (or document `BASE_URL`). 2. Run: `npx playwright test`. 3. If failures, isolate with `--repeat-each=5` to distinguish flaky from broken. 4. Quarantine confirmed flaky tests with `test.fixme()` and a tracking TODO. Never delete a failing test. Use `test.skip()` only for environment guards. **Gate**: `playwright-results.json` exists and parses as valid JSON. ### Phase 4: VALIDATE 1. Deterministic checks first: parse JSON, extract counts, identify `unexpected` and `flaky` entries. 2. LLM triage: classify each failure as (a) broken assertion, (b) selector mismatch, (c) timing/async, or (d) application bug. 3. Write `e2e-report.md`. **Gate**: `e2e-report.md` exists. ### E2E Error Recovery | Symptom | Fix | |---|---| | `npx tsc --noEmit` fails | Check `@playwright/test` in devDeps, verify tsconfig includes test dir | | Pass locally, fail CI | `npx playwright install --with-deps` in CI; verify `BASE_URL` | | Results JSON missing | Check JSON reporter in config; check for OOM/process kill | | Locator timeout on existing element | `await expect(locator).toBeVisible()` before interaction; check overlays | | `fill()` appends | `locator.clear()` then `locator.fill()` | | Flaky (4/5 pass) | Quarantine with `test.fixme()`, reproduce with `--repeat-each=10`, check missing `waitFor` | Confirm flaky vs. broken: `--repeat-each=5 --retries=0`. If fails at least once in 5, it is flaky. Fix if root cause is clear; quarantine otherwise. Verify fix with `--repeat-each=10 --retries=0` (must pass 10/10). --- ## Verification Defense-in-depth verification before declaring any task complete. Match checks to affected behavior and repository requirements. ### Steps 1. **Inspect changes.** `git status --short` and `git diff`. Read changed code; check imports, error handling, compatibility, unintended edits. 2. **Run required checks.** Tests, build, lint, format per repository config. Start with relevant tests; run full affected suite when shared behavior changed. Do not substitute syntax checks for behavior tests. 3. **Verify artifacts.** Check generated artifacts at expected paths. For integrations, verify four levels: **EXISTS** on disk, **SUBSTANTIVE** implementation, **WIRED** into callers, real **DATA FLOWS** through it. An unused file or hardcoded empty result is not a working feature. 4. **Inspect diff for problems.** Debug code, secrets, placeholders, unfinished work. Review in context: an intentional `pass` is not automatically a stub. 5. **Fix and rerun.** Fix failures within authorized scope; rerun affected checks. A failed required build or test blocks a success claim. 6. **Report.** Commands, observed status, counts, limitations. Retain full logs; show actionable excerpts. Distinguish automated, manual, and unrun checks. ### Default Commands | Language | Tests | Build/syntax | Lint | |---|---|---|---| | Python | `pytest -v` | `python -m py_compile {files}` | `ruff check {files}` | | Go | `go test ./... -v -race` | `go build ./...` | `golangci-lint run ./...` | | JavaScript | `npm test` | `npm run build` | `npm run lint` | | TypeScript | `npm test` | `npx tsc --noEmit` | `npm run lint` | | Rust | `cargo test` | `cargo build` | `cargo clippy` | ### Evidence Reuse Reuse a passing result when it covers the current task, checked files, dependencies, and environment. Keep its command, scope, state, and log path. After edits, rerun affected checks. Do not claim inherited results as your own. Required CI checks still apply to the delivered commit. ### Verification Recovery | Problem | Fix | |---|---| | No tests | Manual checks; state coverage gap; add regression test if warranted | | Missing dependencies | Use repo environment; report missing tool; unrun checks are not passes | | Build/test failure | Retain failing command and diagnostic; identify cause, fix, rerun | | Missing wiring or data flow | Name where integration stops; repair it | ### Anti-Rationalization | Rationalization | Required Action | |---|---| | "I loaded the patterns, that's enough" | Loading is not applying. Check against patterns at each gate. | | "This task is simple, full rigor is overkill" | Apply proportionate rigor, never zero. | | "The gate basically passes" | Either it passes with evidence or it does not. | Completion self-check: Did I verify or assume? Did I run tests or just read code? Did I complete everything or just the "important" parts? Can I show evidence? --- ## Deep References Load when the signal applies. | Signal | Load | Content | |---|---|---| | TDD phase steps, language commands | `references/tdd-phase-guidance.md` | RED-GREEN-REFACTOR steps per language | | TDD walkthroughs | `references/tdd-examples.md` | Go, Python, JavaScript worked examples | | BAD/GOOD code per failure mode | `references/patterns-preferred-pattern-catalog.md` | Code examples per pattern per language | | Failure mode classification | `references/patterns-quality-catalog.md` | 10 failure mode descriptions | | Language-specific fix strategies | `references/patterns-fix-strategies.md` | Fix patterns and tooling per language | | Test blind spots | `references/patterns-blind-spot-taxonomy.md` | 6-category gap taxonomy | | Load test scenarios | `references/patterns-load-test-scenarios.md` | Smoke, stress, spike, soak configs | | New-test gate, test audits, pruning campaigns | `references/audit-test-value.md` | Authoring gate, junk patterns, retention bar, evidence, validation | | Agent dispatch patterns | `references/agents-testing-patterns.md` | Dispatch, negative, A/B | | Agent testing examples | `references/agents-examples-and-errors.md` | Worked examples and error cases | | E2E async patterns | `references/e2e-async.md` | Promise.all, race conditions, teardown | | E2E auth testing | `references/e2e-auth.md` | Login, storageState, OAuth, SSO, JWT | | E2E config templates | `references/e2e-templates.md` | playwright.config.ts, POM, CI/CD | | E2E POM and waiting | `references/e2e-playwright-patterns.md` | POM examples, multi-browser | | E2E Web3 wallet | `references/e2e-wallet-testing.md` | MetaMask testing patterns | | E2E financial flows | `references/e2e-financial-flows.md` | Payment flow testing | | Stub detection | `references/verify-adversarial-methodology.md` | Four-level checks, goal-backward verification | | Domain checklists | `references/verify-checklist.md` | Schema change, compatibility checks | | Verification examples | `references/verify-verification-examples.md` | Bug fix, refactor, migration walkthroughs | --- ## Quick Reference: Red Flags - `@skip`, `@ignore`, `xit`, `.skip` without expiration date - `time.sleep()`, `setTimeout()` in test code - Test names with sequential numbers (`test1`, `test2`) - Global mutable state accessed by multiple tests - Mock setup spanning 20+ lines - Empty catch blocks in tests - Assertions like `!= nil`, `> 0`, `toBeTruthy()` without value checks Strict TDD prevents most quality issues: RED catches incomplete assertions, GREEN minimum prevents over-specification, watching failure confirms you test behavior not mocks, incremental cycles prevent interdependence, refactor phase reveals implementation coupling.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.