systematic-debugging
Use when a bug is non-obvious or has resisted quick fixes — applies a 4-phase root cause analysis to find and permanently resolve the issue rather than patching symptoms
Install
npx skills add https://github.com/drvoss/everything-copilot-cli/tree/main/skills/development/systematic-debugging
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install drvoss-everything-copilot-cli@llmmart
git clone https://github.com/drvoss/everything-copilot-cli.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole drvoss/everything-copilot-cli collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Systematic Debugging
A structured 4-phase methodology for diagnosing and fixing non-obvious bugs. Prevents the common failure modes of "shotgun debugging" (random changes hoping something sticks) and "symptom patching" (fixing the visible error without understanding the cause).
When to Use
- A bug has resisted one or more quick-fix attempts
- The bug only appears in certain environments (prod but not dev, intermittent)
- The cause is unclear after an initial look
- A "fixed" bug keeps coming back
- Multiple engineers have looked at it without resolution
When NOT to Use
| Instead of systematic-debugging | Use |
|---|---|
| Obvious typo or off-by-one | Fix directly |
| Build/compilation failure | fix-build-errors skill |
| Performance problem (not a bug) | profiling tools |
| Security vulnerability | security-scan skill |
The 4-Phase Process
Phase 1 — Reproduce
A bug you cannot reproduce reliably cannot be debugged reliably.
Goal: Get a 100% reproducible test case that triggers the bug on demand.
# 1. Note the exact symptoms: error message, stack trace, request/response
# 2. Identify the minimum inputs that trigger it
# 3. Write a failing test that captures the reproduction
# Find the code path involved
Select-String -Path "src\\**\\*.ts" -Pattern "<error keyword>"
# Check recent changes
git --no-pager log --oneline -20
git --no-pager diff HEAD~5 -- <suspect file>
Reproduction criteria:
- Bug triggers on demand with a specific input or sequence
- Bug does NOT trigger with a slightly different (correct) input
- Reproduction is encoded as a failing test
When it is safe, run the minimal reproduction at least twice and record both commands and outputs;
one observation is not proof of reproducibility. If outcomes vary, label the reproduction status
intermittent rather than hiding the variation.
Before handing the issue to diagnosis or another session, produce a reproduction brief with:
- target and commit
- runtime, OS, lockfile, and feature flags
- expected behavior as an observation, without a cause hypothesis
- actual behavior as an observation, without a cause hypothesis
- minimal steps and fixtures
- status:
yes,no, orintermittent - commands, logs, traces, or other evidence
- unknowns and the next hypothesis to test
The brief is the handoff artifact at the reproduce-before-repair boundary already enforced here
and in fix-github-issue. Never alter production data to obtain it,
and redact secrets from evidence.
If you cannot reproduce it in < 15 minutes, move to Phase 2 anyway — understanding the system often reveals the trigger.
Phase 2 — Understand the System
Bugs live in the gap between what you think the code does and what it actually does. Close that gap before guessing at solutions.
# Read the code path end-to-end, not just the error location
# Follow data from entry point to crash point
# Trace the call stack
Select-String -Path "src\\**\\*.ts" -Pattern "<function name>"
# Check state at each step — add temporary logs if needed
# Never modify production behavior during debugging
Questions to answer:
- What is the expected behavior? (From spec, tests, or docs)
- What is the actual behavior? (From error, logs, or reproduction)
- Where exactly does the actual diverge from expected? (Specific line or state)
- What assumptions does this code make? Which one is violated?
Tools:
# Read the function signature and its callers
Select-String -Path "src\\**\\*.ts" -Pattern "functionName"
# Check what data looks like going in
# Add a temporary debug log (remember to remove before committing)
# console.log('[DEBUG]', JSON.stringify(input, null, 2));
# Check recent git changes to the file
git --no-pager log --follow -p -- src/path/to/file.ts | Select-Object -First 80
Phase 3 — Hypothesize and Verify
Generate one specific hypothesis about the root cause, then verify or falsify it. Do not try multiple fixes simultaneously — you will not know which one worked.
Hypothesis template:
Hypothesis: The bug is caused by [specific condition].
Prediction: If I [specific change], then [specific observable outcome].
Test: [How I will verify the prediction]
Example:
Hypothesis: The JWT expiry check compares Unix timestamp (seconds) to Date.now() (ms).
Prediction: If I divide Date.now() by 1000, the check will pass for valid tokens.
Test: The reproduction test case will pass with this change.
Verify with a targeted change:
# Make only the minimal change needed to test the hypothesis
# Run the reproduction test
npm test -- --testPathPattern="reproduction-test"
# If the test passes: hypothesis confirmed → write a proper fix
# If the test fails: hypothesis falsified → form a new hypothesis
Repeat until a hypothesis is confirmed. Do not patch symptoms.
Phase 4 — Fix and Prevent Recurrence
The root cause is confirmed. Now fix it properly and prevent it from returning.
# 1. Write the real fix (not the hypothesis test hack)
# 2. Ensure the reproduction test passes with the fix
# 3. Write additional regression tests for edge cases
# 4. Run the full test suite — no regressions
npm test
# 5. Remove any temporary debug logs
Select-String -Path "src\\**\\*.ts" -Pattern "\\[DEBUG\\]"
# 6. Write a clear commit message explaining root cause
git commit -m "fix: <what was wrong and why>
Root cause: <explanation of the actual bug>
Fix: <what was changed and why it resolves the root cause>
Regression test: <name of the test that now covers this>"
Prevention checklist:
- Root cause is documented in the commit message
- At least one regression test covers the specific scenario
- Related code paths are checked for the same pattern
- If the bug was a class of error (e.g., "timestamp unit mismatch"), check for other instances
Common Rationalizations
| Rationalization | Reality |
|---|---|
| "I'll just try a few things and see what sticks" | Shotgun debugging wastes hours and leaves you without understanding. A methodical approach finds the bug faster. |
| "It's probably a framework/library bug" | It's almost never a framework bug. Blame your code first; verify framework behavior before blaming it. |
| "I can't reproduce it, so I'll just add more logging" | Logging without a hypothesis produces noise. Reproduce first, then verify a specific hypothesis. |
| "I understand the code well enough" | Write down what you think the code does before reading it. The gap between your mental model and reality is where the bug lives. |
| "The fix is obvious, I don't need a regression test" | Obvious fixes regress. The regression test is how the next engineer (or future you) learns the lesson without re-debugging. |
Red Flags
- You've made more than 3 changes without a confirmed hypothesis
- The bug "disappeared" without a clear explanation
- Your fix makes the test pass but you don't understand why
- The same bug has been "fixed" before
- No regression test was added after the fix
Escalation — The Rule of Three
If you have made 3 or more hypothesis-driven fix attempts without resolution:
- Stop making further fixes
- Collect the evidence — record each hypothesis, the test you ran, and why it was falsified
- Escalate to architectural review using the
councilskill:- Architect — is the design itself creating the failure?
- Skeptic — what assumption is everyone making without checking?
- Pragmatist — what is the fastest path back to a working system?
- Critic — would a redesign be cheaper than more incremental fixes?
The Rule of Three usually means the problem is architectural, not an implementation detail. A fourth fix attempt without escalation is often just another symptom patch.
Verification
- Bug is reproducible with a specific test case
- Root cause is identified (not just the symptom)
- Fix addresses the root cause, not the symptom
- Regression test exists and passes
- Full test suite passes with no new failures
- Commit message explains the root cause
Tips
Binary search the bug: use
git bisectto find the commit that introduced itgit bisect start git bisect bad HEAD git bisect good <last-known-good-commit> # git bisect will check out commits for you to testRubber duck debugging: explain the bug out loud (or to an agent). The act of articulating often reveals the assumption you didn't know you were making.
Check the obvious last: timestamp units, off-by-one errors, null vs undefined, async/await missing, wrong environment variable — these are common root causes.
Use the
exploreagent to trace a call path across many files simultaneously.
See Also
fix-build-errors— compilation and build failurestdd-workflow— write regression tests as part of the fixsecurity-scan— if the bug is security-relatedcouncil— escalate persistent failures to a structured architectural review
Files (everything-copilot-cli)
-
SKILL.md 9.5 KB
--- name: systematic-debugging license: MIT description: Use when a bug is non-obvious or has resisted quick fixes — applies a 4-phase root cause analysis to find and permanently resolve the issue rather than patching symptoms metadata: category: development agent_type: general-purpose --- # Systematic Debugging A structured 4-phase methodology for diagnosing and fixing non-obvious bugs. Prevents the common failure modes of "shotgun debugging" (random changes hoping something sticks) and "symptom patching" (fixing the visible error without understanding the cause). ## When to Use - A bug has resisted one or more quick-fix attempts - The bug only appears in certain environments (prod but not dev, intermittent) - The cause is unclear after an initial look - A "fixed" bug keeps coming back - Multiple engineers have looked at it without resolution ## When NOT to Use | Instead of systematic-debugging | Use | |---------------------------------|-----| | Obvious typo or off-by-one | Fix directly | | Build/compilation failure | `fix-build-errors` skill | | Performance problem (not a bug) | profiling tools | | Security vulnerability | `security-scan` skill | ## The 4-Phase Process ### Phase 1 — Reproduce A bug you cannot reproduce reliably cannot be debugged reliably. **Goal**: Get a 100% reproducible test case that triggers the bug on demand. ```powershell # 1. Note the exact symptoms: error message, stack trace, request/response # 2. Identify the minimum inputs that trigger it # 3. Write a failing test that captures the reproduction # Find the code path involved Select-String -Path "src\\**\\*.ts" -Pattern "<error keyword>" # Check recent changes git --no-pager log --oneline -20 git --no-pager diff HEAD~5 -- <suspect file> ``` **Reproduction criteria:** - [ ] Bug triggers on demand with a specific input or sequence - [ ] Bug does NOT trigger with a slightly different (correct) input - [ ] Reproduction is encoded as a failing test When it is safe, run the minimal reproduction at least twice and record both commands and outputs; one observation is not proof of reproducibility. If outcomes vary, label the reproduction status `intermittent` rather than hiding the variation. Before handing the issue to diagnosis or another session, produce a reproduction brief with: 1. target and commit 2. runtime, OS, lockfile, and feature flags 3. expected behavior as an observation, without a cause hypothesis 4. actual behavior as an observation, without a cause hypothesis 5. minimal steps and fixtures 6. status: `yes`, `no`, or `intermittent` 7. commands, logs, traces, or other evidence 8. unknowns and the next hypothesis to test The brief is the handoff artifact at the reproduce-before-repair boundary already enforced here and in [`fix-github-issue`](../fix-github-issue/SKILL.md). Never alter production data to obtain it, and redact secrets from evidence. If you cannot reproduce it in < 15 minutes, move to Phase 2 anyway — understanding the system often reveals the trigger. ### Phase 2 — Understand the System Bugs live in the gap between what you think the code does and what it actually does. Close that gap before guessing at solutions. ```powershell # Read the code path end-to-end, not just the error location # Follow data from entry point to crash point # Trace the call stack Select-String -Path "src\\**\\*.ts" -Pattern "<function name>" # Check state at each step — add temporary logs if needed # Never modify production behavior during debugging ``` **Questions to answer:** 1. What is the **expected** behavior? (From spec, tests, or docs) 2. What is the **actual** behavior? (From error, logs, or reproduction) 3. Where exactly does the actual diverge from expected? (Specific line or state) 4. What **assumptions** does this code make? Which one is violated? **Tools:** ```powershell # Read the function signature and its callers Select-String -Path "src\\**\\*.ts" -Pattern "functionName" # Check what data looks like going in # Add a temporary debug log (remember to remove before committing) # console.log('[DEBUG]', JSON.stringify(input, null, 2)); # Check recent git changes to the file git --no-pager log --follow -p -- src/path/to/file.ts | Select-Object -First 80 ``` ### Phase 3 — Hypothesize and Verify Generate one specific hypothesis about the root cause, then verify or falsify it. Do **not** try multiple fixes simultaneously — you will not know which one worked. **Hypothesis template:** ```text Hypothesis: The bug is caused by [specific condition]. Prediction: If I [specific change], then [specific observable outcome]. Test: [How I will verify the prediction] ``` **Example:** ```text Hypothesis: The JWT expiry check compares Unix timestamp (seconds) to Date.now() (ms). Prediction: If I divide Date.now() by 1000, the check will pass for valid tokens. Test: The reproduction test case will pass with this change. ``` **Verify with a targeted change:** ```powershell # Make only the minimal change needed to test the hypothesis # Run the reproduction test npm test -- --testPathPattern="reproduction-test" # If the test passes: hypothesis confirmed → write a proper fix # If the test fails: hypothesis falsified → form a new hypothesis ``` **Repeat until a hypothesis is confirmed.** Do not patch symptoms. ### Phase 4 — Fix and Prevent Recurrence The root cause is confirmed. Now fix it properly and prevent it from returning. ```powershell # 1. Write the real fix (not the hypothesis test hack) # 2. Ensure the reproduction test passes with the fix # 3. Write additional regression tests for edge cases # 4. Run the full test suite — no regressions npm test # 5. Remove any temporary debug logs Select-String -Path "src\\**\\*.ts" -Pattern "\\[DEBUG\\]" # 6. Write a clear commit message explaining root cause git commit -m "fix: <what was wrong and why> Root cause: <explanation of the actual bug> Fix: <what was changed and why it resolves the root cause> Regression test: <name of the test that now covers this>" ``` **Prevention checklist:** - [ ] Root cause is documented in the commit message - [ ] At least one regression test covers the specific scenario - [ ] Related code paths are checked for the same pattern - [ ] If the bug was a class of error (e.g., "timestamp unit mismatch"), check for other instances ## Common Rationalizations | Rationalization | Reality | |----------------|---------| | "I'll just try a few things and see what sticks" | Shotgun debugging wastes hours and leaves you without understanding. A methodical approach finds the bug faster. | | "It's probably a framework/library bug" | It's almost never a framework bug. Blame your code first; verify framework behavior before blaming it. | | "I can't reproduce it, so I'll just add more logging" | Logging without a hypothesis produces noise. Reproduce first, then verify a specific hypothesis. | | "I understand the code well enough" | Write down what you think the code does before reading it. The gap between your mental model and reality is where the bug lives. | | "The fix is obvious, I don't need a regression test" | Obvious fixes regress. The regression test is how the next engineer (or future you) learns the lesson without re-debugging. | ## Red Flags - You've made more than 3 changes without a confirmed hypothesis - The bug "disappeared" without a clear explanation - Your fix makes the test pass but you don't understand why - The same bug has been "fixed" before - No regression test was added after the fix ## Escalation — The Rule of Three If you have made 3 or more hypothesis-driven fix attempts without resolution: 1. **Stop making further fixes** 2. **Collect the evidence** — record each hypothesis, the test you ran, and why it was falsified 3. **Escalate to architectural review** using the `council` skill: - Architect — is the design itself creating the failure? - Skeptic — what assumption is everyone making without checking? - Pragmatist — what is the fastest path back to a working system? - Critic — would a redesign be cheaper than more incremental fixes? > The Rule of Three usually means the problem is architectural, not an implementation detail. > A fourth fix attempt without escalation is often just another symptom patch. ## Verification - [ ] Bug is reproducible with a specific test case - [ ] Root cause is identified (not just the symptom) - [ ] Fix addresses the root cause, not the symptom - [ ] Regression test exists and passes - [ ] Full test suite passes with no new failures - [ ] Commit message explains the root cause ## Tips - **Binary search the bug**: use `git bisect` to find the commit that introduced it ```powershell git bisect start git bisect bad HEAD git bisect good <last-known-good-commit> # git bisect will check out commits for you to test ``` - **Rubber duck debugging**: explain the bug out loud (or to an agent). The act of articulating often reveals the assumption you didn't know you were making. - **Check the obvious last**: timestamp units, off-by-one errors, null vs undefined, async/await missing, wrong environment variable — these are common root causes. - **Use the `explore` agent** to trace a call path across many files simultaneously. ## See Also - [`fix-build-errors`](../fix-build-errors/SKILL.md) — compilation and build failures - [`tdd-workflow`](../tdd-workflow/SKILL.md) — write regression tests as part of the fix - [`security-scan`](../../security/security-scan/SKILL.md) — if the bug is security-related - [`council`](../../workflow/council/SKILL.md) — escalate persistent failures to a structured architectural review
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.