# Baselines and diagnostic data

Read when comparing branches or builds, using external searches, or sharing diagnostic artifacts. Preserve the working tree and redact secrets regardless of mode.

**Pre-existing failure proof:** run the same check against an explicit base SHA in an isolated worktree or scratch checkout with comparable dependencies and build state. Stashing removes local edits but does not switch a committed feature branch to its base. Compare failure names, assertions, and relevant output, not just exit status; matching failures can still have different causes. Preserve the user's working tree and any concurrent agent's state.

**A cross-branch A/B needs a pristine baseline.** `git checkout <baseline>` does not discard a dirty working-tree edit that merges cleanly -- it carries it into the checked-out tree, so the "before" build silently contains the fix and the experiment runs patched-vs-patched. The tell is *before == after* to the byte when a delta was expected. Commit the fix on its branch first, then run `git status --porcelain` after the baseline checkout and before the baseline build; non-empty output means the comparison is already poisoned. When restoring a single file, name the source (`git checkout HEAD -- <file>` for the commit, `git checkout <base> -- <file>` for the baseline) -- the bare `git checkout -- <file>` restores from the *index*, which may hold neither -- and assert the expected diff before rebuilding.

**Before external searches** (web, docs, forums): strip hostnames, IPs, file paths, SQL fragments, and customer data from the query. Raw stack traces leak privacy and return noise.

**Redaction also applies to what gets shown back, not just what goes out.** This skill has the agent paste commands, probe output, and captured artifacts into the conversation, and those carry credentials -- `Authorization` headers in a captured request, connection strings in a repro command, tokens in an environment dump. Replace each with `<REDACTED>` before it appears. Better, remove the need: build the reproduction loop so every credential is read from the environment (`$API_TOKEN`, `$DATABASE_URL`) rather than typed into the command, which keeps the value out of the transcript entirely, and quote only the lines of a captured artifact that carry signal. If redacting leaves too little to diagnose from, say so and ask for what is missing rather than pasting it raw.

# Analysis of Competing Hypotheses (ACH)

When the root cause is unclear -- especially across multiple components -- systematic hypothesis analysis prevents premature commitment to an incorrect explanation.

## When to Use

- Multiple plausible explanations for a failure
- Bug spans component boundaries (API -> service -> DB)
- Three-Fix Threshold reached (3 failed attempts)
- Intermittent failures with no clear reproduction pattern

## Six Failure Categories

Generate hypotheses across these categories. Most bugs start as one category but have root causes in another.

| Category | Symptoms | Example |
|----------|----------|---------|
| **Logic error** | Wrong output for valid input, off-by-one, incorrect branching | `<=` vs `<` in loop boundary |
| **Data issue** | Unexpected null, wrong type, stale cache, encoding mismatch | JSON field renamed upstream, cached value from previous schema |
| **State problem** | Race condition, leaked global state, order-dependent initialization | Test passes alone, fails in suite due to shared DB state |
| **Integration failure** | Contract mismatch at boundary, wrong endpoint, auth expired | Service A sends `user_id`, service B expects `userId` |
| **Resource exhaustion** | Timeout, OOM, connection pool depleted, disk full | DB pool max hit under load, queries queue indefinitely |
| **Environment** | Config drift, wrong version, missing dependency, OS difference | Works on macOS, fails on Linux due to case-sensitive filesystem |

## Evidence Strength Scale

Not all evidence is equal. Rank each piece:

| Strength | Type | Example |
|----------|------|---------|
| **Strong** | Direct observation | Stack trace pointing to exact line, failing test output |
| **Medium** | Correlational | Bug appeared after deploy X, timing correlates with load spike |
| **Weak** | Testimonial | "I think I saw this before when..." (no logs or reproduction) |
| **Variable** | Absence of evidence | "This component has no errors in its logs" (absence != proof) |

## The ACH Process

### 1. List hypotheses

For each failure category, generate at least one hypothesis. Be specific: "data issue" is not a hypothesis; "the `user.email` field is null because the upstream API changed its response format" is.

### 2. Collect evidence

For each hypothesis, gather evidence FOR and AGAINST:

```
H1: Race condition in session initialization
  FOR:  Intermittent (strong), only under concurrent requests (medium)
  AGAINST: Single-threaded test also fails (strong)
  → WEAKENED by counter-evidence

H2: Stale config cached after deploy
  FOR:  Timestamp of first failure matches deploy (medium), restart fixes it (strong)
  AGAINST: None found
  → STRONGEST candidate
```

### 3. Compare evidential support

| Support | Meaning | Action |
|------------|---------|--------|
| **Strong** | Concrete causal path, competing explanations checked | Test the smallest supported remedy |
| **Mixed** | Consequential premises unresolved | Choose a discriminating probe |
| **Weak** | Little evidence or direct contradiction | Gather evidence or reject the hypothesis |

These are judgment labels, not calibrated probabilities. State the observations and remaining uncertainty instead of manufacturing numerical precision.

### 4. Investigate the top hypothesis

Test the highest-confidence hypothesis first. One change at a time. Fully revert if wrong.

When two hypotheses remain plausible, design a probe that distinguishes them. Consider a compound cause only when evidence supports their interaction; similar confidence alone does not establish it.

## Anti-Patterns

- **Anchoring**: committing to the first hypothesis that seems plausible
- **Confirmation bias**: only looking for evidence that supports your preferred hypothesis
- **Premature closure**: stopping investigation after one piece of supporting evidence
- **Ignoring absence**: "no errors in this component" doesn't mean the component is innocent

# Defense-in-Depth Validation

When you fix a bug caused by invalid data, adding validation at one place feels sufficient. But that single check can be bypassed by different code paths, refactoring, or mocks.

**Core principle:** Validate at EVERY layer data passes through. Make the bug structurally impossible.

## Why Multiple Layers

Single validation: "We fixed the bug."
Multiple layers: "We made the bug impossible."

Different layers catch different failure modes:
- Entry validation catches most invalid input
- Business logic catches domain-specific edge cases
- Environment guards prevent context-specific dangers (e.g., destructive operations in test)
- Debug instrumentation captures forensic context when other layers fail

## The Four Layers

### Layer 1: Entry Point Validation

Reject obviously invalid input at the API/function boundary. This is the first line of defense.

```php
function createProject(string $name, string $workingDirectory): Project
{
    if (empty($workingDirectory)) {
        throw new \InvalidArgumentException('workingDirectory cannot be empty');
    }
    if (!is_dir($workingDirectory)) {
        throw new \InvalidArgumentException("workingDirectory does not exist: {$workingDirectory}");
    }
    // ... proceed
}
```

### Layer 2: Business Logic Validation

Ensure data makes sense for this specific operation, even if it passed entry validation.

```php
function initializeWorkspace(string $projectDir, string $sessionId): void
{
    if (empty($projectDir)) {
        throw new \RuntimeException('projectDir required for workspace initialization');
    }
    // ... proceed
}
```

### Layer 3: Environment Guards

Prevent dangerous operations in specific contexts (test, staging, CI).

```python
import os
import tempfile


async def git_init(directory: str) -> None:
    if os.environ.get("NODE_ENV") == "test":
        normalized = os.path.realpath(directory)
        tmp_dir = os.path.realpath(tempfile.gettempdir())
        if normalized == tmp_dir or os.path.commonpath([normalized, tmp_dir]) != tmp_dir:
            raise RuntimeError(
                f"Refusing git init outside temp dir during tests: {directory}"
            )
    # ... proceed
```

Resolve symlinks before comparing path components; a string prefix also accepts sibling paths such as `/tmp-other`. Restrict the operation to a child of the temporary root, not the root itself. This check assumes the test owns the directory and no concurrent actor can replace its path components; use an isolated workspace or descriptor-relative operations when that assumption does not hold.

### Layer 4: Debug Instrumentation

Capture context for forensics when the other layers fail.

```typescript
async function gitInit(directory: string) {
  const stack = new Error().stack;
  console.error('About to git init', { directory, cwd: process.cwd(), stack });
  // ... proceed
}
```

Use `console.error()` in tests (not logger, which may be suppressed). Log BEFORE the dangerous operation, not after it fails. Include context: cwd, env vars, timestamps, stack trace.

## Applying the Pattern

When you fix a bug:

1. **Trace the data flow** -- where does the bad value originate? Where is it consumed?
2. **Map all checkpoints** -- list every function/boundary data passes through
3. **Add validation at each layer** -- entry, business logic, environment, instrumentation
4. **Test each layer independently** -- bypass layer 1, verify layer 2 catches it

## Key Insight

All four layers are typically necessary. During testing, each layer catches bugs the others miss:
- Different code paths bypass entry validation
- Mocks bypass business logic checks
- Edge cases on different platforms need environment guards
- Debug logging identifies structural misuse patterns

Don't stop at one validation point.

# Repair, escalation, and diagnostic traps

Read when implementing a remedy, assessing a failed experiment, or deciding whether to stop and reassess. Diagnosis-only work remains read-only.

**5. Hypothesize and test** -- one change at a time. If a hypothesis is wrong, fully revert before testing the next. Use `git bisect` to pinpoint the exact commit that introduced a regression. **Scope lock**: after forming a hypothesis, identify the narrowest affected directory or file set; do not edit code outside that scope during the debug session. If the fix requires changes elsewhere, update the hypothesis first.

**6. Fix and verify** -- create a failing test FIRST, then fix. Run the test. Confirm the original reproduction case passes. No completion claims without fresh verification evidence (see `ia-verification-before-completion`).

**No reachable seam for the bug as it actually triggered** (a race or timing window, hardware- or platform-specific behavior, a defect that only appears against production data) is not license to write a test anyway. A test built around a seam that cannot exercise the real trigger passes for the wrong reason and reads as coverage that does not exist. Record the missing seam as a finding in the Debug Report instead.

**Reproduce-passes is not fixed.** The bad state is often still reachable from a nearby variant when the fix landed at the crash site, not the root cause. Before declaring done, run the **bypass self-check**: name one input variation that reaches the same bad state without tripping the change -- if one exists, the fix is at the wrong layer; return to root cause. **Suppression is not a fix**, and the check assumes the fix attacks the bug: swallowing the error (`try/except: pass`, a blanket catch), disabling the failing assertion, or special-casing the reproduction input hides the signal while the defect lives on (a global swallow even *passes* the bypass check). Change behavior at the root cause, not the symptom. For security-relevant bugs, escalate to an **adversarial re-attack**: a fresh-context agent attacks the patched code ([specialized-patterns.md](./specialized-patterns.md)).

**Trim to the minimal diff.** After the fix verifies, simplify to the smallest change that fixes the root cause -- best done as a fresh-context pass ([specialized-patterns.md](./specialized-patterns.md)).

**On a failed fix:** return to Step 5 and identify what the result actually tests: the causal hypothesis, the remedy, the exercised trigger, or the build identity. Reject or revise the hypothesis when the evidence contradicts it; an incomplete remedy does not itself disprove the cause. Change a named variable before another experiment. The Three-Fix Threshold counts complete hypothesis-test cycles.

## Three-Fix Threshold

After 3 failed fix attempts, stop editing and reassess. The count is a retry budget, not evidence of an architectural cause. An attempt = one complete hypothesis-test cycle (form hypothesis, make minimal change, verify). Then:

1. Stop editing.
2. Re-read the failing code path end-to-end instead of spot-checking, questioning assumptions about how the system works.
3. Write down which assumption each failed fix relied on.
4. Escalate with those findings via the ask mechanism in Step 1 (subagents with no user channel: record them in the final report).

In diagnosis-only work, the equivalent budget is 3 investigation cycles without narrowing the component under suspicion -- emit interim findings and the next instrumentation step instead of continuing.

**When reasoning reaches the same contradiction twice, instrument it.** Re-deriving the same impossible conclusion from the source is not a new cycle -- it produces no edit, so it never trips the threshold -- and it is the signal that a value in the mental model is wrong. Log the two divergent values side by side at the point they disagree. One build usually collapses what repeated re-reading cannot.

**Architectural problem indicators** -- signals the bug is structural, not a surface fix: each fix reveals unexpected shared state or coupling; fixes require massive refactoring to implement correctly; each fix creates new symptoms elsewhere in the system.

**No root cause found:** if investigation is exhausted without a clear root cause, say so explicitly. Document what was checked, what was ruled out, and what instrumentation to add for next occurrence. An honest "unknown" with good diagnostics beats a fabricated cause.

## Escalation: Competing Hypotheses

When the cause is unclear across multiple components, use Analysis of Competing Hypotheses: generate hypotheses across failure categories, collect evidence FOR and AGAINST each, rank by confidence, investigate the strongest first. Full methodology in [competing-hypotheses.md](./competing-hypotheses.md).

## Pattern Comparison

When the cause isn't obvious, compare the failing path with a working reference and test relevant differences in code, inputs, state, timing, and environment. Similar source may behave differently because of undefined behavior or hidden state; neither a visible difference nor identical code settles causality alone.

## Specialized Patterns

In [specialized-patterns.md](./specialized-patterns.md) unless noted:

- **Intermittent issues** -- races, deadlocks, resource exhaustion, timing. Key signals: shared mutable state, check-then-act, circular lock acquisition, pool exhaustion under load.
- **Performance regressions** -- slow, latency, or throughput symptoms. Measure a numeric baseline before reading code for the cause.
- **Defense-in-depth validation** -- after fixing, validate at every layer, not just where the bug appeared: [defense-in-depth.md](./defense-in-depth.md).
- **Common bug patterns and triage** -- async ordering, stale state, stale build artifacts, recurring fix site; severity-vs-priority triage.
- **Off-track signals** -- user phrases ("stop guessing", "we're going in circles") that mean the systematic process was abandoned.

## Anti-Patterns and Red Flags

When you catch yourself doing or thinking these things, **stop and return to Step 1 (Reproduce)**:

| What You're Doing / Thinking | What It Really Means |
|-----------------------------|---------------------|
| Shotgun debugging / "I see the problem, let me fix it" / "It's probably X" | Reasoning is not evidence. Form a hypothesis, make one change, test, revert if wrong. Trace the actual execution path. |
| Ignoring intermittent failures ("works on my machine") | Instrument and reproduce under load. Isolation success doesn't explain integration failure. |
| "I'll clean up the debugging later" | Remove diagnostic code now or it ships to production. |
| "This failure is pre-existing, not related to our changes" | Prove it: run the test suite on the base branch. No receipts = no claim. |
| "The tool truncated the output" / "the runner must be broken" | Check local state first -- a moved HEAD, a stale context, or a dirty tree explains this far more often than tool misbehavior. Proving a tool bug means reproducing it at a known commit. A report filed from stale context wastes the fix and costs the tool its credibility for the next session. |
| "The test is wrong, not the code" | Verify before dismissing. Read the test's intent. If the test is genuinely wrong, fix it with a clear rationale, not a silent update. |
| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read the working example completely and apply it exactly. |
| "The experiment came back negative, so the hypothesis is dead" | A bounded experiment bounds itself first. State the coverage the run achieved before a negative retires a hypothesis. |
| "I falsified the trigger, so that area is ruled out" | Falsifying one trigger of a mechanism does not falsify the mechanism. Re-test through a different trigger. |

## Verify

- Causal conclusion supported by source/runtime evidence, or the remaining uncertainty explicitly reported
- For repaired bugs with a reachable test seam, regression test fails without the fix and passes with it; otherwise report the verification used and the missing seam
- Bypass self-check run: no variant input reaches the same bad state without tripping the fix (for security-relevant fixes, adversarial re-attack found no bypass)
- Debug Report emitted with all seven fields (SYMPTOM, ROOT CAUSE, FIX, EVIDENCE, REGRESSION, RELATED, STATUS)
- No diagnostic instrumentation left in code (`git diff` shows no leftover logging)

# Reproduction and investigation

Read when establishing a reproducer, reducing a failure, instrumenting a path, or reconciling contradictory evidence. For baseline/build comparisons, also use the dedicated baseline reference.

**0. Read the error.** Read the full error message, stack trace, and line numbers before doing anything. Error messages frequently contain the exact fix. Don't skim -- read the entire output.

**Read the totals before the matches.** A grep for failures shows only the failures matching the pattern, and a tool printing a capped sample prints the cap while the real count sits elsewhere. A hypothesis built on "only 8 failed" when the summary says 89 aims every later step at the wrong target. Find the tally line first and compare the sample size against the total.

**After a session resume or compaction, confirm HEAD before trusting file content carried in context.** Context survives compaction; the repository does not freeze while it does. A file body, class shape, or pipeline order in context describes the tree at the moment it was read, and commits from another session, a background agent, a teammate, or your own earlier push can have landed since -- nothing marks the drift. Run `git log --oneline -1` as the first call after any resume and compare it to the SHA the context assumes. Tells that HEAD moved: a deterministic count changes with no cause (test count, symbol count, file count), an edit's `old_string` is missing from a file you "just read", or a constant you remember is absent from it. Re-read a file before editing it after a resume; that is the only way staleness tracking engages.

**1. Reproduce** -- build the cheapest feedback loop that exercises the reported failure. Prefer a deterministic broken/fixed signal; for intermittent failures, record conditions, attempts, and observed frequency. A finite run without failure does not prove a race or platform-dependent defect absent.

**Match the exact reported symptom, not a nearby one.** The loop is not reproduction until it fails with the same error text or the same failing assertion the report names -- a different exception, a different failing test, or a generic error on the same feature is a different bug wearing the same file path. Treating it as the same defect sends every later step against the wrong target.

**A loop already provided? Run it before touching source.** If the workspace has a test file, or the report says "run X to see the failure," that command *is* the feedback loop: run it after Step 0 and record the RED output before reading source, forming hypotheses, or editing -- without an observed failing run this session, nothing proves the fix changed anything.

Pick the cheapest loop that triggers the bug:

- Failing test (preferred -- becomes the regression test in step 6)
- `curl` script or `httpie` invocation against a local server
- CLI harness or REPL session
- Headless browser script (Playwright, Puppeteer)
- Throwaway harness in `/tmp/` -- delete when done
- Any other scripted signal: a property-based test, log replay against a captured request body, or a manual bash session with documented reproduction steps (human-in-the-loop)

If the bug is intermittent, use a bounded run under relevant stress or simulated conditions. If it does not trigger, report the limit and investigate traces, dumps, or static paths without claiming reproduction.

**Cannot build a loop?** State exactly what is missing -- access, credentials, artifacts, or repro steps. Continue independent source and artifact investigation, distinguishing observed facts from hypotheses and untested behavior. Ask for material missing input through the active harness's supported question tool (AskUserQuestion in Claude Code, request_user_input in Codex where appropriate), otherwise in chat. A subagent reports the missing input to its orchestrator.

**2. Form initial hypotheses** -- form 2-3 hypotheses from the reproduction before investigating broadly: the most likely causes given the symptoms. This focuses investigation on plausible paths. Cite **at least one concrete observation** per hypothesis: a runtime value, a log line, a boundary capture, a behavior delta against a working case, or a specific code reference. "X seems off" is not evidence; "X is null at line 42 because Y never ran under condition Z" is. A hypothesis without a grounding observation is theorizing -- instrument until there is a signal (extend the Step 1 loop, or add Step 4 boundary captures).

**3. Reduce** -- strip the reproduction to the minimal failing case. Remove unrelated code, data, and configuration until removing one more piece makes the bug disappear. That remaining piece is the trigger.

**4. Investigate** -- trace the failing path far enough to explain the violated invariant and the conditions that cause it ([root-cause-tracing.md](./root-cause-tracing.md): stack instrumentation, test pollution detection). Compare code version, data, environment, timing, and configuration in working versus broken cases. Capture relevant environment state with [collect-diagnostics.sh](../scripts/collect-diagnostics.sh). A causal explanation may span several layers or be local to one expression; neither the earliest observed divergence nor a fixed number of stack levels proves root cause.

**Route the first move by bug class before instrumenting.** Visual/rendering bugs want a static read of the render path and computed styles, not logs; behavioral/async/state/lifecycle bugs want a probe added *now* as part of the hypothesis; pure-logic bugs need only a careful read. Before adding any probe, state the yes/no question it answers and the decision rule. CI check failed? See [specialized-patterns.md](./specialized-patterns.md) for the CI-failure workflow (and full routing detail).

**A uniform probe result is a claim about the probe, not about the mechanism.** The smallest inputs that express the shape are also the usual way to measure something else: a normalization step or an empty-input short-circuit upstream can decide the output for every case. Print the intermediate the mechanism itself emits, choose inputs where every other mechanism is non-degenerate, and keep the uniform case in the same run as a control.

**Multi-component systems** (CI -> build -> deploy, API -> service -> DB): before proposing fixes, log what data enters and exits each component boundary and verify env/config propagation across it. Run once to see WHERE it breaks, then investigate that component. Write probes unbuffered to stderr (`console.error`, `fwrite(STDERR, ...)`, `print(..., file=sys.stderr)`); application loggers may be suppressed in tests. Log BEFORE the dangerous operation, not after it fails. Include context: cwd, env vars, `new Error().stack`.

**When two evidence sources contradict and one is executable, execute it.** A comment against the code, a docstring against the callee, a recorded observation against the current build -- recency and authorship are not tie-breakers, and weighing them settles nothing. Whichever side can become a command settles it in one pass. Once "one of these is stale" is written down, the next action is the probe.

**Three completeness checks live in [specialized-patterns.md](./specialized-patterns.md):** branches sharing a destination can hide different causes, build-variant differences can expose latent source defects, and both confirming and refuting probes need valid controls.

# Root Cause Tracing

Bugs often manifest deep in the call stack. The instinct is to fix where the error appears, but that treats a symptom.

**Core principle:** Trace backward through the call chain until you find the original trigger, then fix at the source.

## When to Use

- Error happens deep in execution (not at entry point)
- Stack trace shows long call chain
- Unclear where invalid data originated
- Need to identify which test/caller triggers the problem

## The Tracing Process

### 1. Observe the symptom

```
Error: git init failed in <repo-root>/packages/core
```

### 2. Find the immediate cause

What code directly triggers this?

```typescript
await execFileAsync('git', ['init'], { cwd: projectDir });
```

### 3. Trace callers upward

```
WorktreeManager.createSessionWorktree(projectDir, sessionId)
  <- Session.initializeWorkspace()
  <- Session.create()
  <- test at Project.create()
```

### 4. Track the bad value

At each level, ask: what value was passed, and where did it come from?

- `projectDir = ''` (empty string)
- Empty string as `cwd` resolves to `process.cwd()`
- That's the source code directory, not the temp dir

### 5. Find the original trigger

Where did the empty string originate?

```typescript
const context = setupCoreTest(); // Returns { tempDir: '' }
Project.create('name', context.tempDir); // Accessed before beforeEach ran!
```

Root cause: top-level variable initialization accessing a value that isn't set until `beforeEach`.

## Adding Stack Traces for Instrumentation

When manual tracing hits a dead end, add instrumentation:

```typescript
async function gitInit(directory: string) {
  console.error('DEBUG git init:', {
    directory,
    cwd: process.cwd(),
    nodeEnv: process.env.NODE_ENV,
    stack: new Error().stack,
  });
  await execFileAsync('git', ['init'], { cwd: directory });
}
```

Use `console.error()` in tests (not logger -- may be suppressed). Log BEFORE the operation, not after failure. Capture and filter:

```bash
npm test 2>&1 | grep 'DEBUG git init'
```

## Finding Test Pollution

When a test passes in isolation but fails in the suite, another test is polluting shared state.

**Bisection approach:** Run tests one at a time until the polluter is found.

```bash
# Run each test file individually, check if artifact appears after each
for f in $(find src -name '*.test.ts'); do
  npx jest "$f" --forceExit 2>/dev/null
  if [ -d ".git/worktrees/phantom" ]; then
    echo "POLLUTER: $f"
    break
  fi
done
```

Analyze stack traces from instrumentation to find the pattern (same test? same parameter? same setup function?).

## Key Principle

Never fix just where the error appears. Trace back to find the original trigger. After finding the source, also add defense-in-depth validation at each layer the data passes through (see [defense-in-depth.md](./defense-in-depth.md)).

# Specialized Debugging Patterns

## Environment Diagnostics

Before investigating, capture the environment state using [collect-diagnostics.sh](../scripts/collect-diagnostics.sh):

```bash
bash collect-diagnostics.sh           # print to stdout
bash collect-diagnostics.sh diag.md   # write to file
```

Collects system info, language versions, git state, project files, and environment variables. Use during differential analysis to compare working vs broken environments, or attach to bug reports.

## Intermittent Issues

- Track with correlation IDs across distributed components
- Race conditions: look for shared mutable state, check-then-act patterns, missing locks. In async code (Node.js, Python asyncio): interleaved `.then()` chains, unguarded shared state between concurrent tasks, missing transaction isolation in DB operations
- Deadlocks: check for circular lock acquisition (DB row locks held across multiple queries), circular `await` dependencies in async code, connection pool exhaustion blocking queries that would release other connections
- Resource exhaustion: monitor memory growth, connection pool depletion, file descriptor leaks. Under load: check pool size vs concurrent request count, verify connections are returned on error paths (finally/dispose)
- Timing-dependent: replace arbitrary `sleep()` with condition-based polling -- wait for the actual state, not a duration

## Performance Regressions

For slow, latency, or throughput symptoms, code reading is not the reproduction step -- a numeric measurement is.

- Establish a baseline before touching anything: time the same input, in the same environment, across N runs. That baseline is the failing test for a perf bug -- it stands in for Step 1's reproduction and Step 6's pass/fail check.
- Attribute before optimizing: a profiler run or per-stage timing that shows where the time actually goes. A hot-spot guess is a hypothesis, not evidence -- optimizing an unmeasured suspect is shotgun debugging with extra steps.
- If the slowness is a regression, bisect commits against the measurement (rerun the baseline at each candidate commit), not by reading diffs for code that looks expensive.
- The fix is verified by re-running the same baseline measurement, not by reasoning that the change should be faster.
- When several independent costs sit on one hot path, fixing the first moves the profile instead of flattening it, which reads as "the fix did nothing" unless the new stack is compared against the old. Compare stacks, not totals; once the same stack arrives a third time, stop bisecting configuration and get symbols.

## CI Failures

When a CI check fails on a PR or branch:

1. **Fetch logs**: `gh run view <run_id> --log` (extract run ID from the checks URL). If `detailsUrl` points to a non-GitHub provider (Buildkite, CircleCI), don't attempt to fetch logs -- report the URL and let the user investigate.
2. **Classify the failure**: build error (compilation/dependency), test failure (which test, what assertion), lint/type error (which rule, which file), timeout (which step exceeded limits), or infrastructure (runner OOM, network, flaky service).
3. **Reproduce locally**: run the same command from the CI config (`cat .github/workflows/*.yml` to find it). A local pass is a comparison result, not proof of an environmental cause; compare inputs, revision, configuration, timing, and build state.
4. **Fix and verify**: fix the issue, then suggest re-running the relevant checks: `gh pr checks <pr> --watch` or `gh run rerun <run_id> --failed`.

**A fast red suggests an early failure.** Check the failed step and first relevant error before deciding whether setup, compilation, or a fast test failed. Duration alone does not identify the phase.

**Failure labels need discriminating evidence.** A bounded retry can measure recurrence, but two identical failures do not rule out flakiness and a later green does not repair it. Compare the exact head/base revisions, failed assertions, inputs, and runner conditions. An identical tree behaving differently in two environments identifies a relevant axis, not necessarily an external cause: timing or allocator differences can expose a source defect.

**A parallel runner's diff output can attach to the wrong test.** With N workers and per-failure diffs, interleaving puts a diff block above a `FAIL <name>` line that belongs to a neighbor. Match the diff's content against that test's expected output before acting on it.

## Post-Fix Passes

- **Adversarial re-attack** (security-relevant fixes): after the bypass self-check, spawn a fresh-context agent, blind to the fix reasoning, and have it attack the patched code to find a variant input that still triggers the bad state. The fixing session cannot attack its own patch objectively -- it knows too much about the intended fix path.
- **Fresh-context trim pass**: after the fix verifies, run a fresh-context pass asked only to "simplify to the smallest change that fixes the root cause." The fixing session is anchored to its own reasoning and over-reaches; a blind pass reliably finds the trim points without reintroducing the bug.

## Postmortem

After resolving non-trivial bugs, document a lightweight postmortem:

1. **Timeline**: when introduced, when detected, when resolved (include commit SHAs)
2. **Root cause**: one sentence -- the actual cause, not the symptom
3. **Impact**: what broke, for how long, who was affected
4. **Fix**: what changed and why this fix addresses the root cause
5. **Prevention**: what test, monitor, or process change prevents recurrence

## Common Bug Patterns

- **Async ordering** -- missing `await`, unhandled promise rejection, callback firing before setup completes. The temporal gap between setup and callback is where bugs hide.
- **Stale state** -- cached values, stale closures, outdated config, old build artifacts. When behavior contradicts the code you're reading, verify you're running what you think you're running.
- **Stale build artifacts** -- a test failure whose source path is provably correct and untouched by your diff is the tell: the source on disk is right, but an incremental build relinked a stale object. A clean working tree (`git status`) does not mean a clean build tree -- build outputs are typically gitignored. Baseline the *build*, not the commit: rebuild from clean (`make clean`, fresh `target/`) before debugging the code. Checking out an old commit inherits the same stale objects and proves nothing.
- **A crash and a leak reported against the same tests** -- two reports from one toolchain upgrade usually trace to a single defect with a single fix, and the closing commit credits only one of them. Search the tracker for reports naming the same test files, read the fix commit, then check per branch whether it landed and whether the buggy code exists there at all.
- **A platform-specific fault is rarely a platform-specific bug** -- when a wrong pointer lands in benign mapped memory under one allocator and on the zero page under another, the defect is latent everywhere. A red/green comparison on the forgiving platform proves nothing, and shrinking the build changes the layout that decides whether the fault appears. Reproduce on the hostile platform with the full component set.
- **Recurring fix site** -- repeated fixes in one file justify examining shared invariants and prior remedies. Redesign requires evidence of structural coupling; a history count alone is not that evidence.
- **A metric pinned at a clean extreme** -- an aggregate landing on exactly 0%, 100%, or all-zero across a correlated set is usually the failure path feeding the metric, not a result. Check whether the error handler emits a value the aggregator accepts as real: a type-valid placeholder verdict, a default score, a swallowed exception returning the neutral case. Ask what number comes out when the dependency fails for *every* item at once; if it is indistinguishable from a genuine one, that is the bug. Size- or batch-correlated zeros (small inputs fine, large ones uniformly zero) point at the scoring call raising before it ever ran.
- **Container-local id used as a global key** -- an id minted per parent (row N of a batch, finding N of a review, id 1 within a tenant) collides silently when flattened into one map. If that map feeds a completeness or coverage check, the failure is a *passing* gate rather than a wrong value: the erased information (which parent) is exactly what the check was measuring. Cheapest probe: the keyed map holds fewer entries than the source rows, with no error raised. Put the parent in the key, and make a collision that "cannot happen" assert instead of overwrite.

## Diagnosis Completeness

**When several branches share a destination, the artifact cannot say which one fired.** A guard chain or a validator whose branches reach one terminal produces a byte-identical symptom for each, so the failing artifact stays true after a fix and says nothing about completeness. Enumerate every branch that reaches that destination and establish each one at the fixed revision. A branch that is fail-open on absence (`!x || x.ok`) passes for the wrong reason on whatever environment was measured, and a cleanup step the fix's own instructions prescribe can remove the state that made a branch pass.

**A working build variant does not rule the source out.** Compare the named source files and build axes: static versus dynamic modules, distribution, compiler flags, allocator, and layout. Identical source can contain undefined behavior or a race exposed only by one variant. Use the comparison to design the next probe, not to dismiss the source hypothesis.

## First Move by Bug Class

The first debugging move depends on the bug class. "Add logging" is the default reflex, but for some classes it is the wrong first move -- it captures nothing and burns a cycle.

- **Visual / rendering / layout** -- read statically first. Instrumentation cannot capture what the compositor, layout engine, or cascade actually did. Read the render path and inspect computed styles (resolved values, not the source rule) instead of logging. A log fires before paint and says nothing about the rendered result.
- **Behavioral / lifecycle / async / state** -- instrument first, before writing any fix. Add the probe (a log or assertion) as part of forming the hypothesis, not after a fix has already failed. These bugs live in values and ordering that are invisible from a static read; the probe is how the hypothesis becomes observable.
- **Pure logic** (off-by-one, wrong branch, bad comparison) -- a careful static read is sufficient. No instrumentation needed; the defect is on the page once the path is read end to end.

**Write the question before the log.** Before adding any probe, state the yes/no question it answers and pre-commit the decision rule: "if this prints X before Y, hypothesis A survives; if not, A is dead." A log with no question attached is noise -- it produces output, not evidence.

**A log that changes the behavior is itself evidence.** If adding or removing a probe makes the bug appear, disappear, or move, that signals a timing, lifecycle, or concurrency defect -- the observation is perturbing the very ordering that is broken. Do not chase the now-hidden symptom; treat the sensitivity as the lead and investigate the race.

**Weight a confirming probe more suspiciously than a refuting one.** Familiar oracle failures return empty; this one returns the answer you hoped for. A false negative costs a finding never filed, while a false positive kills a live hypothesis and cites executed output as proof. Check that the matched token cannot be derived from a fixture value you control -- name fixtures orthogonally, never by interpolating the loop variable -- and require any "no problem found" probe to have printed at least one subject's actual value.

**The flag that makes the bug visible can disable behavior another test asserts.** An option that routes allocations around the runtime allocator so a memory checker can see them also removes that allocator's limits, so every test expecting an exhaustion error produces empty output or a timeout and reads as a regression. Re-run any failure whose expectation is a resource limit without the instrumentation flag.

**When the probe kills the repro, switch to non-perturbing capture.** Once two of these hold -- fires under the real harness but not under a debugger, vanishes when print-style logging is added, vanishes under a built-in verbose dump, or crash-or-not flips across rebuilds of identical source -- stop trying I/O-based observation; every heavier tool makes it less reproducible. Record into a preallocated in-memory buffer using plain stores in the hot path (no formatting, no syscalls, no flush) and dump the buffer only from the failure or crash handler, where I/O is free. Keep the instrument behind a single build flag, add a per-entry invocation counter so re-dispatch within one call is distinguishable from a fresh entry, and commit each instrument increment -- shared automation can reset a worktree mid-task and take an uncommitted probe with it.

**To settle whether a path executes at all, interpose the libc symbol rather than instrumenting the build.** At a 1-2% fault rate, before/after counts are weak evidence. An `LD_PRELOAD` shim that wraps the suspect entry point (`exit`, `close`, `free`), writes one marker, and forwards to `dlsym(RTLD_NEXT, ...)` answers "does this run" deterministically in a single execution, with no rebuild.

## Bug Triage

When multiple bugs exist, prioritize by:
- **Severity** (data loss > crash > wrong output > cosmetic) separately from **Priority** (blocking release > customer-facing > internal)
- Reproducibility: always > sometimes > once. "Sometimes" bugs need instrumentation before fixing.
- Quick wins: if a fix is < 5 minutes and unblocks others, do it first

## Signals You're Off Track

Watch for these signs from the user -- they indicate you've left the systematic process:

- "Is that not happening?" -- you assumed behavior without checking
- "Will it show us...?" -- you're not gathering enough evidence
- "Stop guessing" -- you're proposing fixes without root cause
- "We're going in circles" -- same hypothesis repackaged, not a new approach
- Repeating the same type of fix with slight variations -- that's not a new hypothesis

#!/usr/bin/env bash
# collect-diagnostics.sh — Gather environment diagnostics for debugging
# Usage: bash collect-diagnostics.sh [output-file]
#
# Collects system info, language versions, git state, and project metadata.
# Outputs structured report to stdout or optional file.

set -euo pipefail

OUTPUT="${1:-}"

collect() {
    local buf=""

    buf+="# Diagnostic Report"$'\n'
    buf+="**Collected:** $(date -u +%Y-%m-%dT%H:%M:%SZ)"$'\n\n'

    # --- System ---
    buf+="## System"$'\n\n'
    buf+="| Property | Value |"$'\n'
    buf+="|----------|-------|"$'\n'
    buf+="| OS | $(uname -s) $(uname -r) |"$'\n'
    buf+="| Arch | $(uname -m) |"$'\n'
    buf+="| Shell | ${SHELL:-unknown} |"$'\n'
    if command -v bash &>/dev/null; then
        buf+="| Bash | $(bash --version | head -1) |"$'\n'
    fi
    buf+="| User | $(whoami) |"$'\n'
    buf+="| PWD | $(pwd) |"$'\n'
    buf+=$'\n'

    # --- Disk / Memory ---
    buf+="## Resources"$'\n\n'
    buf+='```'$'\n'
    buf+="Disk (pwd): $(df -h . 2>/dev/null | tail -1 | awk '{print $4 " available of " $2}')"$'\n'
    if command -v free &>/dev/null; then
        buf+="Memory: $(free -h 2>/dev/null | awk '/^Mem:/{print $7 " available of " $2}')"$'\n'
    fi
    buf+='```'$'\n\n'

    # --- Git ---
    if git rev-parse --is-inside-work-tree &>/dev/null; then
        buf+="## Git"$'\n\n'
        buf+="| Property | Value |"$'\n'
        buf+="|----------|-------|"$'\n'
        buf+="| Branch | $(git branch --show-current 2>/dev/null || echo 'detached') |"$'\n'
        buf+="| Last commit | $(git log -1 --format='%h %s' 2>/dev/null || echo 'none') |"$'\n'
        buf+="| Dirty files | $(git status --porcelain 2>/dev/null | wc -l | tr -d ' ') |"$'\n'
        buf+="| Remote | $(git remote get-url origin 2>/dev/null || echo 'none') |"$'\n'
        buf+=$'\n'
    fi

    # --- Language Versions ---
    buf+="## Languages & Runtimes"$'\n\n'
    buf+="| Tool | Version |"$'\n'
    buf+="|------|---------|"$'\n'
    for cmd in node python python3 php ruby go java rustc; do
        if command -v "$cmd" &>/dev/null; then
            local ver
            case "$cmd" in
                node)    ver=$("$cmd" --version 2>/dev/null) ;;
                python|python3) ver=$("$cmd" --version 2>/dev/null) ;;
                php)     ver=$("$cmd" --version 2>/dev/null | head -1) ;;
                ruby)    ver=$("$cmd" --version 2>/dev/null) ;;
                go)      ver=$("$cmd" version 2>/dev/null) ;;
                java)    ver=$("$cmd" -version 2>&1 | head -1) ;;
                rustc)   ver=$("$cmd" --version 2>/dev/null) ;;
                *)       ver="installed" ;;
            esac
            buf+="| ${cmd} | ${ver} |"$'\n'
        fi
    done
    buf+=$'\n'

    # --- Package Managers ---
    buf+="## Package Managers"$'\n\n'
    buf+="| Tool | Version |"$'\n'
    buf+="|------|---------|"$'\n'
    for cmd in npm pnpm yarn bun pip uv composer cargo bundler gem; do
        if command -v "$cmd" &>/dev/null; then
            local ver
            ver=$("$cmd" --version 2>/dev/null | head -1) || ver="installed"
            buf+="| ${cmd} | ${ver} |"$'\n'
        fi
    done
    buf+=$'\n'

    # --- Project Detection ---
    buf+="## Project Files Detected"$'\n\n'
    for f in package.json composer.json pyproject.toml Cargo.toml Gemfile go.mod build.gradle pom.xml Makefile Dockerfile docker-compose.yml .env.example; do
        if [ -f "$f" ]; then
            buf+="- \`${f}\`"$'\n'
        fi
    done
    buf+=$'\n'

    # --- Environment Variables (safe subset) ---
    buf+="## Environment (safe subset)"$'\n\n'
    buf+="| Variable | Value |"$'\n'
    buf+="|----------|-------|"$'\n'
    for var in NODE_ENV APP_ENV RAILS_ENV FLASK_ENV ENVIRONMENT CI TERM; do
        local val="${!var:-}"
        if [ -n "$val" ]; then
            buf+="| ${var} | ${val} |"$'\n'
        fi
    done
    buf+=$'\n'

    echo "$buf"
}

report=$(collect)

if [ -n "$OUTPUT" ]; then
    echo "$report" > "$OUTPUT"
    echo "Diagnostics written to ${OUTPUT}"
else
    echo "$report"
fi