calibrate

<!-- Step 1 in SKILL.md dispatches to this mode file. Steps here continue from Step 2. -->

## Mode: agents

### Domain table

Problem domain by agent:

- `foundry:sw-engineer` → Python bugs: type errors, logic errors, anti-patterns, bare `except:`, mutable defaults
- `foundry:qa-specialist` → coverage gaps: uncovered edge cases, missing exception tests, Machine Learning (ML) non-determinism
- `foundry:linting-expert` → violations: ruff rules, mypy errors, annotation gaps
- `foundry:curator` → config issues: broken cross-refs, missing workflow blocks, wrong model, step gaps; handover compliance: malformed JSON envelopes; context discipline: spawn prompt bloat, AgentSpeak v2 violations
- `foundry:doc-scribe` → docs gaps: missing docstrings, missing Google style sections, broken examples
- `foundry:perf-optimizer` → perf issues: unnecessary loops, repeated computation, wrong dtype, missing vectorisation
- `oss:cicd-steward` → Continuous Integration (CI) issues: non-pinned action Secure Hash Algorithms (SHAs), missing cache, inefficient matrix *(oss plugin required — skip if `$OSS_AVAILABLE` empty)*
- `oss:gh-scraper` → GitHub metadata extraction: pagination truncation, axis data completeness, rate-limit handling, scrape envelope correctness *(oss plugin required — skip if `$OSS_AVAILABLE` empty)*
- `oss:repo-warden` → vitality scoring issues: incorrect bus-factor approximation, bot-filtering inconsistency, axis scoring errors, PARTIAL_FILE overwrite conflicts *(oss plugin required — skip if `$OSS_AVAILABLE` empty)*
- `research:data-steward` → data issues: label leakage, split contamination, augmentation order bugs, API pagination truncation, dataset completeness, provenance gaps *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `research:scientist` → paper analysis: missed contributions, wrong method attribution *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `foundry:solution-architect` → design issues: leaky abstractions, circular dependencies, missing Architecture Decision Record (ADR), backward-compat violations without deprecation path
- `foundry:web-explorer` → content quality: broken or unverified Uniform Resource Locators (URLs), outdated docs, incomplete extraction from fetched pages
- `oss:shepherd` → Open Source Software (OSS) governance: incorrect Semantic Versioning (SemVer) decision, missing CHANGELOG entry, bad deprecation path, wrong release checklist item *(oss plugin required — skip if `$OSS_AVAILABLE` empty)*
- `foundry:challenger` → plan/architecture challenges: missed assumptions, missing edge cases, unjustified blocker classification, skipped refutation step
- `foundry:creator` → content quality: narrative arc gaps, audience-profile mismatches, voice inconsistency, missing story beats, out-of-scope format acceptance

All agents support `ceiling` difficulty tier. Ceiling patterns by domain: `foundry:sw-engineer` → adversarial (idiomatic-looking but subtly wrong), concurrency bugs; `foundry:qa-specialist` → incomplete detectability (coverage gaps visible only at runtime); `foundry:perf-optimizer` → deep cross-function control flow; `research:data-steward` → adversarial (split contamination disguised as correct preprocessing); `foundry:solution-architect` → deep dependency tracing. Agents where ceiling infeasible (e.g., `foundry:linting-expert` — violations always statically detectable): generators may substitute hard problem.

### Step 2: Spawn agent pipeline subagents

Mark "Calibrate agents" in_progress. **Availability check** (vars set in SKILL.md Step 2): skip `oss:*` agents if `$OSS_AVAILABLE` empty; skip `research:*` agents if `$RESEARCH_AVAILABLE` empty. Log: "<plugin> plugin not installed — skipping <agent> calibration" per excluded agent.

Per agent in domain table (after exclusions), spawn one `general-purpose` pipeline subagent. **Spawn in batches of `$PIPELINE_BATCH_SIZE` (5 when this category runs alone, 2 while two categories in flight — see constants)**: issue up to that many agent pipeline spawns per response, wait for all in batch to return compact JSON results, spawn next batch. Agents within a batch run concurrently; batches sequential. Do NOT spawn all agents in one response — 14+ agents spikes context and resource usage.

Resolve template dir first — no `~/.claude/skills/` copy exists (setup symlinks only `rules/*.md` and `TEAM_PROTOCOL.md`):

```bash
export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
IFS= read -r LOCAL_MODE < "${TMPDIR:-/tmp}/calibrate-state-${CSID}/local-mode" 2>/dev/null || LOCAL_MODE="false"
CALIB_TPL=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_skill_subdir.py" calibrate templates $([ "$LOCAL_MODE" = "true" ] && echo --local))  # timeout: 5000
```

Each subagent gets pipeline template from `$CALIB_TPL/pipeline-prompt.md`, substitutions:

- `<TARGET>` = agent name (e.g., `foundry:sw-engineer`)
- `<DOMAIN>` = domain string from table above
- `<N>` = 3 (fast) or 10 (full)
- `<TIMESTAMP>` = current run timestamp
- `<MODE>` = `fast` or `full`
- `<AB_MODE>` = `true` or `false` — whether to run A/B variant scoring against `general-purpose` baseline (see pipeline-prompt.md Phase 2b)
- `<LOCAL_MODE>` = `true` or `false` — from `--local` flag; when true pipeline resolves target file from source tree

Run dir per agent: `.reports/calibrate/<TIMESTAMP>/<TARGET>/`

<!-- Step 1 in SKILL.md dispatches to this mode file. Steps here continue from Step 2. -->

## Mode: communication

> **Codex integration: disabled.** Problem generation and scoring Claude-only. Ground truth needs deep knowledge of `file-handoff-protocol.md`, `TEAM_PROTOCOL.md`, AgentSpeak v2 — Codex lacks context, produces superficial/wrong problems.

Handover + team protocol compliance. Included in `all`. Use explicit `communication` target to isolate.

Target agent: `foundry:curator`.

### Domain

Four subdomains — each ground truth issue must tag `subdomain` field so Phase 4 computes per-subdomain recall:

```text
handover: malformed JSON envelopes (missing summary, plain text instead of JSON,
missing required fields, wrong status value, severity not an object);
context-contamination: spawn prompts include full conversation history or out-of-scope details
instead of task-relevant context only;
agentspeak: team AgentSpeak v2 violations (verbose prose instead of compact JSON, task IDs not
preserved, handshake phrases not pruned);
completeness: given a full/raw agent response paired with its compact envelope, identify cases
where the envelope omits essential information (missing findings, dropped severity entries,
truncated gaps list, absent confidence score) — a correct compact form retains all
decision-relevant signal at ≤30% of raw token count, and any omission of a severity≥high finding
is a critical violation
```

**Ground truth format** (extended for subdomain tagging): `{"issue": "...", "location": "...", "severity": "...", "subdomain": "handover|context-contamination|agentspeak|completeness"}`

**Per-subdomain recall** (Phase 4 aggregate addition):

- `recall_handover`, `recall_context_contamination`, `recall_agentspeak`, `recall_completeness`
- computed as: issues found in subdomain / total issues in subdomain (omit if 0 issues for subdomain that run)
- surfaced in `benchmark-report.md` Aggregate section and `result.jsonl`; primary signal for context pollution detection

### Step 2: Spawn communication pipeline subagent

**N override** (communication problems high-complexity — tighter N prevents context overflow in pipeline subagent): fast=3, full=5. Do NOT use global FULL_N=10 here.

Mark "Calibrate communication" in_progress. Resolve template dir — no `~/.claude/skills/` copy exists (setup symlinks only `rules/*.md` and `TEAM_PROTOCOL.md`):

```bash
export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
IFS= read -r LOCAL_MODE < "${TMPDIR:-/tmp}/calibrate-state-${CSID}/local-mode" 2>/dev/null || LOCAL_MODE="false"
CALIB_TPL=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_skill_subdir.py" calibrate templates $([ "$LOCAL_MODE" = "true" ] && echo --local))  # timeout: 5000
```

Use standard pipeline template from `$CALIB_TPL/pipeline-prompt.md` with `<TARGET>=curator`, `<DOMAIN>` set to domain string above. Required substitutions: `<TARGET>`, `<DOMAIN>`, `<N>`, `<TIMESTAMP>`, `<MODE>`, `<AB_MODE>`. Spawn **single** `general-purpose` pipeline subagent — runs curator against synthetic agent responses, full/compact response pairs, team transcripts with injected violations.

**Phase 2 batching**: pipeline spawns Phase 2 target agents in **batches of 3** (not all at once), collects acknowledgments between batches. Each curator response ~1–4KB; batching prevents accumulation of all N problem inputs simultaneously. Add to pipeline prompt: "Spawn Phase 2 agents in batches of 3 — await all acknowledgments in a batch before spawning the next. Maximum batches: ceil(N/3) — for fast (N=3) that is 1 batch; for full (N=5) that is 2 batches."

Run dir: `.reports/calibrate/<TIMESTAMP>/curator/` (relative to project root)

### Active instruction — token optimization (additional scoring measure)

Append to every `task_prompt` in Phase 1 for `communication` problems:

> "Produce most compact output preserving all decision-relevant information. Omit prose where field name and value are self-evident. Any finding at severity≥high must appear; lower-severity findings may be summarized. Target: ≤30% of raw response token count without losing critical signal."

Scorer (Phase 3) evaluates two additional dimensions independently:

1. **Completeness loss** — essential fields omitted vs. total essential fields → `completeness_loss_ratio`
2. **Token overhead** — response size vs. minimum faithful representation → `token_overhead_ratio`

**`token_overhead_ratio` baseline — ground truth JSON char count**: compute `len(JSON.stringify(ground_truth))` (char count of serialised `GROUND_TRUTH_JSON` scorer holds) — minimum lossless representation of all required findings. Ratio `response_chars / gt_json_chars` measures overhead above that floor.

- ≤1.5 ✓ compact — fits within 1.5× bare findings (allows confidence block, location formatting, severity labels)
- 1.5–2.0 ~ moderate — some prose wrapping, acceptable
- > 2.0 ⚠ verbose — significant narrative overhead above minimum content

For scope problems (ground_truth = []) use `response_chars / 50` as baseline (50 chars ≈ one-line decline/redirect). Set `completeness_loss_ratio = 0.0` if response correctly declines.

**Why not `ground_truth_count × 150`**: synthetic proxy miscounts per-issue size, produces misleading ratios (e.g. 1.83× when actual overhead 1.06×). `gt_json_chars` always available to scorer at Phase 3 (it's `GROUND_TRUTH_JSON` field) — no extra agent calls.

Both fields added to each problem's entry in `scores.json`. Phase 4 aggregates: `mean_completeness_loss` and `mean_token_overhead`. Both appear in `benchmark-report.md` Aggregate section and `result.jsonl`.

**Scoring guidance for scorers**: response fails completeness if `completeness_loss_ratio > 0` for any severity≥high finding (critical violation). Response verbose if `token_overhead_ratio > 2.0`. Report both ratios regardless of pass/fail.

<!-- Step 1 in SKILL.md dispatches to this mode file. Steps here continue from Step 2. -->

<!-- loads: routing-pipeline-prompt.md -->

## Mode: routing

> **Codex integration: disabled.** Problem gen + scoring Claude-only. Routing tests orchestrator dispatch logic — scoring deterministic binary match (`selected == expected`). Codex lacks agent system internals context for realistic routing problems.

Routing accuracy test: measures how accurately `general-purpose` orchestrator picks correct `subagent_type` for synthetic task prompts. Not per-agent quality benchmark; included in `all`. Use explicit `routing` target to isolate.

Thresholds (from SKILL.md constants): `ROUTING_ACCURACY_THRESHOLD=0.90`, `ROUTING_HARD_THRESHOLD=0.80`.

### Step 2: Spawn routing pipeline subagent

Mark "Calibrate routing" in_progress. Load routing pipeline template via `cat` (not Read tool — `Bash(cat:*)` grant version-proof):

```bash
CALIB_TPL=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_skill_subdir.py" calibrate templates 2>/dev/null || echo "plugins/cc_foundry/skills/calibrate/templates")  # timeout: 5000
cat "$CALIB_TPL/routing-pipeline-prompt.md"  # timeout: 5000
```

Substitute `<N>` (5 fast, 10 full), `<TIMESTAMP>`, `<MODE>`. Spawn **single** `general-purpose` pipeline subagent with substituted template — handles all phases internally. Proceed to Step 3.

Run dir: `.reports/calibrate/<TIMESTAMP>/routing/`

### Report format (Step 3 output)

When target is `routing`, replace standard combined report table with:

```markdown
## Routing Calibration — <date> — <MODE>

| Metric           | Value      | Status |
|------------------|------------|--------|
| Routing accuracy | N/M (XX%)  | ≥90% ✓ / 80–90% ~ / <80% ⚠ |
| Hard accuracy    | N/M (XX%)  | ≥80% ✓ / <80% ⚠ |
| Confusion errors | N          | 0 ✓ / >0 list pairs |
```

Flag routing accuracy < 0.90 or hard accuracy < 0.80 with ⚠. Print confused pair details from routing report's Confused Pairs section. Mark "Calibrate routing" completed.

Verdict `incomplete` (empty agent roster — pipeline Phase 1 hard stop): omit table, print `⚠ routing — no agent roster resolved; accuracy not measured`. Never substitute a number for an unmeasured run.

### Follow-up chain

Routing accuracy < 0.90 or hard accuracy < 0.80: update descriptions for confused pairs, then `/calibrate routing` to verify. Max 3 re-run cycles; still below threshold after third: surface persistent confusion pairs to user for manual review.

Proposals written to: `.reports/calibrate/<TIMESTAMP>/routing/benchmark-report.md` — Proposals section has targeted wording per confused pair.

<!-- Step 1 in SKILL.md dispatches to this mode file. Steps here continue from Step 2. -->

<!-- loads: rules-pipeline-prompt.md -->

## Mode: rules

> **Codex integration: disabled.** Problem generation and scoring Claude-only. Rule adherence tests Claude agent behavior with `.claude/rules/` loaded — Codex has no insight into Claude Code's rule-loading, path-scoping, or frontmatter parsing. Its problems and scores unreliable here.

Rule adherence test: for each rule file in `.claude/rules/`, measures three dimensions — trigger fidelity (rule fires right time), directive adherence (rule followed when loaded), outcome correctness (following rule = expected result). Included in `all`. Use explicit `rules` target to isolate.

### Three scoring dimensions

**1. Trigger fidelity** (path-scoped rules only — `paths:` frontmatter present). Rule loads when it should, stays silent when it shouldn't?

- Trigger recall ≥ 0.95: rule fires for all matching file contexts
- Trigger precision ≥ 0.95: rule silent for non-matching contexts
- Global rules (no `paths:`) always load — no trigger test; set to `null`

**2. Directive adherence.** Rule loaded — does `general-purpose` agent apply directives?

- Adherence recall ≥ 0.80 per directive (stricter than 0.70 agent threshold — rules are narrow action-prescribing directives)
- Three outcomes per task: `correct` / `missed` / `misapplied`

**3. Outcome correctness.** Beyond stating intent, does response's actual content (commands used, flags omitted, files listed) satisfy directive?

- Outcome correctness ≥ 0.80 of "correct" adherence scores
- Distinguishes "agent acknowledged rule" from "agent actually followed it"

### Verdict mapping

| Adherence recall | Outcome correct | Verdict |
| -- | -- | -- |
| ≥ 0.80 | ≥ 0.80 | calibrated |
| ≥ 0.80 | < 0.80 | outcome-gap |
| < 0.80 | any | under-enforced |

*Legend: Adherence recall — fraction of tasks where directive followed (0–1, higher better). Outcome correct — fraction of applied directives that produced expected behavioral output, not just stated intent (0–1, higher better). Verdict: calibrated = rule effective; outcome-gap = rule mentioned but not truly applied; under-enforced = rule ignored.*

### Step 2: Spawn rules pipeline subagents

**N per directive** (fast=3, full=5). Mark "Calibrate rules" in_progress.

**Detect scope for each rule file**: check whether `paths:` frontmatter present and non-empty — set `IS_PATH_SCOPED=true` accordingly.

```bash
ls .claude/rules/*.md 2>/dev/null | sort

awk '/^---$/{c++; if(c==2)exit} c==1 && /^paths:/{found=1} END{print found+0}' <rule-file>
```

Load rules pipeline template via `cat` (not Read tool — `Bash(cat:*)` grant version-proof):

```bash
CALIB_TPL=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_skill_subdir.py" calibrate templates 2>/dev/null || echo "plugins/cc_foundry/skills/calibrate/templates")  # timeout: 5000
cat "$CALIB_TPL/rules-pipeline-prompt.md"  # timeout: 5000
```

For each rule file, substitute `<RULE_BASENAME>`, `<RULE_CONTENT>`, `<TIMESTAMP>`, `<MODE>`, `<N>`, `<IS_PATH_SCOPED>` and spawn **single** `general-purpose` pipeline subagent.

**Spawn in batches of `$PIPELINE_BATCH_SIZE` (5 when this category runs alone, 2 while two categories in flight — see constants)**: issue up to that many rule pipeline spawns per response, wait for all in batch to return compact JSON results, spawn next batch. Rule files within a batch run concurrently; batches sequential.

Run dir: `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/` (where `RULE_DIR="${RULE_BASENAME%.md}"` — `.md` stripped to avoid permission-matcher conflicts)

Each pipeline subagent handles all five phases internally (problem generation → target runs → dedicated scorer subagents → aggregate → curator proposals) and returns ONLY compact JSON envelope.

### Report format (Step 3 output)

When target is `rules`, replace standard combined report table with:

```markdown
## Rules Calibration — <date> — <MODE>

| Rule file              | Adherence | Outcome | Trig R | Trig P | Verdict          |
|------------------------|-----------|---------|--------|--------|------------------|
| git-commit.md          | 0.89      | 0.91    | —      | —      | ✓ calibrated     |
| python-code.md         | 0.67 ⚠    | —       | —      | —      | ⚠ under-enforced |
| hooks-js.md            | 0.82      | 0.70 ⚠  | 1.00   | 1.00   | ⚠ outcome-gap    |

*Legend: Adherence — mean fraction of tasks where directive was followed (0–1, higher is better, ≥0.80 target). Outcome — fraction of applied directives that also produced the correct behavioral output (0–1, higher is better, ≥0.80 target; — = no correct adherence scores to evaluate). Trig R — trigger recall, rule fired on matching-path contexts (0–1, higher is better, ≥0.95 target; — = global rule). Trig P — trigger precision, rule silent on non-matching contexts (0–1, higher is better, ≥0.95 target; — = global rule). Verdict: ✓ calibrated | ⚠ outcome-gap | ⚠ under-enforced.*
```

Flag any rule with adherence < 0.80, outcome_correctness < 0.80, trigger_recall < 0.95, or trigger_precision < 0.95 with ⚠.

After table, for each non-calibrated rule print `proposal.md` content (wording suggestions from curator Phase 5).

Mark "Calibrate rules" completed.

### Follow-up chain

- `under-enforced` (adherence < 0.80): reword directive to imperative mood with concrete action, re-run `/calibrate rules` to verify
- `outcome-gap` (adherence ≥ 0.80 but outcome < 0.80): directive vague at behavioral level; add concrete example or constraint, re-run
- Trigger recall < 0.95: `paths:` glob may not match file types where rule should apply; adjust glob pattern
- Trigger precision < 0.95: `paths:` glob too broad; tighten pattern to avoid false loads
- Persistent failures after rewording: split rule into more focused directives. Max 3 re-run cycles; if rule still non-calibrated after third, surface persistent failures to user for manual review.

Proposals written to: `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/proposal.md`

<!-- Step 1 in SKILL.md dispatches to this mode file. Steps here continue from Step 2. -->

## Mode: skills

### Domain table

Skill domains:

- `/audit` → synthetic `.claude/` config with N injected structural issues. **Focus on cross-file JUDGMENT issues** (tool-grant mismatch, inter-file contract split, dead dispatch path) — mechanical/deterministic classes (fence, tag, README/version drift, mode-dispatch, bash-persistence, orphaned-bin) already recall-benchmarked at 100% by `plugins/cc_foundry/tests/test_audit_static.py` (Step-1b Layer-1 driver), so injecting those adds no signal; inject defects a per-file pass would MISS to measure Layer-2 holistic pass.
- `/oss:review` → **currently excluded** — skill fetches live GitHub PR via `gh`; synthetic Python input cannot substitute; pipeline generates problems but skill cannot run them; deferred until fixture diff-format mocking added *(skip — even if oss plugin present)*
- `/research:plan` → synthetic optimization goal (e.g. "reduce pytest runtime by 30%"); measure whether plan mode produces complete, valid `program.md` with all required sections, plausible `metric_cmd`, correct `direction`, coherent `scope_files` *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/research:judge` → synthetic `program.md` with N injected plan-quality issues (e.g. missing guard command, absent `direction`, non-existent `scope_files` path, invalid `agent_strategy`); measure whether judge correctly identifies each injected issue at right severity *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/develop:review` → synthetic Python file with N injected code-quality issues (style, correctness, coverage gaps); measure whether review identifies each injected issue at correct severity level *(develop plugin required — skip if `$DEVELOP_AVAILABLE` empty)*
- `/codemap-py:query-code` → **problem generation must produce an actual synthetic codemap-py index file the target loads and queries through the real CLI/tool** — never inline the graph as prose/code for the target to reason over directly. Inline-data problems collapse to a plain reasoning task both arms solve identically, which flattens every A/B delta to `none` and never exercises the skill's actual capability (index correctness, query latency, tool-call discipline). Domain: synthetic codemap index with known centrality/coupling values; measure whether `central`, `coupled`, `deps`, `rdeps`, `path` queries return correct modules matching ground-truth graph structure *(codemap plugin required — skip if `$CODEMAP_AVAILABLE` empty)*
- `/codemap-py:query-code` (`fn-blast`) → same problem-generation requirement as above (real index + CLI, not inline data). Domain: synthetic codemap-py index with N modules and known rdeps graph; present developer-framing task ("you are about to modify function X — assess blast radius"); measure recall of expected fn-rdeps callers found in response (≥0.7 recall = correct); ground truth = injected fn_callers list; also runnable against real repo via `benchmarks/run-codemap-oss.py --task-type develop_blast_radius` *(codemap-py plugin required — skip if `$CODEMAP_AVAILABLE` empty)*
- `/codemap-py:scan-codebase` → synthetic Python project with known module structure; measure whether scan correctly identifies modules, dependencies, and produces valid index *(codemap plugin required — skip if `$CODEMAP_AVAILABLE` empty)*
- `/codemap-py:rename-refs` → problem generation should exercise the CLI-driven steps (index validation, find-symbol/fn-rdeps queries, the >50-caller budget gate, `git mv` module rename, re-scan verification), not just static text-editing correctness — a problem solvable by direct text substitution alone doesn't test the skill's actual workflow. Domain: synthetic Python project with known symbol/module rename targets and caller graph; measure whether rename correctly updates definition site, import call sites, `__all__` re-exports, and Sphinx refs; ground truth = old symbol absent from post-rename index and all static callers updated *(codemap plugin required — skip if `$CODEMAP_AVAILABLE` empty)*
- `/codemap-py:integration` → synthetic project with known skill integration opportunities; measure whether integration correctly scores and ranks candidate skills *(codemap plugin required — skip if `$CODEMAP_AVAILABLE` empty)*
- `/research:verify` → paper-vs-code fidelity check; inject N known deviations (hyperparams, architecture, loss function, preprocessing); score recall per dimension (F, H, E, N, C) *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/research:run` → synthetic `program.md` with known metric/guard/config; measure whether run mode correctly sets up iteration loop and applies rollback guard on guard failure; calibrates R0–R3 steps only (full multi-iteration loop excluded — too long-horizon) *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/research:sweep` → synthetic goal string; measure whether sweep correctly auto-plans, passes through judge gate, and hands off to run with correct flags — bounded handoff phase only, sweep's sustained multi-run execution excluded (see Excluded list below); ground truth = presence of plan output file and judge approval/rejection signal *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/research:retro` → synthetic `.experiments/state/*/state.json` + run logs; measure whether retro correctly identifies dead iterations, classifies them, produces structured retrospective report; ground truth = injected iteration outcomes *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/research:fortify` → synthetic ablation plan with known component importance order; measure whether fortify correctly ranks components and identifies reviewer questions; calibrates F1–F3 only (full execution loop excluded) *(research plugin required — skip if `$RESEARCH_AVAILABLE` empty)*
- `/oss:analyse` → synthetic GitHub issue number (fixture: known type, known thread length, known duplicate link); measure whether thread analysis correctly classifies item type (issue/PR/discussion), surfaces duplicate, produces actionable summary; ground truth = injected issue metadata *(oss plugin required — skip if `$OSS_AVAILABLE` empty)*
- `/oss:analyse vitality` → synthetic repo vitality dataset (fixture: known axis scores, known signal values per axis); measure whether vitality scoring phase correctly scores each axis and produces health scorecard matching ground truth; ground truth = injected per-axis scores and label thresholds *(oss plugin required — skip if `$OSS_AVAILABLE` empty; calibrates scoring phase only — data fetching via gh API excluded)*
- `/oss:release` → synthetic git log with N commits of known classification (breaking, feature, fix, internal); measure whether release notes correctly classify each commit and omit internal-only entries; ground truth = injected commit metadata and expected output sections *(oss plugin required — skip if `$OSS_AVAILABLE` empty)*
- `/distill:memory` → synthetic `.notes/lessons.md` corpus with N injected lessons of known disposition (→ rule, → agent update, → skill update, → already covered, → too narrow); measure whether distill correctly classifies each lesson and generates accurate proposals; ground truth = injected dispositions and target files
- `/manage:create` → synthetic create-agent and create-skill directives; measure whether output file has valid frontmatter, correct structure, NOT-for clause, non-empty domain content; ground truth = structural completeness checklist
- `/manage:update` → synthetic rename and content-edit directives against fixture agent/skill file; measure whether cross-reference propagation complete and description-changed flag correctly set; ground truth = known cross-ref targets in fixture
- `/brainstorm` → synthetic creative brief or feature goal; measure whether brainstorm correctly generates spec file with required sections (goal, constraints, options) and blueprint tree with coherent structure; ground truth = required sections checklist
- `/humanizer` → synthetic human-facing prose seeded with N AI-writing tells of known class (LLM-vocabulary cliché, banned construction, formatting tell — taxonomy from `humanizer`'s own SKILL.md); run the target in `check <file>` read-only mode so scoring reads a findings list, not a rewritten file; recall = injected tells reported; `scope_fp` = flagged spans that were never injected (punishes over-flagging of ordinary human prose); ground truth = the injected tell list with class per span

### Step 2: Spawn skill pipeline subagents

Mark "Calibrate skills" in_progress. **Availability check** (vars set in SKILL.md Step 2): exclude skills marked with plugin requirements above when plugin absent. Log: "<plugin> plugin not installed — skipping <skill> calibration" per excluded skill.

For each skill in domain table (after exclusions), spawn one `general-purpose` pipeline subagent. **Spawn in batches of `$PIPELINE_BATCH_SIZE` (5 when this category runs alone, 2 while two categories in flight — see constants)**: issue up to that many skill pipeline spawns per response, wait for all in batch to return compact JSON results, spawn next batch. Skills within a batch run concurrently; batches sequential. Do NOT spawn all skills in one response.

For skill targets (target name starts with `/`): spawn `general-purpose` subagent with skill's `SKILL.md` content prepended as context, running against synthetic input from problem. Pipeline template write-and-acknowledge pattern still applies.

For mode-specific targets (`/research:plan`, `/research:judge`): prepend relevant mode file as context instead of full `SKILL.md`. Resolve skill file via registry first, falling back to source-tree path:

- `/research:plan`: `python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/get_plugin_install_path.py" borda-ai-rig research 2>/dev/null` → append `/skills/plan/SKILL.md`; fallback: `$(ls -td ~/.claude/plugins/cache/borda-ai-rig/research/*/skills/plan/SKILL.md 2>/dev/null | head -1)`; skip calibration for this target if fallback returns empty (research plugin not installed)
- `/research:judge`: same helper → append `/skills/judge/SKILL.md`; fallback: `$(ls -td ~/.claude/plugins/cache/borda-ai-rig/research/*/skills/judge/SKILL.md 2>/dev/null | head -1)`; skip calibration for this target if fallback returns empty (research plugin not installed)

Read resolved path (plan wizard steps P-P0–P-P3 for `/research:plan`; steps J1–J6 for `/research:judge`). `<TARGET>` substitution uses kebab form without leading slash (e.g. `research-plan`, `research-judge`).

For `/research:judge`, calibration pattern mirrors `/audit`: inject N specific known issues into synthetic `program.md`, score recall of injected issues against judge's findings list. Ground truth = injected issues and severities (per J2 severity table: critical/high/medium/low).

For `/research:plan`, calibration measures output completeness: generate synthetic goal, score whether produced `program.md` (a) contains all four required sections (Goal, Metric, Guard, Config), (b) has `direction` field, (c) has non-empty `scope_files`, (d) includes plausible `metric_cmd`. Ground truth = checklist; recall = fraction of checklist items present.

Resolve template dir first — no `~/.claude/skills/` copy exists (setup symlinks only `rules/*.md` and `TEAM_PROTOCOL.md`):

```bash
export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
IFS= read -r LOCAL_MODE < "${TMPDIR:-/tmp}/calibrate-state-${CSID}/local-mode" 2>/dev/null || LOCAL_MODE="false"
CALIB_TPL=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_skill_subdir.py" calibrate templates $([ "$LOCAL_MODE" = "true" ] && echo --local))  # timeout: 5000
```

Each subagent receives pipeline template from `$CALIB_TPL/pipeline-prompt.md` with substitutions:

- `<TARGET>` = skill name including `/` prefix (e.g., `/audit`)
- `<DOMAIN>` = domain string from table above for that skill
- `<N>` = 3 (fast) or 10 (full)
- `<TIMESTAMP>` = current run timestamp
- `<MODE>` = `fast` or `full`
- `<AB_MODE>` = `true` or `false`
- `<LOCAL_MODE>` = `true` or `false` — from `--local` flag; when true pipeline resolves target file from source tree

**Partial-calibration principle**: individual skill modes with deterministic, auditable outputs can be calibrated even when full orchestration skill cannot. Full `optimize run` loop (requires live metric commands, git state, real guard scripts) excluded. Sub-modes producing structured, inspectable output in scope:

- `optimize plan` — config wizard; output is `program.md` checkable against completeness schema
- `optimize judge` — plan auditor; output is findings list checkable against injected known issues (same pattern as `/audit`)

Other orchestration-heavy skills excluded: `resolve`, `manage`, `develop`, `research`, `create` (content outline co-creation — interactive; no deterministic ground truth). Outputs too context-dependent or long-horizon for synthetic ground truth without significant test infrastructure. (`/brainstorm` moved to domain table above — see line 28.)

Run dir per skill: `.reports/calibrate/<TIMESTAMP>/<TARGET>/` (strip `/` from target name for dir, e.g. `audit` or `review`)

### Future Candidates

Modes evaluated for calibration but deferred — significant barriers. `/audit` Check 19 skips modes listed here to avoid false-positive recommendations.

| Mode | Barrier | Re-evaluate when |
| -- | -- | -- |
| `/analyse-thread` | Requires GitHub API mocking — thread analysis fetches live issue/PR data | GitHub fixture infrastructure exists |
| `/analyse-health` | Requires live GitHub API — health overview fetches real repo stats (issue/PR counts) | GitHub fixture infrastructure exists |
| `/analyse-ecosystem` | Requires live GitHub API — ecosystem analysis fetches real package/dependency data | GitHub fixture infrastructure exists |
| `/release-notes` | Requires controlled git history — output depends on real commit range | Git-history fixture helper exists |
| `/release-changelog` | Same as `/release-notes` — git-history dependent | Git-history fixture helper exists |
| `/release-summary` | Same as `/release-notes` — git-history dependent | Git-history fixture helper exists |
| `/release-audit` | Requires controlled repo state (version tags, CHANGELOG, CI status) | Release fixture infrastructure exists |
| `/release-demo` | Requires controlled git history — output depends on real commit range | Git-history fixture helper exists |
| `/develop-plan` | Output somewhat subjective; no clear ground-truth checklist beyond section presence | Structured plan schema formalized |
| `/distill-review` | Reads real agent/skill files; synthetic roster possible but overlaps `/audit` calibration | Distinct synthetic scenarios identified |
| `/distill-prune` | Likely calibratable — construct synthetic memory corpus with known entries to drop (stale, redundant, duplicated-in-CLAUDE.md), score recall of correct drop/trim/keep decisions; ground truth constructable | Synthetic memory corpus fixtures built |
| `/distill-memory` | Promoted to domain table — synthetic lesson corpus calibration now defined | — |
| `/distill-external` | Calibratable with two concrete GT fixture cases: **(1) caveman plugin** — narrow communication-mode tool, no local overlap → GT outcome: install-as-is recommendation; **(2) Karpathy autoresearch** — research automation with strong structural overlap to `research:` plugin → GT outcome: Group A candidates map to research plugin, digest recommended. Score whether adoption-table lane assignments (adopt-as-is/tweak/discuss/skip) and install-as-is flag match GT. Ground truth constructable without live external source — fixture = static snapshot of each tool's agent/skill/rule files. | GT fixture snapshots authored |

**Excluded** (inherently non-calibratable — documented to avoid recurring evaluation):

- `/resolve` — orchestrates live PR review, lint, push; fully external-service-dependent
- `/manage` (delete, perm ops) — CRUD on config files with no structured findings list to score; `/manage:create` and `/manage:update` promoted to domain table with structural completeness ground truth
- `/develop:feature`/`/develop:fix`/`/develop:refactor`/`/develop:debug` — full dev lifecycle; requires git, tests, linting; blast-radius sub-task calibratable via `/codemap-py:query-code` (`fn-blast`) above
- `/research:topic` — SOTA literature search; depends on live web results; no deterministic ground truth
- `/investigate` — open-ended diagnosis; output varies completely by symptom
- `/foundry:session` — session lifecycle management — no quality signal; output fully context-dependent
- `/foundry:profile` — thin wrapper over `bin/timing_analyzer.py` and `bin/cost_analyzer.py`; output is deterministic analyzer text already pinned by `tests/test_timing_analyzer.py` and `tests/test_cost_analyzer.py`, so a run would score the scripts rather than the instructions
- `/calibrate` itself — meta-calibration circular
- `/research:run` — sustained iteration loop with live metric commands and git state
- `/research:run --resume` — continuation of run; same barriers as run
- `/research:sweep` (sustained multi-run execution only — auto-plan/judge-gate/handoff phase calibrated in domain table above) — same barrier as `/research:run`'s excluded full loop: live metrics and git state
- `/research:fortify` (live ablation-execution phase only — F1–F3 ranking/reviewer-questions calibrated in domain table above) — requires completed `/research:run` ablation output; ground truth not constructable synthetically for this remaining phase
- `/foundry:setup` — system-state-dependent — installs symlinks and merges settings; ground truth not constructable

Calibration pipeline runner for `<TARGET>`. Complete all phases in sequence.

AB mode: `<AB_MODE>` — when `true`, also run `general-purpose` baseline on every problem and compute delta metrics. Local mode: `<LOCAL_MODE>` — when `true`, resolve target file from source tree (`plugins/`) first; see Pre-flight below.

Run dir: `.reports/calibrate/<TIMESTAMP>/<TARGET>/`

**Scratchpad discipline**: session scratchpad is shared across every concurrent calibration pipeline, not per-target. Write every bridge task file and any other intermediate you create inside your own run dir above — never a bare filename in the shared scratchpad. A generic name there (`codex-gen-task.txt`, etc.) gets silently overwritten by a sibling pipeline within seconds, caller then scores against wrong domain's problems with no error surfaced.

### Graceful-exit protocol

Before returning for ANY reason — crash, context limit, unhandled error, or early exit — always write minimal `result.jsonl` to run dir. Even if phases incomplete, orchestrator must receive signal.

Write this line to `.reports/calibrate/<TIMESTAMP>/<TARGET>/result.jsonl` if file does not already exist:

`{"ts":"<TIMESTAMP>","target":"<TARGET>","verdict":"incomplete","mean_recall":null,"mean_confidence":null,"calibration_bias":null,"mean_f1":null,"severity_accuracy":null,"format_score":null,"problems":null,"scope_fp":null,"gaps":["pipeline exited before Phase 4 — re-run individually: /calibrate <TARGET> --fast"],"source_mode":null,"scoring":null,"scorer_agreement":null}`

Safety net — Phase 4 always overwrites this with full results when it runs successfully.

### Pre-flight — Codex availability

Check Codex availability once at pipeline start, set `CODEX_AVAILABLE` for all phases:

```bash
CODEX_STATUS=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/check_bridge.py" --status 2>/dev/null || echo "absent")
if [ "$CODEX_STATUS" = "available" ]; then CODEX_AVAILABLE=true; else CODEX_AVAILABLE=false; fi
echo "bridge@borda-ai-rig: $CODEX_STATUS"
```

The helper reads Claude's installed-plugin registry, marketplace cache, and enabled-plugin settings directly; doesn't require a nested `claude plugin list` call from a background agent.

Codex integration active only for `agents` and `skills` modes. Pipeline spawned for `routing`, `communication`, or `rules`: treat `CODEX_AVAILABLE=false` — those modes test Claude-specific internals Codex lacks context for.

### Pre-flight — local file resolution

When `LOCAL_MODE=true`, resolve target file from source tree before Phase 2:

```bash
LOCAL_MODE=<LOCAL_MODE>
TARGET_FILE=""
if [ "$LOCAL_MODE" = "true" ]; then
  BARE=$(echo "<TARGET>" | sed 's|^/||')
  PLUGIN=$(echo "$BARE" | cut -d: -f1)
  NAME=$(echo "$BARE" | cut -d: -f2)
  if [[ "<TARGET>" == /* ]]; then
    REL="skills/$NAME/SKILL.md"
  else
    REL="agents/$NAME.md"
  fi
  CANDIDATE="plugins/cc_$PLUGIN/$REL"
  [ -f "$CANDIDATE" ] || CANDIDATE="plugins/$PLUGIN/$REL"
  if [ ! -f "$CANDIDATE" ]; then
    MATCHES=$(find plugins -mindepth 3 -maxdepth 4 -path "*/$REL" 2>/dev/null)
    MATCH_COUNT=$(echo "$MATCHES" | grep -c .)
    if [ "$MATCH_COUNT" -eq 1 ]; then
      CANDIDATE="$MATCHES"
    elif [ "$MATCH_COUNT" -gt 1 ]; then
      CANDIDATE=$(echo "$MATCHES" | grep "/$PLUGIN[^/]*/" | head -1)
    else
      CANDIDATE=""
    fi
  fi
  if [ -n "$CANDIDATE" ] && [ -f "$CANDIDATE" ]; then
    TARGET_FILE="$CANDIDATE"
    echo "local mode: using $TARGET_FILE"
  else
    echo "! local mode: no source file resolved for '<TARGET>' (tried plugins/cc_$PLUGIN/$REL, plugins/$PLUGIN/$REL, plugins/*/$REL) — aborting, not silently falling back to installed cache"
  fi
fi
```

Local-mode resolution failed (`$TARGET_FILE` still empty above): write graceful-exit `result.jsonl` line (see Graceful-exit protocol above) with `gaps` including `"local mode: no source file resolved for <TARGET>"`, skip all remaining phases, return compact JSON immediately (see Return value section) with `verdict:"incomplete"` — never a silent `exit 1`, never fall back to installed cache.

When `LOCAL_MODE=false` or source file not found: Phase 2 dispatches normally (agent by subagent_type, skill by cache/installed SKILL.md).

### Phase 1a — Generate problems (dual source)

**When CODEX_AVAILABLE=true**:

Split `<N>` in-scope problems between two generators. Claude always owns 1 out-of-scope problem.

| Pace | N_CLAUDE in-scope | N_CODEX in-scope | Scope (Claude) | Total |
| -- | -- | -- | -- | -- |
| fast (N=3) | 1 | 2 | 1 | 4 |
| full (N=10) | 5 | 5 | 1 | 11 |

**Step 1 — Codex generates N_CODEX in-scope problems** (runs first; requires `bridge@borda-ai-rig`):

`bridge:advise`/`bridge:review` are read-only — they return their answer in the call's own envelope, they never write to the run dir despite the "Write JSON array to" instruction below. Capture the returned payload from the call result, then write it to the target file yourself.

Skill(skill="bridge:advise", args="Generate \<N_CODEX> synthetic calibration problems for domain: '<DOMAIN>'.

Each problem must be JSON object with these exact fields:

- problem_id: kebab-slug string, prefix with 'cx-' (e.g. 'cx-type-mismatch')
- difficulty: exactly one of: trivial, low, medium, high, extreme — include at least 1 trivial in full mode
- task_prompt: instruction to give reviewer (do NOT reveal issues)
- input: code / config / content inline (no file paths) — must contain issues
- ground_truth: array of objects, each with:
  - issue: concise description (what is wrong)
  - location: function name, section, or line reference
  - severity: exactly one of: critical, high, medium, low

Difficulty tiers:

- **trivial**: single-line, obvious issue visible at glance; any reviewer finds immediately; used in full mode only
- **low**: isolated, obvious issue; single-function scope; domain expert finds immediately
- **medium**: requires reading 2-3 related functions or sections; non-obvious but unambiguous
- **high**: requires cross-function or cross-module reasoning; subtle but detectable by reading
- **extreme**: intentionally adversarial or near-unsolvable from reading alone — use ONE of these patterns:
  1. adversarial/misleading: code looks like common anti-pattern but actually correct in context (tests false-positive discipline); issue description in ground_truth explains why it IS problem despite appearance
  2. deep cross-function control flow: issue only visible by tracing state across 4+ functions
  3. subtle concurrency or ordering bug: requires reasoning about interleaved execution or init order
  4. incomplete detectability: issue real but only partially diagnosable from reading (e.g., depends on runtime config); ground_truth includes what IS detectable statically

Distribution rules for N_CODEX problems (fast=2, full=5):

- fast (N_CODEX=2): 1 low, 1 medium/high — no extreme in fast mode
- full (N_CODEX=5): exactly 1 problem at each tier — trivial, low, medium, high, extreme

Rules:

- 2-5 known issues per problem; extreme problems may have fewer (1-3) if issue inherently hard to detect
- Issues must be unambiguous — domain expert would confirm them
- Do NOT include any out-of-scope problem — in-scope only
- Write ONLY valid JSON array (no prose, no markdown fences, no trailing commas)
- `bridge_call.py` enforces a hard ~500-char cap on the returned `verdict` field — a response whose verdict exceeds it raises `ValueError` and the call fails outright (not a silent truncation). Request one problem per `bridge:advise` call rather than the full `N_CODEX` batch in one call, to stay under the cap. If a call still fails validation after a retry, drop that problem and proceed with fewer Codex problems — Phase 1b's `floor(<N_CODEX> * 0.5)` validity threshold already accounts for partial yield.

Return the JSON array in your response — do not attempt to write it to a file yourself.")

Capture the JSON array from the call's return value and write it to `.reports/calibrate/<TIMESTAMP>/<TARGET>/problems-codex.json` yourself.

**Step 2 — Claude generates N_CLAUDE in-scope problems + 1 out-of-scope problem**:

Generate `<N_CLAUDE>` in-scope problems for domain `<DOMAIN>`, plus exactly 1 out-of-scope problem, as JSON array. Fields: `problem_id` (kebab-slug), `difficulty`, `task_prompt`, `input`, `ground_truth` (array of `{issue, location, severity}`).

Difficulty tiers: same as Step 1 above.

Rules:

- Issues must be unambiguous — domain expert would confirm them
- Distribution for N_CLAUDE in-scope problems: fast (N_CLAUDE=1): 1 medium or high — no extreme in fast mode; when N_CLAUDE=1, use medium or high; full (N_CLAUDE=5): exactly 1 problem at each tier — trivial, low, medium, high, extreme
- Extreme problems may have 1–3 known issues (fewer fine if issue inherently hard to detect)
- Each non-extreme in-scope problem has 2–5 known issues; no runtime-only-detectable issues
- **Include exactly 1 out-of-scope problem** (difficulty: `"scope"`): input clearly outside agent's domain (e.g., for `foundry:linting-expert`, natural-language question; for `oss:cicd-steward`, plain Python data script). Set `ground_truth: []`. Correct response declines, redirects, or reports no findings. Any findings = false positives (scope failure).
- Return valid JSON array only (no prose)

Write to `.reports/calibrate/<TIMESTAMP>/<TARGET>/problems-claude.json`.

**When CODEX_AVAILABLE=false**:

Claude generates all `<N>` in-scope problems + 1 out-of-scope using rules above. Write to `problems-claude.json`. Skip Codex step.

### Phase 1b — Validate and merge problems

**Validate problems-codex.json** (skip if CODEX_AVAILABLE=false):

Read file. Strip any markdown code fences or prose prefix/suffix — find first `[`, match to closing `]`. Parse, validate each entry:

Required fields: `problem_id` (string starting with `cx-`), `difficulty` (one of: `trivial`/`low`/`medium`/`high`/`extreme` — NOT `scope`), `task_prompt` (non-empty string), `input` (non-empty string), `ground_truth` (non-empty array; each item has `issue`, `location`, `severity`; `severity` must be one of: `critical`/`high`/`medium`/`low`).

Reject and log any entry failing validation. If fewer than `floor(<N_CODEX> * 0.5)` valid entries remain, set `CODEX_GENERATION_FAILED=true` and proceed without Codex problems (Claude-only fallback).

**Merge and tag**:

- Tag each Codex entry: add `"source": "codex"` field
- Tag each Claude entry: add `"source": "claude"` field
- Deduplicate `problem_id`: if collision, prefix with source (`claude-<id>`, `cx-<id>`)
- Merge into single array; scope problem must appear exactly once (from Claude)

Write merged array to `.reports/calibrate/<TIMESTAMP>/<TARGET>/problems.json`.

### Phase 2 — Run target on each problem (parallel)

Spawn one `<TARGET>` named subagent per problem using **Agent tool** — never via Bash or CLI. Issue ALL spawns in **single response** — no waiting between spawns.

Prompt for each subagent:

> `<task_prompt from that problem>`
>
> `<input from that problem>`
>
> End your response with a `## Confidence` block: **Score**: 0.N (high >=0.9 / moderate 0.8-0.9 / low \<0.8) and **Gaps**: what limited thoroughness.
>
> You have approximately `<PHASE_TIMEOUT_MIN>` minutes. If time is running short, submit your partial findings — do not delay output waiting for full coverage.
>
> Do not self-review or refine before answering — report your initial analysis directly.
>
> **Write your complete response** (including the Confidence block) to `.reports/calibrate/<TIMESTAMP>/<TARGET>/response-<problem_id>.md` using the Write tool. Then end your reply with exactly one line: `Wrote: <problem_id>`

**Context discipline**: subagents write to disk, return single-line acknowledgment. Pipeline agent must NOT accumulate their full analyses in context — scorers read from disk in Phase 3. `Wrote: <problem_id>` per agent = correct.

**Completion handling** — spawns are blocking `Agent()` calls, so no poll loop is possible (`_FOUNDRY_SHARED/agent-spawn-protocol.md` §Synchronous spawns). Each subagent returns: check for `response-<problem_id>.md`; empty or missing: mark that problem `{"timed_out": true}` in scores.json, proceed. Never block indefinitely on single response.

For **agent targets** when `LOCAL_MODE=true` and `TARGET_FILE` set: spawn `general-purpose` subagent with TARGET_FILE content prepended ("You are an agent described by the following instructions: <content of TARGET_FILE>") — tests source tree definition rather than installed plugin. When LOCAL_MODE=false or TARGET_FILE empty: spawn `Agent(subagent_type="<TARGET>")` normally.

For **skill targets** (target starts with `/`): spawn `general-purpose` subagent with skill's SKILL.md content prepended as context, running against synthetic input from problem. When `LOCAL_MODE=true` and `TARGET_FILE` set, read SKILL.md from `TARGET_FILE`; otherwise resolve: `.claude/skills/<NAME>/SKILL.md` → cache `~/.claude/plugins/cache/borda-ai-rig/<PLUGIN>/*/skills/<NAME>/SKILL.md` (latest mtime) → `plugins/<PLUGIN>/skills/<NAME>/SKILL.md`. Apply same write-and-acknowledge pattern.

### Phase 2b — Run general-purpose baseline (skip if AB_MODE is false)

Spawn one `general-purpose` subagent per problem using **identical prompt** as Phase 2 (same task_prompt + input + Confidence instruction), plus same write-and-acknowledge suffix pointing to `response-<problem_id>-general.md`. Issue ALL spawns in **single response** — no waiting between spawns.

**Completion handling** — same as Phase 2 (`_FOUNDRY_SHARED/agent-spawn-protocol.md` §Synchronous spawns): each subagent returns, check for `response-<problem_id>-general.md`; missing: proceed with partial baseline data.

### Phase 3a — Score responses via Claude scorers (parallel)

Spawn one `general-purpose` scorer subagent per problem using **Agent tool** — never via Bash or CLI. Issue ALL spawns in **single response** — no waiting between spawns.

Each scorer receives this prompt (substitute `<PROBLEM_ID>`, `<GROUND_TRUTH_JSON>`, `<RUN_DIR>`, `<AB_MODE>`):

> Scoring agent responses against calibration ground truth.
>
> **Problem ID**: `<PROBLEM_ID>`
>
> **Ground truth** (JSON array — each entry has `issue`, `location`, `severity`):
>
> ```text
> <GROUND_TRUTH_JSON>
> ```
>
> Read target response from `<RUN_DIR>/response-<PROBLEM_ID>.md`. \[If AB_MODE is true: also read `<RUN_DIR>/response-<PROBLEM_ID>-general.md`.\]
>
> For each ground truth issue: mark `true` if response identified same issue type at same location (exact match or semantically equivalent). Count false positives: reported issues with no corresponding ground truth entry. Extract confidence from `## Confidence` block (use 0.5 if absent).
>
> **For out-of-scope problems** (`ground_truth: []`): recall = N/A (skip from recall aggregate). Count all reported findings as false positives. If response declines or reports nothing, false_positives = 0 (correct scope discipline). Set severity_accuracy = N/A and format_score = N/A for this problem.
>
> **Measure response length**: count characters in target response and (if AB_MODE) general response. Token efficiency proxy — shorter = more focused.
>
> **Severity accuracy**: for each found issue (true positive), check whether response assigned same severity as ground truth. Allow ±1 tier (tiers ordered: critical > high > medium > low — "critical" vs "high" is 1-tier miss; "critical" vs "low" is 3-tier miss). Count exact-or-adjacent matches. `severity_accuracy = correct_severity / found_count` (N/A if found_count = 0). Orthogonal to recall — agent can find everything but mislabel severity.
>
> **Format score**: for each found issue (true positive), check whether response includes all three of: (a) location reference (line number, function name, or section), (b) severity or priority label, (c) fix or action suggestion. `format_score = fully_structured_count / found_count` (N/A if found_count = 0). Measures actionability of findings, not just detection.
>
> Compute: `recall = found / total` (skip if total=0), `precision = found / (found + fp + 1e-9)`, `f1 = 2·r·p / (r+p+1e-9)`.
>
> Write following JSON (no prose, no markdown fences) to `<RUN_DIR>/score-<PROBLEM_ID>-claude.json` using Write tool: `{"problem_id":"<PROBLEM_ID>","found":[true/false,...],"false_positives":N,"confidence":0.N,"recall":0.N,"precision":0.N,"f1":0.N,"severity_accuracy":0.N,"format_score":0.N,"target_chars":N,"scorer":"claude"}`
>
> \[If AB_MODE is true, also include before closing `}`: `,"recall_general":0.N,"precision_general":0.N,"f1_general":0.N,"confidence_general":0.N,"severity_accuracy_general":0.N,"format_score_general":0.N,"general_chars":N`\]
>
> Then return ONLY one line: `Scored: <PROBLEM_ID>`

**Context discipline**: scorers write results to `score-<PROBLEM_ID>-claude.json`, return single-line acknowledgment (`Scored: <PROBLEM_ID>`). Do NOT accumulate inline JSON in pipeline context — Phase 3c reads from disk.

### Phase 3b — Score responses via Codex (skip when CODEX_AVAILABLE=false — requires `bridge@borda-ai-rig`)

For each problem, call the read-only Codex bridge skill. Run **sequentially** (not parallel — scoring writes shared result files).

`bridge:review` is read-only — it returns its answer in the call envelope, it never writes the output file itself despite the "Output file:" instruction below. Also note: `bridge_call.py` enforces a hard ~500-char cap on the returned `verdict` field — a scoring response that exceeds it fails validation (`ValueError`) rather than returning truncated JSON; if the call fails, retry once, then fall back to Claude's score alone for that problem (`scorer_mode: "single"`, per Phase 3c below) rather than looping.

Skill(skill="bridge:review", args="Score a calibration response against ground truth.

```text
Problem ID: \<PROBLEM_ID>
Ground truth (JSON array): \<GROUND_TRUTH_JSON>
```

Read response from: .reports/calibrate/<TIMESTAMP>/<TARGET>/response-\<PROBLEM_ID>.md \[If AB_MODE is true: also read .reports/calibrate/<TIMESTAMP>/<TARGET>/response-\<PROBLEM_ID>-general.md\]

For each ground truth issue: mark true if response identified same issue type at same location (exact or semantically equivalent). Count false positives: reported issues with no ground truth match. Extract confidence from ## Confidence block (use 0.5 if absent).

For out-of-scope problems (ground_truth is []): set recall=null, all reported findings are FPs, set severity_accuracy=null, format_score=null.

```text
Severity accuracy: for found issues, check severity match (allow +-1 tier; tiers: critical>high>medium>low).
Format score: for found issues, check for all three of: location reference, severity label, fix suggestion.
```

Compute: recall=found/total (null if total=0), precision=found/(found+fp+1e-9), f1=2*r*p/(r+p+1e-9).

Write ONLY this JSON (no prose, no markdown fences, no trailing commas) to file below: {"problem_id":"\<PROBLEM_ID>","found":[true/false,...],"false_positives":N,"confidence":0.N,"recall":0.N,"precision":0.N,"f1":0.N,"severity_accuracy":0.N,"format_score":0.N,"scorer":"codex"} [If AB_MODE is true, append before closing }: ,"recall_general":0.N,"confidence_general":0.N,"precision_general":0.N,"f1_general":0.N,"severity_accuracy_general":0.N,"format_score_general":0.N]

Return the JSON object in your response — do not attempt to write it to a file yourself.")

Substitute `<PROBLEM_ID>` and `<GROUND_TRUTH_JSON>` per problem. Capture JSON from call's return value, write it to `.reports/calibrate/<TIMESTAMP>/<TARGET>/score-<PROBLEM_ID>-codex.json` yourself. Returned payload missing or unparsable after one retry: set `scorer_mode: "single"` for that problem — Phase 3c uses Claude's score only.

### Phase 3c — Consensus merge

For each problem, read `score-<PROBLEM_ID>-claude.json` (Phase 3a output) and `score-<PROBLEM_ID>-codex.json` (Phase 3b output) from `<RUN_DIR>` and merge:

**When both scores present**:

- `found[]` — per-issue boolean: both agree → use; disagree → Claude's value (51% tiebreak)
- `false_positives` — Claude's count wins on disagreement
- `recall`, `precision`, `f1` — recomputed from consensus `found[]` and consensus `false_positives`
- `severity_accuracy`:
  - Both agree → use
  - Disagree within 1 tier → use harsher (more conservative) severity
  - Disagree by >1 tier → mark that issue as `severity_disputed`; exclude from `severity_accuracy` aggregate
- `format_score` — weighted average: 0.51 × Claude + 0.49 × Codex
- `confidence` — unchanged (from target agent's response, not scorers)
- `scorer_agreement` = (issues where both scorers agreed on found/not-found) / total_issues; N/A for scope problems
- `scorer_mode` = `"dual"`

**When only Claude score present** (Codex unavailable or failed for this problem):

Use Claude score directly; `scorer_agreement` = null; `scorer_mode` = `"single"`

**A/B mode**: apply same consensus logic independently to general-purpose baseline scores.

Write all merged scores to `.reports/calibrate/<TIMESTAMP>/<TARGET>/scores.json` as JSON array. Each entry includes `"source"` (from problems.json: `"claude"`/`"codex"`), `"scorer_mode"`, `"scorer_agreement"`, and `"severity_disputed_count"` (count of `severity_disputed` issues for this problem).

### Phase 4 — Aggregate, write report and result

Compute aggregates (exclude out-of-scope problem from recall/F1/severity/format averages; include in FP count):

- `mean_recall` = mean of `recall` values for in-scope, **non-extreme** problems only (trivial through high included; extreme excluded — reported separately as `extreme_recall`)
- `extreme_recall` = mean of `recall` values for extreme problems only (null if none present); partial performance (0.4–0.7) informative, not alarming
- `mean_confidence` = mean of all `confidence` values (extreme problems included — confidence calibration applies across all tiers)
- `calibration_bias` = `mean_confidence − mean_recall` (uses non-extreme `mean_recall`; extreme problems don't affect verdict)
- `mean_f1` = mean of `f1` values for in-scope, non-extreme problems only
- `scope_fp` = false_positives from out-of-scope problem (0 = correct discipline, >0 = scope failure)
- `mean_severity_accuracy` = mean of `severity_accuracy` for in-scope problems with found_count > 0 (exclude `severity_disputed` issues from numerator and denominator; extreme problems included if found_count > 0)
- `mean_format_score` = mean of `format_score` for in-scope problems with found_count > 0
- `token_ratio` = mean(target_chars) / mean(general_chars) across all problems — if AB_MODE, else omit
- Recall by difficulty: `recall_trivial`, `recall_low`, `recall_medium`, `recall_high`, `recall_extreme` (omit if 0 problems at that level)

**Additional aggregates (populate when applicable; use null when not)**:

- `mean_scorer_agreement` = mean `scorer_agreement` across dual-scored problems (null if all single-scored)
- `severity_disputed_count` = total issues flagged `severity_disputed` across all problems
- `codex_problems_pct` = fraction of in-scope problems with `source: "codex"` (0.0 if claude-only)
- `recall_claude_problems` = mean recall on in-scope problems where `source: "claude"` (null if none)
- `recall_codex_problems` = mean recall on in-scope problems where `source: "codex"` (null if none)
- `generator_recall_delta` = `recall_claude_problems − recall_codex_problems` (null if either is null)
- `source_mode` = `"dual"` if CODEX_AVAILABLE and generation succeeded, else `"claude-only"`
- `scoring` = `"dual"` if any problem dual-scored, else `"single"`
- `codex_generation_failed` = true if Codex generation attempted but failed, else false

Verdict:

- `|bias| < 0.10` → `calibrated`
- `0.10 ≤ |bias| ≤ 0.15` → `borderline`
- `bias > 0.15` → `overconfident`
- `bias < −0.15` → `underconfident`

Write tool refuses any file whose exact basename is `report.md` from a subagent context ("Subagents should return findings as text, not write report files") — use `benchmark-report.md` instead. Write full report to `.reports/calibrate/<TIMESTAMP>/<TARGET>/benchmark-report.md` using this structure:

```markdown
## Benchmark Report — <TARGET> — <date>
Mode: <MODE> | Problems: <N> (in-scope) + 1 (out-of-scope) | Total known issues: M
Source: dual (claude+codex) | Scorer: dual | Scorer agreement: X.XX [consistent ≥0.85 / moderate 0.70–0.85 / divergent ⚠ <0.70]
[OR: Source: claude-only | Scorer: single — Codex unavailable or generation failed]

### Per-Problem Results
| Problem ID | Source | Difficulty | Recall | Precision | SevAcc | Fmt  | Confidence | Cal. Δ | Agreement |
| ...
| <scope-id> | claude | scope      | —      | —         | —      | —    | —          | scope_fp=N | — |

*Recall: issues found / total. Precision: found / (found + FP). Source: which model generated the problem (claude/codex). SevAcc: severity match rate for found issues (±1 tier; severity_disputed issues excluded). Fmt: fraction of found issues with location + severity + fix. Cal. Δ: confidence − recall (negative = conservative). Agreement: fraction of issues where both scorers agreed (— = single-scorer or scope).*

### Aggregate
| Metric             | Value | Status |
| ...
| Severity accuracy  | X.XX  | high ≥0.80 / moderate 0.60–0.80 / low <0.60 |
| Format score       | X.XX  | high ≥0.80 / moderate 0.60–0.80 / low <0.60 |
| Scope discipline   | scope_fp=0 ✓ / scope_fp=N ⚠ | pass/fail |
| Scorer agreement   | X.XX  | consistent ≥0.85 ✓ / moderate 0.70–0.85 ~ / divergent ⚠ <0.70 |
| Disputed severities | N    | excluded from SevAcc (scorers disagreed >1 tier) |

Recall by difficulty: trivial=X.XX | low=X.XX | medium=X.XX | high=X.XX (omit levels with 0 problems)
Extreme recall: X.XX (extreme problems excluded from mean_recall and verdict — partial performance 0.4–0.7 is expected)

### Recall by Problem Source (dual source mode only)
| Source | Problems | Mean Recall |
|--------|----------|-------------|
| claude | N | X.XX |
| codex  | N | X.XX |
| delta  | — | ±X.XX (+ = harder codex problems; − = codex problems easier) |

### A/B Comparison — specialized vs. general-purpose (AB mode only)
| Metric            | Specialized | General | Delta  | Verdict   |
|-------------------|-------------|---------|--------|-----------|
| Mean Recall       | X.XX        | X.XX    | ±X.XX  | significant ✓ / marginal ~ / none ⚠ |
| Mean F1           | X.XX        | X.XX    | ±X.XX  |           |
| Severity accuracy | X.XX        | X.XX    | ±X.XX  | better ✓ / similar ~ / worse ⚠ |
| Format score      | X.XX        | X.XX    | ±X.XX  | better ✓ / similar ~ / worse ⚠ |
| Token ratio       | X.XX        | 1.00    | ±X.XX  | concise ✓ / verbose ⚠ |
| Scope FP          | N           | N       | —      | pass/fail |

*ΔRecall: specialist recall − general recall. SevAcc: severity match rate (±1 tier). Fmt: actionability score. Token ratio: specialist chars / general chars (below 1.0 = more focused). Scope FP: findings on out-of-scope input (0 = correct discipline).*
Verdict: `significant` (delta_recall or delta_f1 > 0.10) / `marginal` (0.05–0.10) / `none` (<0.05)

### Systematic Gaps (missed in ≥2 problems)
...

### Improvement Signals
...
```

Write single-line JSONL result to `.reports/calibrate/<TIMESTAMP>/<TARGET>/result.jsonl`: (one line per pipeline run — orchestrating skill concatenates these across runs into `.notes/logs/calibrations.jsonl`)

`{"ts":"<TIMESTAMP>","target":"<TARGET>","mode":"<MODE>","mean_recall":0.N,"extreme_recall":0.N_or_null,"mean_confidence":0.N,"calibration_bias":0.N,"mean_f1":0.N,"severity_accuracy":0.N,"format_score":0.N,"problems":<N>,"scope_fp":N,"verdict":"...","gaps":["..."],"source_mode":"dual|claude-only","scoring":"dual|single","scorer_agreement":0.N_or_null,"recall_trivial":0.N_or_null,"recall_low":0.N_or_null,"recall_medium":0.N_or_null,"recall_high":0.N_or_null,"recall_extreme":0.N_or_null,"recall_claude_problems":0.N_or_null,"recall_codex_problems":0.N_or_null,"generator_recall_delta":0.N_or_null,"severity_disputed_count":N,"codex_generation_failed":false}`

**If AB_MODE is true**, append these fields to same JSON line: `"delta_recall":0.N,"delta_f1":0.N,"delta_severity_accuracy":0.N,"delta_format_score":0.N,"token_ratio":0.N,"scope_fp_general":N,"ab_verdict":"significant|marginal|none"`

### Phase 5 — Propose instruction edits

Determine target file path for curator proposals:

When `LOCAL_MODE=true` and `TARGET_FILE` set: use `TARGET_FILE` directly.

Otherwise resolve (first match wins):

- Agent: `.claude/agents/<NAME>.md` → `~/.claude/plugins/cache/borda-ai-rig/<PLUGIN>/*/agents/<NAME>.md` (latest mtime) → `plugins/<PLUGIN>/agents/<NAME>.md`
- Skill: `.claude/skills/<NAME>/SKILL.md` → `~/.claude/plugins/cache/borda-ai-rig/<PLUGIN>/*/skills/<NAME>/SKILL.md` (latest mtime) → `plugins/<PLUGIN>/skills/<NAME>/SKILL.md`

(`<PLUGIN>` and `<NAME>` parsed from `<TARGET>` as in pre-flight: strip `/`, split on `:`)

Spawn **foundry:curator** subagent using **Agent tool** — never via Bash or CLI. Pass only **file path** and **report path** — do NOT paste file contents into prompt; foundry:curator reads files itself:

> Reviewing calibration benchmark result and proposing instruction improvements.
>
> **Files to read** (use Read tool on each):
>
> 1. Target file: `<AGENT_OR_SKILL_FILE_PATH>`
> 2. Benchmark report: `.reports/calibrate/<TIMESTAMP>/<TARGET>/benchmark-report.md` — focus on **Systematic Gaps** and **Improvement Signals** sections
>
> Propose specific, minimal instruction edits that directly address each systematic gap (issues missed in ≥2/N problems) and each false-positive pattern. Conservative: one targeted change per gap. Don't refactor sections unrelated to findings.
>
> If no actionable systematic gaps (target calibrated with recall ≥ 0.70 and no repeated misses), write: `## Proposed Changes — <TARGET>\n\nNo changes needed — target is calibrated.`
>
> Otherwise format each change as:
>
> ```
> ## Proposed Changes — <TARGET>
>
> ### Change 1: <gap name>
> **File**: `<file path>`
> **Section**: `<antipatterns-to-flag>` / `<workflow>` / `<notes>` / etc.
> **Current**: [exact verbatim text to replace; or "none" if inserting new content]
> **Proposed**: [exact replacement text]
> **Rationale**: one sentence — why this closes the gap
> ```

Write foundry:curator response verbatim to `.reports/calibrate/<TIMESTAMP>/<TARGET>/proposal.md`. Ask foundry:curator to end proposed changes with `## Confidence` block per CLAUDE.md output standards.

### Return value

**CRITICAL — context discipline**: return ONLY compact JSON line below. No prose, no report summary, no table, no Confidence block, no additional output. Every extra byte accumulates in orchestrator context, can cause synthesis hang. All report content already on disk from Phase 4.

Return **only** this compact JSON (no prose before or after):

`{"target":"<TARGET>","mean_recall":0.N,"extreme_recall":0.N_or_null,"mean_confidence":0.N,"calibration_bias":0.N,"mean_f1":0.N,"severity_accuracy":0.N,"format_score":0.N,"scope_fp":N,"verdict":"calibrated|borderline|overconfident|underconfident","gaps":["..."],"proposed_changes":N,"source_mode":"dual|claude-only","scoring":"dual|single","scorer_agreement":0.N_or_null,"generator_recall_delta":0.N_or_null}`

If AB_MODE is true, also include: `"delta_recall":0.N,"delta_f1":0.N,"delta_severity_accuracy":0.N,"delta_format_score":0.N,"token_ratio":0.N,"scope_fp_general":N,"ab_verdict":"significant|marginal|none"`

You routing calibration pipeline runner. Complete all phases in sequence.

<!-- Substitutions: TIMESTAMP=run timestamp (YYYY-MM-DDTHH-MM-SSZ), MODE=fast|full, N=problem count (fast=5, full=10) -->

```text
Mode: `<MODE>`
Run dir: `.reports/calibrate/<TIMESTAMP>/routing/`
```

<!-- All paths relative to project root. Pipeline runner must have project root as working dir. -->

### Phase 1 — Collect agent descriptions

Enumerate roster file set. Source tree (`plugins/*/agents/*.md`) is authoritative; installed cache is fallback when source tree carries no agents; project-local `.claude/agents/*.md` is an override tier empty in most setups — `/foundry:setup` never creates that directory, purges stale entries from it, so it must never be the sole source:

```bash
RUN_DIR=".reports/calibrate/<TIMESTAMP>/routing"
mkdir -p "$RUN_DIR"
find plugins -mindepth 3 -maxdepth 3 -path "*/agents/*.md" 2>/dev/null | sort > "$RUN_DIR/roster-files.txt"
if [ ! -s "$RUN_DIR/roster-files.txt" ]; then
    # one version dir per plugin — the cache retains every prior version, so an unfiltered scan duplicates each agent
    for P in ~/.claude/plugins/cache/borda-ai-rig/*/; do
        V=$(find "$P" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | grep -v '\.orphaned_at' | sort -Vr | head -1)
        [ -n "$V" ] && find "$V/agents" -maxdepth 1 -name "*.md" 2>/dev/null
    done | sort > "$RUN_DIR/roster-files.txt"
fi
find .claude/agents -maxdepth 1 -name "*.md" 2>/dev/null >> "$RUN_DIR/roster-files.txt"
grep -c . "$RUN_DIR/roster-files.txt"
```

**Empty-roster hard stop** — zero lines: do NOT generate problems, do NOT score, do NOT reconstruct a roster from memory. A fabricated roster yields a measured-looking `routing_accuracy` for a run that measured nothing, and that number lands in `calibrations.jsonl` history. Write this line to `.reports/calibrate/<TIMESTAMP>/routing/result.jsonl`, return it as compact JSON, skip Phases 2–4:

`{"ts":"<TIMESTAMP>","target":"routing","mode":"<MODE>","routing_accuracy":null,"confusion_rate":null,"hard_accuracy":null,"auto_invoke_accuracy":null,"problems":0,"verdict":"incomplete","confused_pairs":[],"gaps":["no agent files found under plugins/*/agents/, the installed plugin cache, or .claude/agents/ — routing accuracy not measured"]}`

Read each file listed in `roster-files.txt`. Per file, extract `name:` and `description:` from YAML frontmatter (between `---` delimiters).

Roster entries must carry the **dispatch name**, not the bare frontmatter `name:` — that's what `expected_agent` and every selector answer are matched against: for `plugins/<dir>/agents/<n>.md` and cache `<plugin>/<ver>/agents/<n>.md`, use `<plugin>:<n>` (strip any leading `cc_` from `<dir>`); for `.claude/agents/<n>.md`, use bare `<n>`. Same dispatch name from more than one tier: keep `.claude/` entry, else source-tree entry.

Build roster string, one line per agent:

```text
<dispatch-name>: <description>
```

Write roster to `.reports/calibrate/<TIMESTAMP>/routing/roster.txt`.

### Phase 2 — Generate routing problems

Generate `<N>` synthetic task prompts across all agents. Per problem, produce JSON with these fields:

- `problem_id`: kebab-slug string
- `task_prompt`: realistic user request to orchestrator (no hint at expected agent)
- `expected_agent`: correct `subagent_type` from roster (or `"general-purpose"` if no specialist needed)
- `difficulty`: `"easy"` (single-domain, obvious match), `"medium"` (2 domains, one primary), `"hard"` (ambiguous, requires NOT-for clauses or fine distinctions)
- `confusion_pair`: most likely wrong agent for medium/hard; `null` for easy
- `auto_invoke_test`: `true` if problem tests TRIGGER or SKIP-guard coverage; `false` otherwise

Rules:

- Cover every agent ≥1 in `expected_agent` (distribute evenly given N)
- Include ≥2 hard problems testing high-overlap pairs: e.g., sw-engineer vs qa-specialist, doc-scribe vs oss:shepherd, linting-expert vs sw-engineer, solution-architect vs sw-engineer, web-explorer vs sw-engineer (look up docs to implement vs implement directly), challenger vs sw-engineer (critique plan vs implement), oss:analyse vs oss:review (analyze thread vs code review)
- Include exactly 1 `expected_agent: "general-purpose"` problem (general question, no specialist)
- Difficulty distribution: ~40% easy, ~40% medium, ~20% hard (adjust to cover all agents)
- **Auto-invocation coverage**: include ≥3 problems where `task_prompt` uses exact TRIGGER phrasing for an agent with a TRIGGER block (e.g. "what does the requests docs say about retries", "write tests for the auth module", "add docstrings to utils.py") — these are easy/medium; the TRIGGER phrase is the signal
- **SKIP-guard coverage**: include ≥2 problems where `task_prompt` superficially resembles a TRIGGER but a SKIP guard applies — `expected_agent` must be `"general-purpose"` or a different specialist, NOT the TRIGGER agent; add field `"skip_guard_test": true` to these problems
- Add boolean field `"auto_invoke_test": true` to problems covering TRIGGER/SKIP scenarios
- Return valid JSON array only (no prose)

Write JSON array to `.reports/calibrate/<TIMESTAMP>/routing/problems.json`.

### Phase 3 — Run routing selection (parallel)

Read roster from `.reports/calibrate/<TIMESTAMP>/routing/roster.txt`.

Per problem in `problems.json`, spawn `general-purpose` selector subagent. Issue ALL spawns in **single response** — no waiting between spawns.

Each selector gets this prompt (substitute `<ROSTER>`, `<TASK_PROMPT>`, `<PROBLEM_ID>`, `<RUN_DIR>`):

> Select specialized agent for task. Available agents:
>
> ```
> <ROSTER>
> ```
>
> Task: `<TASK_PROMPT>`
>
> Select one agent. If no specialist fits, select `general-purpose`.
>
> Write response to `<RUN_DIR>/selection-<PROBLEM_ID>.md` via Write tool. File must contain ONLY valid JSON (no prose):
>
> `{"selected":"<agent-name>","reasoning":"<one sentence>"}`
>
> Then end reply with exactly one line: `Wrote: <PROBLEM_ID>`

**Context discipline**: subagents write to disk, return single-line ack. Pipeline agent must NOT accumulate full analyses — scorers read from disk in Phase 3. `Wrote: <PROBLEM_ID>` per agent correct.

**Completion handling** — spawns are blocking `Agent()` calls, so no poll loop is possible (`_FOUNDRY_SHARED/agent-spawn-protocol.md` §Synchronous spawns). Each subagent returns: check for `selection-<PROBLEM_ID>.md`; missing: mark that problem `{"selected":null,"timed_out":true}` with ⏱ in report.

### Phase 4 — Score

<!-- Design note: N=5/10, selection files tiny (~100 bytes), under 2K inline threshold. Inline reading intentional. If N>~20, refactor Phase 4 to use consolidator subagent. -->

Per problem, read `selection-<problem_id>.md` from `.reports/calibrate/<TIMESTAMP>/routing/`. Parse JSON, extract `selected` and `reasoning`. Compare vs `expected_agent` from `problems.json`:

- `selected` == `expected_agent` → `correct: true`, `error_type: null`
- `selected` == `confusion_pair` → `correct: false`, `error_type: "confusion"`
- Other mismatch → `correct: false`, `error_type: "wrong"`
- `timed_out: true` → `correct: false`, `error_type: "timeout"`

Compute aggregates:

- `routing_accuracy` = correct_count / total_count
- `confusion_rate` = confusion_error_count / total_count
- `hard_accuracy` = correct hard / total hard (omit if no hard problems)
- `auto_invoke_accuracy` = correct on `auto_invoke_test: true` problems / total `auto_invoke_test: true` problems (omit if no such problems)
- Confusion list: per incorrect selection, record `(expected → selected, task_prompt, reasoning)`

Verdict:

- `routing_accuracy ≥ 0.90` AND `hard_accuracy ≥ 0.80` → `calibrated`
- `routing_accuracy ≥ 0.80` but below threshold OR `hard_accuracy < 0.80` → `borderline`
- `routing_accuracy < 0.80` → `needs-improvement`

The Write tool refuses any file whose exact basename is `report.md` from a subagent context — use `benchmark-report.md` instead. Write full report to `.reports/calibrate/<TIMESTAMP>/routing/benchmark-report.md`:

```markdown
## Routing Benchmark — <date> — <MODE>

### Per-Problem Results
| Problem ID | Difficulty | Expected | Selected | Correct |
|------------|------------|----------|----------|---------|
| ...

### Aggregate
| Metric           | Value     | Status |
|------------------|-----------|--------|
| Routing accuracy | X/N (XX%) | ≥90% ✓ / 80–90% ~ / <80% ⚠ |
| Hard accuracy    | X/N (XX%) | ≥80% ✓ / <80% ⚠ |
| Confusion errors | N         | 0 ✓ / >0 list pairs |
| Auto-invoke accuracy | X/N (XX%) | ≥90% ✓ / <90% ⚠ (auto_invoke_test problems only) |

### Confused Pairs
| Task Prompt | Expected → Selected | Reasoning |
|-------------|---------------------|-----------|
| ...

(omit this section if no confusion errors)

### Proposals
For each confused pair: suggest specific wording improvements to the relevant agent
descriptions that would disambiguate the routing decision. Reference the NOT-for clause
pattern when applicable — adding "NOT for X" to one agent in the pair is often the
minimal effective fix.
```

Write result JSONL to `.reports/calibrate/<TIMESTAMP>/routing/result.jsonl`:

`{"ts":"<TIMESTAMP>","target":"routing","mode":"<MODE>","routing_accuracy":0.N,"confusion_rate":0.N,"hard_accuracy":0.N,"auto_invoke_accuracy":0.N,"problems":<N>,"verdict":"calibrated|borderline|needs-improvement|incomplete","confused_pairs":["expected→selected",...]}`

### Return value

Return **only** compact JSON (no prose):

`{"target":"routing","routing_accuracy":0.N,"confusion_rate":0.N,"hard_accuracy":0.N,"auto_invoke_accuracy":0.N,"problems":<N>,"verdict":"calibrated|borderline|needs-improvement|incomplete","confused_pairs":["expected→selected",...]}`

Rules calibration pipeline runner for rule file `<RULE_BASENAME>`. Complete all phases in sequence.

<!-- Substitutions before spawning: RULE_BASENAME=filename (e.g. commit-and-git.md), RULE_CONTENT=full rule file text verbatim, TIMESTAMP=YYYY-MM-DDTHH-MM-SSZ, MODE=fast|full, N=_tasks per directive (fast=3, full=5), IS_PATH_SCOPED=true|false (true if rule has a non-empty paths: frontmatter field) -->

<!-- Derive RULE_DIR before spawning: RULE_DIR="${RULE_BASENAME%.md}" (strips .md extension — used as the run directory name to avoid permission-matcher conflicts with .md-suffixed paths) -->

Mode: `<MODE>` Run dir: `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/`

```bash
mkdir -p .reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/
```

**Rule under test** (loaded as context for all Phase 2 agents):

```text
<RULE_CONTENT>
```

### Phase 1 — Extract directives and generate problems

**Step 1a — Extract directives**: identify 2–3 key directives from rule content above. Key directive = specific, action-prescribing sentence in imperative mood with concrete, observable required behaviour (e.g. `"Never use git add -A"`, `"Always append a Legend block after any results table"`). Skip section headers, explanatory prose, context-setting sentences.

Write to `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/directives.json`:

```json
[
  {
    "directive_id": "dir-1",
    "text": "<directive text verbatim>",
    "expected_behavior": "<observable property of a compliant response — what must be present or absent>"
  }
]
```

**Step 1b — Generate adherence problems** (`<N>` per directive): for each directive, create `<N>` tasks where correct response MUST apply directive, ignoring it produces detectably wrong result. Cover realistic user requests; vary surface form across `<N>` problems per directive.

Problem format:

```json
{
  "problem_id": "<rule-basename-without-extension>-dir-1-1",
  "type": "adherence",
  "directive_id": "dir-1",
  "task_prompt": "<realistic user request>",
  "context": "<additional context the agent needs, or empty string>",
  "expected_behavior": "<what a compliant response must show or avoid>"
}
```

**Step 1c — Generate trigger problems** (only if `<IS_PATH_SCOPED>` is `true`): for each `paths:` glob pattern in rule frontmatter, generate 2 problems:

- One with **matching** file context (`expected_trigger: true`) — rule should be active
- One with **non-matching** file context (`expected_trigger: false`) — rule should stay silent

```json
{
  "problem_id": "<rule-basename-without-extension>-trigger-1",
  "type": "trigger",
  "file_context": "<filename with extension, e.g. main.py>",
  "expected_trigger": true,
  "task_prompt": "Working on `<file_context>`: <generic task that the rule could apply to>"
}
```

Write all problems to `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/problems.json` as JSON array.

### Phase 2 — Run tasks (parallel)

Spawn one `general-purpose` subagent per problem. **Issue ALL spawns in single response — no waiting between spawns.**

Each subagent receives this prompt (substitute `<PROBLEM_ID>`, `<TASK_PROMPT>`, `<CONTEXT>`, `<RUN_DIR>` before spawning):

<!-- BEGIN SPAWN PROMPT -->

General-purpose coding assistant. Following rule is in effect — apply it in your response:

```text
<RULE_CONTENT>
```

Task: `<TASK_PROMPT>`

`<CONTEXT>`

Write complete response to `<RUN_DIR>/response-<PROBLEM_ID>.md` using Write tool. End reply with exactly one line: `Wrote: <PROBLEM_ID>`

<!-- END SPAWN PROMPT -->

**Context discipline**: subagents write to disk, return single-line acknowledgment. Pipeline agent must NOT accumulate their full analyses in context — scorers read from disk in Phase 3. Receiving only `Wrote: <PROBLEM_ID>` per agent is correct, expected.

**Completion handling** — spawns are blocking `Agent()` calls, so no poll loop is possible (`_FOUNDRY_SHARED/agent-spawn-protocol.md` §Synchronous spawns). Each subagent returns: check for `response-<PROBLEM_ID>.md`; missing: mark that problem `{"timed_out": true}` in scores.json, proceed.

### Phase 3 — Score (parallel scorer subagents)

Spawn one `general-purpose` scorer per problem. **Issue ALL spawns in single response.**

Each scorer receives this prompt (substitute `<PROBLEM_ID>`, `<PROBLEM_TYPE>`, `<DIRECTIVE_TEXT>`, `<EXPECTED_BEHAVIOR>`, `<EXPECTED_TRIGGER>`, `<RUN_DIR>` before spawning):

<!-- BEGIN SPAWN PROMPT -->

Scoring rule compliance test. Read response from `<RUN_DIR>/response-<PROBLEM_ID>.md` using Read tool.

**Problem type**: `<PROBLEM_TYPE>`

**[For adherence problems]**

Directive under test: `<DIRECTIVE_TEXT>` Expected behavior: `<EXPECTED_BEHAVIOR>`

Score on two dimensions:

1. **Directive outcome** — assign exactly one of:

   - `correct` — directive applied; expected behavior present in response
   - `missed` — response otherwise reasonable but directive ignored
   - `misapplied` — wrong directive applied, or directive applied where it should not be

2. **Outcome correctness** — beyond whether directive was mentioned or acknowledged, check whether response's *actual content* (commands used, flags omitted, files listed, patterns followed) satisfies directive's intent:

   - `true` — behavioral output correct, not just stated intent
   - `false` — agent says it will follow rule but concrete output violates it (e.g. says "I'll stage specific files" then writes `git add -A`)

Return ONLY this JSON (no prose): `{"problem_id":"<PROBLEM_ID>","type":"adherence","outcome":"correct|missed|misapplied","outcome_correct":true|false,"reasoning":"<one sentence>"}`

**[For trigger problems]**

Expected trigger: `<EXPECTED_TRIGGER>`

Determine whether rule's directives visible in response:

- `triggered: true` — rule clearly active (response applies or references rule's constraints)
- `triggered: false` — response shows no sign of rule being active

Return ONLY this JSON (no prose): `{"problem_id":"<PROBLEM_ID>","type":"trigger","triggered":true|false,"expected_trigger":<EXPECTED_TRIGGER>,"correct":true|false,"reasoning":"<one sentence>"}`

<!-- END SPAWN PROMPT -->

Collect all scorer compact JSONs. Write to `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/scores.json` as JSON array.

### Phase 4 — Aggregate and write report

Compute aggregates from `scores.json`:

**Adherence** (type == "adherence" problems only):

- Per directive: `adherence_recall_dir_N` = correct_count / total tasks for that directive
- `mean_adherence_recall` = mean across all directives
- `outcome_correctness` = outcome_correct_count / correct_count (N/A if correct_count = 0)
- `misapplied_rate` = misapplied_count / total adherence tasks

**Trigger fidelity** (type == "trigger" problems, skip if none):

- `trigger_recall` = (triggered=true AND expected=true) / total expected=true problems
- `trigger_precision` = (triggered=false AND expected=false) / total expected=false problems

**Verdict**:

- `mean_adherence_recall ≥ 0.8` AND `outcome_correctness ≥ 0.8` → `calibrated`
- `mean_adherence_recall ≥ 0.8` AND `outcome_correctness < 0.8` → `outcome-gap` (directive followed in word, not in effect)
- `mean_adherence_recall < 0.8` → `under-enforced`

The Write tool refuses any file whose exact basename is `report.md` from a subagent context — use `benchmark-report.md` instead. Write full report to `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/benchmark-report.md`:

```markdown
## Rules Benchmark — <RULE_BASENAME> — <date>
Mode: <MODE> | Directives: D | Adherence tasks: D×N | Trigger tests: T (or 0 if global rule)

### Per-Directive Results
| Directive (truncated) | Adherence Recall | Outcome Correct | Misapplied | Status    |
|-----------------------|-----------------|----------------|------------|-----------|
| Never git add -A      | 0.89            | 0.90           | 0          | ✓         |
| Always append legend  | 0.67 ⚠          | —              | 1          | ⚠         |

*Legend: Adherence Recall — directive followed / total tasks (0–1, higher is better, ≥0.8 target). Outcome Correct — fraction of "correct" scores where actual output also matched expected behavior, not just stated intent (0–1, higher is better, ≥0.8 target). Misapplied — tasks where wrong directive was applied (lower is better; ≥1 suggests ambiguous wording). Status: ✓ calibrated | ⚠ under-enforced.*

### Trigger Fidelity (path-scoped rules only — omit section if global)
| Metric            | Value | Status               |
|-------------------|-------|----------------------|
| Trigger recall    | 0.XX  | ≥0.95 ✓ / <0.95 ⚠   |
| Trigger precision | 0.XX  | ≥0.95 ✓ / <0.95 ⚠   |

*Legend: Trigger recall — rule fired on all matching-path contexts (0–1, higher is better, ≥0.95 target). Trigger precision — rule stayed silent on non-matching contexts (0–1, higher is better, ≥0.95 target).*

### Aggregate
| Metric               | Value | Status                |
|----------------------|-------|-----------------------|
| Adherence recall     | 0.XX  | ≥0.80 ✓ / <0.80 ⚠    |
| Outcome correctness  | 0.XX  | ≥0.80 ✓ / <0.80 ⚠    |
| Misapplied rate      | 0.XX  | 0 ✓ / >0 ⚠            |
| Verdict              |       | calibrated / outcome-gap / under-enforced |

*Legend: Adherence recall — mean fraction of tasks where directive was applied across all directives (0–1, higher is better, ≥0.80 threshold). Outcome correctness — mean fraction of applied directives that also produced the correct behavioral output (0–1, higher is better, ≥0.80 target). Misapplied rate — fraction of tasks where wrong directive applied (0–1, lower is better, 0 = ideal).*

### Systematic Gaps
<missed directives or outcome failures recurring in ≥2 problems>

### Wording Improvement Opportunities
<for under-enforced directives: original text and suggested rewording>
```

Write result JSONL to `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/result.jsonl`:

`{"ts":"<TIMESTAMP>","target":"rules/<RULE_BASENAME>","mode":"<MODE>","mean_adherence_recall":0.N,"outcome_correctness":0.N,"misapplied_rate":0.N,"trigger_recall":0.N,"trigger_precision":0.N,"problems":<N>,"verdict":"calibrated|outcome-gap|under-enforced","gaps":["..."]}`

Use `null` for `trigger_recall`/`trigger_precision` when `IS_PATH_SCOPED` is `false`.

### Phase 5 — Propose wording improvements

Spawn **foundry:curator** subagent using Agent tool. Pass only file paths — do NOT paste file contents into prompt:

> Read these files using Read tool:
>
> 1. Rule file: `.claude/rules/<RULE_BASENAME>`
> 2. Benchmark report: `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/benchmark-report.md` — focus on Systematic Gaps and Wording Improvement Opportunities sections
>
> For each under-enforced or outcome-gap directive, propose minimal rewording making directive more specific, action-prescribing, unambiguous. Keep surrounding context unchanged. If all directives calibrated, write: `## Proposed Changes — <RULE_BASENAME>\n\nNo changes needed — all directives calibrated.`
>
> Format each change as:
>
> ```
> ## Proposed Changes — <RULE_BASENAME>
>
> ### Change N: <directive summary>
> **File**: `.claude/rules/<RULE_BASENAME>`
> **Current**: [exact verbatim text]
> **Proposed**: [exact replacement]
> **Rationale**: one sentence — what failure mode this prevents
> ```
>
> Write to `.reports/calibrate/<TIMESTAMP>/rules/<RULE_DIR>/proposal.md`. End with `## Confidence` block per CLAUDE.md output standards.

### Return value

Return **only** this compact JSON (no prose before or after):

`{"rule":"<RULE_BASENAME>","mean_adherence_recall":0.N,"outcome_correctness":0.N,"misapplied_rate":0.N,"trigger_recall":0.N,"trigger_precision":0.N,"problems":<N>,"verdict":"calibrated|outcome-gap|under-enforced","gaps":["..."],"proposed_changes":N}`

Use `null` for `trigger_recall`/`trigger_precision` when `IS_PATH_SCOPED` is `false`.