# Output Section Formats

## Coherence Analysis — Naive Mode

```markdown
## Coherence Analysis

**Mode:** Naive (structural validation only)
**Coherence category:** Not scored (weight redistributed)

### Structural Findings

| # | Type | Detail | Line |
|---|------|--------|------|
| {per-issue rows} |

**Structural Issues:** {count}
```

## Coherence Analysis — Naive Mode: Reference Consistency (split-body)

Only rendered when `references/` directory exists alongside SKILL.md.

```markdown
### Reference Consistency (split-body)

| # | Reference File | Export | Issue | SKILL.md Line | Reference Line |
|---|---------------|--------|-------|---------------|---------------|
| {per-mismatch rows} |

**Exports Cross-Checked:** {count}
**Mismatches Found:** {count}
```

## Coherence Analysis — Contextual Mode

```markdown
## Coherence Analysis

**Mode:** Contextual (full reference validation)
**References Found:** {count}
**References Valid:** {count}
**Broken References:** {count}

### Reference Validation

| Reference | Type | Line | Target Exists | Accurate | Issues |
|-----------|------|------|--------------|----------|--------|
| {per-reference rows} |

### Integration Pattern Completeness

| Pattern | Complete | Issue |
|---------|----------|-------|
| {per-pattern rows} |

### Coherence Score

- **Reference Validity:** {valid}/{total} ({percentage}%)
- **Integration Completeness:** {complete}/{total} ({percentage}%)
- **Combined Coherence:** {percentage}%
```

## Gap Report Section

```markdown
## Gap Report

**Total Gaps:** {N}
**Blocking (Critical + High):** {N}
**Non-blocking (Medium + Low + Info):** {N}

### Remediation Summary

| Severity | Count | Estimated Effort |
|----------|-------|-----------------|
| Critical | {N} | {description} |
| High | {N} | {description} |
| Medium | {N} | {description} |
| Low | {N} | {description} |
| Info | {N} | {description} |
| **Total** | **{N}** | |
```

## Gap Entry Format

```markdown
### GAP-{NNN}: {Brief title}

**Severity:** {Critical|High|Medium|Low|Info}
**Category:** {Coverage|Coherence|Structural}
**Source:** {file:line or section reference}

**Issue:** {Precise description of what is wrong or missing}

**Remediation:** {Exact action to fix this gap}
```

## Remediation Quality Rules

- **Good:** "Add documentation for `formatDate(date: Date, format?: string): string` exported from `src/utils.ts:42`. Include the optional `format` parameter with default value `'YYYY-MM-DD'`."
- **Bad:** "Document the missing function."
- **Good:** "Update signature in SKILL.md line 78 from `(date: Date) => string` to `(date: Date, format?: string) => string` to match source at `src/utils.ts:42`."
- **Bad:** "Fix the signature mismatch."

## Effort Estimation Guidelines

- Critical/High gaps: typically require reading source code and writing documentation
- Medium gaps: typically require adding type definitions or interface docs
- Low gaps: typically require adding examples or metadata
- Info: optional improvements, no action required

---
nextStepFile: 'external-validators.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
outputFormatsFile: '{outputFormatsPath}'
scoringRulesFile: '{scoringRulesPath}'
coherenceAggregationScript: 'scripts/aggregate-coherence.py'
migrationSectionRules: 'references/migration-section-rules.md'
scanSkillMdStructureProbeOrder:
  - '{project-root}/_bmad/skf/shared/scripts/skf-scan-skill-md-structure.py'
  - '{project-root}/src/shared/scripts/skf-scan-skill-md-structure.py'
---

<!-- Config: communicate in {communication_language}. -->

# Step 4: Coherence Check

## STEP GOAL:

Validate internal consistency of the skill documentation. In contextual mode (stack skills): verify that all cross-references in SKILL.md point to real files, types match their declarations, and integration patterns are complete. In naive mode (individual skills): perform basic structural validation only.

### 1. Check Test Mode

Read `testMode` from `{outputFile}` frontmatter.

**IF naive mode → Execute Naive Coherence (Section 2)**
**IF contextual mode → Execute Contextual Coherence (Sections 3-5)**

### 2. Naive Mode: Concrete Structural Validation

Perform the following explicit checks (no hand-waving — most use a single deterministic script; severity assignments are binding; do not relax them).

**Resolve `{scanSkillMdStructureHelper}`** from `{scanSkillMdStructureProbeOrder}`; first existing path wins. HALT if no candidate exists.

**2.0 Run the structural scan.** Invoke `{scanSkillMdStructureHelper}` twice and parse the JSON outputs. These results back §§2.1, 2.2, 2.3, and 2.6 — do not re-implement those checks with grep/sed/awk loops.

```bash
uv run {scanSkillMdStructureHelper} scan {skill-md} --required-sections
uv run {scanSkillMdStructureHelper} scan {skill-md}
```

The first call returns `{ description: {satisfied, matched_synonym, tried[]}, usage: {...}, api_surface: {...} }`. The second returns `{ unbalanced_fences, fence_count, bare_opening_fences[{line,text}], table_drift[{line,section,expected_cols,actual_cols,row}] }`. Hold both JSON blobs for the checks below.

**2.1 Required sections present.** Read the first JSON blob from §2.0. For each of the three families (`description`, `usage`, `api_surface`):

- `satisfied: true` → no finding.
- `satisfied: false` AND family is `description` AND the SKILL.md frontmatter has a non-empty `description` field → no finding (the frontmatter alternative satisfies the family per the original rule).
- Otherwise → **High severity** finding: `naive-coherence — missing required section: {family}` (the `tried[]` list from the JSON identifies which synonyms were checked: `Description`/`Overview`/`Purpose`/`Summary` for description; `Usage`/`Usage Patterns`/`Examples`/`How to use`/`Quickstart`/`Quick Start`/`Getting Started`/`Common Workflows`/`Adoption Steps` for usage; `API`/`API Surface`/`Exports`/`Key Exports`/`Public API`/`Interface`/`Reference`/`Key API Summary`/`Pattern Surface` for api_surface).

The script matches case-insensitively and tolerates `##`/`###` heading levels. SKF-template skills' headings are first-class synonyms baked into the script — the Deep/create-skill `## Quick Start`, `## Common Workflows`, and `## Key API Summary`, the quick-skill `## Usage Patterns` and `## Key Exports`, and the reference-app overrides `## Adoption Steps` (usage) and `## Pattern Surface` (api_surface) — so they all surface with `satisfied: true` and the corresponding `matched_synonym` field.

**2.2 Code fence balance.** Read `unbalanced_fences` from the second JSON blob. **`true` → High severity** finding: `naive-coherence — unbalanced code fence (unclosed block)` (the JSON's `fence_count` may be cited in the detail).

**2.3 Language tags on opening fences.** Read `bare_opening_fences[]` from the second JSON blob. The script already runs the stateful open/close scan — closing fences are never reported. For each entry, emit a **Medium severity** finding: `naive-coherence — opening code fence at line {entry.line} missing language tag`.

**2.4 Exports cross-used in a usage-family section.** For each function name reported in the step 3 subagent inventory (`exports[].name` where `kind == "function"` or `kind == "method"`):
- Determine the usage-family search scope:
  - **Single-body skill** (no `references/` directory, or `## Full*` sections carry real content): the span from §2.1's `matched_synonym` anchor to the next `^## ` anchor.
  - **Split-body skill** (a `references/` directory exists alongside SKILL.md AND the SKILL.md `## Full*` sections are stubs/pointers): the union of EVERY usage-family heading present in SKILL.md (`Usage`/`Usage Patterns`/`Examples`/`How to use`/`Quickstart`/`Quick Start`/`Getting Started`/`Common Workflows`/`Adoption Steps`/`Key API Summary`/`Pattern Surface`/`Key Exports`), each from its anchor to the next `^## ` anchor, PLUS the full text of every file under `references/`.
- `grep -c "{export.name}"` across that scope and sum the counts.
- **Zero occurrences across the entire scope → High severity** finding: `naive-coherence — exported {kind} \`{name}\` is not referenced in any usage-family section or reference file`. This catches the "documented but unused" failure mode that trivially fails discovery testing. A method referenced in any usage-family section OR any `references/` file satisfies the check.

**2.5 Async/sync consistency.** For every export with `async` in its description prose (grep for `\basync\b` in the description segment), check the corresponding code example segment for `await` / `async` keywords:
- Description says async + example shows no `await` → **High severity** finding: `naive-coherence — \`{name}\` described as async but example lacks \`await\``
- Description says sync + example uses `await {name}` → **High severity** finding: `naive-coherence — \`{name}\` described as sync but example awaits it`

**2.6 Table syntax.** Read `table_drift[]` from the second JSON blob (§2.0). The script normalizes escaped pipes (`\|`, used inside TypeScript union types such as `string \| undefined`) before splitting and compares each row against its header's column count, so a plain `split on |` false-positive cannot occur here. For each entry, emit a **Medium severity** finding: `naive-coherence — table row at line {entry.line} has {entry.actual_cols} columns; header has {entry.expected_cols}` (the `entry.section` and `entry.row` fields populate the detail when present).

**2.7 Scripts & Assets section.** If `{skillDir}/scripts/` or `{skillDir}/assets/` exists, `grep -n '^## Scripts' SKILL.md`:
- Directory exists AND no `## Scripts` section → **Medium severity** finding: `naive-coherence — scripts/assets directory exists but Scripts & Assets section missing` (per `{scoringRulesFile}`)

**Hard rule:** 0 findings across §§2.1–2.7 = naive coherence PASS. ≥1 finding = rerank per the severity rubric above; the count and severity list are appended to the Coherence Analysis output in §6.

Build the findings list:

```json
{
  "structural_issues": [
    {"type": "missing_section", "severity": "High", "detail": "No 'Usage' section found", "line": null},
    {"type": "unbalanced_fence", "severity": "High", "detail": "3 opening fences, 2 closing", "line": null},
    {"type": "export_not_in_usage", "severity": "High", "detail": "exported function `formatDate` never referenced in Usage section", "line": 42},
    {"type": "async_mismatch", "severity": "High", "detail": "`fetchData` described async but example lacks await", "line": 67}
  ],
  "issues_found": 4
}
```

**After naive coherence → Execute Section 2b if gate conditions met, then skip to Section 6 (Append Results)**

### 2b. Migration/Deprecation Verification (Mode-Independent)

Apply rules from `{migrationSectionRules}`. That file is the single source of
truth for the gate, scope, and case rules; §5b below applies the same rules on
the contextual path.

**After Section 2b (naive path) → Skip to Section 6 (Append Results)**

### 3. Contextual Mode: Extract References

Scan SKILL.md for all cross-references:

**Reference types to extract:**
- File path references (`./path/to/file.ts`, `../shared/types.ts`)
- Skill references (`See SKILL.md for {other-skill}`, `Integrates with {package}`)
- Type imports (`import { Type } from './module'`)
- Integration pattern references (middleware chains, plugin hooks, shared state)
- Script/asset references (`scripts/{file}`, `assets/{file}`) in SKILL.md body

Delegate to a subagent that grep/regexes SKILL.md for reference patterns and returns only this JSON shape — no prose, no commentary, no markdown fences: `{"references_found": [{"line": N, "type": "file-path|skill|type-import|integration-pattern|script-asset", "target": "..."}]}`. Parent strips wrapping markdown fences (if present) before parsing. If subagent unavailable, scan in main thread.

### 4. Contextual Mode: Validate Each Reference

For EACH reference found, delegate to a subagent that:

1. Checks if the target exists (file exists, skill exists, type is declared)
2. If target exists, validates the reference is accurate:
   - File path references: file exists at specified path
   - Type imports: type is actually exported from the referenced module
   - Skill references: referenced skill exists in skills output folder
   - Integration patterns: documented pattern matches actual implementation
   - Script/asset references: verify the referenced file exists in the skill's `scripts/` or `assets/` directory
3. Returns only this JSON shape per reference — no prose, no commentary, no markdown fences: `{"reference": "...", "line": N, "target_exists": <bool>, "type_match": <bool>, "signature_match": <bool>, "issues": ["..."]}`

Parent strips wrapping markdown fences (if present) before parsing. If subagent unavailable, validate each reference in main thread.

4. **Scripts/assets directory check:** If a `scripts/` or `assets/` directory exists alongside SKILL.md, verify that a "Scripts & Assets" section (Section 7b) is present in SKILL.md. This directory-level check applies in both modes (naive mode performs it in Section 2; contextual mode performs it here alongside per-reference validation). Flag absence as Medium severity gap per `{scoringRulesFile}`.

5. **Path containment:** for every resolved reference target, compute its canonical path (`os.path.realpath`) and require that it lives inside `{skillDir}`, inside `{source_path}` (the extraction tree recorded in metadata.json), OR — for stack skills — inside `{skills_output_folder}`. The third root applies only here in contextual mode: a stack's constituent cross-references legitimately resolve to `skills/{name}/active` under `{skills_output_folder}`, which lies outside `{skillDir}`, and a stack's metadata.json records no single `source_path` to anchor them. References whose canonical path escapes all applicable roots (e.g. `../../../etc/passwd`, absolute paths to unrelated dirs, symlink redirections outside the skill, its source, or — for a stack — the skills output tree) are **High severity** findings: `coherence — reference escapes skill/source sandbox: {raw_ref} → {canonical_path}`. Canonicalization happens before the root check, so a symlink that points outside every applicable root is still caught. Do not validate the target's contents for escaping references — the escape itself is the finding.

### 5. Contextual Mode: Check Integration Pattern Completeness

For stack skills, verify integration patterns are complete:

- **All documented integration points have corresponding code examples**
- **Shared types are consistently used across referenced components**
- **Middleware/plugin chains show complete flow, not fragments**
- **Event handlers reference valid event types**

Build integration completeness findings:

```json
{
  "patterns_documented": 5,
  "patterns_complete": 4,
  "incomplete_patterns": [
    {
      "pattern": "Auth middleware chain",
      "issue": "Shows middleware registration but not the handler function signature",
      "line": 95
    }
  ]
}
```

**Zero integration patterns:** If no integration patterns are documented in SKILL.md (e.g., a contextual-mode skill that uses shared types but has no middleware chains, plugin hooks, or event flows): record `patterns_documented: 0`, `patterns_complete: 0`. The coherence score will use reference validity alone — see `{scoringRulesFile}` Coherence Score Aggregation: "If no integration patterns exist, combined coherence equals reference validity."

### 5b. Migration/Deprecation Verification (Contextual Path)

Apply rules from `{migrationSectionRules}`. Same rules as §2b — the reference
file is the single source of truth. Append findings to the coherence analysis
results.

### 5c. Calculate Coherence Scores

**Contextual mode only.** The reference-validity ratio, the integration-completeness ratio, and their fixed 0.6 / 0.4 weighted mean are pure arithmetic — the judgment (which references are valid in §4, which patterns are complete in §5) has already happened. Do not compute these percentages by hand; they are aggregated by `{coherenceAggregationScript}` (the formulas it encodes are documented in `{scoringRulesFile}` — Coherence Score Aggregation).

Tally the counts from the §4 per-reference JSON (`valid_references` = references with `target_exists && type_match && signature_match && no issues`; `total_references` = references extracted in §3) and the §5 integration JSON (`patterns_documented`, `patterns_complete`), then invoke:

```bash
echo '{"valid_references": <V>, "total_references": <T>, "patterns_documented": <PD>, "patterns_complete": <PC>}' | uv run {coherenceAggregationScript} --stdin
```

The script also accepts the JSON as a positional argument or via `--json-input`. Parse its output and read:
- `referenceValidity` — reference-validity percentage
- `integrationCompleteness` — integration-completeness percentage (`null` when no patterns are documented)
- `combinedCoherence` — the combined coherence percentage passed to score.md §3a as the `coherence` input

The script handles both edge cases the formula requires: `patterns_documented == 0` → `combinedCoherence` equals `referenceValidity` (no divide-by-zero); `total_references == 0` → `referenceValidity` is 100.0 (no references means no broken references). These values fill the `{percentage}%` placeholders in the output template loaded in Section 6.

### 6. Append Coherence Analysis to Output

Load `{outputFormatsFile}` and use the appropriate Coherence Analysis section format (naive or contextual) to append findings to `{outputFile}`.

### 7. Report Coherence Results

Report the coherence result to the user, then proceed to external validation:

- **Naive mode:** the count of structural issues found (the coherence category is not scored — its weight redistributes to coverage).
- **Contextual mode:** the reference-validity ratio, the integration-completeness ratio, the combined coherence percentage, and the issue count — full details are in the Coherence Analysis section.

Update stepsCompleted, then load and execute {nextStepFile}.

---
nextStepFile: 'coherence-check.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
scoringRulesFile: '{scoringRulesPath}'
sourceAccessProtocol: 'references/source-access-protocol.md'
reconcileScript: 'scripts/reconcile-coverage.py'
coherenceScript: 'scripts/check-metadata-coherence.py'
numeratorVerifyScript: 'scripts/verify-declared-numerator.py'
---

<!-- Config: communicate in {communication_language}. -->

# Step 3: Coverage Check

## STEP GOAL:

Compare the exports, functions, classes, types, and interfaces documented in SKILL.md against the actual source code API surface. Identify missing documentation, undocumented exports, and signature mismatches. Analysis depth scales with forge tier.

### 0. Check for Docs-Only Mode

**If all SKILL.md citations are `[EXT:...]` format (no local source citations):**

Set `docs_only_mode: true` in context for step 5 scoring. Coverage scoring adapts: instead of comparing SKILL.md against source code exports, compare SKILL.md documented items against themselves for internal completeness (every documented function has a description, parameters, and return type). Score based on documentation completeness rather than source coverage.

**Quick-tier weight adjustment:** If `confidence_tier` is also `"Quick"`, apply Quick-tier weight redistribution (zeroing Signature Accuracy and Type Coverage) as an additional step per `{scoringRulesFile}`.

"**Docs-only skill detected.** Coverage check evaluates documentation completeness rather than source code coverage."

**If source-based skill:** Continue with standard coverage check below.

### 0b. Load Source Access Protocol

Load `{sourceAccessProtocol}` and follow both sections:
1. **Source API Surface Definition** — determines what counts as the public API for coverage denominator
2. **Source Access Resolution** — 5-state waterfall to determine how source files will be read and sets `analysis_confidence`

### 1. Extract Documented Exports from SKILL.md

<!-- Subagent delegation: read SKILL.md + references/*.md, return compact JSON inventory -->

Delegate reading of the skill under test to a subagent. The subagent receives the path to SKILL.md (and the `references/` directory path if it exists) and must:
1. Read SKILL.md
2. If a `references/` directory exists alongside SKILL.md and SKILL.md's `## Full` headings are absent or stubs, also read all `references/*.md` files
3. only return this compact JSON inventory — no prose, no extra commentary:

```json
{
  "exports": [
    {"name": "functionName", "kind": "function", "params": "...", "return_type": "...", "description": "..."},
    {"name": "ClassName", "kind": "class", "methods": ["..."], "properties": ["..."]},
    {"name": "TypeName", "kind": "type", "fields": ["..."]},
    {"name": "CONST_NAME", "kind": "constant", "values": ["..."]},
    {"name": "useHook", "kind": "hook", "usage_signature": "..."}
  ],
  "capabilities": ["brief capability descriptions from the skill overview"],
  "references": ["references/api-reference.md", "references/type-definitions.md"],
  "cross_check_mismatches": [
    {
      "export": "functionName",
      "skill_md_line": 42,
      "reference_file": "references/api-reference.md",
      "reference_line": 18,
      "issue": "description of the signature mismatch"
    }
  ]
}
```

**Parent uses this JSON summary as the documented inventory.** Do not load SKILL.md or references file contents into parent context.

**If subagent delegation is unavailable:** the parent performs the read itself in the main thread — read SKILL.md (and, per step 2 above, all `references/*.md` when a `references/` directory exists and SKILL.md's `## Full` headings are absent or stubs) and assemble the same compact JSON inventory. The §1a schema validation and ground-truth spot-check still run on the parent-built inventory — a quality gate does not skip its own hallucination guards just because the extraction ran in-thread. This mirrors the §2 fallback ("perform ast-grep analysis in main thread") so §1 degrades gracefully instead of stalling.

#### 1a. Parent-Side Schema Validation + Spot-Check

test-skill is a quality gate — it must not trust subagent output blindly. Before any downstream step consumes the inventory, the parent runs a schema validator and a grep spot-check, and HALTs on any failure.

**Schema validation (required keys + types) — delegated to `scripts/validate-inventory.py`.** Fence-stripping, JSON parsing, and the required-keys / per-entry-type / `kind`-enum / mismatch-field contract are structural validation with one correct verdict per input, so they run in the script, not in-prompt. Pipe the subagent's **raw** response (fence and all) to it exactly as §2c pipes the reconcile input:

```bash
echo '<subagent raw response>' | uv run scripts/validate-inventory.py --stdin
```

The script strips a wrapping markdown fence (a leading line of three backticks with an optional language tag like `json`, and a trailing line of three backticks — subagents frequently return fenced JSON despite instructions), parses the inner content, and enforces exactly this contract, returning a `violations[]` entry for each breach:

- Parses as JSON after fence stripping (parse failure → `not valid JSON`).
- Required keys present with correct types: `exports` (list), `cross_check_mismatches` (list — may be empty). Note: the parent already knows the skill name from workflow context (`{resolved_skill_package}` from step 1) — the subagent is not required to echo it back, and doing so introduces a contract-drift surface without improving verification.
- Each `exports[]` entry is a dict with at minimum `name` (non-empty string) and `kind` (one of `function|class|type|constant|hook|interface|method|struct|enum|trait|macro|adapter`). The enum spans the constructs SKF actually documents across languages and skill types: JS/TS (`function`/`class`/`type`/`constant`/`hook`/`interface`/`method`), Rust public-API items (`struct`/`enum`/`trait`/`macro` — alongside the shared `type`/`constant`/`function`), and stack-composition scaffolds (`adapter`). Malformed entries are counted in `rejectedCount`.
- Each non-empty `cross_check_mismatches[]` entry carries `export`, `skill_md_line`, `reference_file`, `reference_line`, `issue`.

The script returns `{"valid": bool, "violations": [...], "rejectedCount": N, "exportsCount": N, "inventory": {...}|null}`. **If `valid` is false → HALT** `coverage-check: subagent inventory failed schema validation — {violations joined}` (do not downgrade to a warning; a grader must not trust malformed subagent output). When `valid` is true, consume the script's returned `inventory` object as the documented inventory for the spot-check and all downstream steps — do **not** re-parse the raw response by hand.

**Spot-check (ground-truth verification, zero-hallucination guard):** operate on the validated `inventory` from the script.

1. If `inventory.exports` is empty (`exportsCount == 0`): skip the spot-check (no names to verify). Zero-exports policy is handled in the §2b zero-exports guard.
2. Otherwise, sample `min(3, exportsCount)` exports deterministically — by default take indices `[0, len//2, len-1]` (first, middle, last) from `inventory.exports` after a stable sort by `name`.
3. For each sampled export, grep for the name across SKILL.md **and every reference file the subagent listed in `inventory.references`** (the documented surface of a split-body skill spans both): `grep -n "{export.name}" {resolved_skill_package}/SKILL.md {resolved_skill_package}/{each references[] path}` in the parent context. The name must appear at least once somewhere in that file set. Greping SKILL.md alone would false-HALT a split-body skill whose sampled export is documented only in a `references/*.md` file (a legitimate placement per §1 step 2 and the split-body note below).
4. If a sampled name returns zero matches across SKILL.md **and** all listed reference files, HALT "coverage-check: subagent inventory failed ground-truth spot-check — `{name}` claimed as export but absent from SKILL.md and the listed reference files".

These checks catch two hallucination classes: schema-shape drift (subagent paraphrased or dropped the contract) and fabricated exports (subagent invented names not in the document). Both are disqualifying for a grader skill — do not downgrade to a warning.

**Split-body traversal** is handled inside the subagent: if `references/` exists and `## Full` headings are absent or stubs in SKILL.md, the subagent extends its scan to all `references/*.md` files and includes them in the `exports` array. After split-body, Tier 2 content (Full API Reference, Full Type Definitions) lives in reference files — the inventory must reflect the full skill content regardless of where it resides.

### 1b. Cross-Check Split-Body Consistency

**Only execute if the subagent's `references` array is non-empty** (detected during split-body traversal in Section 1). Skip silently otherwise.

The subagent has already read both SKILL.md body and `references/*.md` files. For each function, class, type, or interface that appears in both the SKILL.md body AND any `references/*.md` file, instruct the subagent (or perform in the same subagent call from Section 1) to compare the documented signatures and include mismatches in its JSON output as a `cross_check_mismatches` array:

- **Parameters:** name, type, order, optionality
- **Return types:** exact type match
- **Description:** no contradictions (brief vs detailed is acceptable; conflicting semantics is not)

**SKILL.md body is authoritative.** When a mismatch is found, the reference file is the one that needs updating.

Parent reads `cross_check_mismatches` from the subagent JSON summary. Build the split-body consistency findings list:

```json
{
  "cross_check_mismatches": [
    {
      "export": "formatDate",
      "skill_md_line": 42,
      "reference_file": "references/api-reference.md",
      "reference_line": 18,
      "issue": "SKILL.md shows (date: Date) => string, reference shows (date: Date, format?: string) => string"
    }
  ],
  "exports_cross_checked": 12,
  "mismatches_found": 1
}
```

Flag each mismatch as **High severity** — signature inconsistency between SKILL.md body and reference files undermines agent trust. These findings feed into the gap report (step 6).

### 2. Analyze Source Code (Tier-Dependent)

Start from the package entry point (see 0b) and identify the public API surface. Then analyze those exports at the appropriate tier depth.

**Quick Tier (no tools):**
- Read the entry point file(s) directly
- Identify public exports by scanning for `export` keywords, `module.exports`, `__init__.py` imports, or language-specific export patterns
- Compare against documented inventory by name matching
- Cannot verify signatures — note as "unverified" in report

**Forge Tier (ast-grep available):**

**Before delegating, the parent builds a `documented_signatures` map** from the §1 inventory: `{ "{name}": {"params": "...", "return_type": "..."} }` for every documented export that carries a signature. Pass this map as a structured input alongside the per-file ast-grep instructions. Without it the subagent has only the bare source — it cannot diff against the documented signatures, so `signature_mismatches[]` collapses to empty and Signature Accuracy defaults to 100% on a name-only pass (a silent false positive). The subagent must populate `documented_sig` from this map, not invent it.

For EACH source file that defines public API exports, delegate to a subagent that:
1. Uses ast-grep to extract all exported symbols with their full signatures (the `source_sig`)
2. Matches each export against the `documented_signatures` map supplied by the parent, comparing params (name, type, order, optionality) and return type
3. Returns only the JSON object below — no prose, no commentary, no markdown fences:

```json
{
  "file": "src/utils.ts",
  "exports_found": ["formatDate", "parseConfig", "ConfigType"],
  "exports_documented": ["formatDate", "parseConfig"],
  "missing_docs": ["ConfigType"],
  "signature_mismatches": [
    {
      "name": "formatDate",
      "source_sig": "(date: Date, format?: string) => string",
      "documented_sig": "(date: Date) => string",
      "issue": "missing optional parameter 'format'"
    }
  ]
}
```

Parent strips wrapping markdown fences (if present) before parsing, same as §1a. If subagent unavailable, perform ast-grep analysis in main thread per file.

**Deep Tier (ast-grep + gh + QMD):**
- All Forge tier checks, plus:
- Use gh CLI to verify source repository matches documented version
- Cross-check type definitions against their source declarations
- Verify re-exported symbols trace to their original source

### 2b. Zero-Exports Guard

After the source-code analysis (§2) completes, compute `total_exports` — the count of exports discovered in the source / provenance-map / metadata.json, per the stratified-scope and State 2 rules resolved in §4.

**Stack-skill branch (`metadata.json.skill_type == "stack"`):** A stack skill's own barrel is empty by design — it composes constituent skills rather than exporting a proprietary surface — so `total_exports` derived from its own barrel is `0` for a *correctly* built stack, and its `[from skill: …]` citations never trip §0's `[EXT:…]`-only docs-only trigger. The zero-exports HALT below targets individual source-based skills and must not fire for stacks. Derive the stack's coverage denominator (`stack_denominator`) from its composition surface, in priority order, and use it as `total_exports` for the rest of coverage scoring:

1. Provenance-map cited-contract count — when `{forge_data_folder}/{skill_name}/provenance-map.json` exists **and its `entries[]` is non-empty**: count the named cited contracts, **excluding entries whose `export_name` contains `::`** (impl-block methods roll up under an already-counted type). Use the same exclusion as §4b's named-export rule so the §2b and §4b stack denominators agree.
2. Otherwise the composition surface from `metadata.json`: `len(libraries) + len(integration_pairs)`.

**If `stack_denominator == 0`** (no provenance-map or empty `entries[]`, AND `libraries` and `integration_pairs` are both empty): HALT with `Error: stack composition surface empty — {skill_name} cites no contracts, libraries, or integration pairs, so Export Coverage is undefined. Verify the stack was compiled from at least one constituent skill.` Do not write the Coverage Analysis section; this is an indeterminate state, not a FAIL.

Otherwise carry `stack_denominator` forward as `total_exports`, skip the HALT below, and continue — §2c and §4b consume this same denominator via their own stack-skill branches. (Stacks route to contextual mode per `detect-mode.md` and have a dedicated section in `scoring-rules.md`, so they are first-class — the indeterminate-surface HALT is an individual-skill guard only.)

**If `total_exports == 0` AND `docs_only_mode == false` AND `metadata.json.skill_type != "stack"`:** HALT with:

```
Error: indeterminate API surface — 0 exports discovered in source for {skill_name}.

A source-based skill with zero exports cannot be meaningfully tested:
Export Coverage is undefined (division by zero) and downstream scoring
would yield a vacuous PASS.

Fix one of:
  - Set `scope.include` in the brief to point at the package's entry point(s)
  - Add `[EXT:]` citations if this is actually a docs-only skill
  - Verify the skill's source_path / source_ref resolve to the intended tree
```

Do not write the Coverage Analysis section. Do not proceed to scoring. This is a true indeterminate state, not a FAIL — no score should be attached.

**If `docs_only_mode == true` and the documented inventory is empty:** HALT with the analogous docs-only message ("docs-only skill declares zero items — no API surface to test").

### 2c. Reconcile Documented vs Source Surface (Deterministic Intersection)

On a split-body skill the §1 inventory (documented surface) and the §2 AST output (source barrel) are two independent lists, so the `Documented` count must be their **intersection**, not a parent estimate. That reconciliation — set intersection/difference/cardinality plus the grep-verified numerator of the scalar/stack branches — is deterministic arithmetic with one correct answer per input, so it is performed by `uv run {reconcileScript}`, not by hand.

**Which branch applies — and therefore which `denominatorSource` the script runs — is the policy decision made here.** The denominator itself is resolved by §4/§4b; the script consumes the *already-resolved* denominator and does only the counting (`documented_set` is derived inside the script from the §1 `exports[]`, de-duplicated and with `kind: "method"` excluded — methods are members of an already-counted class/type, not top-level barrel exports). Pick exactly one branch:

1. **Enumerated path (`denominatorSource: "barrel"`)** — the common split-body case. `barrel_set` is the union of `exports_found[]` across every §2 per-file result. **When a stratified-scope or State-2 denominator applies (see §4) and it resolves to an enumerated name set** — the priority-2/3 re-derivation from `scope.tier_a_include` / `scope.include` globs — pass that resolved name set as `barrelSet` so the script intersects against it instead of the raw per-file union. The script computes `Documented := |documented_set ∩ barrel_set|`, `Missing := barrel_set − documented_set` (in source, not documented), `Stale := documented_set − barrel_set` (documented, not in source), and `Export Coverage = |Documented| / |barrel_set| * 100`.

2. **Scalar-denominator branch (`denominatorSource: "scalar"`)** — §4 priority 1, when the resolved denominator is the scalar `metadata.json.stats.effective_denominator` (a count with **no enumerated name set**). There is no `barrel_set` to intersect, so the script instead greps each `documented_set` name across `SKILL.md ∪ references/*.md` and counts appearances. Pass `denominatorValue: effective_denominator` and `skillPackagePath: {resolved_skill_package}`; the script sets `Missing := max(0, effective_denominator − Documented)` and returns an empty `Stale` (not enumerable without a barrel name set). When the grep count overshoots the denominator the script reports the surplus instead of a negative residual — see the surplus note under the returns list. If §4b's numerator-ground-truth arm fires (it triggers only when `exports_documented == effective_denominator`), its verified count is authoritative and **overrides** this numerator — do not apply both.

3. **Stack-skill branch (`denominatorSource: "stack"`, `metadata.json.skill_type == "stack"`)** — a stack's source barrel is empty by design, so intersecting against it would divide by zero. Pass `denominatorValue: stack_denominator` (the §2b composition-surface denominator) and `compositionNames`: the provenance-map cited-contract names (`::`-excluded) when the map exists and is non-empty, else the `libraries` and `integration_pairs` names. The script greps each composition name across `SKILL.md ∪ references/*.md` for the numerator, sets `Missing := max(0, stack_denominator − Documented)`, and omits `Stale` (no source barrel to enumerate against). The same surplus reporting as the scalar branch applies.

**Build the reconciliation input and run the script:**

```bash
echo '<JSON>' | uv run {reconcileScript} --stdin
```

Input JSON (one object — supply only the fields the chosen branch needs; `{reconcileScript}` is resolved relative to the skill root):

```json
{
  "denominatorSource": "barrel | scalar | stack",
  "exports": [ /* §1 inventory exports[] — used for barrel + scalar; kind:"method" entries are excluded automatically */ ],
  "barrelSet": [ /* barrel: resolved enumerated name set, when §4 supplies one */ ],
  "perFileResults": [ /* barrel: the §2 per-file results, unioned into barrel_set when no barrelSet */ ],
  "denominatorValue": 0,
  "compositionNames": [ /* stack: composition-surface names to grep */ ],
  "skillPackagePath": "{resolved_skill_package}"
}
```

The script returns (read these — **do not re-derive them by hand**):

- `documented` — the numerator (`|documented_set ∩ barrel_set|` for barrel; grep-verified count for scalar/stack). This is always the true count, never capped.
- `missing` / `missingCount` — source names not documented (barrel enumerates the names; scalar/stack give only the residual count). Never negative.
- `stale` / `staleCount` — documented names not in source (barrel only; empty with `staleApplicable: false` for scalar/stack)
- `denominator` — `|barrel_set|` (barrel) or the resolved scalar/stack denominator
- `exportCoverage` — `documented / denominator * 100`, already rounded; capped at 100 on the scalar/stack branches
- `numeratorSurplus` / `coverageUncapped` / `coverageCapped` — scalar/stack only; see the surplus note below

**Surplus on the grep branches.** The scalar and stack branches grep a name set against the skill body, so their numerator and denominator measure independent sets and `documented` can exceed `denominator` — a consumer-surface denominator counts one surface while the documented body may also name migration aliases or re-exported sibling symbols. When that happens the script reports `numeratorSurplus > 0` and `coverageCapped: true`, holds `exportCoverage` at 100, floors `missingCount` at 0, and preserves the raw ratio in `coverageUncapped`. **A surplus is a signal, not a pass:** it means the two sets disagree, so state it in the Coverage Analysis section (§5) alongside both counts. A large surplus on a skill whose brief carries no `scope.tier_a_include` is the deflated-denominator signature the `source-access-protocol.md` deflation guard describes — check that guard before accepting the 100.

Carry these into §3's table/summary and §4's Export Coverage, and record the counts in the Coverage Analysis section (§5) so the numerator is auditable. The `exportCoverage` recorded here is the value step 5 feeds to `compute-score.py` — it is the script's value, not a parent estimate.

### 3. Build Coverage Results

Aggregate findings across all source files:

**Per-export status table:**

| Export | Type | Documented | Signature Match | File:Line | Status |
|--------|------|-----------|-----------------|-----------|--------|
| {name} | function/class/type | yes/no | yes/no/unverified | src/file.ts:42 | PASS/FAIL/WARN |

**Summary counts** (read from the §2c reconciliation script's JSON — not re-estimated here):
- Total exports in source: `denominator`
- Documented in SKILL.md: `documented`
- Missing documentation: `missingCount`
- Signature mismatches: {N}
- Undocumented in SKILL.md but not in source (stale docs): `staleCount`

When the script reports `coverageCapped: true`, add the surplus to the summary — `Documented (surplus over denominator): {numeratorSurplus}` and `Uncapped coverage: {coverageUncapped}%` — so the reader can see that `Missing documentation: 0` is a floored residual rather than a verified-complete surface.

### 4. Load Scoring Rules

Load `{scoringRulesFile}` to determine category scores:

- **Export Coverage:** the `exportCoverage` value returned by the §2c reconciliation script (`documented / denominator * 100`) — read it from the script's JSON, do not re-compute it here
- **Signature Accuracy:** (matching_signatures / total_documented) * 100 (Forge/Deep only, "N/A" for Quick)
- **Type Coverage:** (documented_types / total_types) * 100 (Forge/Deep only, "N/A" for Quick)

**Resolve the coverage denominator per `{sourceAccessProtocol}` (already loaded in §0b) — do not re-derive its ladders here.** Determine which §Source API Surface Definition clause matches this skill and apply that clause exactly as written: **stratified-scope** (monorepo curated subset), **multi-entry (exports-map)**, **specific-modules**, **pattern-reference**, or the **State 2** provenance-vs-metadata cross-reference (union on divergence). Each clause fixes its own resolution priority (prefer `metadata.json.stats.effective_denominator` → `scope.tier_a_include` → `scope.include` / subpath union), its deflation and inflation guards, the umbrella-barrel exclusion, and provenance-map canonicalization (including the fold summary the canonicalization records). Use the clause's resolved value as `total_exports`; when no clause matches, use the standard barrel-based denominator. Record the denominator source in the Coverage Analysis section using the exact `Denominator: {barrel | stratified (…) | multi-entry (…) | specific-modules (…) | pattern-reference (…)}` annotation string the matching clause specifies.

**Record the two non-chosen candidate values alongside the chosen one.**
Stratified-scope resolution picks ONE of three denominator candidates
(`stats.effective_denominator`, `tier_a_include` union, `scope.include` union)
per the priority above. To make the choice auditable, append a
`Denominator Candidates` block immediately after the `Denominator:` line listing
all three values — the chosen one explicitly marked and the other two recorded
as-observed (or `absent` when the candidate was not present for this skill):

```markdown
**Denominator Candidates** (stratified-scope audit trail):
- `stats.effective_denominator`: {N | absent}  {← chosen if priority (1) applied}
- `scope.tier_a_include` union: {N | absent}    {← chosen if priority (2) applied}
- `scope.include` union: {N | absent}           {← chosen if priority (3) applied}
- exports-map subpath union: {N | absent}        {← chosen if the multi-entry clause applied}
- root barrel: {N}                               {secondary candidate — root-barrel-vs-subpath-union audit}
```

Readers can then spot-check whether the chosen denominator is reasonable
against the other two without re-running the extraction. A future reviewer who
suspects denominator gaming has the evidence inline. The `multi-entry` clause
requires the root-barrel named-export count in the `root barrel` row so the
root-barrel-vs-subpath-union choice is auditable.

### 4b. Metadata Export-Count Coherence Cross-Check

After the denominator has been resolved (standard, stratified, or State 2), cross-check export counts *within each semantic cluster* to detect extraction drift without false-positiving on intentional multi-denominator reporting. Picking the denominator silently when sources disagree is a known friction — the tester cannot tell whether to trust the pick, ignore the drift, or report it. Make it explicit, but only for counts that are authored to measure the *same* surface.

**Stack-skill branch (`metadata.json.skill_type == "stack"`):** Skip the intra-cluster and cross-cluster count comparisons, the `confidence_distribution` sum check, AND the numerator ground-truth check below — none apply to a stack. For a stack the three counts measure intentionally *different* surfaces, so comparing them yields only false drift: `exports_documented` / `exports[]` measure the stack's own barrel (empty by design → `0`), the provenance-map enumerates the cited *constituent* contracts, and `confidence_distribution` bins those constituents — so for a stack treat `confidence_distribution` as a per-constituent count (it sums to the constituent count, not to `exports_documented`) and do not assert it against `exports_documented`. The numerator-inflation arm below targets an *individual* skill whose `exports_documented` was padded to equal `effective_denominator`; a stack's numerator is instead computed by full-grep in §2c's stack-skill branch and stack `metadata.json.stats` carries no `effective_denominator`, so the arm is not run. Record the denominator using the §2b source — `Denominator: stack composition ({N} cited contracts)` when the provenance-map supplied it, or `Denominator: stack composition ({N} libraries + integration pairs)` when the composition surface did — then proceed.

**Reference-app branch (`metadata.json.scope_type == "reference-app"`):** Skip the intra-cluster and cross-cluster count comparisons, the `confidence_distribution` sum check, AND the numerator ground-truth check below — none apply to a reference app, whose counts measure intentionally *different* surfaces. A reference app documents wiring/construct **pattern surfaces**, not a library export barrel, so `metadata.json.exports[]` is empty by design (`0`), `stats.exports_public_api` / `stats.pattern_surfaces_documented` count the documented pattern surfaces, and `confidence_distribution` bins the per-citation provenance entries (it sums to the citation count, not to `pattern_surfaces_documented`). Comparing these yields only false drift: a spurious Cluster-A "barrel drift" (`exports_public_api` vs `exports[].length == 0`) and a Cluster-B "documented-surface drift" (`pattern_surfaces_documented` vs the larger `confidence_distribution` sum). The numerator-inflation arm also does not apply — a reference app carries no `effective_denominator` (see the `skf-create-skill` reference-app carve-out), so the `exports_documented == effective_denominator` signature never fires. Record the denominator as `Denominator: pattern-surface ({pattern_surfaces_documented})` and proceed. (`referenceApp` and a stack skill are distinct signals — a skill is one or the other, never both, so only one of these two branches applies.)

**Collect available counts (skip any that are absent) and bin them into two clusters:**

**Cluster A — public-barrel surface** (what `__init__.py` / `index.ts` / `lib.rs` re-exports):

1. `metadata.json.stats.exports_public_api` — the declared public API count
2. `metadata.json.exports[]` array length — the enumerated public export list

**Cluster B — documented surface** (what was extracted and documented, including methods and submodule members):

3. `metadata.json.stats.exports_documented` — the declared documented count
4. Provenance-map **named-export count** (if `{forge_data_folder}/{skill_name}/provenance-map.json` exists) — pass the raw `export_name` values as `provenanceExportNames`; the script counts top-level named exports, **excluding entries whose `export_name` contains `::`** (impl-block methods like `Type::method`, which roll up under an already-counted type and are not separate barrel exports). Comparing the raw entry count instead false-positives on any method-enumerating provenance map (common for Rust/TS type-heavy skills): e.g. a map with 88 named exports + 48 `Type::method` entries reports 136 against `exports_documented` ≈ 92 → a spurious ~32% Cluster-B drift, while the comparable named-export count (88) agrees within ~4%.
5. `confidence_distribution` (`t1`, `t1_low`, `t2`, `t3`, when present in `metadata.json.stats`) — pass the tiers as `confidenceDistribution`; the script sums them. Every extracted/documented export is binned into exactly one confidence tier, so the distribution must sum to the documented-surface total; a divergence (e.g., distribution sums to 91 while `exports_documented` is 85) is an internal-consistency defect even when the two clusters look fine

Cluster assignment is canonical: `skf-create-skill` step 5 derives `exports_public_api` from entry-point validation and writes the `exports[]` array from the same barrel surface (see `skf-create-skill/references/compile.md:105`), while `exports_documented` tracks the broader documented surface that the provenance-map also enumerates.

**Delegate the drift arithmetic to `uv run {coherenceScript}`.** The intra-cluster / cross-cluster `>10%` comparisons are deterministic count arithmetic with one correct answer per input, so the script — not the prompt — owns the binning, the divergence percentages, and every skip condition (a cluster with fewer than two present counts, and clusters that agree within the threshold, are skipped inside the script). Build the input from the counts collected above and run it (`{coherenceScript}` resolves relative to the skill root):

```bash
echo '<JSON>' | uv run {coherenceScript} --stdin
```

Input JSON (omit any count that is absent; `driftThresholdPct` defaults to `10`):

```json
{
  "skillType": "{metadata.json.skill_type or null}",
  "scopeType": "{metadata.json.scope_type or null}",
  "clusterA": {"exports_public_api": 0, "exports_length": 0},
  "clusterB": {"exports_documented": 0},
  "provenanceExportNames": [ /* raw provenance-map export_name values — the script excludes `::` impl-block methods and counts the rest */ ],
  "confidenceDistribution": {"t1": 0, "t1_low": 0, "t2": 0, "t3": 0}
}
```

The script returns `{"skipped": bool, "clusterACounts": {...}, "clusterBCounts": {...}, "findings": [...]}`. Read `findings[]` and append each entry directly — **do not re-derive the percentages by hand.** Each entry carries `severity`, `title`, `detail` (the enumerated counts + drift %), and `category: structural/metadata coherence`:

- A **Medium** `metadata drift — {barrel|documented-surface} export counts diverge` is the real drift signal — the two sources should mirror the same surface and they don't, so upstream extraction or compilation produced inconsistent output that a re-compile should reconcile. It is classified under structural/metadata coherence regardless of naive/contextual mode.
- An **Info** `multi-denominator reporting — barrel vs documented surface` is expected for skills whose documented surface intentionally exceeds the barrel (methods, submodule members, re-exported classes) — it is not drift. The note exists so the test report makes the dual-denominator design visible and auditable without demanding action.

(The stack and reference-app branches above already skip this delegation; the script also returns `skipped: true` when passed their `skillType` / `scopeType`, so an unconditional call stays safe.)

**Numerator ground-truth — force a full grep on the inflation signature.** The intra/cross-cluster checks above only compare *counts*; they cannot tell whether the declared exports actually appear in the skill. When `metadata.json.stats.exports_documented == effective_denominator` exactly (numerator equals denominator — the signature of a numerator padded to match the full surface), do not trust the documented count: grep the full declared set (the `metadata.exports[]` / provenance-map names, not the §1a 3-sample) against `SKILL.md ∪ references/*.md`. That grep + present/absent set-diff + count has one correct answer per input, so it runs in `uv run {numeratorVerifyScript}`, not in-prompt (`{numeratorVerifyScript}` resolves relative to the skill root):

```bash
echo '{"declaredNames": [ /* full declared set */ ], "skillPackagePath": "{resolved_skill_package}"}' | uv run {numeratorVerifyScript} --stdin
```

Read the script's output — do not re-derive it by hand:

- `inflated: false` (`verified == declared`) → the skill is genuinely fully documented; no finding, coverage stands.
- `inflated: true` (`verified < declared`) → emit a **High**-severity gap `numerator inflation — {declared − verified} of {declared} declared exports absent from SKILL.md/references` listing the script's `absent[]` names, and use `verified` as the Export Coverage numerator (overriding `exports_documented`). A numerator padded to equal the denominator otherwise produces a tautological 100% that passes the gate.

The grep runs only on the exact-equality signature, so it adds no cost to the common case where the numerator is already below the denominator. Unlike the count-coherence findings above, this arm is authoritative — it changes the numerator used for scoring.

Append any findings (Medium gaps, the Info note, and/or the High numerator-inflation gap) to the Coverage Analysis section's gap list (built in section 5) so they surface in the final test report alongside coverage and signature findings. The count-coherence findings are informational about data quality and do not change the denominator chosen above; the numerator ground-truth arm is the one exception that overrides the numerator.

### 5. Append Coverage Analysis to Output

Append the **Coverage Analysis** section to `{outputFile}`:

```markdown
## Coverage Analysis

**Tier:** {forge_tier}
**Source Access:** {analysis_confidence} (full | provenance-map | metadata-only | remote-only | docs-only)
**Source Path:** {source_path}
**Files Analyzed:** {count}
**Denominator:** {barrel | stratified ({effective_denominator | scope.include union}, {N} files matched)}

### Export Coverage

| Export | Type | Documented | Signature | Source Location | Status |
|--------|------|-----------|-----------|-----------------|--------|
| ... per-export rows ... |

### Coverage Summary

- **Exports Found:** {N}
- **Documented:** {N} ({percentage}%)
- **Missing Documentation:** {N}
- **Signature Mismatches:** {N}
- **Stale Documentation:** {N}
- **Numerator Surplus:** {N} — omit this row unless `coverageCapped: true`; when present, also give the uncapped percentage

### Category Scores

| Category | Score |
|----------|-------|
| Export Coverage | {N}% |
| Signature Accuracy | {N}% or N/A |
| Type Coverage | {N}% or N/A |

Note: Weight application is deferred to step 5 where all category weights are calculated after external validation availability is known.
```

### 6. Report Coverage Results

Report the coverage result to the user: the {forge_tier}-tier analysis of {file_count} source files, the documented ratios for exports / signatures / types (signatures and types are N/A for Quick tier), and the issue count — full details are in the Coverage Analysis section. Then proceed to the coherence check.

Update stepsCompleted, then load and execute {nextStepFile}.

---
nextStepFile: 'coverage-check.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
---

<!-- Config: communicate in {communication_language}. -->

# Step 2: Detect Mode

## STEP GOAL:

Examine the skill metadata to determine whether this is an individual skill (naive mode — API surface coverage only) or a stack skill (contextual mode — full coherence validation including cross-references and integration patterns).

### 1. Determine Test Mode

Read the skill metadata (loaded in step 01) and branch on its `skill_type` field — a single deterministic lookup:

- `skill_type: 'single'` → **Naive Mode** (API-surface coverage; coherence is structural only, no coherence category in scoring)
- `skill_type: 'stack'` → **Contextual Mode** (full coherence validation — cross-references resolve, types match, integration patterns complete; coherence category scored)
- unset or unclear → default to **Naive Mode** (conservative — fewer checks, less chance of false negatives from missing context) and note the default in the report

What each mode actually checks and how category weights are distributed is owned by `scoring-rules.md` (Tier-Dependent Scoring) and `coherence-check.md` — do not restate it here.

**Quick-tier adjustment (applies to both modes):** If `forge_tier` is `Quick`, Signature Accuracy and Type Coverage are skipped during scoring (no AST available); their weights are redistributed proportionally to the remaining active categories per `scoring-rules.md` Tier-Dependent Scoring.

### 2. Update Output Document

Update `{outputFile}` frontmatter:
- Set `testMode: '{naive|contextual}'`

Append the **Test Summary** section to `{outputFile}`:

```markdown
## Test Summary

**Skill:** {skill_name}
**Test Mode:** {naive|contextual}
**Forge Tier:** {detected_tier}

**Mode Rationale:** {brief explanation of why this mode was selected}

**Analysis Plan:**
- Coverage Check: {what will be checked based on mode + tier}
- Coherence Check: {what will be checked based on mode + tier}
```

### 3. Report Mode Detection

Report the detected mode ({naive|contextual}) and why it was selected (individual skill → naive, stack → contextual), then proceed to the coverage check.

Update stepsCompleted, then load and execute {nextStepFile}.

---
nextStepFile: 'step-hard-gate.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
externalScoreScript: 'scripts/combine-external-scores.py'
---

<!-- Config: communicate in {communication_language}. -->

# Step 4b: External Validators

## STEP GOAL:

Run external validation tools (`skill-check` and `tessl`) against the skill directory, capture their scores and findings, and append results to the test report. These tools catch complementary issues that internal coverage and coherence checks miss: `skill-check` validates spec compliance while `tessl` evaluates content quality and actionability.

### 1. Resolve Skill Directory

Read {outputFile} frontmatter to get the skill directory path (`skillDir`).

### 1b. Check for Recent Validation Results (Auto-Reuse)

Before running external validators, check if `{forge_data_folder}/{skill_name}/evidence-report.md` contains validation results (a `## Validation Results` section with quality scores).

**Staleness check:** Determine whether SKILL.md has changed since the evidence report was generated. Walk through these checks in order:

**Pre-check (untracked or staged-only file):** Run `git ls-files --error-unmatch {skillDir}/SKILL.md 2>/dev/null`.
- If the command fails (exit code non-zero) or git is not available, the file is either **untracked** (new, never committed) or we're in a **non-git environment**:
  - Check if `{skillDir}/metadata.json` exists and has a `generation_date` field
  - Compare `metadata.json` `generation_date` against the evidence report's generation date (from its frontmatter `generated` field or the `## Validation Results` timestamp)
  - **Precision guard (mirror of the git-path Primary-cross check):** date-granularity equality is not proof of same-session generation. A same-day `update-skill` that regenerates SKILL.md *after* the cached evidence report was produced yields the same calendar date (e.g. `metadata.generation_date: 2026-05-23T00:00:00Z` vs evidence `generated: 2026-05-23`), so reusing on date-equality alone would publish pre-update scores for post-update content. Auto-reuse is safe **only** when both timestamps carry a real time-of-day component — neither a date-only string (`2026-05-23`) nor a midnight-coerced `…T00:00:00Z` — AND they match to the minute. In that case auto-reuse: the evidence report was generated from the same SKILL.md content.
  - Otherwise — if either timestamp is date-only or midnight-coerced, if they differ, or if `metadata.json` is missing or has no `generation_date` — treat as stale and proceed to section 2 for a fresh run. Forcing a fresh run on ambiguous precision matches the git path's bias toward freshness over reusing possibly-stale scores.
  - Note: "Staleness check: SKILL.md is untracked/non-git — using metadata.json timestamp comparison (date-only/midnight timestamps force a fresh run)."
- If the command succeeds (file is tracked by git), continue to Primary check below.

**Primary (git-tracked):** Run `git log -1 --format=%cI -- {skillDir}/SKILL.md` to get the last commit date of SKILL.md. Compare against the evidence report's generation date (from its frontmatter or the `## Validation Results` timestamp). If SKILL.md's last commit is newer, results are stale.

**Primary-cross (single-commit bundle detection):** The git-commit-timestamp comparison can return a false "fresh" when `update-skill` commits a regenerated SKILL.md alongside an unchanged `evidence-report.md` in the same commit — both files share the same `%cI` even though the cached validation results inside the evidence report were produced during an earlier run. To catch this, after the Primary check also compare `{skillDir}/metadata.json`'s `generation_date` field against the evidence report's internal `## Validation Results` timestamp (or its frontmatter `generated` field). If `metadata.json.generation_date` is strictly newer than the evidence report's internal validation timestamp, treat results as stale regardless of git commit parity — SKILL.md was regenerated after the cached validation ran, so the scores no longer reflect current content. If `metadata.json` is missing or has no `generation_date`, skip this cross-check and rely on the git comparison alone.

**Secondary (uncommitted changes):** Run `git diff --name-only -- {skillDir}/SKILL.md`. If output is non-empty, SKILL.md has uncommitted changes — treat results as stale regardless of commit dates. Also check `git diff --cached --name-only -- {skillDir}/SKILL.md` for staged-but-uncommitted changes — if non-empty, SKILL.md has been staged since last commit, treat results as stale.

If SKILL.md was modified after the evidence report was generated (e.g., after update-skill), the cached results are stale — skip auto-reuse and proceed to section 2 for a fresh run.

If recent, non-stale results exist (from a create-skill run that just completed), auto-reuse them — skip re-running validators and use the existing scores. Record: "External validation: reused from create-skill evidence report." Skip to section 5 (append results).

If no evidence report exists, it contains no validation section, or results are stale, proceed to section 2 (fresh run).

### 2. Run skill-check

**Check availability (short probe — 15s timeout):**

```bash
timeout 15s npx --no-install skill-check -h 2>/dev/null
```

Use `--no-install` so the probe never triggers a slow cold-cache download (npx
would otherwise fetch the package before printing help). Wrap in `timeout 15s`
so a hung probe cannot stall the workflow — consistent with the 120s cap used
on the actual validator run below. If the probe exits non-zero OR the 15s
timeout trips (exit code `124`), record `skill_check_score: N/A` and skip to
section 3.

**Run validation (120s timeout):**

```bash
timeout 120s npx skill-check check {skillDir} --format json --no-security-scan
```

If the command exits non-zero AND the exit code is `124` (GNU timeout's signal for the 120s wall-clock expiring), record `skill_check_score: N/A` with reason `timeout-120s`, log a warning, and skip to section 3. Other non-zero exits fall through to the regular JSON-parse path per the note below.

**Parse JSON output** to extract:
- `scores[].score` — overall score (0-100); match the entry by `relativePath` (or `skillId`) to the validated skill dir. Older skill-check builds exposed this as a top-level `qualityScore` — fall back to that if `scores[]` is absent.
- `diagnostics[]` — any remaining issues
- `summary.errorCount` and `summary.warningCount` — issue counts (the counts live under `summary`, not at the top level)

**Note:** `skill-check` may return a non-zero exit code even when `summary.errorCount` is 0. Always rely on the parsed JSON output, not the shell exit code.

Store in context: `skill_check_score`, `skill_check_diagnostics`

**If skill-check fails entirely:** Record `skill_check_score: N/A`, log warning, continue.

### 3. Run tessl

**Check availability (short probe — 15s timeout):**

```bash
timeout 15s npx --no-install -y tessl --version 2>/dev/null
```

Same rationale as the skill-check probe above: `--no-install` + `timeout 15s`
prevent a cold-cache fetch from stalling the workflow. If the probe exits
non-zero OR the 15s timeout trips (exit code `124`), record
`tessl_score: N/A` and skip to section 4.

**Run review (120s timeout):**

The §2 probe (`npx --no-install -y tessl --version`) already resolved tessl via the caller's npm cache or a locally-installed binary on `$PATH`. Invoke the same binary for the review — do not re-pin to a registry-published version.

```bash
# Use the tessl binary the §2 probe just verified. `--no-install` keeps
# the review execution path identical to the probe; no fresh registry
# fetch needed.
timeout 120s npx --no-install -y tessl skill review {skillDir}
```

Timeout handling mirrors skill-check: exit `124` → `tessl_score: N/A` with reason `timeout-120s`. If the percentage regex (`/(Description|Content|Review Score):\s*(\d+)%/`) returns fewer than three matches, record `tessl_score: N/A` with reason `parse-failure` and include the first 200 chars of output in evidence-report for debugging.

**Registry-404 branch:** if the invocation emits `npm error 404 Not Found` or the npx wrapper exits with a not-found condition, record `tessl_score: N/A` with reason `pin-not-on-registry` and continue. tessl has historically shipped under shifting scope/tag combinations, so a missing registry entry does not HALT the workflow.

**Parse the output** to extract:
- `description_score` — percentage (e.g., 100%)
- `content_score` — percentage (e.g., 45%)
- `review_score` — percentage (e.g., 73%)
- `validation_result` — PASSED/FAILED
- `judge_suggestions[]` — list of improvement suggestions

The tessl output is human-readable text, not JSON. Parse the percentage values from lines like "Description: 100%", "Content: 45%", "Review Score: 73%".

Store in context: `tessl_description_score`, `tessl_content_score`, `tessl_review_score`, `tessl_suggestions`

**If tessl content score < 70%:** Flag a warning:

"**Content quality warning:** tessl scored content at {score}%. This often indicates SKILL.md lacks inline actionable content (e.g., after split-body). If this is a split-body skill, the score drop is expected — tessl evaluates only SKILL.md body, not `references/*.md` (see scoring-rules.md). Consider using selective split to keep actionable content inline."

**If tessl fails entirely:** Record `tessl_score: N/A`, log warning, continue.

### 4. Calculate Combined External Score

The combined external score feeds `externalValidation` into the scoring script (step 5), so its mean is computed by a script, not in-prompt. Both scores are on the same 0-100 scale (skill-check quality score; tessl review percentage). Pass each score, or `null` when its tool did not run or returned N/A (`{externalScoreScript}` resolves relative to the skill root):

```bash
echo '{"skillCheckScore": <score or null>, "tesslReviewScore": <score or null>}' | uv run {externalScoreScript} --stdin
```

Read the result — do not re-average by hand:

- `externalScore` — the combined score: the mean when both tools ran, the single score when one ran, or `null` when neither ran (the scoring step redistributes the external-validation weight on `null`).
- `toolsUsed[]` — the tools that contributed.

Record `external_score: N/A` when `externalScore` is `null`.

### 5. Append External Validation to Output

Append to `{outputFile}`:

```markdown
## External Validation

### skill-check
- **Available:** {yes/no}
- **Quality Score:** {score}/100
- **Errors:** {count}
- **Warnings:** {count}
- **Diagnostics:** {list or "none"}

### tessl
- **Available:** {yes/no}
- **Validation:** {PASSED/FAILED}
- **Description Score:** {score}%
- **Content Score:** {score}%
- **Review Score:** {score}%
- **Suggestions:**
{bulleted list of judge suggestions}

### Combined External Score
- **External Validation Score:** {external_score}%
- **Tools used:** {list of tools that ran}
```

### 6. Report Results

Report the external validation result to the user: the per-tool scores and availability (skill-check out of 100, tessl as a percentage average, each `available` or `skipped`), the combined external score, and a content-quality warning if tessl content is below 70%. Then proceed to scoring.

Update stepsCompleted, then load and execute {nextStepFile}.

---
# `{nextStepFile}` is resolved by probing both candidate roots in order.
# HALT if neither exists — step 6 §7 should have caught this already, but
# this step re-asserts the invariant at dispatch time.
nextStepFileProbeOrder:
  - '{project-root}/_bmad/skf/shared/health-check.md'
  - '{project-root}/src/shared/health-check.md'
---

<!-- Config: communicate in {communication_language}. -->

# Step 7: Workflow Health Check

Probe `{nextStepFileProbeOrder}` in order; load and execute the first path that exists as `{nextStepFile}`, else HALT with a diagnostic naming both candidate paths. This is the terminal step of test-skill.

---
nextStepFile: 'detect-mode.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
templateFile: '{testReportTemplatePath}'
sidecarFile: '{sidecar_path}/forge-tier.yaml'
skillsOutputFolder: '{skills_output_folder}'
# frontmatterScript resolves deterministically by probing two candidate
# paths from `{project-root}` in order. There is NO silent manual fallback —
# if neither candidate exists, the step HALTs with a diagnostic.
frontmatterScriptProbeOrder:
  - '{project-root}/_bmad/skf/shared/scripts/skf-validate-frontmatter.py'
  - '{project-root}/src/shared/scripts/skf-validate-frontmatter.py'
versionPathsKnowledge: 'knowledge/version-paths.md'
---

<!-- Config: communicate in {communication_language}. -->

# Step 1: Initialize Test

## STEP GOAL:

Discover and validate the target skill, load forge tier state to determine analysis depth, and create the test report document from template.

### 1. Receive Skill Path

If skill path was provided as workflow argument, use it directly.

**Recognized flags on the invocation:**
- `--allow-workspace-drift` — bypass the section 5b pre-flight guard that halts when local workspace HEAD does not match `metadata.source_commit`. Store `allow_workspace_drift: true` in workflow context when present. No effect when `source_commit` is unpinned or the source is not a git working tree.
- `--no-discovery` — skip the §4b Discovery Testing block in step 6 (report). Store `no_discovery: true` in workflow context when present.
- `--no-health-check` — skip the §7 health-check dispatch in step 6 (report). Store `no_health_check: true` in workflow context when present.
- `--tier=<Quick|Forge|Forge+|Deep>` — bypass the §4 forge-tier.yaml sidecar HALT. Store `tier_flag: '<value>'` in workflow context when present; §4 will set `detected_tier` directly from this value and skip the sidecar probe.
- `--threshold=<N>` — override the pass threshold for this run. Consumed by `references/score.md` §1; CLI wins over per-pipeline defaults (§1b) and the `workflow.default_threshold` scalar.

If no path provided, ask:

"**Which skill would you like to test?**

Provide the skill path or name. I'll search in `{skillsOutputFolder}`.

**Path or name:**"

### 1b. Resolve Per-Pipeline Quality Threshold

If `{pipeline_alias}` is set in the workflow data context (forwarded by the forger when TS runs inside a pipeline — see `shared/references/pipeline-contracts.md` Pipeline State), look up the alias in the per-pipeline threshold defaults table:

| Pipeline Alias | Default Threshold |
|----------------|-------------------|
| `forge-auto`   | 90                |
| `forge`        | 80                |
| `forge-quick`  | 80                |
| `campaign`     | 90                |

- **If `{pipeline_alias}` is present AND found in the table:** store the corresponding value as `{pipeline_default_threshold}` in workflow context. This variable is consumed by `references/score.md` §1 as a precedence layer between CLI `--threshold` and `{defaultThreshold}`.
- **If `{pipeline_alias}` is present but NOT in the table:** `{pipeline_default_threshold}` remains unset. Score.md falls through to `{defaultThreshold}`.
- **If `{pipeline_alias}` is absent** (standalone TS invocation, not running inside a pipeline): `{pipeline_default_threshold}` remains unset. Score.md falls through to `{defaultThreshold}`.

### 2. Validate Skill Exists (version-aware)

Resolve the skill path using version-aware resolution (see `{versionPathsKnowledge}`):

1. Read `{skillsOutputFolder}/.export-manifest.json` and look up the skill name in `exports` to get `active_version`
2. **Manifest-lag guard.** If the skill is in the manifest, also read the `active` symlink target at `{skillsOutputFolder}/{skill_name}/active`. If that symlink resolves to a *different* version than `active_version`, prefer the **symlink target** as `{resolved_version}` and emit an Info note: "manifest active_version {M} lags the active symlink {N} — testing the symlink target (the just-forged version); run export-skill to reconcile the manifest." This is the canonical SS→TS→EX case: create-stack-skill flipped `active` to the new version, but the manifest only advances when export-skill runs — so a bare manifest-first resolution would test the *previously exported* version and report a PASS for the wrong version (silent false confidence). When the symlink matches `active_version` (or no `active` symlink exists), use `active_version`. See `{versionPathsKnowledge}` "Reading Workflows".
3. If found: resolve to `{skill_package}` = `{skillsOutputFolder}/{skill_name}/{resolved_version}/{skill_name}/`
4. If not in manifest: check for `active` symlink at `{skillsOutputFolder}/{skill_name}/active` — resolve to `{skill_group}/active/{skill_name}/`
5. If neither: fall back to flat path `{skillsOutputFolder}/{skill_name}/`. If SKILL.md exists at the flat path, auto-migrate per `{versionPathsKnowledge}` migration rules
6. Store the resolved path as `{resolved_skill_package}`

Check that the skill package contains required files:

**Required files:**
- `{resolved_skill_package}/SKILL.md` — the skill documentation
- `{resolved_skill_package}/metadata.json` — skill metadata

**If SKILL.md missing:**
"**Error: SKILL.md not found at `{resolved_skill_package}/SKILL.md`**

This skill has not been created yet. Run the **create-skill** workflow first."

**Headless envelope (if `{headless_mode}`):** emit to **stderr**:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"target-inaccessible"}
```

HALT — do not proceed.

**If metadata.json missing:**
"**Warning:** metadata.json not found. Proceeding with limited metadata. Some checks may be skipped."

### 3. Validate Frontmatter Compliance

**3a. Resolve `{frontmatterScript}` deterministically.** Probe each candidate path in `{frontmatterScriptProbeOrder}` (in order) against the filesystem:

1. `{project-root}/_bmad/skf/shared/scripts/skf-validate-frontmatter.py` (installed module layout)
2. `{project-root}/src/shared/scripts/skf-validate-frontmatter.py` (development-tree layout)

Use the first path that exists as `{frontmatterScript}`. There is no manual fallback.

**If neither path exists, HALT** with the diagnostic below. test-skill is a quality gate; without the deterministic validator it cannot produce a trustworthy frontmatter verdict, and a silent manual check can miss subtle spec drift. The missing helper must be restored before testing continues:

```
Error: cannot locate skf-validate-frontmatter.py at either of:
  - {project-root}/_bmad/skf/shared/scripts/skf-validate-frontmatter.py
  - {project-root}/src/shared/scripts/skf-validate-frontmatter.py

test-skill requires the deterministic frontmatter validator. Install the
SKF module (`skf init`) or run from a development checkout with src/ present.
```

Do not proceed. No partial test report is written.

**3b. Python runtime probe (before invoking the validator).** Confirm both `python3` and `uv` are on `$PATH` (`command -v python3` and `command -v uv`). Both are required: `uv run` shells through to `python3` and honors the script's PEP 723 PyYAML dependency declaration that bare `python3` ignores (bare `python3` fails with `ModuleNotFoundError: No module named 'yaml'` on a fresh interpreter — `docs/getting-started.md` documents uv as the runtime prereq for exactly this). If either is missing, set `analysis_confidence: degraded` in workflow context and carry a **score cap** into step 5: `capped_score = threshold - 1` → forces auto-FAIL until the runtime is restored. Record the reason in evidence-report and the test report frontmatter (`analysisConfidence: degraded`, `toolingStatus: python3-missing` or `uv-missing` as appropriate). `uv` is a documented runtime prerequisite — see `docs/getting-started.md` for install instructions.

**3c. Run the validator (30s timeout — the deterministic validator should finish in <1s; the cap only guards against runaway python).**

```bash
timeout 30s uv run {frontmatterScript} {resolved_skill_package}/SKILL.md --skill-dir-name {skill_name}
```

If the command trips the 30s wall-clock (exit code `124`), set
`analysis_confidence: degraded` and `toolingStatus: frontmatter-validator-timeout`
in workflow context, apply the step 5 tooling-degraded cap (score capped at
`threshold - 1` → auto-FAIL), and record the reason in evidence-report.

Parse the JSON output. Treat each `status` value explicitly:

- `status: "pass"` — continue silently.
- `status: "warn"` — display the warning below, log each issue as a pre-check finding, and continue with testing. Frontmatter issues surface in the gap report alongside coverage/coherence findings.
- `status: "fail"` — **HALT with auto-FAIL.** Frontmatter failure means the skill will be rejected by `npx skills add` and `npx skill-check check`; shipping it would produce a false PASS downstream. Write the halt note into evidence-report and exit non-zero. **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting. The output document does not exist yet (created in §6), so `report_path` is `null` — matching the other pre-report init HALTs (target-inaccessible, forge-tier-missing, workspace-drift, another-run-active). A frontmatter-invalid target is the most common failure this gate exists to catch, so a headless orchestrator must be able to branch on it (route to update-skill) rather than see an unlabelled non-zero exit:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"frontmatter-invalid"}
```

```
**Warning/Error: SKILL.md frontmatter is non-compliant with agentskills.io specification.**

{list issues from the JSON output}

This skill will fail `npx skills add` and `npx skill-check check`. {If warn:} Consider fixing frontmatter before proceeding (run `npx skill-check check <skill-dir> --fix` to auto-fix deterministic issues). {If fail:} test-skill cannot proceed — halt and repair frontmatter, then re-run.
```

### 4. Load Forge Tier State

**`--tier=<...>` flag bypass (precedes the sidecar probe).** If `tier_flag` is set in workflow context (from §1's `--tier=<Quick|Forge|Forge+|Deep>` flag), validate the value against the allowed set. On valid match: set `detected_tier` directly to the flag's value, leave `ast_grep`/`gh_cli`/`qmd` availability flags unset (downstream steps treat unset as "unknown" — analysis proceeds without tool-specific enrichment), log Info note "tier — supplied via --tier flag, sidecar bypassed", and SKIP the sidecar probe and HALT below (jump straight to §4b "Apply Tier Override"). On invalid value (not one of the four), HALT with "Error: --tier=<value> is not one of Quick, Forge, Forge+, Deep".

**Otherwise (no `--tier` flag):** Read `{sidecarFile}` to determine available analysis depth.

**If forge-tier.yaml exists:**
- Read `tier` value (Quick, Forge, Forge+, or Deep)
- Read tool availability flags (ast_grep, gh_cli, qmd)

**If forge-tier.yaml missing:**
"**Cannot proceed.** forge-tier.yaml not found at `{sidecarFile}`. Please run the **setup** workflow first to configure your forge tier (Quick/Forge/Forge+/Deep), or re-run with `--tier=<Quick|Forge|Forge+|Deep>` to bypass the sidecar."

**Headless envelope (if `{headless_mode}`):** emit to **stderr**:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"forge-tier-missing"}
```

HALT — do not proceed.

### 4b. Apply Tier Override (if set)

Read `{sidecar_path}/preferences.yaml`. If `tier_override` is set and is a valid tier value (Quick, Forge, Forge+, or Deep), update `detected_tier` to the override value for use in subsequent steps and output documents.

### 5. Load Skill Metadata

Read `metadata.json` to extract:
- `name` — display name
- `skill_type` — single or stack (needed for mode detection)
- `source_path` — path to source code (if present)
- `source_commit` — pinned commit the skill was extracted against (may be null for docs-only skills, `"local"` for non-git sources, or a per-repo map for stack skills)
- `source_ref` — pinned ref (tag/branch/`HEAD`) used at extraction time
- `generation_date` — when skill was generated
- `confidence_tier` — tier used during creation

If source path override was provided as optional input, use that instead.

### 5b. Verify Workspace HEAD Matches Pinned Commit

Test-skill reads `source_path` during coverage and coherence analysis. If the local workspace has drifted from `metadata.source_commit`, gap and signature-mismatch findings will silently reflect the drifted tree, not the skill's pinned source — producing false positives that downstream update-skill runs may then "repair" by corrupting correct documentation.

- Resolve `pinned_commit` from `metadata.source_commit`.
- **If `pinned_commit` is null, empty, or `"local"`:** skip the guard; log `workspace_drift_check: skipped (no pinned commit)` and continue to section 6.
- **If `pinned_commit` is a per-repo map (stack skills):** iterate each `{repo_path: commit}` entry — for each repo run `git -C "{repo_path}" rev-parse HEAD` and compare to its pinned commit (accept full-SHA or short-SHA-prefix match). If ANY repo diverges and the user did not pass `--allow-workspace-drift`, HALT with exit status `workspace-drift` listing every mismatched repo (in `{headless_mode}`, emit the same `workspace-drift` stderr envelope shown in the single-tree branch below before halting). On all-match: log `workspace_drift_check: ok (stack, {N} repos verified)` and continue to section 6. This guard must iterate every repo — do not skip stack skills.
- **If `source_path` is not a git working tree** (bare checkout, tarball extract, docs-only source) — detect by `git -C "{source_path}" rev-parse --is-inside-work-tree`, non-zero exit means skip: log `workspace_drift_check: skipped (not a git working tree)` and continue to section 6.
- **Otherwise** run `git -C "{source_path}" rev-parse HEAD` and compare to `pinned_commit`. Accept full-SHA or short-SHA-prefix match (stored pins are often 8-char short hashes — see `src/knowledge/provenance-tracking.md`).
  - **On match:** log `workspace_drift_check: ok ({short_sha})` and continue.
  - **On mismatch, AND the user did not pass `--allow-workspace-drift`:** HALT with exit status `workspace-drift`. Display:

    ```
    Workspace HEAD does not match the commit this skill was pinned against.

      pinned (metadata.source_commit): {pinned_commit}
      pinned ref (metadata.source_ref): {source_ref or "unset"}
      workspace HEAD ({source_path}):  {head_sha}

    Test-skill verifies against the source the skill was extracted from.
    Testing against a drifted tree produces false gaps/mismatches. Re-sync:

      git -C "{source_path}" checkout {source_ref or pinned_commit}

    Or re-run test-skill with `--allow-workspace-drift` to test against the
    current workspace (accepts that findings reflect HEAD, not the pin).
    ```

    **Headless envelope (if `{headless_mode}`):** emit to **stderr**:

    ```
    SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"workspace-drift"}
    ```

    Do not proceed. The test report has not been created; no partial writes.
  - **On mismatch WITH `--allow-workspace-drift`:** log `workspace_drift_check: overridden (pinned={pinned_commit}, head={head_sha})`, carry the warning into the final report frontmatter (`workspaceDrift: overridden`), and set `allow_workspace_drift: true` in workflow context (consumed by step 5 §5 drift override — a PASS under drift is demoted to `pass-with-drift` and `nextWorkflow` is forced to `update-skill`, never `export-skill`). Continue.

### 6. Create Output Document

**6a. Generate `{run_id}`**: a per-run identifier of the form `{YYYYMMDDTHHmmssZ}-{pid}-{rand4}` (UTC timestamp + process PID + 4-char random hex). Store in workflow context. All per-run artifacts in this and subsequent steps must carry this suffix; step 6 verifies `testDate` in the resulting report matches the run's stamp and fail-fast otherwise.

**6b. Acquire the per-skill test lock**: `flock {forge_version}/.test-skill.lock` for the duration of this run to serialize concurrent `skf-test-skill` invocations against the same skill. If the lock is already held by another run, HALT with "another test-skill run is active for {skill_name}". **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"another-run-active"}
```

**6c. Create `{outputFile}` from `{templateFile}`** — use `{forge_version}/test-report-{skill_name}-{run_id}.md` Initial frontmatter:

```yaml
---
workflowType: 'test-skill'
skillName: '{skill_name}'
skillDir: '{skill_path}'
runId: '{run_id}'
testMode: ''
forgeTier: '{detected_tier}'
testResult: ''
score: ''
threshold: ''
analysisConfidence: '{full|degraded}'
toolingStatus: '{ok|python3-missing|uv-missing|frontmatter-validator-missing|frontmatter-validator-timeout}'
workspaceDrift: '{not-checked|ok|overridden}'
testDate: '{run_id timestamp ISO-8601 UTC}'
stepsCompleted: ['init']
nextWorkflow: ''
---
```

### 7. Report Initialization Status

Report initialization to the user: the resolved skill name, path, type, forge tier, and source path. Then proceed to mode detection.

Update stepsCompleted, then load and execute {nextStepFile}.

<!-- Config: communicate in {communication_language}. -->

# Migration & Deprecation Section Rules (§2b / §5b)

> **Single source of truth.** Both `coherence-check.md` §2b (naive path)
> and §5b (contextual path) apply the rules in this file verbatim. Update this
> file — not the step file sections — when the rule set changes.

## Gate Check

Execute this check only when both conditions are met:

1. Forge tier is **Deep** (tool-gated)
2. `{forge_data_folder}/{skill_name}/evidence-report.md` exists (data-gated)

If either condition fails, skip silently and proceed to the next section.

The check runs regardless of naive/contextual mode. T2-future annotations are a
property of the source code and enrichment data, not the skill type.

## Scope of Section 4b (Authoring Rule This Gate Enforces)

Section 4b (SKILL.md "Migration & Deprecation Warnings") is scoped to
*forward-looking* breaking changes only — what T2-future annotations capture.
Current-state signature gotchas (e.g. "this function is sync not async") belong
alongside the function in Full API Reference, **not here**.

This scoping is authoritative per `skf-create-skill/assets/skill-sections.md`
("Section 4b (Migration & Deprecation Warnings) is conditional: only emitted
for Deep tier when T2-future annotations exist").

Two legitimate exceptions for the `T2-future = 0 AND Section 4b present` case
are formalized in the rules below:

- **(a) historical migration** content — past, shipped package renames or
  consolidated import paths that remain load-bearing for correcting model
  training-data drift → Info severity, no justification required.
- **(b) other non-migration content** — reviewer may downgrade to Low with
  inline justification.

Do not relax the gate otherwise — that would desync the test workflow from the
authoring rule.

## Case Rules

Check whether SKILL.md contains a "Migration & Deprecation Warnings" section
(Section 4b). Then parse `evidence-report.md`'s **YAML frontmatter** for the
pinned `t2_future_count` field — this is the authoritative count, not the
narrative body.

**Detection contract.** Read the frontmatter deterministically:

```bash
# Extract t2_future_count from frontmatter. Requires a `---` delimiter pair.
awk '/^---$/{c++;next} c==1 && /^t2_future_count:/{print $2; exit}' \
    {forge_data_folder}/{skill_name}/evidence-report.md
```

- **Frontmatter missing OR `t2_future_count` absent** → treat as Case 4 (see
  below) and skip silently. Do not fall back to grepping prose (`grep "T2-future"`) —
  prose drift (heading renames, alt phrasings like "forward-looking
  annotations", capitalization variance) silently breaks detection and can
  invert Case-1 vs Case-2/3 severity.
- **`t2_future_count` parsed** → use its integer value for the Case Rules
  below.

The pinned field is emitted by `skf-create-skill/references/compile.md`
§7 (frontmatter-pinned fields), which always writes `t2_future_count: N`
(including 0). Legacy skills whose `evidence-report.md` predates the pinned
field land in Case 4.

### Case 1 — T2-future > 0 AND Section 4b absent

Flag as **Medium** severity gap:

> "Migration section missing — T2-future annotations exist but Section 4b is
> not present in SKILL.md Tier 1."

### Case 2 — T2-future = 0 AND Section 4b present AND content is historical migration

Flag as **Info** severity (not Medium). Historical migration content covers
completed package renames (e.g. `@oldscope/*` → `@newscope/*`), consolidated
import paths, and shipped API cutovers that still surface in training-data
drift — load-bearing for correcting model knowledge even though no
forward-looking change is pending.

Recognizable patterns: old-name → new-name rewrites, citations to
already-shipped PRs/issues, "migrated in version N" or "consolidated from X to
Y" language.

Recommend in the gap report that a future skill revision rename Section 4b to
"Import Corrections" or "Ecosystem Notes" to free the Migration & Deprecation
heading for its forward-looking contract. No inline justification required —
the historical-migration classification is itself the rationale.

### Case 3 — T2-future = 0 AND Section 4b present AND content is non-migration

Flag as **Medium** severity gap:

> "Migration section unexpected — Section 4b contains non-migration content and
> no T2-future annotations were produced."

Reviewer may downgrade to Low with inline justification on a case-by-case
basis.

### Case 4 — evidence-report.md unavailable

Skip silently. Record the note:

> "Section 4b verification skipped — evidence-report.md not found."

## Output

Append any resulting finding(s) to the coherence analysis results. Both the
naive path (§2b) and the contextual path (§5b) funnel findings into the same
Coherence Analysis section of `{outputFile}`.

---
nextStepFile: 'health-check.md'

outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
scoringRulesFile: '{scoringRulesPath}'
outputFormatsFile: '{outputFormatsPath}'
# outputContractSchema and healthCheck resolve relative to the SKF module root
# (`{project-root}/_bmad/skf/` when installed, `{project-root}/src/` during
# development), NOT relative to this step file. Both paths are probed in
# order; HALT if neither exists.
outputContractSchema: 'shared/references/output-contract-schema.md'
healthCheckProbeOrder:
  - '{project-root}/_bmad/skf/shared/health-check.md'
  - '{project-root}/src/shared/health-check.md'
atomicWriteProbeOrder:
  - '{project-root}/_bmad/skf/shared/scripts/skf-atomic-write.py'
  - '{project-root}/src/shared/scripts/skf-atomic-write.py'
---

<!-- Config: communicate in {communication_language}. Test report prose in {document_output_language}. -->

# Step 6: Gap Report

## STEP GOAL:

Generate a detailed gap report listing every issue found during coverage and coherence analysis, assign severity to each gap, provide specific actionable remediation suggestions, and finalize the test report document. Do not recalculate scores — that ran in step 5. This step chains to the local health-check step via `{nextStepFile}` after completion; the user-facing report is not the terminal step.

### 1. Collect All Issues

Read `{outputFile}` and extract every issue found across all analysis sections:

**From Coverage Analysis (step 03):**
- Missing documentation (exports in source but not in SKILL.md)
- Signature mismatches (documented signature differs from source)
- Stale documentation (documented but no longer in source)
- Type coverage gaps (undocumented types/interfaces)

**From Coherence Analysis (step 04):**
- Broken references (file paths, skill references, type imports that don't resolve)
- Incomplete integration patterns (contextual mode)
- Structural issues (naive mode — missing sections, broken examples)

**From External Validation (step 04b):**
- skill-check diagnostics (unresolved errors and warnings)
- tessl judge suggestions (content quality and actionability improvements)

### 2. Load Severity Rules

Load the **Gap Severity** table from `{scoringRulesFile}` — it is the single source of truth for classifying every gap (Critical / High / Medium / Low / Info). Classify each issue directly against that table in §3; do not restate its rows here, so the table cannot drift between this step and the scoring step that already consumes it.

### 3. Classify and Order Gaps

Load `{outputFormatsFile}` for gap entry format and remediation quality rules.

For each issue, assign severity from `{scoringRulesFile}` and generate a specific remediation following the quality rules in `{outputFormatsFile}`. Remediation suggestions must reference specific files, exports, and line numbers. Order gaps by severity: Critical → High → Medium → Low → Info.

### 4. Generate Remediation Summary and Append Gap Report

Load the Gap Report section format from `{outputFormatsFile}`. Count gaps by severity, estimate effort per the guidelines in `{outputFormatsFile}`, and append the complete **Gap Report** section to `{outputFile}`.

If no gaps found, append a clean pass message recommending **export-skill** workflow.

### 4b. Discovery Testing

**`--no-discovery` flag bypass (precedes the precondition check).** If `no_discovery: true` is set in workflow context (from §1 of `init.md` — `--no-discovery` flag on invocation), record an Info-severity note in the Discovery Quality subsection: `discovery — skipped: --no-discovery flag set`, log the bypass, and SKIP §4b.1–§4b.3. Proceed to §4b.4 (description optimization) only if tessl/skill-check flagged description issues; otherwise skip directly to §4c.

After gap enumeration, perform minimum-viable discovery testing. This is a **Medium-weight** check contributing to the Discovery Quality subsection.

**4b.0 Precondition — catalog size check:**

Count the skills in `{skillsOutputFolder}`: `ls -1d {skillsOutputFolder}/*/ 2>/dev/null | wc -l` (directories only — each skill package lives at `{skillsOutputFolder}/<skill-name>/`).

- If `catalog_size < 2`: **skip §4b.1–§4b.3**. Record an Info-severity note in the Discovery Quality subsection: `discovery — skipped: catalog size N={catalog_size}, requires ≥2 candidates for meaningful routing`. The routing test is vacuous with one candidate (any prompt returns the sole skill); reporting `3/3 PASS` under those conditions inflates the Discovery score and masks genuinely bad description triggers. Proceed to §4b.4 (description optimization) if tessl/skill-check flagged description issues; otherwise skip to §4c.
- If `catalog_size >= 2`: continue with §4b.1 as written.

Optional escape hatch: the workflow accepts `--discovery-catalog=all` to broaden the candidate pool to `{project-root}/.claude/skills/` or `{project-root}/_bmad/agents/` for single-skill repos where the repo-local catalog is trivially too small. When the flag is set, rebuild `catalog_size` from the broader pool before the precondition check.

**4b.1 Extract realistic prompts from the skill under test:**

Parse SKILL.md for the three most "organic" prompts found in its `description`, `Triggers`, or example sections. Prefer prompts that:
- Use conversational phrasing (contractions, casual language, implicit context)
- Omit the skill name or explicit command invocation
- Reflect how a user would actually ask for this capability

If SKILL.md does not contain enough organic examples, synthesize 3 from the skill's exports/capability summary using the patterns from §4b.4 below.

**4b.2 Spawn a discovery subagent:**

**Subagents-unavailable guard (precedes the spawn).** If subagents cannot be spawned in this environment (e.g. a headless/CI pipeline with no subagent capability), do not fall back to answering the routing in the main thread — the main thread knows which skill is under test, so it would self-route to `3/3 PASS` and inflate the Discovery score, the exact false confidence §4b.0 warns against. Instead record an Info-severity note in the Discovery Quality subsection: `discovery — skipped: subagents unavailable, routing test requires isolated context`, exclude the discovery check from Discovery Quality scoring (do not count it PASS or FAIL), and skip to §4c.

For each of the 3 prompts, spawn an isolated subagent with NO prior context about which skill is under test. Provide only:
1. A compact list of ALL skills available in `{skillsOutputFolder}` (name + description line from each skill's SKILL.md frontmatter)
2. The prompt text

Instruction to the subagent:

> "You are an agent selecting the best skill to handle a user request. Here is the catalog: {catalog}. The user says: '{prompt}'. Return JSON: `{\"selected_skill\": \"<name>\", \"confidence\": \"<high|medium|low>\", \"reasoning\": \"<one sentence>\"}`. If no skill fits, return `{\"selected_skill\": null, ...}`. Return only JSON."

**4b.3 Evaluate discovery results:**

Parse the 3 responses. Schema-validate (required: `selected_skill`, `confidence`, `reasoning`). On any parse/schema failure, record the prompt as `discovery_result: error` and continue.

For each prompt, PASS = `selected_skill == skill_name` (the skill under test), FAIL otherwise.

- **3 of 3 PASS** → discovery check PASS (Info severity note, no gap)
- **2 of 3 PASS** → discovery check WARN → **Medium**-severity gap: `discovery — 1/3 realistic prompts misrouted`
- **≤1 of 3 PASS** → discovery check FAIL → **High**-severity gap: `discovery — {N}/3 realistic prompts misrouted; description triggers are not pulling the skill`

Append the prompts, selected skills, and outcomes as a table in the Discovery Quality subsection.

**4b.4 Description optimization (secondary):** If tessl `description_score` (from step 04b) is below 90%, or skill-check flagged description issues, add remediation hints to the Discovery Quality subsection:
- Third-person voice check
- Explicit trigger keywords matching real user phrasing
- Negative triggers ("NOT for: ...") to prevent false positives
- Alternative skill references for excluded use cases

Realistic prompt patterns for synthesis (§4b.1 fallback):
- Vague: "can you help me with this {artifact} my boss sent"
- Implicit: "why did {metric} drop last {period}"
- Abbreviated: "run the {keyword} thing on this data"

### 4c. Result Contract (atomic write)

**Resolve `{atomicWriteHelper}`:** probe `{atomicWriteProbeOrder}`. HALT if neither candidate exists — the contract is a downstream-consumer protocol and must never be written non-atomically. **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"atomic-writer-missing"}
```

Write the result contract per `{outputContractSchema}`:
- Per-run record: `{forge_version}/skf-test-skill-result-{run_id}.json` (the `{run_id}` set in step 1 §6a — already carries UTC timestamp + PID + random suffix, so no same-second collision).
- Latest copy: `{forge_version}/skf-test-skill-result-latest.json` (stable path for pipeline consumers — copy, not symlink).

Both writes must go through the atomic writer so partial writes are never observable:

```bash
# Build the JSON payload in memory, then:
cat payload.json | python3 {atomicWriteHelper} write --target {forge_version}/skf-test-skill-result-{run_id}.json
cat payload.json | python3 {atomicWriteHelper} write --target {forge_version}/skf-test-skill-result-latest.json
```

Payload contents:
- `outputs[]` — include the test report path at `{outputFile}` with its `{run_id}` suffix
- `summary` — `score`, `threshold`, `result` (`"PASS"`, `"PASS_WITH_DRIFT"`, `"FAIL"`, or **`"INCONCLUSIVE"`**), `testMode` (naive/contextual), `activeCategories[]`, `inconclusiveReasons[]` (when present). `PASS_WITH_DRIFT` is set when the workflow observed workspace drift and the user passed `--allow-workspace-drift` — see step 5 §5 drift override. Downstream consumers must treat `PASS_WITH_DRIFT` as a non-exportable result: re-run against the pinned commit before export. When threshold fallback occurred, add `threshold_fallback: true`, `original_threshold: {N}`, and `evidence_report_path: '{path}'` to the summary — these fields are absent (not `false`/`null`) when no fallback occurred.
- `runId` — the workflow's `{run_id}` for downstream correlation
- `healthCheckDispatched` — boolean, set by §7 after the dispatch decision

The `{forge_version}/.test-skill.lock` acquired in step 1 §6b remains held until the end of this step — it guards against concurrent latest-file overwrites.

**Post-finalization hook.** If `{onCompleteCommand}` (resolved in SKILL.md On Activation §3 from `workflow.on_complete` scalar) is non-empty, invoke it as:

```bash
{onCompleteCommand} --result-path={forge_version}/skf-test-skill-result-{run_id}.json
```

Run it with a bounded timeout (default 60s). On success: log Info note "on_complete — invoked: {command}" and continue. On non-zero exit, timeout, or any failure: append the failure reason to `workflow_warnings[]` (e.g. `on_complete — failed (exit {N}): {stderr_first_line}`) and continue. **The hook must never fail the workflow** — its purpose is integration glue (notify a CI router, post to a queue, archive the result) and any failure there is orthogonal to the test verdict. If `{onCompleteCommand}` is empty, this hook is a no-op (no log entry needed).

### 5. Finalize Output Document — Enforce Step Completeness

**Incremental step tracking:** read `stepsCompleted` from the output frontmatter. The expected set is the canonical chain:

```
['init',
 'detect-mode',
 'coverage-check',
 'coherence-check',
 'external-validators',
 'hard-gate',
 'score',
 'report']
```

If any expected entry is missing, HALT with "step completeness violation — missing {list}; workflow state is inconsistent, do not finalize the report". **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"step-completeness-violation"}
```

Only append `'report'` and write back after the check passes.

**Section anchor presence check (companion to stepsCompleted).** The
report template ships six canonical H2 anchors — one per populating step. An
off-sequence run (e.g. a step wrote its section into the wrong anchor, or a
subagent truncated the file) can leave `stepsCompleted` intact while a section
is missing. `grep -n` each anchor below against `{outputFile}`; each must
return ≥1 match:

```
^## Test Summary$
^## Coverage Analysis$
^## Coherence Analysis$
^## External Validation$
^## Completeness Score$
^## Gap Report$
```

On any miss, HALT with "report anchor missing: {anchor} — section was not
appended by its owning step". **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"report-anchor-missing"}
```

Do not append `'report'` and do not
write the result contract. The template at `{testReportTemplatePath}`
declares these anchors as TBD placeholders; a miss means a step silently
skipped its append.

**INCONCLUSIVE as gate:** if `testResult == 'inconclusive'` (from step 5), the report final presentation (§6) and result contract (§4c) have already been written with that verdict. Do not auto-map INCONCLUSIVE to PASS or FAIL. Recommend `manual-review`. The step must still complete (health-check runs unconditionally) — INCONCLUSIVE is a report-time signal, not a workflow abort.

### 6. Present Final Report

"**Test complete for {skill_name}.**

---

**Result:** **{PASS|PASS_WITH_DRIFT|FAIL|INCONCLUSIVE}** — **{score}%** (threshold: {threshold}%)

{If `thresholdFallback` is present in output frontmatter:}
**Threshold fallback:** scored {score}% against {originalThreshold} target — accepted at 80% floor. Evidence report: {evidenceReportPath}

**Gaps Found:** {total_gaps}
- Critical: {N}
- High: {N}
- Medium: {N}
- Low: {N}
- Info: {N}

**Report saved to:** `{outputFile}`

---

**Recommended next step:**

{IF PASS:}
**export-skill** — This skill is ready for export. Run the export-skill workflow to package it for distribution.

{IF PASS_WITH_DRIFT:}
**update-skill** — The skill scored above threshold, but `--allow-workspace-drift` was in effect: the test ran against workspace HEAD, not `metadata.source_commit`. A conditional PASS is not trustworthy enough to export. Re-sync the source tree to the pinned commit (or re-extract against current HEAD) and re-run test-skill without the drift override before exporting.

{IF FAIL:}
**update-skill** — This skill needs remediation. Review the gap report above and run the update-skill workflow to address the {N} blocking issues (Critical + High).

{IF INCONCLUSIVE:}
**manual-review** — The evidence base was too thin to grade automatically. See `inconclusiveReasons` in the Completeness Score section. Typical fixes: upgrade forge tier, enable external validators, or re-extract with a wider scope. Do not export.

---

**See Discovery Quality section in the report for description optimization and realistic prompt testing recommendations.**

**Test report finalized.**"

### 6b. Determine Headless Exit Code

This step only determines the terminal exit code — it does not exit. Both modes then reach §7 (headless auto-proceeds past the menu; non-headless goes through the [C] menu), and the terminal process-exit with this code happens in §7 after the health-check dispatch.

If `{headless_mode}`, map `testResult` to the code the workflow will exit with in §7 and store it as `{headless_exit_code}` in workflow context:
- `testResult: 'pass'` → exit code 0
- `testResult: 'pass-with-drift'` → exit code 4 (distinct from clean pass — see the pass-with-drift row in SKILL.md Exit Codes; exiting 0 under a drift override would wrongly signal a clean pass)
- `testResult: 'fail'` → exit code 2 (the result contract was written in §4c — never exit before it)
- `testResult: 'inconclusive'` → exit code 3 (distinct from fail so orchestrators can route to manual-review queues)

### 6c. Emit Headless Result Envelope (stdout)

If `{headless_mode}`, emit the terminal result envelope to **stdout** as a single line before chaining to §7 — this is the branchable record a headless orchestrator reads for the happy path (PASS / FAIL / INCONCLUSIVE / pass-with-drift). The SKILL.md Result Contract owns the shape and the field rules; the on-disk copy written in §4c is the richer form. Build it from the settled verdict and the values already in the output frontmatter:

```
SKF_TEST_RESULT_JSON: {"status":"success","skill_name":"{skill_name}","verdict":"{PASS|FAIL|INCONCLUSIVE|pass-with-drift}","score":{score},"threshold":{threshold},"report_path":"{outputFile}","next_workflow":{export-skill when PASS | update-skill when FAIL or pass-with-drift | null when INCONCLUSIVE},"exit_code":{headless_exit_code},"halt_reason":null}
```

`verdict` is uppercase for `pass`/`fail`/`inconclusive` (→ `PASS`/`FAIL`/`INCONCLUSIVE`) and the literal `pass-with-drift`. When threshold fallback occurred (frontmatter `thresholdFallback: true`), add `"threshold_fallback":true` and `"original_threshold":{originalThreshold}`; omit both otherwise. Non-headless runs skip this emission — the §6 presentation is their terminal output.

### 7. Health-Check Dispatch + MENU OPTIONS

**`--no-health-check` flag bypass (precedes the health-check resolution).** If `no_health_check: true` is set in workflow context (from §1 of `init.md` — `--no-health-check` flag on invocation), set `health_check_dispatched: false` in the output report frontmatter and mirror `healthCheckDispatched: false` into the result contract written in §4c (re-write atomically via `{atomicWriteHelper}`). Log Info note "health-check — skipped: --no-health-check flag set" and exit the workflow: in `{headless_mode}`, exit with `{headless_exit_code}` (determined in §6b); non-headless, simply terminate after the §6 presentation. Do not resolve `{healthCheckFile}`, do not display the menu, do not chain to `{nextStepFile}`. This flag is the one path where §7 does not dispatch the health-check.

Resolve `{healthCheckFile}`: probe `{healthCheckProbeOrder}` in order. **HALT** if neither candidate exists — the health-check is the true terminal step; without it the workflow cannot complete honestly:

```
Error: cannot locate shared/health-check.md at either of:
  - {project-root}/_bmad/skf/shared/health-check.md
  - {project-root}/src/shared/health-check.md

test-skill delegates its terminal step to the shared health-check. Install
the SKF module or run from a development checkout with src/ present.
```

**Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"health-check-missing"}
```

Before displaying the menu, write the dispatch decision into the output report frontmatter (so the artifact records whether the health-check ran):

- `health_check_dispatched: true` — when the C menu choice will be taken (headless, or user will select C)
- `health_check_dispatched: false` — should be rare (only if operator explicitly skips, e.g. future flag)

Also mirror the boolean into the `healthCheckDispatched` field of the result contract written in §4c (re-write atomically via `{atomicWriteHelper}` if the dispatch decision is made after the initial contract write).

Display: "**Test complete.** [C] Finish"

On [C] (or auto-proceed in `{headless_mode}` — log: "headless: auto-continue past report menu"): set `health_check_dispatched: true` in frontmatter, then load and execute `{nextStepFile}` (the local health-check dispatcher). The test report document at `{outputFile}` contains the full analysis: Test Summary, Coverage Analysis, Coherence Analysis, Completeness Score, and Gap Report. In `{headless_mode}`, once the dispatched health-check completes, the workflow makes its terminal process-exit with `{headless_exit_code}` (determined in §6b) — this is the single terminal exit for the headless happy path.

---
nextStepFile: 'report.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
scoringRulesFile: '{scoringRulesPath}'
sourceAccessProtocol: 'references/source-access-protocol.md'
scoringScript: 'scripts/compute-score.py'
---

<!-- Config: communicate in {communication_language}. -->

# Step 5: Score

## STEP GOAL:

Calculate the overall completeness score by aggregating coverage, coherence, and external validation category scores with the appropriate weight distribution (naive or contextual), apply the pass/fail threshold, and determine the test result.

### 1. Load Scoring Rules

Load `{scoringRulesFile}` to get:
- Category weights (naive vs contextual distribution)
- Tier-dependent scoring adjustments

**Resolve the pass threshold (precedence: CLI > pipeline default > scalar > bundled fallback):**

1. If the workflow received `--threshold=<N>` on invocation, use that integer as `effective_threshold` (CLI wins). Set `threshold_source` = `"CLI override ({N}%)"`.
2. Else if `{pipeline_default_threshold}` is set in workflow context (resolved by init.md §1b from the per-pipeline threshold lookup table when `{pipeline_alias}` is present), use it as `effective_threshold`. Set `threshold_source` = `"pipeline default ({pipeline_alias} → {N}%)"`.
3. Else if the resolved `{defaultThreshold}` workflow-context variable (from SKILL.md On Activation §3 — `workflow.default_threshold` scalar, default `80`) is set, use it as `effective_threshold`. Set `threshold_source` = `"workflow default ({N}%)"`.
4. Else fall back to `80` (the bundled default — this branch should be unreachable when SKILL.md resolution ran correctly, but keeps the step robust if customize.toml resolution failed silently). Set `threshold_source` = `"bundled fallback (80%)"`.

Store `threshold_source` in workflow context for use in the score report section.

Pass `effective_threshold` into the scoring-input JSON's `threshold` field in §3a (the compute-score.py script already honors this field). The CLI flag, pipeline default, and the scalar all feed the same downstream field; the script does not need to know which layer supplied the value.

**Docs-only mode check:** If the Coverage Analysis section in `{outputFile}` notes docs-only mode (set by step 3 for skills with all `[EXT:...]` citations and no local source), apply Quick-tier weight redistribution: Signature Accuracy and Type Coverage are not scored, their weights (22% + 14%) are redistributed proportionally to remaining active categories. Coverage score is based on documentation completeness rather than source coverage (as calculated by step 3).

### 2. Read Category Scores from Output

Read `{outputFile}` and extract the category scores calculated in previous steps:

**From Coverage Analysis (step 03):**
- Export Coverage: {percentage}%
- Signature Accuracy: {percentage}% or N/A (Quick tier)
- Type Coverage: {percentage}% or N/A (Quick tier)

**From Coherence Analysis (step 04):**
- Combined Coherence: {percentage}% (contextual mode only)
- Or: not scored (naive mode — weight redistributed)

**From External Validation (step 04b):**
- External Validation Score: {percentage}% (combined skill-check + tessl average)
- Or: N/A (if neither tool was available — weight redistributed to other categories)

### 3. Apply Weight Distribution

**Read testMode from {outputFile} frontmatter.**

#### 2b. Apply State 2 Undercount Deduction (pre-script)

If `analysis_confidence == 'provenance-map'` (State 2) AND `metadata.json.skill_type != "stack"` AND step 3 recorded a provenance vs metadata divergence > 5% (see §4b in step 3), apply a 10-point deduction to `exportCoverage` BEFORE building the scoring input:

```
exportCoverage_adjusted = max(0, exportCoverage - 10)
```

Record in the report: `scoring_notes: State 2 undercount risk acknowledged — 10% deduction applied to Export Coverage (raw: {N}%, adjusted: {M}%)`. Use the adjusted value as the `exportCoverage` field in §3a below. The deduction is deterministic and does not change category weights or active-category counting.

Stack skills are exempt (the `metadata.json.skill_type != "stack"` guard above; `skill_type` is loaded in step 01 and also surfaces as `stackSkill` in §3a): a stack's own barrel is empty by design, so a provenance-vs-metadata divergence on a stack reflects the barrel-vs-constituent surface difference (suppressed by §4b's stack-skill branch in step 3), not extraction undercount — deducting for it would penalize a correctly built stack.

#### 3a. Construct Scoring Input JSON

Build a JSON object from the data gathered in steps 1-2:

```json
{
  "mode": "{testMode: contextual or naive}",
  "tier": "{forge_tier: Quick, Forge, Forge+, or Deep}",
  "docsOnly": "{true if docs_only_mode detected in step 03, else false}",
  "state2": "{true if analysis_confidence is provenance-map, else false}",
  "stackSkill": "{true if metadata.json.skill_type == 'stack', else false}",
  "referenceApp": "{true if metadata.json.scope_type == 'reference-app', else false}",
  "scores": {
    "exportCoverage": "{export_coverage_percentage}",
    "signatureAccuracy": "{signature_accuracy_percentage or null if N/A}",
    "typeCoverage": "{type_coverage_percentage or null if N/A}",
    "coherence": "{combined_coherence_percentage or null if naive mode}",
    "externalValidation": "{external_validation_score or null if N/A}"
  },
  "threshold": "{effective_threshold from §1 — CLI --threshold wins, then pipeline default, then workflow.default_threshold scalar, then 80}",
  "analysisConfidence": "{resolved analysis confidence — 'degraded' when python3/frontmatter validator missing; else full/provenance-map/metadata-only/remote-only/docs-only; omit if unknown}",
  "toolingStatus": "{a missing-helper marker like 'python3-missing' or 'frontmatter-validator-missing' when a helper is unavailable, else omit}"
}
```

**Important:** Score values must be numbers (not strings). Use `null` (not `"N/A"`) for categories that were not scored. `analysisConfidence` and `toolingStatus` are optional strings — pass them so the script can apply the post-score tooling cap (§3d) deterministically; the script fires Cap 1 when `analysisConfidence == "degraded"` or `toolingStatus` contains `missing`. Read `metadata.json.skill_type` from `{resolved_skill_package}/metadata.json`; if the value is `"stack"`, set `stackSkill: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (the categories will be redistributed per `{scoringRulesFile}` Stack Skills rule). Likewise read `metadata.json.scope_type`; if the value is `"reference-app"`, set `referenceApp: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (redistributed per `{scoringRulesFile}` Reference-App rule — a reference app documents wiring patterns, not library export signatures).

#### 3b. Run the Scoring Script

```bash
echo '<JSON>' | uv run {scoringScript} --stdin
```

Where `{scoringScript}` is the path resolved from the frontmatter variable (relative to the skill root, i.e., the skf-test-skill/ directory). The script also accepts the JSON as a positional argument (`uv run {scoringScript} '<JSON>'`) or via `--json-input '<JSON>'`; `--stdin` is preferred since it avoids shell-quote escaping of nested JSON.

Parse the JSON output. The script returns:
- `weights` — final redistributed weights per category
- `weightedScores` — weighted contribution per category
- `totalScore` — the overall completeness score
- `threshold` — the threshold used
- `result` — `"PASS"`, `"FAIL"`, or **`"INCONCLUSIVE"`** (minimum-evidence floor — see `{scoringRulesFile}`)
- `activeCategories` — list of categories that were scored
- `skippedCategories` — list of categories that were skipped
- `skipReasons` — why each category was skipped
- `weightSum` — sum of final weights (should be ~100)
- `inconclusiveReasons` — only present when `result == "INCONCLUSIVE"`; explains which floor clause tripped
- **Verdict-override group** — present as an atomic set of four keys only when a post-score cap or the threshold fallback engaged (§3d/§4b are applied by the script, not re-derived here):
  - `effectiveResult` — the **final verdict** after caps + fallback (`"PASS"` / `"FAIL"`); read this as the outcome for §4–§8
  - `capReason` — string describing the cap(s) that fired, or `null`
  - `thresholdFallback` — `true` when the FAIL→PASS-at-80-floor fallback fired
  - `originalThreshold` — the pre-fallback threshold when `thresholdFallback` is `true`, else `null`
  - When the group is **absent**, no cap or fallback engaged and `result` is the final verdict.

Use these values for Section 4 (pass/fail/inconclusive) and Section 6 (output formatting). **The script owns the minimum-evidence floor, both post-score caps, and the threshold fallback — everywhere below, read its emitted fields (`result`, `effectiveResult`, `capReason`, `thresholdFallback`) and never recompute a verdict, cap, or threshold decision.** The final verdict is `effectiveResult` when the override group is present, otherwise `result` (INCONCLUSIVE is never overridden).

#### 3c. Fallback (if script execution fails)

**First distinguish "the script could not run" from "the script rejected the input" — they take opposite paths.**

A `{"error": ..., "code": "INVALID_INPUT"}` envelope on stdout means the script ran fine and refused what it was given: a field is missing, mistyped, or out of range (exit 2), or the payload was not parseable JSON at all (exit 1). Either way the *input* is wrong, not the script. **Correct the §3a input and re-run.** Do not fall through to the manual redistribution below — it would hand-compute a total from the very numbers the script just refused, so an out-of-range `exportCoverage` would silently become a score. If the input cannot be corrected, report `score: scoring input rejected — {error}` and leave the score unset rather than emitting a computed total.

The fallback below applies only when the script genuinely could not run — missing file, no `uv`, unreadable interpreter — that is, **no envelope on stdout at all**.

If the script is unavailable, redistribute each skipped category's weight (the `null`-scored categories in the §3a JSON — naive mode already zeroes coherence, and Quick-tier/docsOnly/state2/stackSkill/referenceApp already null out Signature Accuracy + Type Coverage) proportionally across the active categories, then report `total = Σ(weight/100 × category_score)` using the detected mode's weight table in `{scoringRulesFile}`. Report: "**Note:** Scoring script unavailable — calculated manually per scoring-rules.md."

### 3d. Read Post-Score Caps (applied by the script)

The script applies two post-score caps — **Cap 1** (tooling degraded) and **Cap 2** (docs-only with no external validators) — and returns the settled outcome as `capReason` + `effectiveResult` (§3b). Read those fields; never recompute a cap. If `capReason` is non-null, record `scoring_notes: {capReason}` in the report. A fired cap forces the script's `PASS` into `FAIL`, never touches an INCONCLUSIVE verdict, and may still be re-flipped to PASS by the threshold fallback (§4b) — `effectiveResult` reflects that settled outcome. (Both caps exist because a degraded-tooling or docs-only-without-validators run has too thin an evidence base to trust a PASS; §3a passes the fields Cap 1 reads.)

### 4. Determine Result (PASS / FAIL / INCONCLUSIVE)

The scoring script enforces the minimum-evidence floor BEFORE comparing score vs threshold, then applies the post-score caps (§3d) and threshold fallback (§4b). The **settled verdict** is `effectiveResult` when the override group is present, otherwise `result`:

```
IF result == "INCONCLUSIVE" — minimum-evidence floor tripped; not PASS, not FAIL (never overridden)
ELSE settled verdict = effectiveResult if the override group is present, else result   (PASS or FAIL)
```

**INCONCLUSIVE floor clauses** (see `{scoringRulesFile}`):
- `active_categories < 2` (after all redistribution), OR
- `tier == "Quick"` AND Export Coverage is the sole scoring contributor

### 4b. Threshold Fallback and Evidence Report

After §4 determines the result but before §5 recommends the next workflow, the script's threshold fallback may already have converted a FAIL into a PASS at the 80% floor. This step documents that quality compromise in an evidence report.

The rule the script encodes (documented for interpretation): when `result == "FAIL"` AND `totalScore >= 80` AND `effective_threshold > 80`, the script overrides `result` to `"PASS"` at the 80% floor; an `INCONCLUSIVE` verdict is never overridden. Read the script's fields:

- `thresholdFallback == true` → the fallback fired; `effectiveResult` is `"PASS"` and `originalThreshold` holds the pre-fallback threshold.
- `thresholdFallback` absent or `false` → no fallback; the settled verdict from §4 stands.

**When `thresholdFallback` is true:**

1. Record `threshold_fallback: true`, `original_threshold: {originalThreshold}`, `fallback_threshold: 80` in workflow context.
2. Use `effectiveResult` (`"PASS"`) as the settled verdict — do not recompute it.
3. Set `effective_threshold = 80` for use by §5/§6/§7/§8.
4. Generate the evidence report (§4b.1 below).

The cap↔fallback interaction (a cap forcing FAIL that the fallback then re-flips to PASS at the 80 floor) is resolved inside the script — `effectiveResult` reflects the settled outcome. The `{totalScore}` used in the report below is the script's raw `totalScore`.

#### 4b.1 Generate Evidence Report

Write the evidence report to `{forge_version}/evidence-report-fallback.md`. The report documents the quality compromise for audit purposes.

**Read gap entries:** extract findings from the Coverage Analysis and Coherence Analysis sections in `{outputFile}` — list each gap with severity (Critical through Low).

**Check for prior remediation:** glob `{forge_version}/test-report-{skill_name}-*.md` for a prior test report. If found, note the path — this implies remediation was attempted between runs. If not found, note "first test run — no prior remediation cycle".

**Check for post-score cap:** if Cap 1 or Cap 2 (§3d) fired, note the cap reason in the remediation section: "Score capped due to {cap_reason} — address tooling to test at the higher threshold."

**Evidence report template:**

```markdown
# Evidence Report: Threshold Fallback

**Skill:** {skill_name}
**Date:** {ISO-8601 timestamp}
**Run ID:** {run_id}

## Threshold Summary

| Field | Value |
|-------|-------|
| Attempted Threshold | {original_threshold}% |
| Achieved Score | {totalScore}% |
| Threshold Source | {threshold_source} |
| Final Accepted Threshold | 80% |

## Findings Preventing Higher Threshold

{For each gap entry from Coverage Analysis and Coherence Analysis:}
- **{GAP-NNN}: {title}** — Severity: {severity}, Category: {category}

{Count: N critical, M high, P medium, Q low, R info findings}

## Remediation Context

{If prior test report exists:}
A prior test run was found at `{prior_report_path}`, indicating remediation was attempted between runs.

{If no prior test report:}
No prior test report found for this skill version — this is the first test run.

{If post-score cap was active:}
**Note:** Score was capped due to {cap_reason}. Address tooling limitations to test at the higher threshold.

## Conclusion

Skill accepted at 80% floor (original target: {original_threshold}%). The {N} findings above prevented meeting the higher threshold. Review and address findings before the next pipeline run to achieve the {original_threshold}% target.
```

Record `evidence_report_path: '{forge_version}/evidence-report-fallback.md'` in workflow context for use by §6/§7/§8 and by report.md.

### 5. Determine Next Workflow Recommendation

Based on the **settled verdict** (§4 — `effectiveResult` when the override group is present, else `result`; INCONCLUSIVE is never overridden):

**IF PASS:**
- `nextWorkflow: 'export-skill'` — skill is ready for export
- **Drift override:** if workflow context carries
  `allow_workspace_drift: true` (set in step 1 §5b when the user passed
  `--allow-workspace-drift` AND the workspace HEAD did not match
  `metadata.source_commit`), the PASS is a **conditional PASS**:
  - Write `testResult: 'pass-with-drift'` to the output frontmatter instead of
    bare `'pass'`. The result contract (§4c of step 6) mirrors the same
    value.
  - Override `nextWorkflow` to `'update-skill'` — **refuse to recommend
    `export-skill`**. The drift override weakens the workflow's strongest
    false-positive guard (we tested against HEAD, not the pinned source); a
    PASS under drift is not trustworthy enough to promote to export without a
    clean re-test against the pinned commit.
  - Record `scoring_notes: workspace drift overridden — PASS is conditional; re-run against pinned commit before export`.

**IF FAIL:**
- `nextWorkflow: 'update-skill'` — skill needs remediation before export

**IF INCONCLUSIVE:**
- `nextWorkflow: 'manual-review'` — evidence base is insufficient to grade the skill automatically. The test report records `inconclusiveReasons` from the scoring script. Surface to the user — do not auto-recommend export or update.

### 6. Append Completeness Score to Output

Append the **Completeness Score** section to `{outputFile}`:

```markdown
## Completeness Score

### Score Breakdown

| Category | Score | Weight | Weighted |
|----------|-------|--------|----------|
| Export Coverage | {N}% | {W}% | {WS}% |
| Signature Accuracy | {N}% | {W}% | {WS}% |
| Type Coverage | {N}% | {W}% | {WS}% |
| Coherence | {N}% | {W}% | {WS}% |
| External Validation | {N}% | {W}% | {WS}% |
| **Total** | | **100%** | **{total}%** |

### Result

**Score:** {total}%
**Threshold:** {threshold}%
**Result:** **{PASS|FAIL|INCONCLUSIVE}**
{If INCONCLUSIVE:}
**Inconclusive Reasons:**
{bulleted list from script `inconclusiveReasons`}

**Threshold Source:** {threshold_source}
{If threshold_fallback is true:}
**Threshold Fallback:** scored {totalScore}% against {original_threshold}% target — accepted at 80% floor. Evidence report: {evidence_report_path}
**Weight Distribution:** {naive (redistributed) | contextual (full)}
**Tier Adjustment:** {none | Quick tier — signature and type coverage not scored}
**External Validators:** {both available | skill-check only | tessl only | none — weight redistributed}
**Analysis Confidence:** {full | provenance-map | metadata-only | remote-only | docs-only}
```

If `analysis_confidence` is not `full`, append a degradation notice. **The notice must be confidence-aware** — see the degradation notice rules in `{sourceAccessProtocol}`:

```markdown
### Access Degradation Notice

**Resolved via:** {analysis_confidence} {confidence breakdown if provenance-map, e.g., "(T1 AST-verified at compilation time)" or "(12 T1, 3 T1-low)"}
**Impact:** {describe limitation — e.g., "Signature checks limited to name-matching. Source file:line citations from provenance-map, not live AST." — or "Provenance data is at highest confidence; no limitation." for all-T1 provenance-map}
**Recommendation:** {confidence-dependent — see {sourceAccessProtocol} degradation notice rules. Do not recommend local clone when provenance-map entries are already T1.}
```

### 7. Update Output Frontmatter

Update `{outputFile}` frontmatter:
- `testResult: '{pass|pass-with-drift|fail|inconclusive}'` (lowercase; mirrors the **settled verdict** — `effectiveResult` when the override group is present, else `result` — with `pass-with-drift` substituted for `pass` when `allow_workspace_drift` was set and drift was observed — see §5 drift override)
- `score: '{total}%'`
- `threshold: '{threshold}%'`
- `thresholdSource: '{threshold_source}'`
- When `threshold_fallback` is true, add: `thresholdFallback: true`, `originalThreshold: '{original_threshold}%'`, `evidenceReportPath: '{evidence_report_path}'`
- `analysisConfidence: '{full|degraded|provenance-map|metadata-only|remote-only|docs-only}'`
- `nextWorkflow: '{export-skill|update-skill|manual-review}'`
- Append `'score'` to `stepsCompleted`

### 8. Report Score

Report the completeness score to the user: the total percentage and PASS/FAIL verdict, the per-category weighted breakdown (already appended to the report in §6), the threshold, and the recommended next workflow (export-skill on pass, update-skill on fail). When `threshold_fallback` is true, include the fallback notice:

**Threshold fallback:** scored {totalScore}% against {original_threshold}% target — accepted at 80% floor. Evidence report: {evidence_report_path}

Then proceed to the gap report.

Update stepsCompleted, then load and execute {nextStepFile}.

<!-- Config: communicate in {communication_language}. -->

# Scoring Rules

## Default Threshold

**Pass threshold:** 80%

## Category Weights

| Category               | Weight | Description                                                                               |
|------------------------|--------|-------------------------------------------------------------------------------------------|
| Export Coverage        | 36%    | Percentage of source exports documented in SKILL.md                                       |
| Signature Accuracy     | 22%    | Documented signatures match actual source signatures                                      |
| Type Coverage          | 14%    | Types and interfaces referenced are complete                                              |
| Coherence (contextual) | 18%    | Cross-references valid, integration patterns complete                                     |
| Coherence (naive)      | 0%     | Not applicable — weight redistributed to other categories                                 |
| External Validation    | 10%    | Average of skill-check quality score + tessl average score (redistributed if unavailable) |

## Naive Mode Weight Redistribution

The following weights replace the default table for naive mode. The 18% coherence weight from the default table has been proportionally redistributed into these values. Do not re-redistribute for coherence (already handled in this table). Quick-tier redistribution (zeroing Signature Accuracy and Type Coverage) still applies as an additional step.

When running in naive mode (no coherence category):
- Export Coverage: 45%
- Signature Accuracy: 25%
- Type Coverage: 20%
- External Validation: 10%

## External Validation Unavailable

When neither skill-check nor tessl is available, redistribute the 10% external validation weight proportionally to the other active categories. When only one tool is available, use that tool's score as the external validation score.

## tessl and Split-Body Interaction

tessl evaluates SKILL.md body content only — it does not read `references/*.md` files. After split-body extraction, the tessl content score will drop significantly (e.g., 65% to 38%) because Tier 2 content is no longer inline. This is expected behavior and does not reflect actual content quality. When reporting scores for a split-body skill, note: "tessl content score reflects post-split inline content only. Use the pre-split tessl score as the content quality baseline."

## Tier-Dependent Scoring

### Quick Tier (no tools)
- Export Coverage: file/structure existence check only
- Signature Accuracy: skipped (no AST)
- Type Coverage: skipped (no AST)
- Score based on: structural completeness only
- Weight redistribution: skipped categories' weights (Signature Accuracy 22% + Type Coverage 14%) redistributed proportionally to remaining active categories

### Docs-Only Mode (all [EXT:...] citations, any tier)

When `docs_only_mode: true` is set by step 3 (indicating a skill where all SKILL.md citations are `[EXT:...]` format with no local source code):

- **Signature Accuracy:** Not scored (no source to compare against)
- **Type Coverage:** Not scored (no source to compare against)
- **Weight redistribution:** Same as Quick tier — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories
- **Export Coverage basis:** Documentation completeness rather than source coverage. Score = (documented_items_with_complete_descriptions / total_documented_items) * 100. A "complete" item has: description, parameters (if function/method), and return type (if function/method).
- **Coherence:** Standard rules for the detected mode (naive or contextual) apply unchanged

This is functionally identical to Quick tier weight redistribution but with a different coverage denominator (self-consistency instead of source comparison).

**External-validator requirement for docs-only:** docs-only mode removes two categories (Signature Accuracy, Type Coverage) from scoring. If External Validation is also unavailable, the evidence base collapses to Coverage alone (naive) or Coverage + Coherence (contextual) — which in the naive/Quick case trips the minimum-evidence floor (INCONCLUSIVE). To keep docs-only skills gradable when external validators are present but still deterministic when they are missing: **when `docsOnly: true` AND `externalValidation is null`, step 5 must cap `totalScore` at `threshold - 1` (forcing FAIL) before the INCONCLUSIVE floor is evaluated.** This prevents a docs-only skill from PASSing with only one or two redistributed categories carrying all the weight. Implement in step 5 §4 as a pre-compare cap, recorded in the report as `scoring_notes: docs-only without external validators — capped below threshold`.

### Stack Skills (Any Tier)

When `metadata.json.skill_type == "stack"` (set `stackSkill: true` in the scoring input):

- **Signature Accuracy:** N/A — a stack skill's "signature surface" is the external library API it composes (pydantic, SQLAlchemy, FastAPI, etc.), not a proprietary surface the skill authors. Grading signatures against a surface the skill does not own produces meaningless numbers.
- **Type Coverage:** N/A — same rationale; the type surface belongs to the external libraries.
- **Weight redistribution:** Same as Quick tier / docs-only / State 2 — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories (Export Coverage, Coherence, External Validation).
- **Applies regardless of detected tier** (Quick, Forge, Forge+, Deep) and is independent of `docsOnly` and `state2`. A stack skill can also be docs-only or State 2; the skip reasons combine additively (e.g. `"stack skill (external type surface) + State 2 (provenance-map)"`).
- **Detection:** step 5 reads `metadata.json.skill_type` from the skill package. If the value is `"stack"`, set `stackSkill: true` in the scoring input JSON.

### Reference-App Skills (Any Tier)

When `metadata.json.scope_type == "reference-app"` (set `referenceApp: true` in the scoring input):

- **Signature Accuracy:** N/A — a reference app documents wiring patterns (how surfaces are composed), not a library export surface the skill authors. There are no public-export signatures to grade against; the Pattern Surface replaces the Key API Summary (see `skf-create-skill` Reference-App Assembly Overrides).
- **Type Coverage:** N/A — same rationale; a reference app has no library type surface to cover. Coverage is measured as pattern-surface coverage (`stats.pattern_surfaces_documented`), not export/type coverage.
- **Weight redistribution:** Same as Quick tier / docs-only / State 2 / stack — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories (Export Coverage, Coherence, External Validation).
- **Applies regardless of detected tier** (Quick, Forge, Forge+, Deep) and is independent of `docsOnly` and `state2`; skip reasons combine additively. `referenceApp` and `stackSkill` are distinct scope/type signals and should not both be set for the same skill.
- **Detection:** step 5 reads `metadata.json.scope_type` from the skill package. If the value is `"reference-app"`, set `referenceApp: true` in the scoring input JSON. The skip reason recorded is `"reference-app (no library export signatures)"`.

### State 2 Source Access (Any Tier, Provenance-Map Only)

When source is not locally available and analysis resolves to State 2 (provenance-map baseline per source-access-protocol.md):

- **Signature Accuracy:** N/A — provenance-map stores parameters as flat string arrays; verification is string comparison only, not semantic AST verification. Type aliases (`str` vs `String`, `list` vs `List[Any]`) cannot be resolved without live source.
- **Type Coverage:** N/A — cannot verify type completeness without local source access for AST re-parsing.
- **Weight redistribution:** Same as Quick tier — Signature Accuracy (22%) and Type Coverage (14%) weights redistributed proportionally to remaining active categories (Export Coverage, Coherence, External Validation).
- **Applies regardless of detected tier** (including Forge, Forge+, Deep) whenever `analysis_confidence` is `provenance-map` and local source is unavailable.
- **Export Coverage denominator:** Uses the union of provenance-map entry names and metadata.json `exports[]` names (per source-access-protocol.md State 2 rules).

Note: When provenance-map entries are predominantly T1 (AST-verified at compilation time), the coverage and name-matching data is already at highest confidence. The N/A categories reflect the inability to re-verify at test time, not low-quality extraction data.

**State 2 undercount risk acknowledgement:** provenance-map is a cached extraction snapshot — if the source has evolved since extraction, public API adds/removes will not surface in Export Coverage (denominator is frozen to the provenance-map union). When `state2: true` AND step 3 records any provenance vs metadata divergence (e.g. union > either source by >5%), apply a flat **10% deduction** to `exportCoverage` before calling the scoring script, AND set `analysis_confidence: provenance-map` (already set) with a report note: `scoring_notes: State 2 undercount risk acknowledged — 10% deduction applied to Export Coverage`. Rationale: the skill cannot be reliably scored on a frozen denominator when the cache is known to disagree with its own metadata; prefer understating over overstating.

### Forge Tier (ast-grep)
- Export Coverage: AST-backed export comparison
- Signature Accuracy: AST-verified signature matching
- Type Coverage: AST-verified type completeness
- Full scoring formula applied

### Forge+ Tier (ast-grep + ccc)
- Same scoring as Forge tier — ccc provides pre-ranking but does not change scoring weights
- Improved extraction coverage (from ccc pre-discovery) may increase T1 count, but scoring formula is identical to Forge
- Full scoring formula applied

### Deep Tier (ast-grep + gh + QMD)
- All Forge tier checks plus:
- Cross-repository reference verification
- QMD knowledge enrichment for coherence
- Full scoring formula with maximum depth
- **Migration & Deprecation Warnings section:** If T2-future annotations exist in the enrichment data, verify that Section 4b is present in SKILL.md Tier 1 and that each warning traces to a T2 provenance citation. If no T2-future annotations exist, Section 4b should normally be absent (not empty). Presence/absence mismatch is a Medium severity gap — with one Info-severity exception for historical-migration content (completed package renames, consolidated import paths, shipped API cutovers that remain load-bearing for training-data drift remediation). See `references/coherence-check.md` §2b/§5b for the three-case rule.

## Score Calculation

```
score = sum(category_weight * category_score) for each category
category_score = (items_passing / items_total) * 100
```

## Coherence Score Aggregation (Contextual Mode)

```
reference_validity = (valid_references / total_references) * 100
integration_completeness = (complete_patterns / total_patterns) * 100
combined_coherence = (reference_validity * 0.6) + (integration_completeness * 0.4)
```

If no integration patterns exist, combined coherence equals reference validity.

This is the documented contract. The tally + weighted mean is computed by `scripts/aggregate-coherence.py` (invoked from coherence-check.md §5c), not by hand; the per-reference validity judgment (§4) and pattern-completeness judgment (§5) stay in the prompt, only the arithmetic is scripted.

## Result Determination

Three-state gate — **PASS / FAIL / INCONCLUSIVE**. `INCONCLUSIVE` is not PASS and not FAIL; it signals insufficient evidence to grade the skill. Downstream workflows must treat `INCONCLUSIVE` as a hard gate — do not export, do not auto-retry, surface to the human.

- **Minimum-Evidence Floor (applies before PASS/FAIL comparison):**
  - `active_categories` = count of categories with a non-zero final weight *after* all redistribution (Quick tier, docs-only, State 2, external-validator-unavailable). Categories with a redistributed weight of 0 do not count as active, even if they received a score.
  - **If `active_categories < 2`** → force `result: INCONCLUSIVE` with rationale `"insufficient evidence: only {N} active category"`. A single active category cannot cross-validate itself and a PASS would be a false signal.
  - **If `tier == "Quick"` AND the sole active contributor is Export Coverage** → force `result: INCONCLUSIVE` with rationale `"Quick tier: Export Coverage alone is insufficient evidence — add a second active category by upgrading tier or enabling external validators"`. This catches the degenerate case where every signature/type/coherence/external category gets redistributed to 0 and Export Coverage is doing all the work.
  - The floor is enforced by `scripts/compute-score.py`. The step 5 scoring step reads `result` from the script output and writes it into the test report frontmatter unchanged.

- Otherwise:
  - score >= threshold → PASS
  - score < threshold → FAIL

The floor is intentionally conservative: skf-test-skill grades other skills, so a false PASS has catastrophic downstream effects (polluted exports, misleading feasibility data). Falling back to INCONCLUSIVE is always preferred over a low-evidence PASS.

## Gap Severity

| Severity | Criteria                                                                                                       |
|----------|----------------------------------------------------------------------------------------------------------------|
| Critical | Missing exported function/class documentation                                                                  |
| High     | Signature mismatch between source and SKILL.md                                                                 |
| Medium   | Missing type or interface documentation                                                                        |
| Medium   | Migration section present/absent mismatch with T2-future annotation data (Deep tier)                           |
| Medium   | Metadata drift — intra-cluster export counts diverge (barrel: `stats.exports_public_api` vs `exports[].length`; or documented-surface: `stats.exports_documented` vs provenance-map entry count; >10% divergence) |
| Medium   | Denominator inflation — stratified-scope `scope.include` union exceeds provenance-map entry count by >25% (brief missing `scope.tier_a_include`) |
| Medium   | Script/asset directory exists but no Scripts & Assets section in SKILL.md                                      |
| Medium   | Scripts & Assets section references file not found in scripts/ or assets/ directory                            |
| Low      | Script/asset file present without provenance entry in provenance-map.json file_entries                         |
| Low      | Missing optional metadata or examples                                                                          |
| Low      | Description trigger optimization recommended (third-person voice, negative triggers, or keyword coverage gaps) |
| Info     | Style suggestions, non-blocking observations                                                                   |
| Info     | Discovery testing not performed — realistic prompt testing recommended before export                           |
| Info     | Multi-denominator reporting — barrel vs documented-surface clusters diverge by design (>10% cross-cluster)     |

<!-- Config: communicate in {communication_language}. -->

# Source Access Protocol

## Source API Surface Definition

**Source API surface** = the package's top-level public exports. These are the symbols reachable from the primary entry point without importing internal modules:

- **Python:** symbols exported in `__init__.py` (including re-exports) — exclude private (`_prefixed`) names
- **TypeScript/JavaScript:** named exports from `index.ts` / `index.js` — exclude unexported locals
- **Go:** exported identifiers (capitalized) from the package's public-facing files
- **Rust:** items in `pub use` from `lib.rs` or `mod.rs`
- **Empty-barrel packages (copy-paste / subpath-only distribution):** If the primary entry point is empty or re-exports nothing (e.g., `export {};` in `index.ts`, an empty `__init__.py`, `lib.rs` with no `pub use`), the package does not expose a barrel API. Do **not** compute coverage against the empty barrel — the denominator would be zero and the score meaningless. Instead, consult the skill brief's `scope.include` globs (`forge-data/{skill_name}/skill-brief.yaml`) to identify the authorized entry points, and build the public API surface from the **union of named exports across those files**. The skill brief's `scope.notes` field should document this distribution model explicitly; if present, treat it as confirmation that the empty barrel is by design rather than a bug. If no skill brief is available and the barrel is empty, set `analysis_confidence: docs-only` and report that the source API surface could not be determined.

- **Stratified-scope monorepo packages (curated subsets of multi-package repos):** If the source is a monorepo (detect via `packages/` layout, `workspaces` field in root `package.json`, `lerna.json`, `rush.json`, `nx.json`, or Cargo `[workspace]`) AND the skill brief's `scope.include` lists a curated file/directory subset rather than the full workspace, the coverage denominator must reflect only the authored surface, not the monorepo's global export count. This is distinct from the empty-barrel case: each workspace package may have a non-empty barrel, but the skill intentionally documents only a tiered subset.

  **Resolution order:**

  1. **Prefer `metadata.json.stats.effective_denominator`** when present. `skf-create-skill` step 5 §4 writes this field for stratified-scope skills. When set, use it directly as the `exports_public_api` count for coverage scoring — **subject to the deflation guard below**.

     **Denominator deflation guard (when `effective_denominator` is used).** `effective_denominator` is a stored count, and a gap-driven `skf-update-skill` run can lower it — legitimately (a real rescope expressed in the brief) or by gaming (deleting in-scope public exports from `metadata`/provenance and setting `effective_denominator` to the documented total to force 100% coverage). When source is readable (State 1, or State ≥ 2 with remote tools), validate it: re-derive the public surface from `scope.include` filtered by `scope.exclude` — the same union step 2 computes — and compare. If the re-derived source barrel exceeds `effective_denominator` by **more than 25%** AND the brief carries no `scope.tier_a_include` (the only legitimate narrowing mechanism), treat `effective_denominator` as **deflated**: use the re-derived source-barrel count as `total_exports` instead, and emit a **Medium**-severity gap `denominator deflation — effective_denominator below source public surface without tier_a_include` reporting both counts (`effective_denominator: {N}`, `re-derived source barrel: {M}`, `{percent}% below source`). A legitimate scope reduction is expressed in the brief — via `scope.tier_a_include`, or by adding the removed exports to `scope.exclude` so the re-derived barrel shrinks to match (the mechanism `skf-update-skill` gap-driven rescope uses) — **never** by editing `metadata.stats` to equal the documented count. This is the directional counterpart to the inflation check in step 3: inflation fires when the coarse surface is too large, deflation when the stored denominator is suspiciously small. When source is unavailable, skip the guard and annotate the report: `effective_denominator used unverified — no source access to re-derive the barrel`.

     **Rust barrel re-derivation (when ast-grep `pub`-fn extraction is unreliable).** The Rust-skill condition documents the ast-grep `pub-fn` extractor as unreliable, so re-deriving a crate's barrel surface — for this guard or for the live fallback below — falls back to grep. A naive `pub`-token sweep over-counts badly: it pulls in `pub(crate)`/`pub(super)` internals, impl-block associated consts and methods, test modules, and foreign re-export files (such sweeps routinely land 30–40% above the true barrel). Enumerate deterministically instead:

     - Trace the `pub use` re-export chain from the crate root (`lib.rs` / `mod.rs`) to find which modules the barrel actually re-exports; count from those, not from every file containing a `pub` token.
     - Count only **unrestricted, module-level (column-0) `pub`** item declarations: `pub struct` / `pub enum` / `pub trait` / `pub type` / `pub const` / `pub fn` (free functions) that start at column 0.
     - Exclude `pub(crate)` / `pub(super)` / `pub(in …)` (crate-internal, not barrel surface); anything indented (impl-block associated consts and methods are not free barrel items); `*tests.rs` and `#[cfg(test)]` modules; and files listed under `scope.exclude` (e.g. foreign re-export files).

     **TypeScript / JavaScript barrel re-derivation (multi-line re-export blocks).** When re-deriving a TS/JS barrel — for the deflation guard above or the live fallback below — a line-oriented `^export` regex undercounts: a named re-export block spans continuation lines (`export type {\n  A,\n  B,\n} from "./x"`), and a single-line match captures only the opening line, dropping every member listed on the continuation lines. Enumerate the full block instead:

     - For each `export { … }` / `export type { … }` (including the `import`-then-`export` re-export form), accumulate named members across continuation lines until the closing `}` — count every identifier in the brace list, not only those on the opening line. A renamed re-export (`export { A as B }`) counts once under its public name (`B`).
     - Resolve `export * from "…"` to the target module's own public names — a star re-export contributes that module's surface, not a single entry.
     - Prefer a `tsc` / `ts-morph` / AST entry-point resolution over the line regex when a TS toolchain is reachable; it resolves multi-line blocks and star re-exports natively. Fall back to the brace-accumulation grep only when no AST tool is available — the same grep-fallback posture the Rust note above takes.
  2. **Fall back to live re-derivation** when `effective_denominator` is absent (older skills, quick-tier output, or skills compiled before this rule existed). Read the brief's scope globs from `forge-data/{skill_name}/skill-brief.yaml`, resolve them against `source_path`, filter out files matching `scope.exclude`, and compute the source API surface as the **union of named exports across the matched files only**. The skill brief's `scope.notes` field should document the stratification strategy (e.g., "Tier A: fully documented; Tier B: deferred to references; Tier C: excluded") — when present, treat it as confirmation that the curated subset is by design, not a scope gap.

     **Honor `scope.tier_a_include` when present.** When re-deriving, prefer the brief-level `scope.tier_a_include` narrow include list over the coarse `scope.include`. `tier_a_include` is an optional brief field that lists only the authoring surface the brief actually intends to document (tier A), letting the denominator match the brief's authoring-vs-installing intent even when `scope.include` uses coarse globs that also match internal infrastructure. When `tier_a_include` is present, resolve its globs (still filtered by `scope.exclude`), compute the union across those files, and use that count as the denominator. When absent, fall back to resolving `scope.include`.

     **Exclude umbrella barrel files from a `tier_a_include` re-derivation.** A barrel file (`lib.rs`, `mod.rs`, `index.ts`/`index.js`, `__init__.py`) whose public surface is dominated by `pub use` / re-export leaves re-exports the *entire* crate/package surface. Including such a file in the resolved `tier_a_include` set makes the union **grow** rather than shrink — defeating the narrowing and producing a denominator larger than `scope.include` would have. Detect an umbrella barrel as a file whose exported names are mostly (>50%) re-exports of symbols defined in other files. When a `tier_a_include` file set resolves to include an umbrella barrel, drop the barrel and count only the concrete-definition files; if dropping it leaves no meaningful surface, prefer `effective_denominator` (priority 1) instead of the inflated `tier_a_include` union.

  3. **Denominator inflation check (absent `tier_a_include`).** If re-derivation used `scope.include` because no `tier_a_include` was provided, compare the resulting union count against the provenance-map entry count (when provenance-map exists). If the `scope.include` union is more than 25% larger than the provenance-map entry count, the coarse globs are almost certainly sweeping in internal infrastructure that the brief did not intend to document. Emit a **Medium**-severity gap titled `denominator inflation — coarse scope.include union exceeds authored surface` that points the user at the brief for rescoping via `scope.tier_a_include`. **When the package's barrel is an umbrella re-export file** (public surface mostly `pub use` / re-export leaves — see the umbrella-barrel note in step 2), recommend `stats.effective_denominator` (priority 1) as the **primary** remediation instead of `tier_a_include`: a `tier_a_include` that lists the umbrella barrel would recount *larger* than `scope.include`, not smaller, so it cannot clear the inflation. Report both counts (`scope.include union: {N}`, `provenance-map entries: {M}`, `{percent}% inflation`) and state that the coverage score it produced is driven by denominator inflation rather than documentation gaps. The check is skipped when provenance-map is unavailable (there is no baseline to compare against).

  Leave `analysis_confidence` unchanged (still `full` or `provenance-map` per the waterfall) — stratified scope does not degrade confidence, only the denominator. Annotate the coverage report with: `Stratified scope — denominator: {effective_denominator | tier_a_include union | scope.include union} ({N} files matched, {M} exports union)`.

  **When this clause does not apply:** `scope.type: "full-library"` skills, single-package repositories, or stratified briefs where the full monorepo is intentionally in scope. For those, use the standard barrel-based denominator — **unless** the single-package repo is a pattern-reference app (see next bullet) or publishes a multi-subpath `exports` map (use the multi-entry clause below).

- **Pattern-reference apps (non-library source):** If the source is a single-package repo whose purpose is demonstrating an integration pattern rather than distributing a library API — typical markers are `scope.type: "full-library"` **without** a barrel file at any recognized entry-point path (`__init__.py`, `index.ts`/`index.js`, `lib.rs`, `mod.rs`) AND without a monorepo layout — the skill's value lives in wiring patterns, not exports. None of the preceding three clauses fits: there is no barrel to count from, no empty-barrel `scope.include` to consult, and no monorepo stratification to re-derive.

  **Trigger (either fires):**

  1. `scope.notes` in `forge-data/{skill_name}/skill-brief.yaml` flags pattern-reference intent (phrases such as "Reference app, not a library", "pattern-reference", "embedded-pattern skill", or "skill value is the … pattern"). The `scope.notes` field is authoritative when the author wrote it.
  2. Source tree lacks a barrel file at every recognized entry-point path AND the repo is not a monorepo (no `packages/`, `workspaces`, `lerna.json`, `rush.json`, `nx.json`, or Cargo `[workspace]`) AND the package does not declare a multi-subpath `exports` map (those route to the multi-entry clause below). Detected at test time by filesystem inspection of `{source_path}`.

  **Denominator:** canonicalized provenance-map entry count (same canonicalization as the "Provenance-map canonicalization" section below). `skf-create-skill`'s extraction pass has already curated the provenance-map to the authored pattern surface; treat it as the authoritative enumeration of the skill's documented reach.

  **Recommendation — prefer `tier_a_include`:** authors should add `scope.tier_a_include` to the brief listing the files that constitute the authored pattern surface, the same way stratified-scope briefs do. When `tier_a_include` is present, use its re-derived union (filtered by `scope.exclude`) as the denominator exactly as in the stratified-scope clause. When absent, fall back to the canonicalized provenance-map count — do not fabricate a denominator from arbitrary source-tree sweeps.

  **Confidence:** leave `analysis_confidence` unchanged (still `full` or `provenance-map` per the waterfall). Pattern-reference does not degrade confidence — the surface is smaller than a library barrel, not lower quality. Annotate the coverage report with: `Pattern-reference — denominator: {tier_a_include union | canonicalized provenance-map count} ({N} pattern surfaces)`.

  **When this clause does not apply:** any repo with a non-empty barrel file, any monorepo (use the stratified-scope clause), or any single-package repo whose `scope.type` is explicitly `specific-modules` (use the specific-modules clause), `public-api` with a multi-subpath `exports` map (use the multi-entry clause below — a `public-api` package WITHOUT such a map keeps the standard root-barrel rule), `component-library`, or `docs-only`. Also does not apply when `scope.type: "reference-app"` — that scope type carries its own pattern-surface denominator semantics (the brief speaks for itself), so this clause's filesystem trigger is moot.

- **Single-crate curated subset (`scope.type: "specific-modules"`):** If the source is a single-package (non-monorepo) repo whose skill brief sets `scope.type: "specific-modules"` and uses `scope.include`/`scope.exclude` to carve a subset of the crate's public surface, the coverage denominator is the **in-scope reachable barrel** — not the full barrel.

  **Resolution:** Derive the barrel as normal for the language (e.g., Rust: `pub use` chain from `lib.rs`), then filter:

  1. **Exclude** any modules or items that fall under `scope.exclude` patterns.
  2. **Include only** items reachable from the modules listed in `scope.include`.
  3. **Respect module visibility:** items behind `mod` (not `pub mod`) boundaries that are not re-exported through the barrel are unreachable and excluded. For Rust: count only unrestricted, module-level (column-0) `pub` item declarations in barrel-reachable modules; exclude `pub(crate)` / `pub(super)` / `pub(in …)` restricted items.

  The resulting count is the denominator. Annotate the coverage report with: `Specific-modules subset — denominator: in-scope reachable barrel ({N} items from {M} modules, after scope.include/exclude filtering)`.

  **When `effective_denominator` is present:** prefer `metadata.json.stats.effective_denominator` (same priority-1 rule as the stratified-scope clause), subject to the same deflation guard.

  **When this clause does not apply:** monorepo packages (use the stratified-scope clause), `scope.type: "full-library"` (use the standard barrel), or empty-barrel packages (use the empty-barrel clause). This clause is specifically for single-crate repos where the brief intentionally documents a curated subset rather than the full public surface.

- **Multi-entry (exports-map) packages (single-package libraries publishing via a `package.json` `exports` map):** If the in-scope `package.json` declares an `exports` map with **multiple non-root subpath entries** (more than just `"."`) and the repo carries **no** monorepo markers (`packages/` layout, `workspaces` field, `lerna.json`, `rush.json`, `nx.json`, Cargo `[workspace]`), the package's public surface spans every published subpath, not just the root barrel. The standard "named exports from `index.ts`" rule undercounts: it measures only the `"."` barrel while installers reach the full subpath set. This clause covers both `scope.type: "full-library"` AND `scope.type: "public-api"` for such packages.

  **Denominator:** the **union of named exports across the files each NON-WILDCARD `exports` subpath resolves to**. Resolve each subpath to its target file (or its committed `.d.ts` / `.d.mts` declaration), then apply the same multi-line brace-accumulation and `export *` star-resolution rules documented for barrel re-derivation earlier in this file ("TypeScript / JavaScript barrel re-derivation"). **Explicitly exclude wildcard subpaths** (`"./*"` forms — they map to an open-ended file set whose surface is unbounded and uncountable). If the `exports` map has only a root `"."` entry, or only wildcard subpaths, fall back to the standard root-barrel rule.

  **Curation/priority (same ladder the specific-modules clause uses):** prefer `metadata.json.stats.effective_denominator` first (subject to the existing deflation guard), then `scope.tier_a_include` globs (filtered by `scope.exclude`, the umbrella-barrel note applies) when the brief supplies it, else the full subpath union.

  **Audit:** the root-barrel named-export count must be reported as a **secondary candidate** in the Denominator Candidates audit block (coverage-check.md §4) so the root-barrel-vs-subpath-union choice is auditable. Annotate the coverage report with: `Multi-entry (exports-map) — denominator: {effective_denominator | tier_a_include union | subpath union} ({N} subpaths resolved, {M} exports union; root barrel: {R})`.

  **When this clause does not apply:** monorepo packages (use the stratified-scope clause), empty-barrel packages (use the empty-barrel clause), pattern-reference apps (use the pattern-reference clause), `scope.type: "specific-modules"` (use the specific-modules clause), or single-entry / wildcard-only `exports` maps (use the standard root-barrel rule).

Internal module symbols are **excluded** from the coverage denominator unless they are explicitly documented in SKILL.md (in which case they count as documented extras, not missing coverage).

This matches the extraction-patterns.md convention used during skill creation: coverage measures how well SKILL.md documents what users actually import, not the entire internal codebase.

### Provenance-map canonicalization

When the test-side intersects documented SKILL.md exports against a stratified-scope provenance-map, raw provenance-map entry names may include **bookkeeping variants** of the same underlying export. These variants are artifacts of how the source library structures its registry (e.g., Storybook's component-plus-story decomposition, accessibility renderer shadowing, exact-match versus fuzzy-match renderer disambiguation). Counting them as separate exports inflates the denominator and produces false "missing documentation" findings for names that are structurally duplicates of an already-documented base export.

Before intersecting documented names against the provenance-map entry list, **fold bookkeeping variants back to their base name** using the rules below. This matches the convention `skf-create-skill` records in `metadata.json.stats.effective_denominator_source` (e.g., `"provenance-map canonicalized count (ThemesGlobals_def folds with ThemesGlobals under _def convention)"`) — the base form is authoritative; the variant form is a sibling record, not an independent export.

**Folding rules (apply in order, case-sensitive):**

1. **Suffix `_def`** — registry definition twin. `ThemesGlobals_def` folds to `ThemesGlobals`. Common in Storybook-style component registries where the definition object and the rendered component share the same public name.
2. **Suffix `_exact`** — exact-match variant. `ButtonSpec_exact` folds to `ButtonSpec`. Common in matcher/renderer registries where an `_exact` sibling signals a stricter resolution path.
3. **Prefix `a11y_`** — accessibility renderer shadow. `a11y_Checkbox` folds to `Checkbox`. Common in accessibility-wrapper layers that re-export every component under a parallel prefixed namespace.
4. **Other renderer-prefix disambiguation** — when the library uses a prefix-namespace convention (e.g., `mobile_`, `web_`, `ssr_`) to shadow the base export, fold the prefix form back to the base. **Only apply when the base form is also present in the provenance-map** — otherwise the prefix form is the real export and should be kept. Document the specific prefix used in the test report so the rule is auditable.

**How to apply:**

1. Read all entry names from the provenance-map.
2. Build a canonical-name set by applying the folding rules above — each variant maps to its base. Retain the original variant → base mapping for reporting.
3. Intersect the documented SKILL.md export names against the **canonical** set, not the raw entry list.
4. When computing `Export Coverage`, use the **canonical count** as the denominator — not the raw provenance-map entry count. This aligns the denominator with `metadata.json.stats.effective_denominator` (when present), which `skf-create-skill` already writes as the canonicalized count.
5. In the test report, note the fold summary: `Provenance-map canonicalization: {N} raw entries → {M} canonical bases ({N−M} bookkeeping variants folded: _def×{a}, _exact×{b}, a11y_×{c}, other×{d})`. This makes the reduction auditable by future testers and update runs.

**When to skip canonicalization:**

- If the library's public surface genuinely distinguishes the variants (e.g., `a11y_Checkbox` is a separately-documented, separately-installed component and not a shadow), do not fold — the variant is a real export. Check SKILL.md for explicit documentation of the variant before folding. When in doubt, err on the side of not folding and report both forms.
- If `metadata.json.stats.effective_denominator` is present and the provenance-map raw count matches it (no drift), canonicalization is not needed — the denominator is already canonical. Fold only when raw count > `effective_denominator` and the drift corresponds to recognizable bookkeeping suffixes/prefixes.
- If drift remains after folding (e.g., raw 222 → canonical 215 but `effective_denominator` says 216), record the residual 1-count drift as an unexplained-reconciliation note in the test report. Do not fabricate additional fold rules to close the gap.

## Source Access Resolution

Before analysis, determine source access level. Walk through these states in order — use the first that succeeds:

**State 1 — Local source available:**
Check if `{source_path}` (from metadata.json `source_root`) exists on disk. If yes → full analysis at detected tier (AST + signatures). Set `analysis_confidence: full`.

**State 2 — Local absent, provenance-map exists:**
Check `{forge_data_folder}/{skill_name}/provenance-map.json`. If present AND contains at least 1 entry, use it as the baseline export inventory — each entry contains structured fields: `export_name`, `export_type`, `params[]`, `return_type`, `source_file`, `source_line`, `confidence`, and `ast_node_type`. Cross-reference against SKILL.md documented exports for name-matching and param-by-param coverage. Signature verification compares SKILL.md's documented params/return types against provenance-map entries directly.

**Cross-reference with metadata.json:** After loading provenance-map entries, compare the entry count against `metadata.json`'s `exports[]` array length and `stats.exports_public_api` count. If metadata reports more exports than provenance-map entries:
- Compute `gap = metadata.exports.length - provenance_map.entries.length`
- Report: "Provenance-map contains {pmap_count} entries but metadata.json lists {meta_count} exports ({gap} gap). Coverage denominator uses the union."
- Build the coverage denominator from the **union** of provenance-map entry names and metadata.json `exports[]` names. Exports present in metadata but absent from provenance-map are counted as "missing documentation" in the coverage calculation.
- If metadata.json is unavailable or has no `exports[]` array, use provenance-map count alone with a note: "Coverage denominator is provenance-map only — may undercount if extraction was incomplete." If remote reading tools are available (zread, deepwiki, gh API, or similar), supplement by reading the entry point file for live signature verification. Set `analysis_confidence: provenance-map`.

**State 2 limitations:** Signature verification at State 2 is **string comparison only**, not semantic. Provenance-map stores parameters as flat string arrays (e.g., `["data: Union[BinaryIO, list, str]"]`), so `str` vs `String` or `list` vs `List[Any]` would be treated as mismatches even when semantically equivalent. For full type-aware verification (handling type aliases, generic equivalence), State 1 (local source) with AST re-parsing is required. When the SKILL.md was compiled from the same provenance-map (typical for create-then-test flows), most strings will match. However, enrichment (step 4) and doc-fetching (step 3c) during compilation may alter parameter descriptions, add type annotations, or normalize signatures, causing mismatches even in create-then-test flows. Expect some string-level mismatches and treat them as compilation artifacts, not source drift signals, until signature fidelity is enforced by step 5's Signature Fidelity Rule (see `signature_source` field in provenance-map entries).

**State 3 — No provenance-map, metadata exports exist (quick-skill path):**
If no provenance-map.json exists (typical for quick-skill output), fall back to `metadata.json`'s `exports[]` array for the export name list. Coverage check becomes a self-consistency comparison: are all names in `exports[]` documented in SKILL.md with description, parameters, and return type? Signatures cannot be verified. If remote reading tools are available, supplement by reading the entry point for live export comparison. Set `analysis_confidence: metadata-only`.

**State 4 — No local source, no forge-data, remote tools available:**
If neither provenance-map nor metadata exports provide a usable baseline, but remote reading tools (zread, deepwiki, gh API, or similar) are available and `source_repo` is set in metadata.json, read the entry point remotely to build the export inventory from scratch. Name-matching only — no AST. Set `analysis_confidence: remote-only`.

**State 5 — No source access at all:**
If none of the above succeed, fall through to docs-only mode (as defined in coverage-check.md Section 0: pre-analysis source type detection). Set `analysis_confidence: docs-only`. Warn: "**No source access available.** Coverage check evaluates documentation self-consistency only. Re-run with local clone or remote access for source-backed verification."

Set `analysis_confidence` in context for use in Section 2 analysis depth, step 5 output, and step 5 scoring.

**Confidence tier mapping:** `full` = T1, `provenance-map` = T1, `metadata-only` = T1-low, `remote-only` = T1-low, `docs-only` = T3. This aligns with the T1/T1-low/T2/T3 scale used across all SKF workflows.

**Degradation notice rules:** When `analysis_confidence` is `provenance-map`, check the `confidence` field of provenance-map entries before emitting a degradation recommendation:

- **All/most entries T1 (AST-verified):** The provenance-map data is already at highest confidence. Do not recommend re-running with a local clone — it would produce identical results. Use: "Resolved via: provenance-map (T1 AST-verified at compilation time). Local clone not required — provenance data is already at highest confidence."
- **Mixed T1/T1-low entries:** Report the breakdown. Recommend local clone only for the T1-low entries: "Resolved via: provenance-map ({n} T1, {m} T1-low). Re-run with local clone to upgrade T1-low entries to full AST verification."
- **All/most entries T1-low or lower:** Keep the standard recommendation: "Re-run with local clone for full AST-backed verification."

---
nextStepFile: 'score.md'
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
---

<!-- Config: communicate in {communication_language}. -->

# Step 4c: Hard Gate

## STEP GOAL:

Scan accumulated findings from coverage and coherence analysis for critical or high severity gaps. If any exist, block the pipeline with a clear listing of every blocking finding — no coverage score is computed or reported. If only medium, low, or info findings exist, pass through to scoring.

### §1. Read Findings from Output

Read `{outputFile}` and scan the **Coverage Analysis** and **Coherence Analysis** sections for GAP entries. Each GAP entry contains a severity marker in this format:

```
**Severity:** {Critical|High|Medium|Low|Info}
```

Extract every line matching `**Severity:** Critical` or `**Severity:** High`. For each match, also capture the parent GAP heading (`### GAP-{NNN}: {title}`) and the `**Source:**` line to build a blocking-findings list.

### §2. Evaluate Gate

**Count blocking findings** (Critical + High severity).

**IF blocking findings exist → BLOCK (§3)**

**IF no blocking findings → PASS (§4)**

### §3. Block — Critical/High Findings Detected

The hard gate blocks the pipeline. No scoring step runs.

Update `{outputFile}` frontmatter:
- Set `testResult: 'fail'`
- Append `'hard-gate'` to `stepsCompleted`

Report to the user:

"**Hard gate BLOCKED — {N} critical/high finding(s) must be resolved before scoring.**

| # | GAP | Severity | Source |
|---|-----|----------|--------|
{for each blocking finding:}
| {i} | {GAP-NNN}: {title} | {severity} | {source} |

**{M} medium/low/info findings also noted (non-blocking).**

**Action required:** resolve all Critical and High findings, then re-run test-skill.
**Recommended next step:** update-skill"

**Headless envelope (if `{headless_mode}`):** emit to **stderr**:

```
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":"FAIL","score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":"update-skill","exit_code":2,"halt_reason":"hard-gate-blocked"}
```

HALT — do not chain to `{nextStepFile}`.

### §4. Pass — No Critical/High Findings

The hard gate passes. Medium, low, and info findings are documented in the gap report but do not block.

Update `{outputFile}` frontmatter:
- Append `'hard-gate'` to `stepsCompleted`

Report that the hard gate passed, noting the count of non-blocking medium/low/info finding(s), then proceed to scoring.

Update stepsCompleted, then load and execute `{nextStepFile}`.

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""Deterministic contextual-coherence aggregator.

Pure-function tally + weighted mean for the SKF test-skill workflow
(step-04, coherence-check.md §5c). The prompt keeps the judgment — deciding
which cross-references are valid (§4) and which integration patterns are
complete (§5). This script does ONLY the arithmetic those judgments feed:
the reference-validity ratio, the integration-completeness ratio, and their
fixed 0.6 / 0.4 weighted mean. That way the 18%-weight `coherence` input to
compute-score.py is computed once, deterministically, instead of by hand on
every run (this skill grades other skills — a false PASS is catastrophic, so
every scoring input is scripted for run-to-run reproducibility).

Formula (mirrors scoring-rules.md "Coherence Score Aggregation (Contextual
Mode)" — the prose there remains the documented contract):

    reference_validity       = (valid_references / total_references) * 100
    integration_completeness = (complete_patterns / total_patterns) * 100
    combined_coherence       = reference_validity * 0.6 + integration_completeness * 0.4

Field-name mapping: the step-04 §5 integration JSON calls the pattern counts
`patterns_documented` (= total_patterns) and `patterns_complete`
(= complete_patterns); this script's input uses those step-04 names directly
so §5c passes them through unrenamed.

Edge cases (both documented in scoring-rules.md — absence is never penalized):
  * patterns_documented == 0 -> no integration patterns to weigh, so
    combined_coherence == reference_validity and integrationCompleteness is
    null. (Do not divide by zero.)
  * total_references == 0 -> no references means no broken references, so
    reference_validity == 100.0 (vacuously coherent). Unreachable on the normal
    contextual path — §3 always extracts at least one reference — but handled so
    the arithmetic never divides by zero.

Input schema (one JSON object):
  {
    "valid_references":    <int >= 0, <= total_references>,
    "total_references":    <int >= 0>,
    "patterns_documented": <int >= 0>,
    "patterns_complete":   <int >= 0, <= patterns_documented>
  }

Output (JSON):
  {
    "input": { ...echo... },
    "referenceValidity":       <0-100>,
    "integrationCompleteness": <0-100 | null when patterns_documented == 0>,
    "combinedCoherence":       <0-100>,
    "patternsScored":          <bool>
  }
  or {"error": ..., "code": "INVALID_INPUT"} on a schema violation.

Percentages use the same JavaScript-compatible 2-decimal rounding as
compute-score.py so the two scoring scripts agree to the last digit.

CLI usage (mirrors compute-score.py):
  uv run aggregate-coherence.py '<JSON>'                  # positional arg
  uv run aggregate-coherence.py --json-input '<JSON>'     # explicit flag form
  cat input.json | uv run aggregate-coherence.py --stdin  # piped input

Exit codes (same convention as compute-score.py / reconcile-coverage.py):
  0  — a result object was emitted
  1  — input could not be parsed at all (no input provided, or malformed JSON)
  2  — input parsed but schema/semantics invalid

Both 1 and 2 emit an {"error": ..., "code": "INVALID_INPUT"} envelope on stdout,
so the envelope's presence — not the specific code — tells a caller the input was
refused rather than scored.
"""

from __future__ import annotations

import argparse
import json
import math
import sys

# Weights for the combined-coherence mean (from scoring-rules.md). Kept as
# named constants so the 0.6 / 0.4 split lives in exactly one place.
REFERENCE_VALIDITY_WEIGHT = 0.6
INTEGRATION_COMPLETENESS_WEIGHT = 0.4

COUNT_FIELDS = (
    "valid_references",
    "total_references",
    "patterns_documented",
    "patterns_complete",
)


def round2(value):
    """Round to 2 decimals with JavaScript-compatible half-up rounding.

    Matches compute-score.py.round2 so both scoring scripts produce identical
    numbers. JS Math.round rounds .5 away from zero for positives; Python's
    built-in round uses banker's rounding. We replicate JS: floor(x*100+0.5)/100.
    """
    return math.floor(value * 100 + 0.5) / 100


def make_error(message):
    return {"error": message, "code": "INVALID_INPUT"}


def validate_input(inp):
    if inp is None or not isinstance(inp, dict):
        return "Input must be a JSON object"

    for field in COUNT_FIELDS:
        if field not in inp or inp[field] is None:
            return f"Missing required field: {field} (non-negative integer)"
        value = inp[field]
        # bool is a subclass of int; reject it so true/false can't pose as 1/0.
        if isinstance(value, bool) or not isinstance(value, int):
            return (
                f"Field `{field}` must be a non-negative integer, "
                f"got {type(value).__name__}: {value!r}"
            )
        if value < 0:
            return f"Field `{field}` must be >= 0, got {value}"

    if inp["valid_references"] > inp["total_references"]:
        return (
            "valid_references cannot exceed total_references "
            f"({inp['valid_references']} > {inp['total_references']})"
        )

    if inp["patterns_complete"] > inp["patterns_documented"]:
        return (
            "patterns_complete cannot exceed patterns_documented "
            f"({inp['patterns_complete']} > {inp['patterns_documented']})"
        )

    return None


def aggregate_coherence(inp):
    """Compute reference validity, integration completeness, and combined coherence.

    Weighted mean of the two ratios (0.6 / 0.4), JS-compatible 2-decimal rounding:
    >>> out = aggregate_coherence({"valid_references": 6, "total_references": 7,
    ...                            "patterns_documented": 5, "patterns_complete": 4})
    >>> out["referenceValidity"], out["integrationCompleteness"], out["combinedCoherence"]
    (85.71, 80.0, 83.43)
    >>> out["patternsScored"]
    True

    Zero integration patterns -> combined equals reference validity, no divide-by-zero:
    >>> out = aggregate_coherence({"valid_references": 9, "total_references": 10,
    ...                            "patterns_documented": 0, "patterns_complete": 0})
    >>> out["integrationCompleteness"] is None
    True
    >>> out["patternsScored"]
    False
    >>> out["referenceValidity"], out["combinedCoherence"]
    (90.0, 90.0)

    Zero references -> vacuously perfect reference validity, never a ZeroDivision:
    >>> aggregate_coherence({"valid_references": 0, "total_references": 0,
    ...                      "patterns_documented": 2, "patterns_complete": 1})["referenceValidity"]
    100.0

    A schema violation returns an error object, not an exception:
    >>> aggregate_coherence({"valid_references": 5, "total_references": 3,
    ...                      "patterns_documented": 0, "patterns_complete": 0})["code"]
    'INVALID_INPUT'
    """
    validation_error = validate_input(inp)
    if validation_error:
        return make_error(validation_error)

    valid_references = inp["valid_references"]
    total_references = inp["total_references"]
    patterns_documented = inp["patterns_documented"]
    patterns_complete = inp["patterns_complete"]

    # Reference validity — vacuously 100 when there are no references at all.
    if total_references == 0:
        reference_validity = 100.0
    else:
        reference_validity = round2((valid_references / total_references) * 100)

    # Integration completeness — absent when no patterns are documented.
    patterns_scored = patterns_documented > 0
    if patterns_scored:
        integration_completeness = round2(
            (patterns_complete / patterns_documented) * 100
        )
        combined_coherence = round2(
            reference_validity * REFERENCE_VALIDITY_WEIGHT
            + integration_completeness * INTEGRATION_COMPLETENESS_WEIGHT
        )
    else:
        integration_completeness = None
        combined_coherence = reference_validity

    return {
        "input": {field: inp[field] for field in COUNT_FIELDS},
        "referenceValidity": reference_validity,
        "integrationCompleteness": integration_completeness,
        "combinedCoherence": combined_coherence,
        "patternsScored": patterns_scored,
    }


def _build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        prog="aggregate-coherence",
        description=(
            "Deterministic contextual-coherence aggregator. Input is a single "
            "JSON object of reference + integration-pattern counts; output is the "
            "reference-validity, integration-completeness, and combined-coherence "
            "percentages that feed the `coherence` scoring input."
        ),
        epilog=(
            "Example:\n"
            "  uv run aggregate-coherence.py "
            "'{\"valid_references\":6,\"total_references\":7,"
            "\"patterns_documented\":5,\"patterns_complete\":4}'"
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "json_input",
        nargs="?",
        help="JSON object as a positional argument (single-quote it on the shell).",
    )
    src.add_argument(
        "--json-input",
        dest="json_input_flag",
        help="JSON object passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the JSON object from stdin.",
    )
    return parser


def _resolve_input(args: argparse.Namespace) -> str:
    if args.stdin:
        return sys.stdin.read()
    if args.json_input_flag is not None:
        return args.json_input_flag
    if args.json_input is not None:
        return args.json_input
    return ""


def main(argv: list[str] | None = None) -> int:
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print(
            "error: no input provided (positional arg, --json-input, or --stdin)",
            file=sys.stderr,
        )
        return 1

    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        print(json.dumps(make_error(f"Invalid JSON: {exc.msg}"), indent=2))
        return 1

    result = aggregate_coherence(data)
    print(json.dumps(result, indent=2))
    # Same convention as compute-score.py / reconcile-coverage.py. This script
    # produces the coherence percentage that score.md §3a feeds to
    # compute-score.py, so a silently-rejected run here would hand a bogus or
    # absent number to the gate one layer downstream.
    if isinstance(result, dict) and result.get("code") == "INVALID_INPUT":
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# ///
"""Deterministic metadata export-count coherence cross-check.

Count-drift helper for the SKF test-skill workflow (coverage-check.md §4b).
Distinct from reconcile-coverage.py (which owns the coverage *numerator*
intersection) and compute-score.py (weights + INCONCLUSIVE floor): this script
performs the §4b *count-drift* arithmetic — binning the collected export counts
into the two canonical clusters and computing the intra-cluster / cross-cluster
percentage divergences — so the "13% drift → emit / 4% → skip" decision no
longer swings between runs on an in-prompt eyeball.

It reproduces exactly the §4b operationalized checks:

  * Cluster A — public-barrel surface (`__init__.py` / `index.ts` / `lib.rs`
    re-exports): `stats.exports_public_api`, `exports[].length`.

  * Cluster B — documented surface (extracted + documented, incl. methods and
    submodule members): `stats.exports_documented`, the provenance-map
    **named-export count** (entries whose `export_name` contains `::` are
    excluded — impl-block methods roll up under an already-counted type), and
    the `confidence_distribution` sum (`t1 + t1_low + t2 + t3`).

  * Intra-cluster divergence (Medium): within a cluster with >=2 present counts,
    if the largest and smallest disagree by more than the drift threshold
    (default 10%) of the larger, emit `metadata drift — {barrel|documented-surface}
    export counts diverge`.

  * Cross-cluster divergence (Info): if both clusters resolved to a
    representative count (the higher of each cluster's present counts) and they
    differ by more than the threshold, emit `multi-denominator reporting —
    barrel vs documented surface` (expected for skills whose documented surface
    intentionally exceeds the barrel — not drift, just made auditable).

Stack skills (`skill_type == "stack"`) and reference apps
(`scope_type == "reference-app"`) are skipped: their three counts measure
intentionally *different* surfaces, so comparing them yields only false drift.

CLI usage:
  uv run check-metadata-coherence.py '<JSON>'                  # positional
  uv run check-metadata-coherence.py --json-input '<JSON>'     # explicit flag
  cat input.json | uv run check-metadata-coherence.py --stdin  # piped

Input schema (one object; omit any count that is absent for the skill):
  {
    "skillType": "stack" | ... | null,          # metadata.json.skill_type
    "scopeType": "reference-app" | ... | null,   # metadata.json.scope_type
    "clusterA": {
      "exports_public_api": <int|null>,
      "exports_length": <int|null>
    },
    "clusterB": {
      "exports_documented": <int|null>
    },
    "provenanceExportNames": ["Type::method", "foo", ...] | null,  # raw entry names
    "confidenceDistribution": {"t1": N, "t1_low": N, "t2": N, "t3": N} | null,
    "driftThresholdPct": 10                       # optional, default 10
  }

Output (stdout, one object):
  {
    "skipped": <bool>,
    "skipReason": <str|null>,
    "driftThresholdPct": <int>,
    "clusterACounts": {label: count, ...},        # present counts only
    "clusterBCounts": {label: count, ...},        # present counts only, incl. derived
    "findings": [
      {"severity": "Medium|Info", "title": "...", "detail": "...",
       "category": "structural/metadata coherence", "driftPct": <int>}
    ]
  }

Exit codes:
  0  — cross-check emitted successfully (findings may be empty)
  1  — no input / input could not be parsed as JSON
  2  — input parsed but schema/semantics invalid (error object emitted as JSON)
"""

from __future__ import annotations

import argparse
import json
import sys

DEFAULT_DRIFT_PCT = 10
CATEGORY = "structural/metadata coherence"

# Stable source labels used in finding bodies (match coverage-check.md §4b examples).
LABEL_EXPORTS_PUBLIC_API = "stats.exports_public_api"
LABEL_EXPORTS_LENGTH = "exports[].length"
LABEL_EXPORTS_DOCUMENTED = "stats.exports_documented"
LABEL_PROVENANCE_NAMED = "provenance named-exports"
LABEL_CONFIDENCE_SUM = "confidence_distribution sum"


def make_error(message):
    return {"error": message, "code": "INVALID_INPUT"}


def _as_count(value, label, errors):
    """Return value if it is a non-negative int (not bool), else record an error."""
    if value is None:
        return None
    if isinstance(value, bool) or not isinstance(value, int) or value < 0:
        errors.append(f"{label} must be a non-negative integer (got {value!r})")
        return None
    return value


def _provenance_named_count(names, errors):
    """Count provenance entry names, excluding `::` impl-block methods."""
    if names is None:
        return None
    if not isinstance(names, list):
        errors.append("provenanceExportNames must be a list of strings")
        return None
    return sum(1 for n in names if isinstance(n, str) and n and "::" not in n)


def _confidence_sum(dist, errors):
    """Sum t1 + t1_low + t2 + t3 (each defaulting to 0 when absent)."""
    if dist is None:
        return None
    if not isinstance(dist, dict):
        errors.append("confidenceDistribution must be an object")
        return None
    total = 0
    for tier in ("t1", "t1_low", "t2", "t3"):
        v = dist.get(tier, 0)
        if v is None:
            v = 0
        if isinstance(v, bool) or not isinstance(v, int) or v < 0:
            errors.append(
                f"confidenceDistribution.{tier} must be a non-negative integer (got {v!r})"
            )
            continue
        total += v
    return total


def drift_pct(counts):
    """Percentage divergence of the extreme counts: (max - min) / max * 100."""
    hi = max(counts)
    lo = min(counts)
    if hi == 0:
        return 0.0
    return (hi - lo) / hi * 100.0


def _intra_cluster_finding(present, cluster_label, title, threshold):
    """Emit an intra-cluster Medium finding when the extremes diverge > threshold."""
    if len(present) < 2:
        return None
    values = [c for _, c in present]
    pct = drift_pct(values)
    if pct <= threshold:
        return None
    enumerated = ", ".join(f"{label}={count}" for label, count in present)
    pct_display = round(pct)
    return {
        "severity": "Medium",
        "title": title,
        "detail": f"{enumerated} → {pct_display}% drift",
        "category": CATEGORY,
        "driftPct": pct_display,
    }


def check(inp):
    """Pure count-drift cross-check. Returns the result object (or an error object)."""
    if inp is None or not isinstance(inp, dict):
        return make_error("Input must be a JSON object")

    threshold = inp.get("driftThresholdPct", DEFAULT_DRIFT_PCT)
    if isinstance(threshold, bool) or not isinstance(threshold, (int, float)) or threshold < 0:
        return make_error("driftThresholdPct must be a non-negative number")

    skill_type = inp.get("skillType")
    scope_type = inp.get("scopeType")
    if skill_type == "stack":
        return {
            "skipped": True,
            "skipReason": "stack skill — the three counts measure intentionally "
            "different surfaces (empty own barrel, cited constituent contracts, "
            "per-constituent confidence bins); comparing them yields only false drift",
            "driftThresholdPct": threshold,
            "clusterACounts": {},
            "clusterBCounts": {},
            "findings": [],
        }
    if scope_type == "reference-app":
        return {
            "skipped": True,
            "skipReason": "reference app — counts measure pattern surfaces vs "
            "per-citation provenance, not a shared export barrel; comparing them "
            "yields only false drift",
            "driftThresholdPct": threshold,
            "clusterACounts": {},
            "clusterBCounts": {},
            "findings": [],
        }

    cluster_a_in = inp.get("clusterA") or {}
    cluster_b_in = inp.get("clusterB") or {}
    if not isinstance(cluster_a_in, dict) or not isinstance(cluster_b_in, dict):
        return make_error("clusterA and clusterB must be objects when present")

    errors: list[str] = []

    exports_public_api = _as_count(
        cluster_a_in.get("exports_public_api"), LABEL_EXPORTS_PUBLIC_API, errors
    )
    exports_length = _as_count(
        cluster_a_in.get("exports_length"), LABEL_EXPORTS_LENGTH, errors
    )
    exports_documented = _as_count(
        cluster_b_in.get("exports_documented"), LABEL_EXPORTS_DOCUMENTED, errors
    )
    provenance_named = _provenance_named_count(inp.get("provenanceExportNames"), errors)
    confidence_sum = _confidence_sum(inp.get("confidenceDistribution"), errors)

    if errors:
        return make_error("; ".join(errors))

    # Present-count lists preserve a stable cluster-canonical order for reporting.
    cluster_a = [
        (LABEL_EXPORTS_PUBLIC_API, exports_public_api),
        (LABEL_EXPORTS_LENGTH, exports_length),
    ]
    cluster_a = [(label, c) for label, c in cluster_a if c is not None]

    cluster_b = [
        (LABEL_EXPORTS_DOCUMENTED, exports_documented),
        (LABEL_PROVENANCE_NAMED, provenance_named),
        (LABEL_CONFIDENCE_SUM, confidence_sum),
    ]
    cluster_b = [(label, c) for label, c in cluster_b if c is not None]

    findings = []

    a_finding = _intra_cluster_finding(
        cluster_a, "barrel", "metadata drift — barrel export counts diverge", threshold
    )
    if a_finding:
        findings.append(a_finding)

    b_finding = _intra_cluster_finding(
        cluster_b,
        "documented-surface",
        "metadata drift — documented-surface export counts diverge",
        threshold,
    )
    if b_finding:
        findings.append(b_finding)

    # Cross-cluster: representative = the higher of each cluster's present counts.
    if cluster_a and cluster_b:
        repr_a = max(c for _, c in cluster_a)
        repr_b = max(c for _, c in cluster_b)
        pct = drift_pct([repr_a, repr_b])
        if pct > threshold:
            findings.append(
                {
                    "severity": "Info",
                    "title": "multi-denominator reporting — barrel vs documented surface",
                    "detail": f"barrel={repr_a}, documented={repr_b}",
                    "category": CATEGORY,
                    "driftPct": round(pct),
                }
            )

    return {
        "skipped": False,
        "skipReason": None,
        "driftThresholdPct": threshold,
        "clusterACounts": {label: count for label, count in cluster_a},
        "clusterBCounts": {label: count for label, count in cluster_b},
        "findings": findings,
    }


# --- CLI --------------------------------------------------------------------


def _build_parser():
    parser = argparse.ArgumentParser(
        prog="check-metadata-coherence",
        description=(
            "Deterministic metadata export-count coherence cross-check "
            "(coverage-check.md §4b). Bins the collected export counts into the "
            "two canonical clusters and emits the intra-cluster / cross-cluster "
            "drift findings so the count-drift decision is reproducible."
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=(
            "Example:\n"
            "  uv run check-metadata-coherence.py "
            "'{\"clusterA\":{\"exports_public_api\":55,\"exports_length\":48},"
            "\"clusterB\":{\"exports_documented\":114}}'"
        ),
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "json_input",
        nargs="?",
        help="JSON object as a positional argument (single-quote it on the shell).",
    )
    src.add_argument(
        "--json-input",
        dest="json_input_flag",
        help="JSON object passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the JSON object from stdin.",
    )
    return parser


def _resolve_input(args):
    if args.stdin:
        return sys.stdin.read()
    if args.json_input_flag is not None:
        return args.json_input_flag
    if args.json_input is not None:
        return args.json_input
    return ""


def main(argv=None):
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print(
            "error: no input provided (positional arg, --json-input, or --stdin)",
            file=sys.stderr,
        )
        return 1

    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        print(json.dumps(make_error(f"Invalid JSON: {exc.msg}"), indent=2))
        return 1

    result = check(data)
    print(json.dumps(result, indent=2))
    if isinstance(result, dict) and result.get("code") == "INVALID_INPUT":
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""Deterministic combined external-validation score.

Arithmetic helper for the SKF test-skill workflow (external-validators.md §4).
The combined external score feeds compute-score.py as the `externalValidation`
category input, so — like every other scoring input in this skill — the mean is
computed once by a script rather than by hand, keeping the verdict reproducible
run-to-run (an odd-sum average such as (80 + 73) / 2 = 76.5 is exactly where an
in-prompt round swings).

Rule (external-validators.md §4):
  * both tools ran  -> mean of skill-check + tessl review scores
  * one tool ran    -> that tool's score
  * neither ran     -> null (scoring step redistributes the external weight)

Both scores are on the same 0-100 scale (skill-check quality score; tessl review
percentage). Rounding matches compute-score.py (JS-compatible half-up) so the
number this script emits and the one compute-score.py weights agree to the digit.

Input schema (one JSON object; a tool that did not run is null or omitted):
  {
    "skillCheckScore":  <0-100 | null>,
    "tesslReviewScore": <0-100 | null>
  }

Output (stdout, one object):
  {
    "externalScore": <0-100 float | null>,   # null when neither tool ran
    "toolsUsed":     ["skill-check", "tessl"],  # tools that contributed
    "available":     <bool>                    # at least one tool ran
  }
  or {"error": ..., "code": "INVALID_INPUT"} on a schema violation.

CLI usage (mirrors compute-score.py):
  uv run combine-external-scores.py '<JSON>'                  # positional
  uv run combine-external-scores.py --json-input '<JSON>'     # explicit flag
  cat input.json | uv run combine-external-scores.py --stdin  # piped input

Exit codes:
  0  — score emitted successfully
  1  — no input / input could not be parsed as JSON
  2  — input parsed but schema/semantics invalid (error object emitted as JSON)
"""

from __future__ import annotations

import argparse
import json
import math
import sys

SKILL_CHECK = "skill-check"
TESSL = "tessl"


def round2(value):
    """Round to 2 decimals with JS-compatible half-up rounding (matches compute-score.py)."""
    return math.floor(value * 100 + 0.5) / 100


def make_error(message):
    return {"error": message, "code": "INVALID_INPUT"}


def _as_score(value, label, errors):
    """Return value if it is a 0-100 number (not bool), None if absent, else record an error."""
    if value is None:
        return None
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        errors.append(f"{label} must be a number between 0 and 100 or null (got {value!r})")
        return None
    if value < 0 or value > 100:
        errors.append(f"{label} must be between 0 and 100 (got {value})")
        return None
    return value


def combine(inp):
    """Pure combined-external-score computation. Returns the result (or error) object."""
    if inp is None or not isinstance(inp, dict):
        return make_error("Input must be a JSON object")

    errors: list[str] = []
    skill_check = _as_score(inp.get("skillCheckScore"), "skillCheckScore", errors)
    tessl = _as_score(inp.get("tesslReviewScore"), "tesslReviewScore", errors)
    if errors:
        return make_error("; ".join(errors))

    tools_used = []
    present = []
    if skill_check is not None:
        tools_used.append(SKILL_CHECK)
        present.append(skill_check)
    if tessl is not None:
        tools_used.append(TESSL)
        present.append(tessl)

    if not present:
        external_score = None
    elif len(present) == 1:
        external_score = round2(present[0])
    else:
        external_score = round2(sum(present) / len(present))

    return {
        "externalScore": external_score,
        "toolsUsed": tools_used,
        "available": bool(present),
    }


# --- CLI --------------------------------------------------------------------


def _build_parser():
    parser = argparse.ArgumentParser(
        prog="combine-external-scores",
        description=(
            "Deterministic combined external-validation score "
            "(external-validators.md §4). Averages the skill-check and tessl "
            "review scores (or passes a single available score through) into the "
            "`externalValidation` scoring input."
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=(
            "Example:\n"
            "  uv run combine-external-scores.py "
            "'{\"skillCheckScore\":80,\"tesslReviewScore\":73}'"
        ),
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "json_input",
        nargs="?",
        help="JSON object as a positional argument (single-quote it on the shell).",
    )
    src.add_argument(
        "--json-input",
        dest="json_input_flag",
        help="JSON object passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the JSON object from stdin.",
    )
    return parser


def _resolve_input(args):
    if args.stdin:
        return sys.stdin.read()
    if args.json_input_flag is not None:
        return args.json_input_flag
    if args.json_input is not None:
        return args.json_input
    return ""


def main(argv=None):
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print(
            "error: no input provided (positional arg, --json-input, or --stdin)",
            file=sys.stderr,
        )
        return 1

    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        print(json.dumps(make_error(f"Invalid JSON: {exc.msg}"), indent=2))
        return 1

    result = combine(data)
    print(json.dumps(result, indent=2))
    if isinstance(result, dict) and result.get("code") == "INVALID_INPUT":
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""Deterministic Completeness Score Calculator.

Pure-function scoring script for the SKF test-skill workflow (step-05).
Implements the weight tables, skip conditions, and proportional redistribution
defined in scoring-rules.md.

CLI usage:
  uv run compute-score.py '<JSON>'                  # JSON literal as positional arg
  uv run compute-score.py --json-input '<JSON>'     # explicit flag form
  cat input.json | uv run compute-score.py --stdin  # piped input

Input schema (one object):
  {
    "mode": "contextual" | "naive",
    "tier": "Quick" | "Forge" | "Forge+" | "Deep",
    "scores": {
      "exportCoverage": <0-100>,
      "signatureAccuracy": <0-100>,
      "typeCoverage": <0-100>,
      "coherence": <0-100>,
      "externalValidation": <0-100>
    },
    "threshold": <0-100, optional, default 80>,
    "evidenceCount": <int, optional, used for INCONCLUSIVE floor>,
    "analysisConfidence": "<optional string; 'degraded' fires the tooling cap>",
    "toolingStatus": "<optional string; a '*-missing' marker fires the tooling cap>"
  }

Verdict override (post-score caps + threshold fallback — see compute_score):
  When a cap or the threshold fallback engages, the output additionally carries
  `effectiveResult` (the final verdict after caps/fallback), `capReason`
  (string or null), `thresholdFallback` (bool), and `originalThreshold` (the
  pre-fallback threshold, or null). When neither engages, those keys are omitted
  and `result` is the final verdict. `result` itself is never mutated — it is
  always the pre-cap/pre-fallback score-vs-threshold (or evidence-floor) verdict.

Exit codes (same convention as reconcile-coverage.py):
  0  — a score was computed; the verdict may be PASS/FAIL/INCONCLUSIVE
  1  — input could not be parsed at all (no input provided, or malformed JSON)
  2  — input parsed but schema/semantics invalid

Both 1 and 2 emit an {"error": ..., "code": "INVALID_INPUT"} envelope on stdout,
so the presence of that envelope — not the specific code — is what tells a caller
the input was refused. A refused input is distinct from an unavailable script:
the numbers handed in are wrong and must be fixed, NOT hand-computed into a total
from the values this script declined to score. score.md §3c keys its
manual-redistribution fallback on that distinction (no envelope at all).
"""

from __future__ import annotations

import argparse
import json
import math
import sys

# --- Weight Tables (from scoring-rules.md) ---

CONTEXTUAL_WEIGHTS = {
    "exportCoverage": 36,
    "signatureAccuracy": 22,
    "typeCoverage": 14,
    "coherence": 18,
    "externalValidation": 10,
}

NAIVE_WEIGHTS = {
    "exportCoverage": 45,
    "signatureAccuracy": 25,
    "typeCoverage": 20,
    "coherence": 0,
    "externalValidation": 10,
}

CATEGORIES = [
    "exportCoverage",
    "signatureAccuracy",
    "typeCoverage",
    "coherence",
    "externalValidation",
]

DEFAULT_THRESHOLD = 80


# --- Redistribution equivalence classes (moved here from scoring-rules.md) ---
#
# Determinism note for maintainers — NOT reader-model context. The redistributed
# `weights` / `activeCategories` / `skippedCategories` output depends ONLY on
# three things: the base table (contextual vs naive), whether Signature Accuracy
# + Type Coverage are skipped (Quick tier OR docsOnly OR state2 OR stackSkill OR
# referenceApp), and whether External Validation is skipped (externalValidation
# is null). Every (mode × tier × docsOnly × state2 × stackSkill × referenceApp)
# cell with an identical skip set + base table therefore reduces to the same
# equivalence class below and emits identical final weights for identical input
# scores — only the `skipReasons` string differs. The prompt (score.md) never
# names a class or fixture; it sets the flags and reads the script's output back.
# The rightmost column pins each class to a representative fixture in
# test/fixtures/compute-score-contract.json (see test-compute-score-contract.py);
# "(equiv.)" cells reduce algebraically to a listed representative.
#
# |  # | mode       | tier   | docsOnly | state2 | base       | sig/type skip          | ext skip | class | representative fixture           |
# |----|------------|--------|----------|--------|------------|------------------------|----------|-------|----------------------------------|
# |  1 | contextual | Deep   | F        | F      | contextual | —                      | if null  | A     | suite_a_all_active               |
# |  2 | contextual | Forge+ | F        | F      | contextual | —                      | if null  | A     | suite_k_forge_plus               |
# |  3 | contextual | Forge  | F        | F      | contextual | —                      | if null  | A     | suite_p_contextual_forge         |
# |  4 | contextual | Quick  | F        | F      | contextual | Quick tier             | if null  | B     | suite_c_quick_tier               |
# |  5 | contextual | Deep   | T        | F      | contextual | docs-only              | if null  | B     | suite_r_contextual_deep_docsonly |
# |  6 | contextual | Forge+ | T        | F      | contextual | docs-only              | if null  | B     | (equiv.)                         |
# |  7 | contextual | Forge  | T        | F      | contextual | docs-only              | if null  | B     | (equiv.)                         |
# |  8 | contextual | Quick  | T        | F      | contextual | Quick tier + docs-only | if null  | B     | suite_f_docs_only                |
# |  9 | contextual | Deep   | F        | T      | contextual | State 2                | if null  | B     | suite_g_state2                   |
# | 10 | contextual | Forge+ | F        | T      | contextual | State 2                | if null  | B     | (equiv.)                         |
# | 11 | contextual | Forge  | F        | T      | contextual | State 2                | if null  | B     | (equiv.)                         |
# | 12 | contextual | Quick  | F        | T      | contextual | Quick + State 2        | if null  | B     | (equiv.)                         |
# | 13 | contextual | *      | T        | T      | contextual | docs-only + State 2    | if null  | B     | (equiv.)                         |
# | 14 | naive      | Deep   | F        | F      | naive      | —                      | if null  | C     | suite_q_naive_deep               |
# | 15 | naive      | Forge+ | F        | F      | naive      | —                      | if null  | C     | (equiv.)                         |
# | 16 | naive      | Forge  | F        | F      | naive      | —                      | if null  | C     | suite_b_naive                    |
# | 17 | naive      | Quick  | F        | F      | naive      | Quick tier             | if null  | D     | suite_e_triple_skip              |
# | 18 | naive      | Deep   | T        | F      | naive      | docs-only              | if null  | D     | suite_s_naive_deep_docsonly      |
# | 19 | naive      | Forge+ | T        | F      | naive      | docs-only              | if null  | D     | (equiv.)                         |
# | 20 | naive      | Forge  | T        | F      | naive      | docs-only              | if null  | D     | (equiv.)                         |
# | 21 | naive      | Quick  | T        | F      | naive      | Quick + docs-only      | if null  | D     | suite_o_input_echo               |
# | 22 | naive      | Deep   | F        | T      | naive      | State 2                | if null  | D     | suite_t_naive_state2             |
# | 23 | naive      | Forge+ | F        | T      | naive      | State 2                | if null  | D     | (equiv.)                         |
# | 24 | naive      | Forge  | F        | T      | naive      | State 2                | if null  | D     | (equiv.)                         |
# | 25 | naive      | Quick  | F        | T      | naive      | Quick + State 2        | if null  | D     | (equiv.)                         |
# | 26 | naive      | *      | T        | T      | naive      | docs-only + State 2    | if null  | D     | (equiv.)                         |
#
# Stack skills (stackSkill) and reference-app skills (referenceApp) share the
# skip set of the docsOnly/state2 rows: contextual → class B, naive → class D;
# only skipReasons differs (fixtures suite_u_stack_skill_deep,
# suite_v_reference_app_deep). Quick-tier rows (4, 8, 12, 17, 21, 25) then feed
# the minimum-evidence floor in the result block below (§7 of compute_score),
# which can force INCONCLUSIVE after this pre-floor redistribution.


# --- Helpers ---


def round2(value):
    """Round to 2 decimal places using JavaScript-compatible rounding.

    JavaScript Math.round rounds .5 up (away from zero for positives).
    Python's built-in round uses banker's rounding (.5 to even).
    We replicate JS behavior: floor(value * 100 + 0.5) / 100.
    """
    return math.floor(value * 100 + 0.5) / 100


def make_error(message):
    return {"error": message, "code": "INVALID_INPUT"}


# --- Validation ---


BOOL_FIELDS = ("docsOnly", "state2", "stackSkill")


def validate_input(inp):
    if inp is None or not isinstance(inp, dict):
        return "Input must be a JSON object"

    if not inp.get("mode") or inp["mode"] not in ("contextual", "naive"):
        return 'Missing or invalid required field: mode (must be "contextual" or "naive")'

    valid_tiers = ["Quick", "Forge", "Forge+", "Deep"]
    if not inp.get("tier") or inp["tier"] not in valid_tiers:
        return f"Missing or invalid required field: tier (must be one of: {', '.join(valid_tiers)})"

    # H3: reject string booleans — bare true/false only.
    # `bool` is a subclass of `int` in Python; accept only actual bools or the
    # absence of the field. Strings like "true"/"false" and ints 0/1 are rejected
    # to catch the common YAML/JSON hand-editing mistake of quoting the value.
    for field in BOOL_FIELDS:
        if field in inp and inp[field] is not None:
            if not isinstance(inp[field], bool):
                return (
                    f"Field `{field}` must be a bare boolean (true/false). "
                    f"Got {type(inp[field]).__name__}: {inp[field]!r}. "
                    "String 'true'/'false' or 0/1 is not accepted — "
                    "this typically indicates a YAML/JSON quoting mistake."
                )

    if "scores" not in inp or not isinstance(inp.get("scores"), dict):
        return "Missing required field: scores"

    if inp["scores"].get("exportCoverage") is None:
        return "scores.exportCoverage is required and cannot be null"

    threshold = inp.get("threshold")
    if threshold is not None:
        if not isinstance(threshold, (int, float)) or threshold < 0 or threshold > 100:
            return "threshold must be a number between 0 and 100"

    for str_field in ("analysisConfidence", "toolingStatus"):
        val = inp.get(str_field)
        if val is not None and not isinstance(val, str):
            return f"{str_field} must be a string or null, got: {type(val).__name__}"

    for cat in CATEGORIES:
        score = inp["scores"].get(cat)
        if score is not None:
            if isinstance(score, bool) or not isinstance(score, (int, float)):
                return f"scores.{cat} must be a number or null, got: {type(score).__name__}"
            if score < 0 or score > 100:
                return f"scores.{cat} must be between 0 and 100, got: {score}"

    return None


# --- Core Scoring Function ---


def compute_score(inp):
    # 1. Validate
    validation_error = validate_input(inp)
    if validation_error:
        return make_error(validation_error)

    mode = inp["mode"]
    tier = inp["tier"]
    docs_only = inp.get("docsOnly") is True
    state2 = inp.get("state2") is True
    stack_skill = inp.get("stackSkill") is True
    reference_app = inp.get("referenceApp") is True
    threshold = inp.get("threshold") if inp.get("threshold") is not None else DEFAULT_THRESHOLD
    scores = inp["scores"]

    # 2. Select base weight table
    base_weights = dict(NAIVE_WEIGHTS if mode == "naive" else CONTEXTUAL_WEIGHTS)

    # 3. Determine skip set
    skip_reasons = {}
    skip_sig_type = tier == "Quick" or docs_only or state2 or stack_skill or reference_app

    if skip_sig_type:
        reasons = []
        if tier == "Quick":
            reasons.append("Quick tier")
        if docs_only:
            reasons.append("docs-only mode")
        if state2:
            reasons.append("State 2 (provenance-map)")
        if stack_skill:
            reasons.append("stack skill (external type surface)")
        if reference_app:
            reasons.append("reference-app (no library export signatures)")
        reason = " + ".join(reasons)
        skip_reasons["signatureAccuracy"] = reason
        skip_reasons["typeCoverage"] = reason

    if scores.get("externalValidation") is None:
        skip_reasons["externalValidation"] = "No external validators available"

    # Collect warnings
    warnings = []
    for cat in skip_reasons:
        if scores.get(cat) is not None:
            warnings.append(
                f"{cat} score provided ({scores[cat]}) but category is skipped — score ignored"
            )

    # Validate active categories have scores
    skipped_set = set(skip_reasons.keys())
    for cat in CATEGORIES:
        is_active = cat not in skipped_set and base_weights[cat] > 0
        score_missing = scores.get(cat) is None
        if is_active and score_missing:
            return make_error(
                f"Category {cat} is active but score is null. "
                "Provide a numeric score or set the appropriate skip condition."
            )

    # 4. Redistribute weights
    adjusted_weights = dict(base_weights)
    for cat in skip_reasons:
        adjusted_weights[cat] = 0

    sum_active_weights = sum(adjusted_weights[cat] for cat in CATEGORIES)

    final_weights = {}
    for cat in CATEGORIES:
        if adjusted_weights[cat] == 0:
            final_weights[cat] = 0
        else:
            final_weights[cat] = round2((adjusted_weights[cat] / sum_active_weights) * 100)

    # 5. Compute weighted scores
    weighted_scores = {}
    for cat in CATEGORIES:
        if final_weights[cat] == 0:
            weighted_scores[cat] = 0
        else:
            weighted_scores[cat] = round2((final_weights[cat] / 100) * scores[cat])

    # 6. Compute total
    total_score = round2(sum(weighted_scores[cat] for cat in CATEGORIES))

    # Weight sum for verification
    weight_sum = round2(sum(final_weights[cat] for cat in CATEGORIES))

    # 7. Determine result — MINIMUM-EVIDENCE FLOOR first, then PASS/FAIL.
    # skf-test-skill grades other skills; a false PASS is catastrophic.
    # If the evidence base is too thin to cross-validate itself, force
    # INCONCLUSIVE (a gate, not a pass/fail). See scoring-rules.md.
    active_categories = [cat for cat in CATEGORIES if final_weights[cat] > 0]
    skipped_categories = [
        cat for cat in CATEGORIES if cat in skipped_set or base_weights[cat] == 0
    ]

    floor_reasons = []
    if len(active_categories) < 2:
        floor_reasons.append(
            f"insufficient evidence: only {len(active_categories)} active category"
        )
    elif tier == "Quick" and active_categories == ["exportCoverage"]:
        # Defensive second check: if somehow there are >= 2 active categories
        # but they collapse to just exportCoverage (shouldn't happen given the
        # first clause, kept for robustness), still force INCONCLUSIVE.
        floor_reasons.append(
            "Quick tier: Export Coverage alone is insufficient evidence "
            "— add a second active category by upgrading tier or enabling "
            "external validators"
        )
    elif tier == "Quick":
        # Cover the case where Export Coverage is the only active category
        # carrying signal in Quick tier even when technically another
        # non-contributing category survived redistribution.
        non_export_active_scores = [
            scores.get(cat) for cat in active_categories if cat != "exportCoverage"
        ]
        # All other active categories have a zero score => Export Coverage is
        # the sole real contributor.
        if active_categories and "exportCoverage" in active_categories and all(
            (s == 0) for s in non_export_active_scores
        ) and non_export_active_scores:
            floor_reasons.append(
                "Quick tier: Export Coverage is the sole scoring contributor "
                "(other active categories scored 0) — insufficient evidence"
            )

    if floor_reasons:
        result = "INCONCLUSIVE"
    else:
        result = "PASS" if total_score >= threshold else "FAIL"

    # 7b. Post-score caps + threshold fallback — deterministic verdict override.
    # Lifted from score.md §3d/§4b so the cap<->fallback interaction is centralized
    # and unit-tested here rather than re-derived in the prompt. The minimum-
    # evidence floor (INCONCLUSIVE) is a gate that NO cap or fallback may override,
    # so all of this is skipped when result == "INCONCLUSIVE".
    analysis_confidence = inp.get("analysisConfidence")
    tooling_status = inp.get("toolingStatus")

    effective_result = result
    cap_reason = None
    threshold_fallback = False
    original_threshold = None
    cap_fired = False

    if result != "INCONCLUSIVE":
        cap_reasons = []
        # Cap 1 — tooling degraded: analysis confidence is "degraded", or a
        # missing-helper marker (e.g. python3-missing, frontmatter-validator-missing).
        tooling_degraded = analysis_confidence == "degraded" or (
            isinstance(tooling_status, str) and "missing" in tooling_status
        )
        if tooling_degraded:
            cap_fired = True
            cap_reasons.append(
                "tooling degraded — capped below threshold until helper restored"
            )
        # Cap 2 — docs-only mode with no external validators available.
        if docs_only and scores.get("externalValidation") is None:
            cap_fired = True
            cap_reasons.append(
                "docs-only without external validators — capped below threshold"
            )
        if cap_fired:
            cap_reason = "; ".join(cap_reasons)
            # A cap only flips a PASS into FAIL; a pre-existing FAIL stays FAIL.
            if result == "PASS":
                effective_result = "FAIL"

        # Threshold fallback — convert a FAIL into PASS at the 80 floor when the
        # RAW totalScore clears 80 and the effective threshold was above 80. The
        # fallback reads the raw totalScore (matching score.md §4b's documented
        # `totalScore >= 80`), not the capped value; because it also requires
        # threshold > 80 (so threshold - 1 >= 80), min(totalScore, threshold - 1)
        # is itself >= 80 whenever totalScore >= 80, so raw-vs-capped is the same
        # decision — raw is used for spec fidelity and testability.
        if effective_result == "FAIL" and total_score >= 80 and threshold > 80:
            threshold_fallback = True
            original_threshold = threshold
            effective_result = "PASS"

    overrides_engaged = cap_fired or threshold_fallback

    # Build scores echo with null preservation
    scores_echo = {}
    for cat in CATEGORIES:
        scores_echo[cat] = scores.get(cat)

    output = {
        "input": {
            "mode": mode,
            "tier": tier,
            "docsOnly": docs_only,
            "state2": state2,
            "stackSkill": stack_skill,
            "threshold": threshold,
            "scores": scores_echo,
        },
        "activeCategories": active_categories,
        "skippedCategories": skipped_categories,
        "skipReasons": skip_reasons,
        "weights": final_weights,
        "weightedScores": weighted_scores,
        "totalScore": total_score,
        "threshold": threshold,
        "result": result,
        "weightSum": weight_sum,
    }

    # Verdict-override fields — emitted as an atomic group only when a post-score
    # cap or the threshold fallback engaged (mirrors the conditional `warnings` /
    # `inconclusiveReasons` fields). When absent, `result` is the final verdict.
    if overrides_engaged:
        output["effectiveResult"] = effective_result
        output["capReason"] = cap_reason
        output["thresholdFallback"] = threshold_fallback
        output["originalThreshold"] = original_threshold

    if warnings:
        output["warnings"] = warnings

    if floor_reasons:
        output["inconclusiveReasons"] = floor_reasons

    return output


# --- CLI Entry Point ---


def _build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        prog="compute-score",
        description=(
            "Deterministic Completeness Score calculator. Input is a single JSON "
            "object describing mode, tier, scores, and optional threshold/evidence "
            "count; output is the verdict + redistributed score breakdown."
        ),
        epilog=(
            "Example:\n"
            "  uv run compute-score.py "
            "'{\"mode\":\"contextual\",\"tier\":\"Deep\","
            "\"scores\":{\"exportCoverage\":92,\"signatureAccuracy\":85,"
            "\"typeCoverage\":100,\"coherence\":80,\"externalValidation\":78}}'"
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "json_input",
        nargs="?",
        help="JSON object as a positional argument (single-quote it on the shell).",
    )
    src.add_argument(
        "--json-input",
        dest="json_input_flag",
        help="JSON object passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the JSON object from stdin.",
    )
    return parser


def _resolve_input(args: argparse.Namespace) -> str:
    if args.stdin:
        return sys.stdin.read()
    if args.json_input_flag is not None:
        return args.json_input_flag
    if args.json_input is not None:
        return args.json_input
    return ""


def main(argv: list[str] | None = None) -> int:
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print("error: no input provided (positional arg, --json-input, or --stdin)", file=sys.stderr)
        return 1

    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        print(json.dumps(make_error(f"Invalid JSON: {exc.msg}"), indent=2))
        return 1

    result = compute_score(data)
    print(json.dumps(result, indent=2))
    # A rejected input exits 2, matching reconcile-coverage.py. Exiting 0 here
    # made a malformed scoring input indistinguishable from a scored run, so the
    # error envelope could be skimmed past and the score hand-computed from the
    # very numbers the script refused. score.md §3c keys the manual-redistribution
    # fallback on this distinction.
    if isinstance(result, dict) and result.get("code") == "INVALID_INPUT":
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# ///
"""Deterministic Documented-vs-Source coverage reconciliation.

Set-reconciliation helper for the SKF test-skill workflow (coverage-check.md
§2c). Distinct from compute-score.py (which owns weight tables, redistribution,
and the INCONCLUSIVE floor): this script performs the coverage *numerator*
arithmetic that §2c calls the "Deterministic Intersection" — turning two
already-extracted name lists (the §1 documented inventory and the §2 source
barrel) into a reproducible {documented, missing, stale, exportCoverage}
result, so the numerator no longer swings between runs on split-body skills.

It covers the three branches §2c enumerates:

  * "barrel"  (enumerated path) — intersect documented_set against the source
    barrel_set: Documented = |documented_set ∩ barrel_set|,
    Missing = barrel_set − documented_set, Stale = documented_set − barrel_set,
    Export Coverage = |Documented| / |barrel_set| * 100.

  * "scalar"  (§4 priority-1 effective_denominator, no enumerated name set) —
    grep each documented name across SKILL.md ∪ references/*.md;
    Documented = count present, denominator = effective_denominator,
    Missing = max(0, denominator − Documented), Stale not enumerable (empty),
    Export Coverage = min(100, Documented / denominator * 100).

  * "stack"   (skill_type == "stack", empty source barrel) — grep each
    composition-surface name (provenance-map cited contracts, ::-excluded, or
    libraries + integration_pairs — resolved upstream) across
    SKILL.md ∪ references/*.md; denominator = stack_denominator,
    Missing = max(0, denominator − Documented), Stale not enumerable,
    Export Coverage = min(100, Documented / stack_denominator * 100).

The two grep branches bound their outputs because the numerator and the
denominator are independent measures of different sets — the grep count can
exceed a consumer-surface denominator without either being wrong. Unbounded,
that produced a negative Missing and a >100% coverage that compute-score.py
rejects as out of range. `documented` is left as the true count (it is reported
verbatim as "Documented in SKILL.md"); the derived ratio is what gets bounded,
and `numeratorSurplus` / `coverageUncapped` / `coverageCapped` report the
overshoot so a deflated denominator stays visible instead of being swallowed.

CLI usage:
  uv run reconcile-coverage.py '<JSON>'                  # JSON literal positional
  uv run reconcile-coverage.py --json-input '<JSON>'     # explicit flag form
  cat input.json | uv run reconcile-coverage.py --stdin  # piped input

Input schema (one object):
  {
    "denominatorSource": "barrel" | "scalar" | "stack",   # required
    "exports": [ {"name": "...", "kind": "..."}, ... ],    # §1 inventory
    "barrelSet": ["a", "b", ...],                          # barrel: resolved name set
    "perFileResults": [ {"exports_found": ["a", ...]} ],   # barrel: union'd if no barrelSet
    "denominatorValue": <int>,                             # scalar/stack: resolved denominator
    "compositionNames": ["lib::x", ...],                   # stack: names to grep
    "skillPackagePath": "/path/to/skill"                   # scalar/stack: SKILL.md ∪ references
  }

Output (stdout, one object):
  {
    "branch": "enumerated" | "scalar" | "stack",
    "denominatorSource": "barrel" | "scalar" | "stack",
    "denominator": <int>,
    "documented": <int>,
    "missing": [names],          # enumerated: source names not documented; scalar/stack: []
    "missingCount": <int>,       # always present; never negative
    "stale": [names],            # enumerated: documented names not in source; scalar/stack: []
    "staleCount": <int>,         # always present
    "staleApplicable": <bool>,   # false for scalar/stack (no barrel to enumerate)
    "exportCoverage": <float>,   # scalar/stack: capped at 100
    "numeratorSurplus": <int>,   # scalar/stack only: max(0, documented − denominator)
    "coverageUncapped": <float>, # scalar/stack only: the ratio before the cap
    "coverageCapped": <bool>     # scalar/stack only: true when the cap bound the ratio
  }

Exit codes:
  0  — reconciliation emitted successfully
  1  — no input / input could not be parsed as JSON
  2  — input parsed but schema/semantics invalid (error object emitted as JSON)
"""

from __future__ import annotations

import argparse
import json
import math
import sys
from pathlib import Path

VALID_SOURCES = ("barrel", "scalar", "stack")


def round2(value):
    """Round to 2 decimals with JS-compatible half-up rounding (matches compute-score.py)."""
    return math.floor(value * 100 + 0.5) / 100


def make_error(message):
    return {"error": message, "code": "INVALID_INPUT"}


# --- Documented-set derivation ---------------------------------------------


def build_documented_set(exports):
    """De-duplicated set of `name` from the §1 inventory, excluding kind:"method".

    Methods are members of an already-counted class/type, not top-level barrel
    exports (coverage-check.md §2c step 1).
    """
    names = set()
    for entry in exports or []:
        if not isinstance(entry, dict):
            continue
        if entry.get("kind") == "method":
            continue
        name = entry.get("name")
        if isinstance(name, str) and name:
            names.add(name)
    return names


def build_barrel_set(inp):
    """barrel_set = explicit `barrelSet` if given, else union of exports_found[]."""
    if inp.get("barrelSet") is not None:
        return {n for n in inp["barrelSet"] if isinstance(n, str) and n}
    barrel = set()
    for result in inp.get("perFileResults") or []:
        if not isinstance(result, dict):
            continue
        for name in result.get("exports_found") or []:
            if isinstance(name, str) and name:
                barrel.add(name)
    return barrel


# --- Grep (documented-name presence) ---------------------------------------


def load_doc_text(skill_package_path):
    """Concatenate SKILL.md ∪ references/*.md text for name-presence grepping.

    Deterministic: references are read in sorted order. Reads as UTF-8 so
    non-ASCII exports do not mojibake on Windows (cp1252 default).
    """
    root = Path(skill_package_path)
    parts = []
    skill_md = root / "SKILL.md"
    if skill_md.is_file():
        parts.append(skill_md.read_text(encoding="utf-8"))
    refs_dir = root / "references"
    if refs_dir.is_dir():
        for ref in sorted(refs_dir.glob("*.md")):
            parts.append(ref.read_text(encoding="utf-8"))
    return "\n".join(parts)


def names_present(names, doc_text):
    """Return the sorted, de-duplicated set of `names` that appear in doc_text.

    Substring containment, case-sensitive — mirrors `grep "{name}"`.
    """
    present = {n for n in names if isinstance(n, str) and n and n in doc_text}
    return sorted(present)


# --- Core reconciliation ----------------------------------------------------


def _validate(inp):
    if inp is None or not isinstance(inp, dict):
        return "Input must be a JSON object"
    source = inp.get("denominatorSource")
    if source not in VALID_SOURCES:
        return (
            "Missing or invalid required field: denominatorSource "
            f"(must be one of: {', '.join(VALID_SOURCES)})"
        )
    if source == "barrel":
        if inp.get("barrelSet") is None and inp.get("perFileResults") is None:
            return "barrel branch requires either `barrelSet` or `perFileResults`"
    else:  # scalar / stack
        dv = inp.get("denominatorValue")
        if not isinstance(dv, int) or isinstance(dv, bool) or dv < 0:
            return f"{source} branch requires integer `denominatorValue` >= 0"
        if not inp.get("skillPackagePath"):
            return f"{source} branch requires `skillPackagePath` to grep SKILL.md ∪ references"
        if source == "stack" and inp.get("compositionNames") is None:
            return "stack branch requires `compositionNames` (composition-surface names to grep)"
    return None


def reconcile(inp, doc_text=None):
    """Pure reconciliation. `doc_text` may be injected (tests); else loaded from disk."""
    err = _validate(inp)
    if err:
        return make_error(err)

    source = inp["denominatorSource"]

    if source == "barrel":
        documented_set = build_documented_set(inp.get("exports"))
        barrel_set = build_barrel_set(inp)
        if not barrel_set:
            return make_error(
                "barrel branch: barrel_set is empty (denominator 0) — "
                "Export Coverage is undefined; upstream §2b zero-exports guard should HALT"
            )
        documented_names = documented_set & barrel_set
        missing = sorted(barrel_set - documented_set)
        stale = sorted(documented_set - barrel_set)
        denominator = len(barrel_set)
        documented = len(documented_names)
        return {
            "branch": "enumerated",
            "denominatorSource": source,
            "denominator": denominator,
            "documented": documented,
            "missing": missing,
            "missingCount": len(missing),
            "stale": stale,
            "staleCount": len(stale),
            "staleApplicable": True,
            "exportCoverage": round2(documented / denominator * 100),
        }

    # scalar / stack — grep-based numerator against SKILL.md ∪ references
    denominator = inp["denominatorValue"]
    if doc_text is None:
        doc_text = load_doc_text(inp["skillPackagePath"])

    if source == "scalar":
        candidates = sorted(build_documented_set(inp.get("exports")))
    else:  # stack
        candidates = [n for n in inp["compositionNames"] if isinstance(n, str) and n]

    present = names_present(candidates, doc_text)
    documented = len(present)

    if denominator == 0:
        return make_error(
            f"{source} branch: denominatorValue is 0 — Export Coverage is undefined; "
            "upstream §2b guard should HALT before reconciliation"
        )

    # The grep numerator and the resolved denominator are independent measures,
    # so `documented` can legitimately exceed `denominator` — a consumer-surface
    # denominator counts one surface while the documented body may also name
    # migration aliases, re-exported sibling symbols, and other extras. Left
    # unbounded that yields a negative Missing and >100% coverage (which
    # compute-score.py then rejects outright as out of range).
    #
    # `documented` stays the true grep count: it is reported as "Documented in
    # SKILL.md", so capping it would print a number that was never measured.
    # The derived ratio is what gets bounded, and the surplus is surfaced rather
    # than swallowed so a deflated denominator stays visible.
    surplus = max(0, documented - denominator)
    coverage_uncapped = round2(documented / denominator * 100)
    return {
        "branch": source,
        "denominatorSource": source,
        "denominator": denominator,
        "documented": documented,
        "missing": [],
        "missingCount": max(0, denominator - documented),
        "numeratorSurplus": surplus,
        "stale": [],
        "staleCount": 0,
        "staleApplicable": False,
        "exportCoverage": min(100.0, coverage_uncapped),
        "coverageUncapped": coverage_uncapped,
        "coverageCapped": surplus > 0,
    }


# --- CLI --------------------------------------------------------------------


def _build_parser():
    parser = argparse.ArgumentParser(
        prog="reconcile-coverage",
        description=(
            "Deterministic Documented-vs-Source coverage reconciliation "
            "(coverage-check.md §2c). Consumes the §1 documented inventory and "
            "the §2 source barrel (or a resolved scalar/stack denominator + the "
            "skill package path) and emits {documented, missing, stale, "
            "exportCoverage} so the coverage numerator is reproducible."
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=(
            "Example (barrel branch):\n"
            "  uv run reconcile-coverage.py "
            "'{\"denominatorSource\":\"barrel\","
            "\"exports\":[{\"name\":\"a\",\"kind\":\"function\"}],"
            "\"barrelSet\":[\"a\",\"b\"]}'"
        ),
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "json_input",
        nargs="?",
        help="JSON object as a positional argument (single-quote it on the shell).",
    )
    src.add_argument(
        "--json-input",
        dest="json_input_flag",
        help="JSON object passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the JSON object from stdin.",
    )
    return parser


def _resolve_input(args):
    if args.stdin:
        return sys.stdin.read()
    if args.json_input_flag is not None:
        return args.json_input_flag
    if args.json_input is not None:
        return args.json_input
    return ""


def main(argv=None):
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print(
            "error: no input provided (positional arg, --json-input, or --stdin)",
            file=sys.stderr,
        )
        return 1

    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        print(json.dumps(make_error(f"Invalid JSON: {exc.msg}"), indent=2))
        return 1

    result = reconcile(data)
    print(json.dumps(result, indent=2))
    if isinstance(result, dict) and result.get("code") == "INVALID_INPUT":
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# ///
"""Deterministic schema validation of the §1 subagent inventory JSON.

Structural validator for the SKF test-skill coverage step (coverage-check.md
§1a "Schema validation" block). test-skill is a quality gate, so it must not
trust subagent output blindly: before any downstream step consumes the
documented inventory, the shape is validated. That validation is pure
structural checking — strip wrapping fences, parse JSON, assert required keys
and types, assert each export entry is a dict with a non-empty `name` and a
`kind` drawn from a fixed enum, and assert each cross-check mismatch carries its
five fields — with exactly one correct pass/fail per input and no
interpretation of meaning. It is therefore script work, not prompt work; the
prompt reads this verdict and owns the HALT decision plus the separate
grep ground-truth spot-check.

Distinct from reconcile-coverage.py (coverage numerator arithmetic) and
compute-score.py (weight tables): this script only validates the *shape* of the
subagent inventory and, on success, echoes the fence-stripped parsed inventory
back so the prompt consumes it directly instead of re-parsing the raw response.

CLI usage (mirrors reconcile-coverage.py):
  uv run validate-inventory.py '<raw response>'                 # positional
  uv run validate-inventory.py --json-input '<raw response>'    # explicit flag
  echo '<raw response>' | uv run validate-inventory.py --stdin  # piped input

Input: the subagent's RAW response text (JSON, optionally wrapped in a markdown
code fence — a leading line of three backticks with an optional language tag and
a trailing line of three backticks are stripped before parsing).

Output (stdout, one object):
  {
    "valid": <bool>,             # true only when parse + every schema check pass
    "violations": [<str>, ...],  # human-readable failures; empty when valid
    "rejectedCount": <int>,      # count of malformed exports[] entries
    "exportsCount": <int>,       # len(exports) when parsed; 0 otherwise
    "inventory": {...} | null    # fence-stripped parsed inventory when valid; null otherwise
  }

Exit codes:
  0  — inventory valid
  1  — no input provided (usage error)
  2  — inventory invalid (parse failure or schema violation; result JSON emitted)
"""

from __future__ import annotations

import argparse
import json
import re
import sys

# Fixed enum for exports[].kind — the constructs SKF documents across languages
# and skill types: JS/TS (function/class/type/constant/hook/interface/method),
# Rust public-API items (struct/enum/trait/macro, alongside the shared
# type/constant/function), and stack-composition scaffolds (adapter). Mirrors the
# enum documented in coverage-check.md §1a.
VALID_KINDS = (
    "function",
    "class",
    "type",
    "constant",
    "hook",
    "interface",
    "method",
    "struct",
    "enum",
    "trait",
    "macro",
    "adapter",
)

REQUIRED_MISMATCH_FIELDS = (
    "export",
    "skill_md_line",
    "reference_file",
    "reference_line",
    "issue",
)

_FENCE_OPEN = re.compile(r"^```[A-Za-z0-9_-]*$")
_FENCE_CLOSE = re.compile(r"^```$")


def strip_fences(text):
    """Remove a wrapping markdown code fence, matching coverage-check.md §1a step 1.

    When the first non-empty line is three backticks (optionally with a language
    tag) and the last non-empty line is three backticks, drop those two lines.
    Otherwise return the text unchanged.
    """
    lines = text.split("\n")
    non_empty_idx = [i for i, ln in enumerate(lines) if ln.strip()]
    if len(non_empty_idx) < 2:
        return text
    first, last = non_empty_idx[0], non_empty_idx[-1]
    if _FENCE_OPEN.match(lines[first].strip()) and _FENCE_CLOSE.match(lines[last].strip()):
        kept = [ln for i, ln in enumerate(lines) if i != first and i != last]
        return "\n".join(kept)
    return text


def validate_inventory(raw):
    """Pure structural validation of the subagent inventory response.

    `raw` is the raw response text. Returns the result dict documented in the
    module docstring. Deterministic: identical input yields an identical verdict.
    """
    violations = []
    inner = strip_fences(raw)
    try:
        data = json.loads(inner)
    except json.JSONDecodeError as exc:
        return {
            "valid": False,
            "violations": [f"subagent response not valid JSON: {exc.msg}"],
            "rejectedCount": 0,
            "exportsCount": 0,
            "inventory": None,
        }

    if not isinstance(data, dict):
        return {
            "valid": False,
            "violations": ["subagent response is not a JSON object"],
            "rejectedCount": 0,
            "exportsCount": 0,
            "inventory": None,
        }

    exports = data.get("exports")
    if not isinstance(exports, list):
        violations.append(
            "missing/typo: `exports` must be present and a list"
        )
        exports = []

    cross = data.get("cross_check_mismatches")
    if not isinstance(cross, list):
        violations.append(
            "missing/typo: `cross_check_mismatches` must be present and a list (may be empty)"
        )
        cross = []

    # Per-entry export validation — count rejections.
    rejected = 0
    for idx, entry in enumerate(exports):
        if not isinstance(entry, dict):
            rejected += 1
            continue
        name = entry.get("name")
        kind = entry.get("kind")
        if not (isinstance(name, str) and name):
            rejected += 1
            continue
        if kind not in VALID_KINDS:
            rejected += 1
            continue
    if rejected > 0:
        subject = "entry does" if rejected == 1 else "entries do"
        violations.append(
            f"{rejected} exports[] {subject} not match schema "
            f"(each needs a non-empty string `name` and a `kind` in "
            f"{{{', '.join(VALID_KINDS)}}})"
        )

    # Non-empty cross-check mismatch entries must carry all five fields.
    for idx, entry in enumerate(cross):
        if not isinstance(entry, dict):
            violations.append(f"cross_check_mismatches[{idx}] is not an object")
            continue
        missing = [f for f in REQUIRED_MISMATCH_FIELDS if f not in entry]
        if missing:
            violations.append(
                f"cross_check_mismatches[{idx}] missing field(s): {', '.join(missing)}"
            )

    valid = not violations
    return {
        "valid": valid,
        "violations": violations,
        "rejectedCount": rejected,
        "exportsCount": len(exports),
        "inventory": data if valid else None,
    }


# --- CLI --------------------------------------------------------------------


def _build_parser():
    parser = argparse.ArgumentParser(
        prog="validate-inventory",
        description=(
            "Deterministic schema validation of the §1 subagent inventory JSON "
            "(coverage-check.md §1a). Strips wrapping fences, parses, and asserts "
            "the required-keys/types/enum/mismatch-field contract, returning "
            "{valid, violations, rejectedCount, exportsCount, inventory}."
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "raw_input",
        nargs="?",
        help="Raw subagent response as a positional argument.",
    )
    src.add_argument(
        "--json-input",
        dest="raw_input_flag",
        help="Raw subagent response passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the raw subagent response from stdin.",
    )
    return parser


def _resolve_input(args):
    if args.stdin:
        return sys.stdin.read()
    if args.raw_input_flag is not None:
        return args.raw_input_flag
    if args.raw_input is not None:
        return args.raw_input
    return ""


def main(argv=None):
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print(
            "error: no input provided (positional arg, --json-input, or --stdin)",
            file=sys.stderr,
        )
        return 1

    result = validate_inventory(raw)
    print(json.dumps(result, indent=2))
    return 0 if result["valid"] else 2


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""Deterministic numerator ground-truth verifier.

Grep-and-count helper for the SKF test-skill workflow (coverage-check.md §4b
"Numerator ground-truth"). It fires only on the inflation signature
(`stats.exports_documented == effective_denominator` exactly): the numerator
equals the denominator, the tell of a documented count padded to force 100%
coverage. On that signature the declared count is not trusted — every declared
export name is greped against the skill's documentation surface and only the
names that actually appear count toward coverage.

Distinct from reconcile-coverage.py (whose scalar/stack branches grep the §1
inventory names and return only a residual missing *count*): this script greps
the full declared metadata/provenance set and enumerates the *absent* names so
§4b can list them in a High-severity gap. Both scripts share the same grep
semantics (SKILL.md ∪ sorted references/*.md, UTF-8, case-sensitive substring
containment — mirroring `grep "{name}"`) so their numerators agree.

Input schema (one JSON object):
  {
    "declaredNames": ["foo", "Bar::baz", ...],  # metadata.exports[] / provenance declared set
    "skillPackagePath": "/path/to/skill"        # SKILL.md ∪ references/*.md
  }

Output (stdout, one object):
  {
    "declared":  <int>,           # de-duplicated declared name count
    "verified":  <int>,           # declared names present in the doc surface
    "present":   [names],         # sorted
    "absent":    [names],         # sorted (declared − verified) — the gap list
    "inflated":  <bool>           # verified < declared (numerator was padded)
  }
  or {"error": ..., "code": "INVALID_INPUT"} on a schema violation.

`verified` is the numerator §4b uses for Export Coverage (it overrides the
declared count when `inflated` is true).

CLI usage (mirrors reconcile-coverage.py):
  uv run verify-declared-numerator.py '<JSON>'                  # positional
  uv run verify-declared-numerator.py --json-input '<JSON>'     # explicit flag
  cat input.json | uv run verify-declared-numerator.py --stdin  # piped input

Exit codes:
  0  — verification emitted successfully
  1  — no input / input could not be parsed as JSON
  2  — input parsed but schema/semantics invalid (error object emitted as JSON)
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path


def make_error(message):
    return {"error": message, "code": "INVALID_INPUT"}


def load_doc_text(skill_package_path):
    """Concatenate SKILL.md ∪ references/*.md text for name-presence grepping.

    Deterministic: references are read in sorted order. Reads as UTF-8 so
    non-ASCII exports do not mojibake on Windows (cp1252 default). Mirrors
    reconcile-coverage.py.load_doc_text so the two numerators agree.
    """
    root = Path(skill_package_path)
    parts = []
    skill_md = root / "SKILL.md"
    if skill_md.is_file():
        parts.append(skill_md.read_text(encoding="utf-8"))
    refs_dir = root / "references"
    if refs_dir.is_dir():
        for ref in sorted(refs_dir.glob("*.md")):
            parts.append(ref.read_text(encoding="utf-8"))
    return "\n".join(parts)


def _validate(inp):
    if inp is None or not isinstance(inp, dict):
        return "Input must be a JSON object"
    names = inp.get("declaredNames")
    if not isinstance(names, list) or not names:
        return "declaredNames must be a non-empty list of strings"
    if not inp.get("skillPackagePath"):
        return "skillPackagePath is required to grep SKILL.md ∪ references"
    return None


def verify(inp, doc_text=None):
    """Pure verification. `doc_text` may be injected (tests); else loaded from disk."""
    err = _validate(inp)
    if err:
        return make_error(err)

    declared_set = sorted({n for n in inp["declaredNames"] if isinstance(n, str) and n})
    if doc_text is None:
        doc_text = load_doc_text(inp["skillPackagePath"])

    present = sorted(n for n in declared_set if n in doc_text)
    absent = sorted(n for n in declared_set if n not in doc_text)

    return {
        "declared": len(declared_set),
        "verified": len(present),
        "present": present,
        "absent": absent,
        "inflated": len(present) < len(declared_set),
    }


# --- CLI --------------------------------------------------------------------


def _build_parser():
    parser = argparse.ArgumentParser(
        prog="verify-declared-numerator",
        description=(
            "Deterministic numerator ground-truth verifier (coverage-check.md "
            "§4b). Greps the full declared export set against SKILL.md ∪ "
            "references/*.md and enumerates the present/absent names so a padded "
            "numerator is caught and the verified count replaces it."
        ),
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=(
            "Example:\n"
            "  uv run verify-declared-numerator.py "
            "'{\"declaredNames\":[\"foo\",\"bar\"],"
            "\"skillPackagePath\":\"/path/to/skill\"}'"
        ),
    )
    src = parser.add_mutually_exclusive_group()
    src.add_argument(
        "json_input",
        nargs="?",
        help="JSON object as a positional argument (single-quote it on the shell).",
    )
    src.add_argument(
        "--json-input",
        dest="json_input_flag",
        help="JSON object passed via flag (overrides positional).",
    )
    src.add_argument(
        "--stdin",
        action="store_true",
        help="Read the JSON object from stdin.",
    )
    return parser


def _resolve_input(args):
    if args.stdin:
        return sys.stdin.read()
    if args.json_input_flag is not None:
        return args.json_input_flag
    if args.json_input is not None:
        return args.json_input
    return ""


def main(argv=None):
    parser = _build_parser()
    args = parser.parse_args(argv)
    raw = _resolve_input(args)
    if not raw.strip():
        parser.print_usage(file=sys.stderr)
        print(
            "error: no input provided (positional arg, --json-input, or --stdin)",
            file=sys.stderr,
        )
        return 1

    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        print(json.dumps(make_error(f"Invalid JSON: {exc.msg}"), indent=2))
        return 1

    result = verify(data)
    print(json.dumps(result, indent=2))
    if isinstance(result, dict) and result.get("code") == "INVALID_INPUT":
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

---
workflowType: 'test-skill'
skillName: ''
skillDir: ''
runId: ''
testMode: ''
forgeTier: ''
testResult: ''
score: ''
threshold: ''
analysisConfidence: ''
toolingStatus: ''
workspaceDrift: ''
health_check_dispatched: false
testDate: ''
stepsCompleted: []
nextWorkflow: ''
---

# Test Report: {{skillName}}

<!--
Section order is LOAD-BEARING: step 5 §5 enforces it, step 6 §5 verifies
stepsCompleted against the canonical chain. Do not reorder or delete anchors.

Anchor / Step mapping:
  Test Summary       → detect-mode
  Coverage Analysis  → coverage-check
  Coherence Analysis → coherence-check
  External Validation→ external-validators
  (hard gate)        → step-hard-gate (reads findings, blocks or passes; no report section)
  Completeness Score → score
  Gap Report         → report (includes Discovery Quality subsection)
-->

## Test Summary

<!-- Populated by detect-mode §3 -->

## Coverage Analysis

<!-- Populated by coverage-check §5 -->

## Coherence Analysis

<!-- Populated by coherence-check §6 (naive or contextual variant) -->

## External Validation

<!-- Populated by external-validators §5 -->

## Completeness Score

<!-- Populated by score §6 -->

## Gap Report

<!-- Populated by report §3-§4b (includes Discovery Quality subsection) -->