# A/B equivalence - a transform-agnostic behavioural-equivalence capability

A thin capability that compares two versions of an LLM-directed document - an `original` (the teacher) and a `candidate` (the student) - across a transfer set, and returns a per-case verdict on whether the candidate still induces the behaviour the original induced.

It is **transform-agnostic**: it judges *behavioural equivalence between two versions* and neither knows nor cares which transform produced the candidate. It therefore serves every optimizer transform that claims to preserve behaviour - compression today, directive-clarity next - not just compression. It is a **library capability** other skills compose: `semantic-compress` invokes it to gate a distillation. It does **not** change skill-forge's own five-lens quality gate hierarchy - that hierarchy judges *absolute quality* ("is this skill good?"); A/B equivalence judges *sameness between two versions* ("does the candidate still do what the original did?"). They are different questions answered by different judges.

## Input contract

| Input | Required | Notes |
|-------|----------|-------|
| `original` | yes | Path to the teacher document - the version whose behaviour is the equivalence target. |
| `candidate` | yes | Path to the student document - the transformed version under test. |
| `transfer_set` | yes | Array of cases drawn from the skill-forge test taxonomy (happy / edge / adversarial / composition - see `test-taxonomy.md`). The transfer set *is* the operational definition of the behaviour being preserved, so its breadth bounds the safety of the conclusion. |

The caller (e.g. `semantic-compress`) owns deriving and confirming the transfer set; this capability consumes it. A thin transfer set yields a weak equivalence claim - the caller is responsible for flagging coverage, and the output records it.

## Mechanism

For each case in the transfer set:

1. Run the **existing** runner (`runner-prompt.md`, unchanged - the same pure-wrapper prompt the forge loop uses) once with `original` as the skill draft, on the case input, producing the **teacher transcript**.
2. Run the same runner once with `candidate` as the skill draft, on the **identical** case input, producing the **candidate transcript**.
3. Hand both transcripts to the **equivalence judge** (`equivalence-judge-prompt.md` - a focused compare-two-transcripts judge, separate from the five quality lenses), which emits the per-case verdict and efficiency signal.

The runner and runner-prompt are reused verbatim; nothing about how a single version is executed changes. The only new component is the equivalence judge, which compares two transcripts rather than scoring one against an intent.

### Baseline caching (the teacher is captured once)

The `original` never changes across a multi-round transform loop, so its transcript per case is **captured once and reused across every round**. Only the candidate is re-run each round. This is a hard rule, not an optimization: re-running the teacher each round wastes runner budget and risks introducing teacher-side noise that the judge would mistake for a candidate change. The caller passes the cached teacher transcripts back in on rounds >= 2; this capability re-runs only the candidate. A budget ceiling on candidate re-runs belongs to the caller's loop, not here.

<!-- chat-skip:start -->
Execution follows the same mode selection as the forge loop (see SKILL.md): the runner pair per case can be spawned as parallel runner subagents (phased mode) or worked sequentially by a single agent (solo mode). The equivalence judge runs as one more focused judge in whatever mode the harness is in. No persistent team is required - the judge holds no across-round memory of its own; the cached teacher transcript is the only state carried between rounds, and the caller owns it.
<!-- chat-skip:end -->

## Verdict categories

Per case, the judge returns exactly one verdict:

| Verdict | Meaning | What it must cite |
|---------|---------|-------------------|
| `equivalent` | The candidate induced every behaviour and discipline the original induced. Incidental wording differences with no behavioural consequence are still `equivalent`. | Nothing required beyond the verdict. |
| `candidate-regressed` | A behaviour or discipline the original induced is **absent** in the candidate. This is the failing verdict. | The **specific behaviour lost** - the discipline, step, or output the original produced and the candidate did not. |
| `candidate-diverged` | The candidate behaves **differently** but no behaviour the original induced was lost - a different-but-not-worse change (including incidental improvements). | The **difference** - what the candidate did differently. Not necessarily worse; documented for the caller's judgement. |

The regressed-vs-diverged decision rule is the load-bearing distinction and is stated authoritatively in `equivalence-judge-prompt.md`:

- **regressed** = a behaviour or discipline the original induced is **missing** from the candidate (essence lost).
- **diverged** = the candidate did something **different**, but every behaviour the original induced is **still present** (nothing lost).

When uncertain whether a delta is a loss or merely a difference, the judge defaults to `candidate-regressed` - a false regression costs one add-back round; a false `equivalent` ships a behaviour-losing transform undetected.

## Efficiency signal (alongside every verdict)

Independent of the verdict, the judge records an **efficiency signal** per case - how directly the runner acted on each version versus how much it had to unpack or reinterpret the instruction before acting:

| Field | Type | Meaning |
|-------|------|---------|
| `original_directness` | integer 1-5 | How directly the runner acted on the **original**: 5 = acted immediately, no reinterpretation; 1 = had to unpack, infer, or work around the instruction heavily before acting. |
| `candidate_directness` | integer 1-5 | The same measure for the **candidate**. |
| `interpretation_notes` | string | What the runner had to unpack or reinterpret on each version - the qualitative evidence behind the two scores. |

The signal is read from the runner self-report (`runner-prompt.md`): *steps followed / skipped*, *ambiguities hit and how resolved*, *improvisation beyond the skill*, and *any point it wanted to deviate but followed literally* all reveal how much interpretive work each version forced.

Why it exists: compression's gate is **strict no-regression** (sameness alone). But the optimizer family includes transforms that claim *behaviour-preserving-but-lighter* - the next one, directive-clarity, rewrites instructions the model has to unpack into directives that name the action. Such a transform can only be validated if the harness **measures the lightness, not just the sameness**: its gate is no-regression **and** a measured efficiency gain (`candidate_directness` > `original_directness` with no `candidate-regressed`). Recording the signal here, on every A/B run, is what lets those future transforms prove a measured gain instead of asserting one. Compression ignores the gain and gates on no-regression alone; both read the same signal.

## Output schema

```json
{
  "cases": [
    {
      "case_id": "string",
      "verdict": "equivalent|candidate-regressed|candidate-diverged",
      "behaviour_delta": "string",
      "efficiency_signal": {
        "original_directness": 1,
        "candidate_directness": 1,
        "interpretation_notes": "string"
      }
    }
  ],
  "summary": {
    "pass": true,
    "regressions": 0,
    "divergences": 0,
    "equivalents": 0
  }
}
```

- `case_id` - the transfer-set case identifier.
- `verdict` - one of the three categories above.
- `behaviour_delta` - for `candidate-regressed`, the specific behaviour lost; for `candidate-diverged`, the difference observed; empty (or `""`) for `equivalent`.
- `efficiency_signal` - the per-case directness scores and notes described above.
- `summary.regressions` / `divergences` / `equivalents` - counts of each verdict across `cases`.
- **`summary.pass` is `true` if and only if zero cases are `candidate-regressed`.** Divergences do not fail the run - they are surfaced for the caller's judgement. This encodes the strict no-regression gate: the candidate is accepted only when it loses nothing.

## What this capability does not do

- It does not derive or confirm the transfer set - the caller does, and signs off on coverage.
- It does not decide whether the candidate is good in absolute terms - that is the five-lens forge gate, a different run.
- It does not add back lost behaviour or regenerate candidates - on a regression it names what was lost; the caller's loop acts on that.
- It does not gate on the efficiency signal - it records it. Whether a measured efficiency gain is required is the calling transform's gate, not this capability's.

# The equivalence-judge prompt

A focused compare-two-transcripts judge, separate from the five quality lenses. The five lenses score one transcript against an intent ("is this skill good?"); this judge compares **two** transcripts against each other ("does the candidate still do what the original did?"). It is the only new component A/B equivalence adds - the runner and runner-prompt are reused unchanged.

It is filled once per transfer-set case: drop in the case input, the teacher transcript, and the candidate transcript, and send it to a fresh-context judge. The judge emits one structured verdict per case, conforming to the output schema in `ab-equivalence.md`.

## The comparison rubric

The judge does **not** read the two transcripts for prose similarity - textual sameness is irrelevant. It compares them on three behavioural dimensions, drawn from the runner self-report fields (`runner-prompt.md`):

1. **Outputs produced.** Did the candidate produce every output the original produced (the *output produced* field on both)? An output the original produced and the candidate did not is a loss.
2. **Disciplines enforced.** Did the candidate enforce every discipline, rule, or guard the original enforced (read from *steps followed / skipped*, *improvisation beyond the skill*, and *any point it wanted to deviate but followed literally*)? A discipline the original held the runner to and the candidate let slide is a loss.
3. **Steps followed.** Did the candidate lead the runner through the same load-bearing steps in a behaviourally-equivalent order (read from *steps followed / skipped*)? A step the original induced and the candidate dropped is a loss; a re-ordering with no behavioural consequence is not.

A behaviour counts as *induced by the original* only if it actually appears in the teacher transcript - the judge compares observed behaviour to observed behaviour, never the candidate against what the original document *says* it should do. (That latter question is Fidelity's, judged against intent, not this judge's.)

## The regressed-vs-diverged decision rule

The single load-bearing decision. Apply it after the three-dimension comparison:

- **`candidate-regressed`** - on any of the three dimensions, a behaviour, discipline, output, or step that the original **induced** is **absent** from the candidate transcript. Essence lost. Cite the specific behaviour lost in `behaviour_delta`. This is the only failing verdict.
- **`candidate-diverged`** - the candidate did something **different** on some dimension, but **every** behaviour the original induced is **still present** in the candidate transcript. Nothing the original did was lost; the candidate merely also did something else, or did it differently (including doing it *better*). Cite the difference in `behaviour_delta`. Not a failure - surfaced for the caller's judgement.
- **`equivalent`** - the candidate induced every behaviour the original induced and introduced no behaviourally-significant difference. Incidental wording differences with no behavioural consequence are `equivalent`, not `diverged`. `behaviour_delta` is empty.

Decision order: first check for any loss (a behaviour present in the teacher transcript, absent in the candidate). **If any loss exists, the verdict is `candidate-regressed`** - even if the candidate also improved elsewhere; a regression is never excused by an unrelated gain. Only if there is no loss do you choose between `diverged` (a behaviourally-significant difference remains) and `equivalent` (none does).

**Tie-break toward regression.** When you cannot decide whether a delta is a genuine loss or merely a difference, default to `candidate-regressed`. A false regression costs the caller one add-back round; a false `equivalent` ships a behaviour-losing transform undetected. The asymmetry is deliberate - the strict no-regression gate is only as trustworthy as this judge's conservatism.

## The efficiency signal

Alongside the verdict, score how directly the runner acted on each version - independent of whether behaviour was preserved:

- `original_directness` (1-5): how directly the runner acted on the **original** - 5 = acted immediately with no reinterpretation; 1 = had to unpack, infer, or work around the instruction heavily before acting.
- `candidate_directness` (1-5): the same measure for the **candidate**.
- `interpretation_notes`: what the runner had to unpack or reinterpret on each version, citing the *ambiguities hit*, *improvisation*, and *wanted to deviate but followed literally* self-report fields as evidence.

Score directness from how much interpretive work each transcript shows the runner doing **before** it could act - not from document length. A shorter document that forced more reinterpretation is *less* direct, not more. The verdict and the efficiency signal are independent: a candidate can be `equivalent` yet more direct (the case a lightness-claiming transform needs), or `equivalent` with no directness change (the typical compression case).

## Structured output

Emit one object per case, conforming to the `cases[]` element schema in `ab-equivalence.md`:

```json
{
  "case_id": "string",
  "verdict": "equivalent|candidate-regressed|candidate-diverged",
  "behaviour_delta": "string",
  "efficiency_signal": {
    "original_directness": 1,
    "candidate_directness": 1,
    "interpretation_notes": "string"
  }
}
```

- `behaviour_delta`: the specific behaviour lost (`candidate-regressed`), the difference observed (`candidate-diverged`), or empty (`equivalent`).
- Every field is required; a missing field makes the case unjudgeable. The caller aggregates these into the run-level `summary` (`pass` = zero `candidate-regressed`).

## Template

```text
You are an equivalence judge. You compare two runner transcripts produced from
the SAME input by two versions of a document - an ORIGINAL (teacher) and a
CANDIDATE (student) - and decide whether the candidate still induces the
behaviour the original induced. You are NOT scoring quality, and NOT comparing
either transcript against what the document says it should do. You compare
observed behaviour to observed behaviour.

--- TEST-CASE INPUT (identical for both runs) ---
<the case input both runners received>
--- END TEST-CASE INPUT ---

--- TEACHER TRANSCRIPT (from the ORIGINAL document) ---
<the original version's runner transcript and self-report>
--- END TEACHER TRANSCRIPT ---

--- CANDIDATE TRANSCRIPT (from the CANDIDATE document) ---
<the candidate version's runner transcript and self-report>
--- END CANDIDATE TRANSCRIPT ---

Compare on three dimensions: outputs produced, disciplines enforced, steps
followed. Then apply the decision rule:

- candidate-regressed: any behaviour/discipline/output/step the TEACHER
  transcript shows is ABSENT from the candidate. Essence lost. The only failing
  verdict. Cite the specific behaviour lost.
- candidate-diverged: the candidate did something different, but EVERY behaviour
  the teacher induced is still present. Nothing lost. Cite the difference.
- equivalent: every teacher behaviour present, no behaviourally-significant
  difference. Empty behaviour_delta.

Check for loss FIRST: any loss => candidate-regressed, even alongside an
unrelated gain. When unsure whether a delta is loss or mere difference, default
to candidate-regressed.

Also score the efficiency signal (independent of the verdict): how directly the
runner acted on each version (1-5), from how much it had to unpack/reinterpret
before acting - read the ambiguities, improvisation, and wanted-to-deviate
fields. Not from document length.

Emit exactly this JSON object (all fields required):

{
  "case_id": "<this case's id>",
  "verdict": "equivalent | candidate-regressed | candidate-diverged",
  "behaviour_delta": "<behaviour lost | difference observed | empty>",
  "efficiency_signal": {
    "original_directness": <1-5>,
    "candidate_directness": <1-5>,
    "interpretation_notes": "<what each version forced the runner to unpack>"
  }
}
```

# The runner prompt

It must be a **pure wrapper**: it never explains why the skill works, adds context beyond the draft, or coaches the runner (SKILL.md carries the rationale for why this is load-bearing).

The lead fills this template per runner - one runner per test case per round. Every self-report field is required; fields 1-5 are the standard report, and field 6 (gates hit) records interactive gates and reads "none encountered" on a run that hit none. The optional `Runner model` header records which model tier is executing, so a caller running a multi-tier sweep (e.g. skill-forge's runner-model knob) can attribute each transcript's verdict to a tier; omit it when no model is pinned. Copy the template, drop in the draft and the case input, send it to a fresh-context runner.

## Two runner variants

The runner has two variants. Both are pure wrappers, both return the **same six self-report fields**, and both carry the same axis-1 role-boundary guard (apply, do not judge). They differ only in *what* the runner is handed and what "apply it" means:

| Variant | When | What the runner is handed | What "apply" means |
|---------|------|---------------------------|--------------------|
| **Skill variant** (default) | The document under test is a skill (`SKILL.md`) or any draft *invoked on demand* against a case input | The skill draft + one test-case input | Execute the skill on the input and produce its output |
| **Instruction-file variant** | The document under test is an *always-loaded* agent instruction file (`CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `.cursor/rules/*`, `.github/copilot-instructions.md`) | **Only** this instruction file as operating context + one realistic repo task | Carry out the repo task the way the instruction file directs - but **read-only / sandboxed** |

The instruction-file variant exists because an always-loaded context file is never "invoked on an input" - it steers whatever the agent already does. So the runner is handed the file as its *only* context (no other repo conventions leaking in) plus a realistic repo task, and the transcript shows how a cold-start agent that trusted only that file would behave. The read-only / sandbox rule is what keeps a behaviour-probe from mutating the target repo: the runner **states the actions it would take** (commands it would run, files it would edit and how) rather than performing them; it may read files to inform those actions. That makes the runner safe to point at any real repo, and it still surfaces the load-bearing signal - the **accuracy** of the file's stated commands and paths - which the Fidelity lens reads from the *output produced* and *ambiguities* fields. The same selector that picks the variant is owned by the caller (skill-forge's artifact-type detection); this file owns both templates.

## Template - skill variant (default)

```text
Runner model: <optional - the model tier executing this runner, e.g. haiku |
sonnet | opus; record it so the transcript's verdict can be attributed to a tier.
Omit if the caller is not pinning a model.>

You are a test runner. Apply the following skill to the following input, exactly
as the skill instructs.

Role boundary:
- Do not add, skip, or reinterpret steps. Follow the skill as written.
- If the skill is ambiguous, note the ambiguity in your self-report, but still
  attempt to follow it as written - do not resolve it by guessing what it
  "should" have said.
- Do not judge the skill. Judging the skill is not your job; another agent does
  that from your report. Your job is to apply it and report honestly.

Gate handling (for A/B equivalence runs):
- If the skill invokes an interactive gate (AskUserQuestion, confirmation prompt),
  consume the next scripted answer from the provided gate_responses array.
- Match the gate's prompt text against each gate_response.pattern in order.
- If a match is found, respond with gate_response.response and continue execution.
- If no match is found (gate reached but no scripted answer available):
  1. Record the gate as a checkpoint: log the gate type, prompt text, and execution state.
  2. STOP execution immediately.
  3. Report the checkpoint in your self-report under a new field: gates_hit.

--- SKILL DRAFT (apply this verbatim) ---
<the full current draft of the skill, fenced exactly as authored>
--- END SKILL DRAFT ---

--- TEST-CASE INPUT ---
<this runner's specific test-case input>
--- END TEST-CASE INPUT ---

Self-report format (all fields required - fill every one):

1. Output produced:
   <the exact output the skill instructed you to produce>

2. Steps followed / skipped + why:
   <each step: followed or skipped, and the reason for any skip>

3. Ambiguities hit + how resolved:
   <each ambiguity in the skill and how you proceeded despite it>

4. Improvisation beyond the skill:
   <anything you did that the skill did not explicitly instruct, and why>

5. Any point you wanted to deviate but followed literally:
   <where following the skill as written felt wrong, but you did it anyway>

6. Gates hit:
   <each interactive gate encountered: gate_type, prompt_text, response_used or "STOPPED (no scripted answer)">
```

## Template - instruction-file variant (read-only)

Use this variant when the document under test is an always-loaded instruction file. It is the same pure wrapper with two changes: the document is the runner's *only* operating context (not a draft invoked on an input), and the runner is **read-only / sandboxed** - it states the actions it would take instead of performing them. The six self-report fields are unchanged so every lens reads them identically; "Output produced" now means the concrete actions and edits the runner would make (plus any artifact text it would author).

```text
Runner model: <optional - the model tier executing this runner, e.g. haiku |
sonnet | opus; record it so the transcript's verdict can be attributed to a tier.
Omit if the caller is not pinning a model.>

You are a test runner. The instruction file below is your ONLY operating context
for this run. Carry out the repo task below the way the instruction file directs.

Read-only / sandbox boundary (do not skip):
- Do NOT mutate the repo. State the actions you WOULD take - the exact commands
  you would run, and the files you would create or edit and how - instead of
  performing them.
- You MAY read any file in the repo to inform those actions. If the instruction
  file names a command or path, check whether it actually exists before relying
  on it, and record what you found.

Role boundary:
- Do not add, skip, or reinterpret what the instruction file directs. Follow it
  as written.
- If the instruction file is ambiguous, incomplete, or names something that does
  not exist, note that in your self-report, but still attempt to follow it as
  written - do not silently fill the gap from outside knowledge of the repo.
- Do not judge the instruction file. Judging it is not your job; another agent
  does that from your report. Your job is to apply it and report honestly.

--- INSTRUCTION FILE (your only operating context) ---
<the full current instruction file under test, fenced exactly as authored>
--- END INSTRUCTION FILE ---

--- REPO TASK ---
<this runner's specific realistic repo task>
--- END REPO TASK ---

Self-report format (all fields required - fill every one):

1. Output produced:
   <the concrete actions and edits you would take - commands you would run, files
   you would create/edit and how - plus any artifact text you would author>

2. Steps followed / skipped + why:
   <each directive you acted on or skipped, and the reason for any skip>

3. Ambiguities hit + how resolved:
   <each ambiguity, gap, or stated-but-missing command/path, and how you proceeded>

4. Improvisation beyond the instruction file:
   <anything you did that the file did not direct, and why>

5. Any point you wanted to deviate but followed literally:
   <where following the file as written felt wrong or build-breaking, but you did
   it anyway>

6. Gates hit:
   <each interactive gate encountered: gate_type, prompt_text, response_used or
   "STOPPED (no scripted answer)"; "none encountered" if none>
```

The accuracy signal lives in fields 1 and 3: a runner that "would run" a command the repo does not define, or "would edit" only the files the instruction file listed while the repo's tooling enforces more, exposes exactly the divergence the Fidelity accuracy sub-check fails on (see `judge-lenses.md`).

## The role-boundary section

The role-boundary section is the axis-1 recursion guard at the runner end: runners apply, lenses judge, the lead amends. A runner that starts judging the document collapses that separation, which is why both variants spell out "do not judge" explicitly. The instruction-file variant adds the read-only / sandbox boundary on top: a behaviour-probe that mutated the target repo would be a second way the runner overstepped its lane.

## Why the self-report fields are required output, not optional notes

Fields 1-5 are what each lens reads; the full self-report-field-to-lens mapping is owned by `judge-lenses.md`. They are required output, not optional notes, because a vague self-report blinds the lenses that depend on them. Field 6 (gates hit) is not lens input - it is read by the distill loop's gate-truncation logic (`skills/semantic-compress/references/distill-loop.md`) to mark a baseline truncated at an unanswered gate.

A runner that returns "I wrote the message" with the other fields blank has produced an unjudgeable transcript. Treat any missing field as a failed runner and re-run it.