# Validation-design reference (model-validation)

Load-on-demand backbone for Phases 2–7 — the leakage taxonomy, the internal-vs-external
tier ladder, comparator design, run variance, test-set sizing, and the reporting map. Anchored
to **Kapoor & Narayanan** (*Patterns* 2023, leakage taxonomy), **Varoquaux & Cheplygina**
(*npj Digital Medicine* 2022, medical-imaging ML failure modes), **Metrics Reloaded**
(Maier-Hein & Reinke et al., *Nature Methods* 2024), **CLAIM 2024**, **TRIPOD+AI** (*BMJ* 2024),
and **STARD-AI** (*Nature Medicine* 2025). It explains *what to check and advise*; the patient-disjointness
verdict is proven by `scripts/check_split_leakage.py`, not by this prose.

## 1. The data-leakage taxonomy

Leakage = any information about a test case that could influence training. Organise the audit by
the three Kapoor-Narayanan (*Patterns* 2023) categories; the deterministic gate covers only the
first row.

| Category (Kapoor-Narayanan) | Imaging-specific leak | How it inflates the metric | How to catch |
|---|---|---|---|
| **No clean train/test separation** | **Patient-level overlap** — the same patient's images straddle train and test | Model memorises patient anatomy, not pathology | `check_split_leakage.py` → `PATIENT_OVERLAP` (set arithmetic on IDs) |
| ″ | **Near-duplicate / repeated-acquisition** — repeat scans, follow-ups, augmented copies, overlapping patches of one volume across splits | Test cases are not independent of training | Split on the **patient**, not the image/slice/series; dedup by patient before partitioning |
| ″ | **Preprocessing-before-split** — normalisation stats, intensity windowing, resampling, feature selection, ComBat harmonisation, or foundation-model embeddings fit on the **whole cohort** | Test statistics bleed into the training pipeline | Fit every transform on the **training fold only**; the test set is touched only at scoring time |
| ″ | **Temporal leakage** — a random split where future and past coexist, under a prognostic/surveillance claim | Model peeks at later-era data | Use a **temporal split** (train on earlier, test on later) when the claim is temporal |
| **Illegitimate features** | **Site / scanner / burned-in-label shortcut** — the model keys on acquisition site, vendor signature, a laterality token, or a body-part marker rather than the finding | Discrimination collapses off-site | Standalone confound check (shortcut-learning, Geirhos et al. *Nat Mach Intell* 2020; DeGrave et al. *Nat Mach Intell* 2021; Zech et al. *PLOS Med* 2018); subgroup-by-site slice |
| **Test set ≠ population of interest** | **Spectrum / selection bias** — test cases curated, enriched, or selected on an optional modality | Reported accuracy does not transfer to deployment | Confirm the test set reflects the intended-use population; see §2 / §6 |

The decisive question to ask of every preprocessing and selection step: **could any value used in
training have been computed only with knowledge of a test case?** If yes, it is leakage even when
the split table itself looks disjoint.

### Tuning-on-test (the test set must be touched once)
Architecture search, hyperparameter sweeps, early-stopping, **and operating-point / threshold
selection** that read the test set are all forms of the first category — the test set has become a
development set, and the headline metric is optimistic. Fix the threshold and select the model on
the **training/validation folds**, then evaluate the frozen model on the test set exactly once.
"Developed with external validation" where the single external set was also used for tuning is no
longer external validation (§2).

## 2. Internal vs genuine external validation

Classify the evidence honestly and let the tier cap the claim. Cross-validation and bootstrap are
development-time **optimism corrections** (apparent-performance debiasing), **not** external
validation — this is the long-standing TRIPOD / prediction-model distinction (Collins et al.,
TRIPOD 2015; TRIPOD+AI, *BMJ* 2024).

```
apparent (train=test, never sufficient)
  → internal random split
    → k-fold cross-validation / bootstrap   ← still internal (optimism correction)
      → temporal split (later era held out)
        → geographic / external (different site, scanner, vendor)
          → multi-site / prospective external      ← retrospective evidence ends here
            → silent / shadow deployment            ┐
              → prospective comparative (impact) RCT  ├ early clinical evaluation + use
                → live deployment + monitoring        ┘  (DECIDE-AI; see §2b)
```

- A **generalisability or deployment-readiness** claim needs at least a genuine external tier
  (different site/scanner/vendor), not an internal split.
- Single-centre external validation supports a narrower claim than multi-site; say which.
- Reusing the external set for any tuning demotes it back to internal — flag the contradiction.
- A **clinical-use, workflow-impact, or patient-outcome** claim needs the prospective tiers
  in §2b (silent trial → impact study), not retrospective external accuracy.

## 2b. Beyond retrospective external — prospective evaluation & deployment monitoring

Retrospective external validation establishes that accuracy transfers; it does **not**
establish that the model is safe and useful *in the clinical workflow*. That is a separate,
higher tier the lane previously stopped short of. Design it explicitly when the claim is
clinical use, not just discrimination:

- **Silent / shadow deployment.** The model runs on live prospective cases without acting on
  care; compare its outputs to the real-time reference and to clinician decisions. Catches
  prospective performance drop, input/workflow mismatch, and edge-case failures before any
  patient is affected. Pre-specify the prospective performance and calibration targets.
- **Prospective comparative (impact) study / RCT.** To claim a workflow or patient-outcome
  benefit, randomise or prospectively compare the AI-assisted pathway against standard care
  on a **clinical** endpoint (time-to-diagnosis, recall rate, downstream outcome), not just
  standalone accuracy. This is the top of the evidence hierarchy for clinical AI.
- **Post-deployment monitoring.** A deployed model is not a finished artifact: pre-specify a
  monitoring plan for **performance drift, dataset / population shift, and calibration
  drift** over time, with trigger thresholds for recalibration or withdrawal, and an audit of
  subgroup performance over time (equity does not hold automatically post-deployment).

**Reporting.** Early-stage live evaluation of decision-support AI is reported against
**DECIDE-AI** (Stage 1–2a, early clinical evaluation); full prospective trials against
**CONSORT-AI / SPIRIT-AI**. Route via `/check-reporting` (Phase 7). State which tier the
study reaches and scope the claim to it — a retrospective external study must not claim
deployment readiness, monitoring adequacy, or clinical-outcome benefit.

## 3. Comparator design

A standalone metric rarely answers the clinical question; decide what the model is measured
*against*, evaluated on the **same** test set. CLAIM 2024 and TRIPOD+AI both ask for comparison to
current practice / an existing model.

| Comparator | When | Hand-off |
|---|---|---|
| **Clinical / no-model baseline** | "does the model beat current standard of care?" | — |
| **Incremental value over an existing score** | model added on top of an established risk score / radiologist read | added-value statistics (NRI / IDI / decision curve) → `/analyze-stats` |
| **Reader comparison (standalone or AI-assisted)** | model vs / with radiologists | rubric, reader panel, inter-rater design → `/design-ai-benchmarking` |

Name whether the claim is **standalone** (model alone) or **assistive** (clinician + model); they
need different comparators and different reporting.

## 4. Single-run vs multi-seed variance

A single training run overstates precision: deep-model metrics move with the random seed
(initialisation, data order, augmentation), and some GPU ops are non-deterministic even with cuDNN
deterministic flags set (Varoquaux & Cheplygina, *npj Digit Med* 2022; reproducibility crisis,
Kapoor & Narayanan 2023). Require the headline metric as **mean ± SD over ≥ 3 seeds / runs**, or a
**single fixed reported seed with the determinism caveat stated**. A point estimate from one run,
presented as if exact, is a reporting defect.

## 5. Test-set sizing

Check **events per class in the test set**, not the cohort total. A metric computed on a sparse
positive set has a confidence interval spanning much of the usable range, so a headline AUROC /
sensitivity can be statistically uninformative even when the cohort is large.

- Size the test set for the **CI width** of the headline metric and for **per-subgroup** estimates
  you intend to report.
- **Calibration** in particular is data-hungry — prediction-model validation guidance uses a rule
  of thumb of roughly ≥ 100 events and ≥ 100 non-events before a reliability assessment is stable
  (treat as a rule of thumb, not a hard cutoff; confirm for the specific design).
- Hand the formal calculation (diagnostic-accuracy precision, AUC precision, agreement, calibration
  sample size) to `/calc-sample-size`.

Metric **selection** (Dice + boundary metric; AUROC + AUPRC under imbalance; FROC/mAP with the IoU
criterion) is owned by `/model-evaluation` (`references/metric_guide.md`, anchored to Metrics
Reloaded); this skill only checks that the chosen metric is task- and prevalence-correct.

## 6. Reporting-guideline fit

Map the study to its standard via `/check-reporting`, and **name both the base instrument and the
AI extension**, citing each at its actual maturity (published guideline vs protocol-stage), never
beyond it. The four standards below are cross-checked against the repository's `check-reporting`
verified checklists.

| Study framing | Primary standard (extension) | Base instrument | Risk of bias |
|---|---|---|---|
| Diagnostic / triage **imaging-AI** study (standalone or assistive) | **CLAIM 2024** update (Tejani et al., *Radiology: AI* 2024) | CLAIM 2020 (Mongan, Moy, Kahn, *Radiology: AI* 2020) | PROBAST+AI |
| **Prediction model** (diagnostic or prognostic; regression or ML) | **TRIPOD+AI** (Collins, Moons et al., *BMJ* 2024) | TRIPOD 2015 (Collins, Reitsma, Altman, Moons) | **PROBAST+AI** (Moons et al., *BMJ* 2025; replaces PROBAST-2019) on base PROBAST (Wolff et al., *Ann Intern Med* 2019) |
| **Diagnostic accuracy** study (index test vs reference standard; sens/spec) | **STARD-AI** (Sounderajah et al., *Nature Medicine* 2025) | STARD 2015 (Bossuyt et al., *BMJ* 2015) | QUADAS-2 / QUADAS-C |

Tie the partition, leakage controls, validation tier, comparator, run variance, and test-set sizing
above to the specific items these standards request (data partition, sample size, model evaluation,
comparison to current practice, reproducibility).

## Hand-offs
- Patient-disjointness proof → `scripts/check_split_leakage.py` (Phase 2, run first).
- Test-set / event sizing → `/calc-sample-size`.
- Reader-comparison rubric + inter-rater design → `/design-ai-benchmarking`.
- Per-case metric computation + reporting gate → `/model-evaluation` → `/analyze-stats`.
- Item-by-item compliance → `/check-reporting`; Methods write-up → `/write-paper`; reviewer-side
  audit of the finished draft → `/self-review` (MD0–MD11 `model_development.md` probe).

## Verification notes (what each claim is grounded on)
- **Leakage taxonomy / three categories, reproducibility crisis** — Kapoor & Narayanan, "Leakage and
  the reproducibility crisis in machine-learning-based science," *Patterns* 2023. Imaging-specific
  failure modes (patient-level split, preprocessing-before-split) — Varoquaux & Cheplygina,
  *npj Digital Medicine* 2022. Both already cited in `check_split_leakage.py`.
- **Site/scanner/shortcut leakage** — shortcut learning, Geirhos et al., *Nature Machine
  Intelligence* 2020; radiographic shortcut example, DeGrave et al., *Nature Machine Intelligence*
  2021; cross-site generalisation failure, Zech et al., *PLOS Medicine* 2018. Used as named
  examples of the "illegitimate features" / spectrum-bias rows, not as numeric claims.
- **Internal vs external, optimism correction, CV ≠ external** — TRIPOD 2015 (Collins, Reitsma,
  Altman, Moons) and TRIPOD+AI (*BMJ* 2024). The tier ladder mirrors the skill's Phase 3.
- **Metric selection deferral** — Metrics Reloaded (Maier-Hein & Reinke et al., *Nature Methods*
  2024); detail lives in `/model-evaluation`.
- **Reporting map** — CLAIM 2024 update (Tejani et al., *Radiology: AI* 2024) on base CLAIM 2020
  (Mongan, Moy, Kahn); TRIPOD+AI (*BMJ* 2024); STARD-AI (Sounderajah et al., *Nature Medicine*
  2025) on base STARD 2015 (Bossuyt et al., *BMJ* 2015); PROBAST+AI (Moons et al., *BMJ* 2025) on
  base PROBAST (Wolff et al., *Ann Intern Med* 2019). All four are cross-checked against this
  repository's `check-reporting` verified checklists (CLAIM 2024 = e240300; TRIPOD+AI = e078378;
  STARD-AI = DOI 10.1038/s41591-025-03953-8, PMID 40954311; PROBAST+AI = e082505).
- **Numbers deliberately not asserted**: the only quantitative figure is the ~100 events/non-events
  calibration rule of thumb, flagged as a rule of thumb to confirm per design — no dataset names,
  thresholds, or performance numbers are invented here. The reporting-standard DOIs/PMIDs above are
  carried from the repository's verified `check-reporting` checklists; still re-confirm the exact
  identifier via `/search-lit` before quoting any of them in a manuscript, and mark any uncertain
  item `[VERIFY]`.

#!/usr/bin/env python3
"""Train/validation/test split-leakage gate for a medical-imaging model (model-validation).

The single most common — and most metric-inflating — defect in an engineer-built
imaging model is a data split that is NOT disjoint at the patient level: the same
patient contributes images to both the training and the test partition, so the model
memorises patient-specific anatomy and every reported metric is optimistic
(Kapoor & Narayanan, Patterns 2023; Varoquaux & Cheplygina, npj Digit Med 2022;
CLAIM 2024 data-partition items). Unlike a prose hygiene linter, this is a fully
*decidable* data check: it reads the emitted split-assignment table and proves, by
set arithmetic on the patient/subject IDs, whether any ID crosses partitions.

It also checks that the split is REPRODUCIBLE — a recorded random seed — because a
split with no seed cannot be regenerated and its disjointness cannot be re-verified.

CHECKS (verdicts):
  1. PATIENT_OVERLAP  (Major)  one or more IDs appear in >= 2 distinct partitions
                               (after collapsing train/training, val/validation,
                               test/testing/holdout synonyms). The decisive leak.
  2. MISSING_SEED     (Major)  no split seed found via --seed, --seed-file, an
                               auto-detected split_seed.txt next to the CSV, or a
                               seed / random_state column. Pass --no-require-seed to
                               downgrade to a flag when the seed is genuinely external.
  3. SINGLE_PARTITION (Minor)  fewer than two distinct partitions — the file is not a
                               train/val/test split (informational, not a leak).

An ID appearing multiple times WITHIN one partition (several images per patient) is
NOT a leak and does not fire; only an ID spanning >= 2 partitions does.

INPUTS
  --splits   split-assignment CSV (required). Columns auto-detected:
               id    : patient_id / subject_id / case_id / id / pid / mrn / studyid
               split : split / partition / set / subset / fold / phase / assignment
             Override with --id-col / --split-col.
  --seed     the split's random seed (int/str), to record reproducibility.
  --seed-file path to a file holding the seed (default: auto-detect split_seed.txt
             alongside --splits).

OUTPUT
  A reconciliation table (stdout) and, with --out, a JSON artifact:
    {splits, id_col, split_col, n_rows, n_subjects, partitions{name:count},
     seed, claims[{verdict, severity, detail, where}], summary}
  PATIENT_OVERLAP / MISSING_SEED are Major candidates.

Stdlib-only (csv / json / re / argparse / pathlib). Exit codes: 0 clean (or
report-only), 1 Major claim(s) found (with --strict), 2 input/usage error.
"""

from __future__ import annotations

import argparse
import csv
import json
import re
import sys
from pathlib import Path

ID_HINTS = ("patient_id", "subject_id", "case_id", "patientid", "subjectid", "caseid",
            "patient", "subject", "id", "pid", "eid", "uid", "mrn", "studyid", "study_id")
SPLIT_HINTS = ("split", "partition", "set", "subset", "fold", "phase", "assignment",
               "split_assignment", "data_split", "group_split")

# Synonym collapse so "train"/"training" do not register as two partitions (a
# conservative choice: real leakage is train-vs-test, not a labelling variant).
SPLIT_CANON = {
    "train": "train", "training": "train", "trn": "train",
    "val": "val", "valid": "val", "validation": "val", "dev": "val", "development": "val",
    "test": "test", "testing": "test", "holdout": "test", "hold-out": "test",
    "hold_out": "test", "eval": "test", "evaluation": "test",
}


def _norm(s: str) -> str:
    return re.sub(r"[^a-z0-9]+", "", (s or "").strip().lower())


def _canon_split(v: str) -> str:
    n = _norm(v)
    return SPLIT_CANON.get(n, n)


def _pick(header: list[str], hints: tuple[str, ...]):
    norm = [_norm(h) for h in header]
    for hint in hints:                       # exact match first
        h = _norm(hint)
        for i, col in enumerate(norm):
            if col == h and h:
                return header[i]
    for hint in hints:                       # then substring
        h = _norm(hint)
        for i, col in enumerate(norm):
            if h and h in col:
                return header[i]
    return None


def _find_seed(splits_path: Path, rows: list[dict], header: list[str],
               seed: str | None, seed_file: str | None) -> str | None:
    if seed is not None and str(seed).strip() != "":
        return str(seed).strip()
    # explicit or auto-detected seed file
    candidates = []
    if seed_file:
        candidates.append(Path(seed_file))
    candidates.append(splits_path.with_name("split_seed.txt"))
    candidates.append(splits_path.parent / "split_seed.txt")
    for c in candidates:
        if c.is_file():
            txt = c.read_text(encoding="utf-8").strip()
            m = re.search(r"-?\d+", txt)
            if m:
                return m.group(0)
    # a seed / random_state column
    col = _pick(header, ("seed", "random_state", "random_seed", "rng_seed"))
    if col:
        for r in rows:
            v = (r.get(col) or "").strip()
            if v:
                return v
    return None


def analyze(splits: str, id_col: str | None, split_col: str | None,
            seed: str | None, seed_file: str | None, require_seed: bool) -> dict:
    p = Path(splits)
    if not p.is_file():
        sys.stderr.write(f"ERROR: --splits not found: {splits}\n")
        sys.exit(2)
    with p.open(encoding="utf-8-sig", newline="") as f:
        rows = [r for r in csv.DictReader(f)]
    if not rows:
        sys.stderr.write(f"ERROR: --splits has no rows: {splits}\n")
        sys.exit(2)
    header = list(rows[0].keys())

    if id_col and id_col not in header:
        sys.stderr.write(f"ERROR: --id-col '{id_col}' not in {header}\n")
        sys.exit(2)
    if split_col and split_col not in header:
        sys.stderr.write(f"ERROR: --split-col '{split_col}' not in {header}\n")
        sys.exit(2)
    idc = id_col or _pick(header, ID_HINTS)
    spc = split_col or _pick(header, SPLIT_HINTS)
    if idc is None:
        sys.stderr.write(f"ERROR: no ID column found (looked for {ID_HINTS[:6]}…); pass --id-col\n")
        sys.exit(2)
    if spc is None:
        sys.stderr.write(f"ERROR: no split column found (looked for {SPLIT_HINTS[:6]}…); pass --split-col\n")
        sys.exit(2)

    # ID -> set of canonical partitions it appears in.
    id_to_splits: dict[str, set[str]] = {}
    part_counts: dict[str, int] = {}
    for r in rows:
        sid = (r.get(idc) or "").strip()
        part = _canon_split(r.get(spc) or "")
        if not sid or not part:
            continue
        id_to_splits.setdefault(sid, set()).add(part)
        part_counts[part] = part_counts.get(part, 0) + 1

    n_subjects = len(id_to_splits)
    overlapping = sorted(sid for sid, parts in id_to_splits.items() if len(parts) > 1)
    seed_val = _find_seed(p, rows, header, seed, seed_file)

    claims: list[dict] = []
    if overlapping:
        shown = ", ".join(overlapping[:8]) + ("…" if len(overlapping) > 8 else "")
        # a representative offender with its partitions
        ex = overlapping[0]
        ex_parts = "/".join(sorted(id_to_splits[ex]))
        claims.append({
            "verdict": "PATIENT_OVERLAP",
            "severity": "Major",
            "detail": (f"{len(overlapping)} of {n_subjects} subjects appear in >= 2 partitions "
                       f"(e.g. '{ex}' in {ex_parts}); the same patient in train and test "
                       f"inflates every metric. Offenders: {shown}"),
            "where": f"--splits id column '{idc}', split column '{spc}'",
        })
    if seed_val is None and require_seed:
        claims.append({
            "verdict": "MISSING_SEED",
            "severity": "Major",
            "detail": ("no split seed found (--seed / --seed-file / split_seed.txt / a seed "
                       "column); the partition cannot be regenerated or re-verified"),
            "where": f"--splits {p.name}",
        })
    if len(part_counts) < 2:
        claims.append({
            "verdict": "SINGLE_PARTITION",
            "severity": "Minor",
            "detail": (f"only {len(part_counts)} distinct partition(s) "
                       f"({', '.join(sorted(part_counts)) or 'none'}); this is not a "
                       f"train/val/test split"),
            "where": f"--splits split column '{spc}'",
        })

    n_major = sum(1 for c in claims if c["severity"] == "Major")
    return {
        "splits": str(p),
        "id_col": idc,
        "split_col": spc,
        "n_rows": len(rows),
        "n_subjects": n_subjects,
        "partitions": dict(sorted(part_counts.items())),
        "seed": seed_val,
        "claims": claims,
        "summary": {
            "n_claims": len(claims),
            "n_major": n_major,
            "n_minor": len(claims) - n_major,
            "n_overlapping_subjects": len(overlapping),
            "verdict": "MAJOR_CANDIDATE" if n_major else "OK",
        },
    }


def render(result: dict) -> str:
    lines = ["| Check | Severity | Detail |", "|---|---|---|"]
    for c in result["claims"]:
        lines.append(f"| {c['verdict']} | {c['severity']} | {c['detail']} |")
    if len(lines) == 2:
        lines.append("| (none) | — | patient-disjoint split with a recorded seed |")
    return "\n".join(lines)


def main() -> int:
    ap = argparse.ArgumentParser(description="Train/val/test split-leakage gate (model-validation).")
    ap.add_argument("--splits", required=True, help="split-assignment CSV (id column + split column)")
    ap.add_argument("--id-col", help="patient/subject ID column (auto-detected if omitted)")
    ap.add_argument("--split-col", help="split/partition column (auto-detected if omitted)")
    ap.add_argument("--seed", help="the split's random seed, to record reproducibility")
    ap.add_argument("--seed-file", help="file holding the seed (default: auto-detect split_seed.txt)")
    ap.add_argument("--no-require-seed", dest="require_seed", action="store_false",
                    help="downgrade a missing seed from Major to no-finding")
    ap.add_argument("--out", help="write JSON artifact to this path")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major claim exists")
    ap.add_argument("--quiet", action="store_true", help="suppress stdout table")
    ap.set_defaults(require_seed=True)
    args = ap.parse_args()

    result = analyze(args.splits, args.id_col, args.split_col,
                     args.seed, args.seed_file, args.require_seed)

    if not args.quiet:
        print("=" * 41)
        print(" Split-Leakage Gate (model-validation)")
        print("=" * 41)
        print(f"  rows={result['n_rows']}  subjects={result['n_subjects']}  "
              f"partitions={result['partitions']}  seed={result['seed']}")
        print(render(result))
        print()
        s = result["summary"]
        if s["n_major"]:
            print(f"MAJOR candidate: {s['n_major']} split-integrity issue(s).")
        else:
            print("OK: patient-disjoint split with a recorded seed.")

    if args.out:
        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
        Path(args.out).write_text(json.dumps({"detector": "check_split_leakage", **result}, indent=2), encoding="utf-8")
        if not args.quiet:
            print(f"\nwrote {args.out}")

    return 1 if (args.strict and result["summary"]["n_major"]) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# Regression test for the split-leakage gate (model-validation).
# Synthetic, PII-free fixtures reproduce: (a) a patient that crosses train/test,
# (b) column auto-detection (subject_id / partition), (c) a missing split seed,
# (d) the --no-require-seed / --seed downgrades, (e) a single-partition file, and
# (f) a seed read from a column. Stdlib-only (python3).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$HERE/../scripts/check_split_leakage.py"
F="$HERE/fixtures"
CHF="$HERE/../scripts/check_split_leakage_challenge/fixture"
OUT="$(mktemp -t spl_XXXX).json"
trap 'rm -f "$OUT"' EXIT

fail=0
check() { local label="$1"; shift
    if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$label"
    else printf '  FAIL  %s\n' "$label"; fail=$((fail+1)); fi
}
has_verdict() { python3 -c "
import json
d=json.load(open('$OUT'))
assert any(c['verdict']=='$1' for c in d['claims']), '$1 not found'
"; }
no_verdict() { python3 -c "
import json
d=json.load(open('$OUT'))
assert not any(c['verdict']=='$1' for c in d['claims']), '$1 unexpectedly present'
"; }

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing" >&2; exit 2; }

# (1) challenge leak fixture: P03/P07 cross partitions, seed auto-detected -> PATIENT_OVERLAP, exit 1
python3 "$SCRIPT" --splits "$CHF/splits_leak.csv" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 1 under --strict (patient overlap)" test "$?" -eq 1
check "PATIENT_OVERLAP detected" has_verdict PATIENT_OVERLAP
check "two overlapping subjects reported" python3 -c "
import json; d=json.load(open('$OUT'))
assert d['summary']['n_overlapping_subjects']==2, d['summary']"
check "seed auto-detected from split_seed.txt" python3 -c "
import json; d=json.load(open('$OUT')); assert d['seed']=='42', d['seed']"

# (2) clean challenge fixture: synonyms collapse, disjoint -> exit 0, no overlap
python3 "$SCRIPT" --splits "$CHF/splits_clean.csv" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 0 on disjoint split (synonyms collapsed)" test "$?" -eq 0
check "no PATIENT_OVERLAP on clean split" no_verdict PATIENT_OVERLAP

# (3) column auto-detection (subject_id / partition) + --seed isolates the overlap
python3 "$SCRIPT" --splits "$F/leak_subject.csv" --seed 1 --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 1 with auto-detected subject_id/partition columns" test "$?" -eq 1
check "PATIENT_OVERLAP via auto-detect (S1)" has_verdict PATIENT_OVERLAP
check "no MISSING_SEED when --seed supplied" no_verdict MISSING_SEED

# (4) explicit --id-col / --split-col also resolve
python3 "$SCRIPT" --splits "$F/leak_subject.csv" --id-col subject_id --split-col partition --seed 1 --out "$OUT" --quiet >/dev/null 2>&1
check "explicit --id-col/--split-col" has_verdict PATIENT_OVERLAP

# (5) missing seed on an otherwise-disjoint split -> MISSING_SEED (Major), exit 1
python3 "$SCRIPT" --splits "$F/noseed_clean.csv" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 1 when seed missing" test "$?" -eq 1
check "MISSING_SEED detected" has_verdict MISSING_SEED
check "no PATIENT_OVERLAP on disjoint split" no_verdict PATIENT_OVERLAP

# (6) --no-require-seed downgrades the missing seed -> exit 0
python3 "$SCRIPT" --splits "$F/noseed_clean.csv" --no-require-seed --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 0 with --no-require-seed" test "$?" -eq 0
check "MISSING_SEED suppressed by --no-require-seed" no_verdict MISSING_SEED

# (7) single-partition file with a seed column -> SINGLE_PARTITION (Minor only), exit 0
python3 "$SCRIPT" --splits "$F/single_partition.csv" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 0 on single-partition (Minor only)" test "$?" -eq 0
check "SINGLE_PARTITION detected" has_verdict SINGLE_PARTITION
check "seed read from column" python3 -c "
import json; d=json.load(open('$OUT')); assert d['seed']=='7', d['seed']"

echo "fail=$fail"; [[ "$fail" -eq 0 ]] && echo "ALL PASS" || echo "FAILURES: $fail"
exit "$fail"