# Combine / adapt / fine-tune existing models — design the comparator as an ablation

For a study whose model is built by **combining, adapting, or fine-tuning existing models** —
nnU-Net, TotalSegmentator, SAM / MedSAM, a pretrained backbone (the models `/architecture-zoo`
helps you choose) — the single decision that most determines acceptance is the **comparator**: you
must show that the combination / adaptation **earns its complexity**. Reverse-engineered from
accepted open-access papers (cited by DOI as design facts). It extends the Phase-3 *fine-tuning
contribution baseline* note (framed there for NLP/LLM) to imaging combine-existing studies, and it
is a **design-time** decision — none of these contrasts can be added after data collection.

## The failure the reviewer expects

*"Our combined model achieved Dice 0.90."* — with no decomposition, the reviewer cannot tell whether
the combination did anything; maybe the best single component alone scores 0.89. A single headline
number for a multi-component model is the **rejected** pattern. The accepted pattern **isolates what
the combination adds**.

## Design the ablation ladder (register the baselines before data collection)

Pre-specify the baselines the combined model must beat — each on the **same test set**, a
**patient-level** split, with a **paired CI** on the difference:

- **Un-adapted / off-the-shelf base** — the foundation model *without* your fine-tuning, or the
  pretrained backbone *without* your head. Proves the **adaptation** helps, not just the base.
  *(PCaSAM, npj Digital Medicine 2025 `10.1038/s41746-025-01756-2`: the fine-tuned model beats the
  un-adapted generalist foundation model and other baselines.)*
- **Best single component** — if you combine A + B, beat the better of A-alone / B-alone. Proves the
  **combination** helps. *(Embed-MedSAM, npj DM 2025 `10.1038/s41746-025-01881-y`: +≈16% Dice over
  the second-best model; VIBESegmentator, Eur Radiol 2025 `10.1007/s00330-025-12035-9`: positioned
  against the off-the-shelf tools — TotalSegmentator + spine + body-composition — it was seeded
  from.)*
- **Direct-train vs transfer** — if you fine-tune a pretrained model, compare against **training the
  same architecture from scratch** on your data. Isolates the pretrained component's contribution.
  *(Medulloblastoma nnU-Net, Radiology: AI 2024 — direct-train vs adult-glioma-pretrained-then-
  fine-tuned as the whole point of the study.)*
- **Nested incremental value** for a clinical/radiomics/DL combination (clinical → +radiomics → +DL
  → integrated) is the same discipline on tabular+imaging features — see Phase 3 *incremental value*
  and `analyze-stats` `incremental_value` table-type.

## Ground the gain in the clinical deliverable

A modest Dice / AUC delta reads as consequential only when tied to the endpoint clinicians use:
a **downstream clinical metric** *(PCaSAM: PI-RADS AUC on the external set)* or a **volume-agreement
statistic** — ICC / Bland–Altman / Lin's concordance *(CRLM total tumor volume, Eur Radiol Exp 2023
`10.1186/s41747-023-00383-4`, ICC 0.98; orbital-lymphoma volumetry, Neuroradiology 2024
`10.1007/s00234-024-03429-5`)* — not the segmentation metric alone.

## The leakage trap specific to reusing pretrained models

The base model you reuse may have been **developed on data that overlaps your test set** — the
`/model-sourcing` trap (evaluating on the benchmark the base was trained or tuned on, so the arm
reads like validation while being closer to a training-set score). **Pin the base-model
revision** and check its training corpus against your external set before you trust the number.
nnU-Net's self-configuration is attractive precisely because hyperparameters are set from the
**training data's fingerprint**, not tuned on the reported test folds.

## Reproducibility is part of the design for a combine-existing study

Reviewers of a "we built on X" paper expect to be able to run it. Decide **up front** to open
weights / code and to pin the base-model revision (`/architecture-zoo` records the licence — mind
non-commercial base weights such as nnInteractive / ConvNeXt V2 / most medical FMs).

## What this does NOT replace

- **metric selection** (Dice + a boundary/agreement metric) → `/model-evaluation`; **validation
  design + the split-leakage gate** → `/model-validation`; **which venue tier** the design lands at
  → `venue_accept_recipe.md`; **the model choice + its licence** → `/architecture-zoo` +
  `/model-sourcing`.

This is the **comparator-design decision** for a combine-existing-models study — the one that turns
"our model scored X" into "the combination is *why* it scored X". Decide it before data collection.

# DAG-based confounder adjustment-set selection

Pre-specify the adjustment set from a **causal DAG**, not from a Table-1 `p<0.05` screen.
A p-value screen adjusts for mediators and colliders (which *introduce* bias) and misses
confounders that happen to be balanced by chance — the single most common
confounding-completeness error reviewers reject.

## The backdoor criterion (what a sufficient set must do)

A set **Z** identifies the causal effect of X on Y if:
1. **Z blocks every backdoor path** (every path from X to Y that starts with an arrow into
   X), and
2. **Z contains no descendant of X** (adjusting for a descendant of the exposure is
   over-adjustment / collider-stratification).

Three roles drive every decision:

- **Confounder (common cause)** — has a path into X *and* an X-free path to Y. **Adjust.**
- **Mediator** — lies on a directed `X → … → Y` path. **Do not adjust** (it is part of the
  effect you want).
- **Collider** — a node with two arrows into it (`A → C ← B`). **Do not adjust** (and do
  not adjust its descendants): conditioning on a collider *opens* a non-causal path (M-bias).

Note: because `X → Y` makes every ancestor of X also an ancestor of Y, "common cause" is
**not** simply "ancestor of both." A node that affects Y *only through* X (an
instrument-like `A → X → Y`) is not a confounder — omitting it is not bias. A confounder
needs a path to Y that does **not** pass through X.

## Step 1 — draw the DAG, then sanity-check covariate roles deterministically

Write the DAG as parent→child edges and run the helper to classify each proposed covariate
and catch the four unambiguous errors before you write the Methods:

```bash
# dag.json: {"edges": [["age","statin"], ["age","CVD"], ["statin","LDL"], ["LDL","CVD"], ["statin","CVD"]]}
python3 scripts/adjustment_set_helper.py \
  --dag dag.json --exposure statin --outcome CVD \
  --adjust "age,LDL" --out qc/adjustment_set.json --strict
```

It flags `MEDIATOR_ADJUSTMENT` (here `LDL`, on `statin→LDL→CVD`),
`DESCENDANT_ADJUSTMENT`, `COLLIDER_ADJUSTMENT`, and `CONFOUNDER_OMITTED`, and prints a
**candidate** sufficient set (the open-backdoor common causes). The helper uses
reachability only and **does not claim minimality** — it catches the errors, not the
optimum.

## Step 2 — derive the minimal sufficient set with dagitty (the validated solver)

For the **minimal** adjustment set (and to verify sufficiency by d-separation), use
`dagitty` — do not hand-roll a d-separation solver. Generate and run:

```r
library(dagitty)
g <- dagitty('dag {
  age -> statin; age -> CVD
  statin -> LDL; LDL -> CVD
  statin -> CVD
}')
exposures(g) <- "statin"; outcomes(g) <- "CVD"
adjustmentSets(g, type = "minimal")     # the minimal sufficient set(s)
adjustmentSets(g, type = "canonical")   # the all-common-causes set
# Falsification: testable conditional independencies implied by the DAG
impliedConditionalIndependencies(g)
```

`ggdag::ggdag_adjustment_set(g)` renders the DAG with the adjustment set highlighted for a
supplement figure. Report the DAG (or its dagitty string) in the supplement so reviewers
can audit the identification assumptions.

## Step 3 — report

- State the adjustment set **and its DAG-based justification** in Methods (not "covariates
  with p<0.05").
- Pre-specify it before looking at outcome associations.
- Plan an **extended-adjustment sensitivity** model: if a measured covariate turns out
  imbalanced by exposure but sits outside the adjustment set, show the primary estimate is
  robust to adding it (`/self-review` Phase 2.5e + `observational_confounding.md` O1–O14
  check this at review time).
- Unmeasured confounding that remains → report an **E-value**.

## Handoff

- Target-trial / comparative design → `references/target_trial_emulation.md`.
- Estimation (IPTW, g-methods, standardization) + E-value → `/analyze-stats`.
- Review-time confounding completeness → `/self-review` Phase 2.5e, `observational_confounding.md`.

## References (cite the originals)

- Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research.
  *Epidemiology* 1999.
- Textor J, van der Zander B, Gilthorpe MS, et al. Robust causal inference using directed
  acyclic graphs: the R package 'dagitty'. *Int J Epidemiol* 2016.
- VanderWeele TJ. Principles of confounder selection. *Eur J Epidemiol* 2019.

# Compare several models head-to-head — design the comparison to be fair

For a study whose contribution is **comparing several models / architectures against each other** on
the same task (CNN vs Transformer vs a foundation-model backbone; N segmentation networks; N fusion
strategies), the single decision that most determines acceptance is **not which model wins — it is
whether the comparison is fair**. Reverse-engineered from accepted open-access papers (cited by DOI as
design facts). It is a **design-time** decision: none of the fairness controls below can be added after
you have run the models. Distinct from `combine_models_ablation_design.md` (an ablation *of one* model
built from parts) and `/design-ai-benchmarking` (AI *versus a human-expert panel*) — this is
**model-vs-model**.

## The failure the reviewer expects

*"Our model beat ResNet, DenseNet, and a transformer."* — with borrowed baseline numbers, only the
home model tuned, one accuracy per cell and no paired test, this is a **"we win" leaderboard**, and it
is the **rejected** pattern. Its purest published form: baselines copied from a benchmark's
documentation (never re-run under matched conditions) while only the authors' own model had its
learning-rate / batch-size / epochs configured, reported as a bare accuracy table with no CIs
*(Sci Rep 2024 `10.1038/s41598-024-63094-9`, included in the corpus as the negative control)*. The
operational proof that this matters: when CNN- vs Transformer- vs Mamba-segmentation is re-run under
**matched** conditions with an adequately-tuned baseline, most claimed architecture "wins" **evaporate**
*(nnU-Net Revisited, arXiv:2404.09556, CC BY — the field's fair-benchmarking argument; the MICCAI 2024
proceedings version `10.1007/978-3-031-72114-4_47` is © Springer, so reuse the arXiv copy)*.

## Design the comparison to be fair (decide before you run anything)

- **One dataset, one frozen patient-level split, one preprocessing pipeline — pushed through every
  model.** Fix the split and preprocessing *once* and reuse them identically; no model gets cleaner
  data, its own resampling, or a private augmentation policy. This is the reviewer's first question.
  *(liver multi-phase MRI, Sci Rep 2025 `10.1038/s41598-025-07084-5`: one N4/z-score/registration
  pipeline and one nested-CV stratification across nnU-Net, ResEnc nnU-Net, and Swin UNETR; mandibular
  canal CBCT, Int Dental J 2026 `10.1016/j.identj.2026.109427`: one fixed 128/20/25 split + identical
  clip/augmentation across UNETR / Swin UNETR / 3D UX-Net.)*
- **A strong, self-configuring reference baseline — not a hobbled U-Net.** If the reference model is
  under-tuned, the whole comparison is void. Instantiate the baseline through **nnU-Net / nnUNetv2** and
  configure the strongest competitor as carefully as your own. *(tooth CBCT, Head & Face Med 2025
  `10.1186/s13005-025-00555-0` [CC BY-NC-ND]: the "3D U-Net" arm is nnUNetv2, self-configuring; nnU-Net
  Revisited names a weak baseline as the mechanism that manufactures novel-method wins.)*
- **Match the training / HPO / compute budget across models — the #1 fairness threat — or disclose the
  gap.** "New ≠ better; the new model was just tuned harder / given more VRAM / trained longer." Either
  **match** the iteration budget *(mandibular canal, tooth: same 30k / 150-epoch budget for all)* or,
  where hardware forbids parity, **disclose the disparity in the open** *(liver: 8.5 vs 28 vs 40 GB VRAM,
  ~11 vs ~32 training-days, stated plainly)* — never silently tune only the home model. nnU-Net Revisited
  makes matched compute a first-class requirement.
- **Report variance over a single run.** A single-run leaderboard ranks by *skill + luck*: train over
  **multiple seeds** and report the spread, so the ranking is shown to be stable rather than a lucky
  draw. nnU-Net Revisited names **inter-publication variance** (the same baseline spanning a wide DSC
  range across papers) as the reason one-number leaderboards are untrustworthy. This is the control the
  accepted papers most often **still omit** — a place a new study can clear the current bar.
- **Pre-specify the primary metric and the primary comparison.** Name — before data — the one metric and
  the one contrast (proposed vs the strong baseline) the claim rests on. Choosing the metric or facet
  where you happen to win, after seeing the results, is cherry-picking; a multi-facet win is credible
  only when the primary was fixed in advance.

## Commit to a paired delta, not a side-by-side table

Run all models on the **same cases** so the between-model difference can be **tested**, not eyeballed.
The accepted comparisons pair and test: **Friedman + pairwise Wilcoxon signed-rank + Bonferroni** for
bounded, non-normal per-case Dice *(tooth)*, **repeated-measures ANOVA + post-hoc** *(mandibular canal)*,
**DeLong + Benjamini–Hochberg** for a paired ΔAUC *(spondylitis MRI, Eur J Med Res 2025
`10.1186/s40001-025-03731-9`)*. Two features recur: the test respects the **paired** structure, and
**multiplicity is corrected** when many pairs are compared. Deciding to pair is a design choice — you
cannot pair after collecting the arms separately. (Size this delta with `calc-sample-size` Test 16;
present it with the leaderboard figure below.)

## Rank honestly — a ranking is not a significance test

Put uncertainty on every model (a CI, or at least a tested delta) and **resist over-ranking** — but
rank against the right object: the **paired Δ and its CI**, not the overlap of two marginal CIs. Two
marginal intervals can overlap heavily while the paired Δ excludes zero, because the covariance the
paired test uses is invisible in the marginals; non-overlap implies a difference, overlap implies
nothing either way. Models inside the critical difference are **not separated by the test** — leave
them unranked rather than reporting a demonstrated tie. Disaggregate
(per-structure / per-class) and show **where the winning model still fails** — the honest negative
(a structure that collapses for all models, a fusion strategy at chance) is an acceptance asset, not a
liability. A corollary the corpus surfaced: a CI must be **believable for the N** — an implausibly tight
interval on a small test set is a red flag, not reassurance.

## What this does NOT replace

- **metric selection** (Dice + a boundary metric; AUROC + AUPRC) → `/model-evaluation`; **validation
  design + the split-leakage gate** → `/model-validation`; **sizing the between-model delta** →
  `calc-sample-size` `references/multi_model_comparison_sample_size.md` (Test 16); **presenting the
  comparison** → `make-figures` `exemplar_plots/model_comparison_leaderboard.md` + `analyze-stats`
  `table-standards/table-types/model_comparison.md`; **which architecture to consider at all** →
  `/architecture-zoo`; **a model built by combining / adapting one set of parts** →
  `combine_models_ablation_design.md`.

This is the **fair-comparison decision** for a model-vs-model study — the one that turns "our model won"
into "the comparison was fair, and our model won". Decide it before data collection.

# Reader / expert-elicitation study design

Load-on-demand companion to `/design-study` Phase 2, section E. Read it when the study
elicits expert ratings — a reader study, an annotation panel, or an AI-output evaluation.
A design with no human-rater arm needs none of it.

It covers rubric design (decoupled axes, anchored Likert points, pre-specified discriminant
validity), planted calibration probes, operational rigor, the human-as-**operator** arm for
interactive / promptable AI, and the six design-stage ceiling decisions for a perceptual /
reader AI study — the ones that fix the acceptance ceiling before a single reader sees an
image, and that no amount of good execution can lift afterwards.

When the study elicits expert ratings (reader study, annotation panel, AI-output evaluation), check
the following before data collection.

**Rubric design**
- **Decouple the axes.** Each rated dimension should measure one construct. Keep "is the finding
  valid/correct" separate from "is it novel", "is it feasible to measure", "does it add value over
  current tools", and "would it change action". A candidate can be high-validity yet low-added-value
  ("real but redundant"); a single blended score hides this.
- **Anchor every Likert point** with a short verbal descriptor; pilot the anchors with at least one
  reviewer before locking.
- **Pre-specify discriminant validity**: hypothesize which dimensions should correlate vs be
  orthogonal, then report the full inter-dimension correlation matrix to confirm the rubric measures
  distinct constructs.

**Calibration probes (planted control items)**
Insert a small number of deliberate control items, blinded and randomized across raters (record who
received which, e.g. a `probe_arm` flag), to (i) anchor the scale, (ii) measure rater drift and
fatigue, and (iii) audit the rubric and pipeline itself. Four useful flavors:
- **Positive control / "too-good" item** — a known-strong or near-tautological item; tests whether
  raters equate "largest effect" with "best", and whether an upstream construct-independence gate works.
- **Known-bad negative control** — an engineered defect (fabricated reference, missing key statistic);
  expected to score low.
- **Instability item** — an estimate that reverses or fails to replicate on holdout; tests caveat handling.
- **Mechanism-contradiction item** — an empirical direction that opposes the proposed mechanism.

Report inter-rater reliability **on the control items separately** as primary evidence of rubric and
scale validity; a low overall ICC is interpretable only if raters at least converge on the controls.

**Operational rigor**
- Randomize item order **per reviewer** (not one global seed); analyze order and fatigue effects.
- Collect reviewer metadata (years of experience, prior AI-evaluation experience, subspecialty) for
  descriptive reporting.
- Define a structured export schema (per-item ratings, free-text justifications, follow-ups, timing) up front.
- Require each item to be judged standalone; discourage cross-item references in free-text, which
  signal non-independent rating.

**Human-as-operator arm (interactive / promptable AI).** The reader-study patterns above assume the
human is a *rater / reference* judging outputs. Interactive / promptable segmentation (SAM2, MedSAM2,
nnInteractive) inverts this: the human is the *operator* who places the prompts, so the measured object
is the human-operated system's **accuracy + interaction count + time + learning curve**, not a rating.
Design for it explicitly:
- Define the operator population and their onboarding; a **learning curve** (performance vs case index)
  is a first-class outcome, not noise to average away.
- Fix the prompting protocol (allowed prompt types, stopping rule, target Dice) identically to any
  simulated-prompting arm so the two are comparable — **protocol fidelity**, checked in `/model-validation`.
- Pre-specify the interaction and timing metrics; their deterministic reporting gate is
  `/model-evaluation --task interactive`. (A design document is free-form prose, so the deterministic
  anchor for these items sits at the reporting stage, not on the protocol text.)

For an AI-system-versus-human-expert benchmark specifically, route to `/design-ai-benchmarking`, which
extends this subsection with arm definition, LLM-as-judge versus human-as-judge adjudication, and a
structured export schema.

**Perceptual / reader AI study — design-stage ceiling gate**

For a reader/observer/perceptual or diagnostic-accuracy AI study (visual Turing test, AI-vs-human
detection, image-provenance/deepfake, observer study), the acceptance ceiling is fixed **at design
time, not at analysis time** — excellent execution cannot lift a ceiling baked into the comparator,
the estimand, or the reader cohort. Walk these six before data lock and, for each, take the
higher-ambition option or record an explicit, defensible reason not to (set each at the impact level
of the journal you actually want):

1. **Comparator realism (biggest lever).** A curated teaching-repository "authentic" arm scopes the
   claim to "teaching-quality", not clinical. Use consecutive, de-identified clinical-acquisition
   images (the real PACS spectrum), or add a clinical-spectrum validation arm.
2. **Format / non-content confound matching.** Match every non-content attribute (aspect ratio,
   resolution, compression, color profile) across arms by construction, and pre-specify a
   confound-classifier ceiling check (format-only AUC must be ≪ reader AUC) as a *primary* gate.
3. **Synthetic / index-arm denominator (survivorship).** Pre-specify how failed/low-quality
   generations are counted; report the full generation denominator rather than evaluating only the
   convincing survivors.
4. **Reader independence and breadth.** Recruit an independent, non-author, multi-site (ideally
   multi-national) reader cohort; collect reader characteristics; blind readers to the hypothesis
   where feasible.
5. **Estimand and power (generalize, don't condition).** Power the reader-AND-case generalization as
   the **primary** estimand from the start, so the two-way interval — not a pool-conditional number —
   supports the headline claim.
6. **Novelty positioning vs scoop, and venue-fit.** Scan for close prior work at design time; if a
   flagship precedent exists, make the differentiation categorical (new modality class, clinical
   spectrum, outcome linkage), not incremental; pick the venue whose audience values the likely
   result (a rigorous null fits a methodology-forward journal better than an impact-first one).

The meta-rule: set the comparator, the confound-matching, the reader cohort, and the estimand at the
target journal's impact level **before** data collection — do not plan to out-write a structural
ceiling in revision.

# Is the segmentation usable — designing past the mean Dice

For a segmentation study whose claim is **clinical usability** rather than benchmark accuracy, the
decisive design question is not which metric you report but **what counts as a failure, who decides,
and whether the reader can see the tail**. A mean DSC of 0.90 with a 5% catastrophic tail and a
uniform 0.90 are different clinical objects, and the mean cannot tell you which one you have.
Reverse-engineered from accepted open-access papers (cited by DOI as design facts). These are
**design-time** decisions: an acceptability rate cannot be recovered from a finished experiment that
never asked a clinician, and a failure taxonomy assigned after seeing the failures is a description,
not a finding. Distinct from `/model-evaluation` (which metric, computed correctly) and
`/uncertainty-imaging` (per-case uncertainty, abstention, risk–coverage).

## The failure the reviewer expects

*"Mean DSC 0.87 across 40 organs-at-risk, therefore ready for clinical use."* — one aggregate number,
no per-case distribution, no clinician judgement, no statement of what a failure would have looked
like. The reviewer's objection is not that 0.87 is low. It is that **0.87 is silent on the question
being claimed**, and the study has no instrument that could have answered it.

How wide that silence is, measured: in a seven-site evaluation of one auto-contouring pipeline,
physician-rated *use-as-is* rates ranged from **89% (head/neck normal tissue) to 40% (head/neck CTV)
to 44% (postmastectomy breast)** — with **the same software** *(JCO Glob Oncol 2024
`10.1200/GO.23.00376`, CC BY: 5-point Likert, 31 radiation oncologists across 16 institutions and 6
countries, ≥3 independent raters per case)*. A single pooled accuracy figure over those structures
would have reported a usable system and concealed that a third of it was not.

## Design the usability question (decide before you run anything)

- **Pre-specify the failure taxonomy.** Name the classes before you look: *boundary drift* (right
  structure, wrong edge), *missed structure* (absent output), *hallucinated structure* (output where
  no structure exists), and *catastrophic / anatomically impossible* (a contour no clinician could
  have drawn). These fail differently in the clinic and a single overlap score maps all four onto one
  axis. Assigning classes after seeing the errors makes the taxonomy a summary of *this* run rather
  than a measurement instrument.
- **Define the acceptability endpoint, the judge, and the adjudication rule.** State the scale, who
  applies it, how many raters see each case, and how disagreement resolves — *before* data. A
  structured scale is what makes the rate reproducible: the JCO evaluation's 5 points (*unusable /
  major edits / minor edits required / stylistic only / use as-is*) yield a number another group can
  reproduce. The contrast is instructive: an otherwise careful head-and-neck validation assessed
  acceptability through **eight physicians' satisfaction categories with no structured scale**,
  concluding the contours needed "no edits of major clinical significance" *(Front Oncol 2023
  `10.3389/fonc.2023.1137803`, CC BY)*. That conclusion may well be right, and it is **not a rate** —
  nobody can reproduce it, meta-analyse it, or size a future study against it.
- **Report the tail, not only the centre.** Commit in advance to the **per-case distribution**, the
  **worst-case percentile**, and the **count of catastrophic failures** alongside the mean. "Only 4
  of 777 structures fell below DSC 0.5" *(Front Oncol 2023, above)* is a tail statement and belongs
  in the design, because a study that never records it cannot produce it later.
- **Treat edit effort as its own endpoint, paired and disaggregated.** If the claim is that the model
  saves work, measure the work: editing time on the **same cases** against manual-from-scratch, per
  structure and per site. The disaggregation is not decoration — a seven-centre evaluation reporting
  an overall **46% time saving** also found **no significant saving for lymph-node levels IA, IB,
  III, IVA and IVB**, with some centres taking **longer to edit than to contour manually** *(npj
  Digit Med 2025 `10.1038/s41746-025-01624-z`, CC BY-NC-ND — learn-only)*. The pooled 46% is true and
  would have hidden every one of those.
- **Stratify failures by what predicts them.** Decide up front which case attributes the failure rate
  will be broken down by — structure size, contrast phase, pathology present, scanner or site — so
  the study can say *where* the model fails rather than only *how often*. Small structures and target
  volumes are where the acceptability rates above collapse; a design that pools them cannot show it.

## Two traps specific to this endpoint

- **The acceptability question determines the answer.** *Use-as-is* and *acceptable after minor
  edits* are different endpoints, and the gap between them is large: 44% → 91% (breast), 40% → 93%
  (head/neck CTV) *(JCO Glob Oncol 2024)*. Both are legitimate; naming which one the headline claim
  rests on is mandatory, and reporting the permissive one while implying the strict one is the
  overclaim reviewers catch.
- **A high global score can coexist with a clinically unusable case.** Overlap computed over a whole
  volume is dominated by the easy interior, so a contour can score well in 3-D while carrying errors
  on the slices that matter. Design the per-case (and where relevant per-slice) read so that such a
  case is *visible* rather than averaged away.

## What this does NOT replace

**Which metric and how it is computed** (Dice + a boundary metric, per structure) →
`/model-evaluation`; **per-case uncertainty, abstention, and risk–coverage failure detection** →
`/uncertainty-imaging`; **split leakage, tuning-on-test, internal vs external** → `/model-validation`;
**comparing several models fairly** → `multi_model_comparison_design.md`; **sizing the acceptability
rate, the failure-rate bound, and the edit-time contrast** → `calc-sample-size`
`references/segmentation_acceptability_sample_size.md` (Test 17); **showing the distribution and its
tail** → `make-figures` `exemplar_plots/segmentation_failure_panel.md`.

This is the **usability decision** for a segmentation study — the one that turns "the mean Dice was
high" into "clinicians accepted N% of cases as-is, these are the failures, and this is where they
fall". Decide it before data collection.

# Target-trial emulation — design module

A target trial is the hypothetical randomized trial you would run if you could. Emulating
it with observational data is the discipline that turns an **association** into a
**defensible causal contrast** — the jump from a specialty paper to a high-impact one, and
the design that high-impact journals now expect for comparative observational questions
(treatment vs treatment, screening vs no screening, drug A vs drug B).

Use this when the question is causal/comparative on routinely-collected data (NHIS,
KNHANES, EHR, registry, health-checkup DB). It is **not** needed for a purely descriptive,
prevalence, or diagnostic-accuracy study.

## Protocol component table (fill every row before data extraction)

Specify all seven, then state how each is **emulated** in the data. Leaving a row blank is
where immortal-time, prevalent-user, and selection bias enter.

| # | Target-trial component | Specify | Emulation in the data |
|---|------------------------|---------|------------------------|
| 1 | **Eligibility criteria** | who could enroll (assessed using only information available at time zero) | the cohort filter; no criterion may use post-baseline information |
| 2 | **Treatment strategies** | the well-defined interventions being compared (dose, duration, start/stop rules) | how each strategy is identified from codes/prescriptions |
| 3 | **Assignment / treatment groups** | randomized in the target trial; here, assigned by observed strategy at time zero | new-user / active-comparator definition (below) |
| 4 | **Time zero (start of follow-up)** | the instant eligibility, treatment assignment, and follow-up **coincide** | the index date — must be identical for all three; misalignment ⇒ immortal-time bias |
| 5 | **Outcome** | the event, its ascertainment, and validation | code algorithm + (ideally) a validation reference |
| 6 | **Causal contrast** | ITT (assignment), per-protocol (sustained adherence), or as-treated | the estimand you report |
| 7 | **Analysis plan** | the model + confounding control + censoring weights | IPTW / g-methods / standardization; pre-specified covariates from a DAG |

## The three alignment errors this prevents

1. **Immortal-time bias.** Time zero must be the moment eligibility, strategy assignment,
   and follow-up start *coincide*. If a strategy is defined by something that can only
   happen after baseline (e.g., "patients who received ≥2 doses"), the period before that
   is immortal — survivors are guaranteed event-free. Fix: assign strategy using only
   information available at time zero; if a strategy needs a window to be met, use a
   **grace period** with cloning + censoring + weighting (below).
2. **Prevalent-user bias.** Including patients already on treatment at baseline conditions
   on having survived/tolerated it. Fix: **new-user (incident-user) design** — time zero =
   treatment initiation.
3. **Confounding by indication.** The reason a treatment was chosen also affects the
   outcome. Fix: **active comparator** (compare two treatments for the same indication, not
   treatment vs nothing) + a DAG-derived adjustment set (`dag_adjustment.md`), with IPTW
   or g-methods rather than a single outcome-regression adjustment.

## Grace period + clone-censor-weight (when a strategy needs a window)

When a strategy is "initiate within N days," a patient's data are consistent with *both*
strategies during the grace window. The standard emulation:
1. **Clone** each eligible person into both strategy arms at time zero.
2. **Censor** a clone when its data diverge from its assigned strategy (e.g., did not
   initiate within the grace period).
3. **Weight** by inverse probability of censoring to remove the selection introduced by
   artificial censoring.
This removes immortal-time bias without discarding the grace window. Report the grace
period and the censoring/weighting model explicitly.

## Estimand: ITT vs per-protocol

- **ITT (assignment) effect** — robust (assignment is as-randomized-as-emulated) but
  diluted by non-adherence; the conservative primary for most emulations.
- **Per-protocol effect** — the effect of *sustained* adherence; needs IP-weighting for
  adherence-related time-varying confounding (a naive per-protocol analysis re-introduces
  selection bias). State which is primary and why.

## Negative controls (falsification)

Pre-specify at least one **negative-control outcome** (an outcome the exposure cannot
plausibly cause through the hypothesized mechanism) and, where possible, a **negative-control
exposure**. A non-null association on a negative control flags residual confounding /
design bias and bounds the credibility of the primary estimate. Report them alongside the
primary result, not only when convenient.

## Reporting + registration

- Report against **STROBE** (+ RECORD for routinely-collected data); state the target-trial
  protocol table in the Methods or a supplement.
- For comparative-effectiveness emulations, pre-register the protocol (the seven components
  above) — a registered target-trial protocol is the strongest defense against post-hoc
  design choices.

## Handoff

- Adjustment set → `references/dag_adjustment.md` + `scripts/adjustment_set_helper.py`.
- Time-origin / survivorship structural checks → design-study Phase 2 §F.
- Analysis (IPTW, g-methods, marginal structural models) → `/analyze-stats`.
- Sample size for the emulated trial → `/calc-sample-size`.

## References (canonical — cite the originals, do not paraphrase as your own)

- Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial
  is not available. *Am J Epidemiol* 2016.
- Hernán MA, Sauer BC, Robins JM, et al. Specifying a target trial prevents immortal time
  bias and other self-inflicted injuries in observational analyses. *J Clin Epidemiol* 2016.
- Hernán MA, Wang W, Leaf DE. Target trial emulation: a framework for causal inference from
  observational data. *JAMA* 2022.

# Venue-tier acceptance recipe — clinical DL / AI-validation studies (design-time)

This is **not** another quality checklist. `design-study` already covers the comparator,
leakage, metric, and reader-study-ceiling decisions (see Phase 3, the Frequent Failure Modes,
and `reader_elicitation_design.md`). This reference answers a **different** question, and answers
it *before data collection*:

> For the study you can **actually run** given your data, labels, and compute, **which venue tier
> will accept it — and which single design move most raises the acceptance probability at the tier
> above it?**

It is reverse-engineered from **accepted open-access papers** (cited by DOI below), and it is a
design-time decision aid, not a post-hoc rescue.

## The decoupling principle (why this is a design decision, not a writing one)

Acceptance of a clinical DL / AI-validation study is set at **design time** by three things — the
**comparator**, the **validation reach**, and the **scope-vs-venue** match — not by how well the
manuscript is written. A structurally single-centre, internal-only study cannot out-write its way
into a flagship; venue-fit is **decoupled from quality** (an internal-only study can be excellent
and still land only at a focused venue). The corollary: pick the tier your design can support, or
pay for one specific "clearing move" to reach the next tier — *before* you collect data, because
none of these moves can be added post hoc.

## The acceptance ladder — design profile → venue tier → the clearing move

| The design you can actually run | Lands at (tier) | The clearing move that got it accepted | Accepted OA exemplar |
|---|---|---|---|
| **Single-centre, internal validation**, but a disciplined **clinical-endpoint** comparator + independent **blinded** readers against an independent reference standard | Focused clinical-radiology (**KJR / AJNR / Eur Radiol**) | Replace the image-metric endpoint with a **clinical diagnostic endpoint**, and read it with **≥2 blinded readers vs an independent reference standard** — the reader discipline substitutes for external data | TLE MRI deep-learning reconstruction, *KJR* 2024 (`10.3348/kjr.2023.0842`, CC BY-NC): 3-arm comparator (routine vs thin-slice vs thin-slice+DLR), 2 blinded neuroradiologists on 351 shuffled studies, EEG/surgical reference, pre-specified subgroup — accepted single-centre on comparator + reader rigour |
| **Focused single-organ / biomarker task**, multi-centre development + a **genuine external** set with an **honestly reported drop**, reporting-guideline named, ideally **open weights/data** | SpringerOpen methods (**Insights into Imaging / European Radiology Experimental**) | Add a **genuine external** test set (different sites/scanners) and **report the degradation**; name the fitting guideline (**CLEAR + METRICS** for radiomics, **STARD / TRIPOD+AI** for diagnostic models); release weights/data | Endometrial-subtype radiomics-DL, *Insights into Imaging* 2025 (`10.1186/s13244-025-01966-y`, CC BY): 3-site, combined external (macro-AUC 0.79→0.74), **nested** clinical→radiomics→DL comparator, CLEAR+METRICS · Inner-ear U-Net, *Eur Radiol Exp* 2024 (`10.1186/s41747-024-00508-3`, CC BY): 4-centre / 3-vendor train → held-out **5th-vendor** external (DSC 0.89→0.83), open weights + data |
| **Full ladder** — multi-centre dev → **multiple external** sets → **reader study** (accuracy **and** reading time) → prospective cohort; **or** a reader study as the centrepiece with washout + specialty/experience subgroups | Flagship / methods-flagship (**npj Digital Medicine / Radiology / Lancet Digital Health**) | Add a **reader study that measures clinical impact** (MRMC/crossover, washout, blinding, readers across experience/specialty, accuracy + reading-time) on top of external validation — the flagship bar is *"the model changes what a clinician does, and for whom"* | PCN CT model, *npj Digital Medicine* 2025 (`10.1038/s41746-025-01970-y`, CC BY): 4-hospital dev + **3 external sets** + 8-reader MRMC crossover (AUC 0.786→0.845, reading time down) + prospective 3-month-follow-up cohort · Pelvic-radiograph AI-support, *npj Digital Medicine* 2025 (`10.1038/s41746-025-01923-5`, CC BY-NC-ND): 26-physician, 3-session (no-AI / alert / heatmap) **washout** reader study as the centrepiece, specialty subgroups |

*Read the exemplars as design patterns, not templates; the DOIs let you inspect the real designs.*

## The five design levers, ranked by acceptance impact

Distilled across the accepted batch above — the recurring decisions that separated them from a
predictable reject:

1. **External / multi-site validation with an honest drop is the single biggest lever.** Every
   top-tier accept tested on data from sites/scanners *outside* development and **reported the
   degradation openly** (0.89→0.83 DSC; 0.79→0.74 AUC). A disclosed generalization gap earns more
   trust than a suspiciously flat external number. Internal-only studies survive only when a very
   strong reader design carries them.
2. **A comparator that is the real clinical alternative, not a strawman.** Accepts benchmark
   against *current practice* — radiologist double-reading, the conventional reconstruction, or a
   **nested** clinical→radiomics→DL ladder that isolates the marginal value of the DL component.
   "Model vs nothing" reads as promotional (see also Phase 3 *incremental value* in SKILL.md).
3. **A reader study that measures clinical impact, not standalone AUC.** MRMC / crossover designs
   with **washout, blinding, multiple readers across experience/specialty, and reading-time +
   accuracy endpoints** recur across the batch. Showing the model changes clinician behaviour (and
   for whom, via subgroups) is what lifts a paper above a leaderboard result. Design mechanics live
   in `reader_elicitation_design.md`; route AI-vs-human benchmarks to `/design-ai-benchmarking`.
4. **Scope matched to the venue tier** (the ladder above). Mismatching an underpowered single-centre
   study to a flagship is a predictable reject; a focused, honest single-organ study is a clean
   accept at a methods venue.
5. **Explicit reporting-guideline conformance + reproducibility artifacts.** Accepts *name* their
   framework (STARD / TRIPOD+AI for diagnostic-accuracy models; CLEAR + METRICS for radiomics)
   rather than gesturing at "standard methods" — it pre-empts the methods-rigour objection. Where
   the cohort is modest, **releasing weights + data + code** is itself the acceptance argument, and
   substitutes for scale at the methods venues.

## Using this at design time (the decision)

Before you collect data:

1. **Name honestly the design you can run** (data reach, label source, whether an external cohort
   and readers are obtainable).
2. **Read its tier** off the ladder.
3. **Decide**: submit at that tier, or invest the **one clearing move** to reach the next tier. The
   cheapest ladder rungs, in order, are usually: *(a) add a genuine external set → (b) add a reader
   study measuring clinical impact → (c) add a prospective cohort.*
4. Do **not** push a structurally-ceilinged design at a higher tier expecting to out-write it —
   that is the cascade-reject trap. If the ceiling is real, either lower the venue or pay for the
   clearing move at design time.

## What this does NOT replace

- **Comparator / incremental-value / endpoint-scope** design quality → Phase 3 of this skill.
- **Reader-study internals** (rubric axes, calibration probes, the six ceiling decisions) →
  `reader_elicitation_design.md`; **AI-vs-human-expert** benchmark → `/design-ai-benchmarking`.
- **The journal shortlist itself** (scope fit, AI-policy, APC) → `/find-journal`.

This reference is the **design → acceptance-tier bridge** you consult *before* those — it turns "is
my study good?" into "which tier is my study *for*, and what one move moves it up?"

#!/usr/bin/env python3
"""Confounder-adjustment-set helper for DAG-based covariate selection.

design-study (and `~/.claude/rules` confounding-completeness) tells authors to
"pre-specify the adjustment set from a DAG, not a Table-1 p<0.05 rule" — but shipped
no scaffold to do it. This is that scaffold. Given a causal DAG, an exposure, an
outcome, and a *proposed* adjustment set, it deterministically classifies each
covariate by its DAG role and flags the adjustment errors reviewers reject:

  - MEDIATOR_ADJUSTMENT   — adjusting for a node on a directed X→…→Y path (blocks part
                            of the effect you are trying to estimate).
  - DESCENDANT_ADJUSTMENT — adjusting for a descendant of the exposure (over-adjustment /
                            collider-stratification on the causal pathway).
  - COLLIDER_ADJUSTMENT   — adjusting for a collider (≥2 parents, not a common cause)
                            opens a non-causal path (M-bias).
  - CONFOUNDER_OMITTED    — a common cause of X and Y is NOT in the adjustment set, so a
                            backdoor path is left open.

It also proposes a *candidate* sufficient set — the pre-exposure common causes of X and
Y — which is a valid (if not always minimal) backdoor adjustment set. Finding the
**minimal** sufficient set in general is a graph problem best left to a validated tool;
this helper prints ready-to-run dagitty code for that and never claims minimality.

Soundness: every classification uses only reachability on the directed graph (ancestors
/ descendants), which is unambiguous. It does NOT implement full d-separation, so it will
not certify an arbitrary set as sufficient — it flags the four common, unambiguous errors
and defers optimal minimisation to dagitty.

DAG input (JSON): {"edges": [["C","X"], ["C","Y"], ["X","Y"]]}  (parent → child).
Stdlib-only. Exit codes: 0 clean (or report-only), 1 a Major flag exists (--strict),
2 input/usage error.
"""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path


def _build(edges):
    children: dict[str, set[str]] = {}
    parents: dict[str, set[str]] = {}
    nodes: set[str] = set()
    for a, b in edges:
        children.setdefault(a, set()).add(b)
        parents.setdefault(b, set()).add(a)
        nodes.update((a, b))
    for n in nodes:
        children.setdefault(n, set())
        parents.setdefault(n, set())
    return nodes, children, parents


def _reach(start: str, adj: dict[str, set[str]]) -> set[str]:
    """Nodes reachable from `start` following `adj` (excludes start)."""
    seen: set[str] = set()
    stack = list(adj.get(start, ()))
    while stack:
        n = stack.pop()
        if n in seen:
            continue
        seen.add(n)
        stack.extend(adj.get(n, ()))
    return seen


def classify(edges, exposure: str, outcome: str, adjust: list[str]) -> dict:
    nodes, children, parents = _build(edges)
    for required in (exposure, outcome):
        if required not in nodes:
            raise ValueError(f"node {required!r} not in DAG")

    desc_X = _reach(exposure, children)          # descendants of exposure
    anc_X = _reach(exposure, parents)            # ancestors of exposure
    anc_Y = _reach(outcome, parents)             # ancestors of outcome (full graph)

    mediators = (desc_X & anc_Y) - {exposure, outcome}        # on a directed X→…→Y path

    # Confounder = common cause with a path to Y that does NOT pass through X (an open
    # backdoor). Because X→Y makes every ancestor of X an ancestor of Y, intersecting on
    # the full graph would mis-flag instrument-like A→X→Y nodes. Compute Y's ancestors on
    # the graph with X removed, so only nodes with an X-free path to Y count.
    parents_noX = {n: (p - {exposure}) for n, p in parents.items() if n != exposure}
    anc_Y_noX = _reach(outcome, parents_noX)
    common_causes = (anc_X & anc_Y_noX) - {exposure, outcome} - desc_X  # open-backdoor confounders
    candidate_set = sorted(common_causes)

    claims: list[dict] = []
    for z in adjust:
        if z not in nodes:
            claims.append({"node": z, "role": "unknown",
                           "verdict": "NODE_NOT_IN_DAG", "severity": "Minor",
                           "detail": f"'{z}' is not a node in the DAG"})
            continue
        if z in mediators:
            claims.append({"node": z, "role": "mediator",
                           "verdict": "MEDIATOR_ADJUSTMENT", "severity": "Major",
                           "detail": f"'{z}' lies on a directed {exposure}→…→{outcome} path; "
                                     f"adjusting for it removes part of the causal effect"})
        elif z in desc_X:
            claims.append({"node": z, "role": "descendant_of_exposure",
                           "verdict": "DESCENDANT_ADJUSTMENT", "severity": "Major",
                           "detail": f"'{z}' is a descendant of the exposure; adjusting for it "
                                     f"is over-adjustment (collider-stratification on the pathway)"})
        elif z not in common_causes and len(parents.get(z, ())) >= 2 \
                and z not in anc_X and z not in anc_Y:
            claims.append({"node": z, "role": "collider",
                           "verdict": "COLLIDER_ADJUSTMENT", "severity": "Major",
                           "detail": f"'{z}' is a collider (≥2 parents, not a common cause); "
                                     f"conditioning on it can open a non-causal path (M-bias)"})
    # backdoor left open: a common cause not adjusted for
    for c in sorted(common_causes - set(adjust)):
        claims.append({"node": c, "role": "confounder",
                       "verdict": "CONFOUNDER_OMITTED", "severity": "Major",
                       "detail": f"'{c}' is a common cause of {exposure} and {outcome} but is not "
                                 f"in the adjustment set; a backdoor path is left open"})
    return {
        "exposure": exposure, "outcome": outcome,
        "proposed_adjustment": sorted(adjust),
        "candidate_sufficient_set": candidate_set,
        "mediators": sorted(mediators),
        "descendants_of_exposure": sorted(desc_X),
        "claims": claims,
        "note": "candidate_sufficient_set is the pre-exposure common-cause set — a valid backdoor "
                "set, not necessarily minimal. Minimise/verify with dagitty (see design-study "
                "references/dag_adjustment.md).",
    }


def main(argv=None) -> int:
    ap = argparse.ArgumentParser(description="DAG-based adjustment-set helper (design-study).")
    ap.add_argument("--dag", required=True, help='JSON: {"edges": [["C","X"], ...]} (parent→child)')
    ap.add_argument("--exposure", required=True)
    ap.add_argument("--outcome", required=True)
    ap.add_argument("--adjust", default="", help="comma-separated proposed adjustment covariates")
    ap.add_argument("--out", help="write the classification JSON here")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major flag")
    ap.add_argument("--quiet", action="store_true")
    args = ap.parse_args(argv)

    try:
        data = json.loads(Path(args.dag).read_text(encoding="utf-8"))
        edges = data["edges"] if isinstance(data, dict) else data
        adjust = [z.strip() for z in args.adjust.split(",") if z.strip()]
        result = classify(edges, args.exposure, args.outcome, adjust)
    except (OSError, ValueError, KeyError, json.JSONDecodeError) as e:
        print(f"INPUT-ERR: {e}", file=sys.stderr)
        return 2

    if args.out:
        Path(args.out).write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding="utf-8")
    majors = [c for c in result["claims"] if c["severity"] == "Major"]
    if not args.quiet:
        print(f"candidate sufficient set (common causes): {result['candidate_sufficient_set'] or '∅'}")
        for c in result["claims"]:
            print(f"  [{c['severity']}] {c['verdict']}: {c['node']} — {c['detail']}")
        print(f"\n{len(majors)} Major adjustment flag(s).")
    return 1 if (args.strict and majors) else 0


if __name__ == "__main__":
    raise SystemExit(main())