# Classification architectures (architecture-zoo)

For "is finding X present / which class" questions — per image or per patient, binary or
multi-label. Almost always start from a **pretrained** backbone (ImageNet via `timm`, or a
medical foundation model — see `foundation_models.md`) and fine-tune; training a large
backbone from scratch on a few thousand medical images underperforms and overfits.

Each card: **paper → core idea → when to use → medical-imaging use → reference impl →
validation/experiment setup → `/model-scaffold` note.** Cite the paper in your decision
note; never quote a benchmark you have not cited.

---

## CNN backbones

### AlexNet / VGG (the baseline lineage)
- **Papers**: Krizhevsky et al., *NeurIPS* 2012 (AlexNet); Simonyan & Zisserman, *ICLR*
  2015 (VGG).
- **Core idea**: deep stacks of small convolutions + ReLU + pooling; VGG showed depth with
  3×3 convs. Historically important, now mostly superseded as backbones.
- **When to use**: rarely as a primary backbone today; VGG features still appear as a
  **perceptual loss** network in synthesis. Use ResNet/EfficientNet instead.

### ResNet (residual learning) — the default workhorse
- **Paper**: He et al., "Deep Residual Learning for Image Recognition," *CVPR* 2016.
- **Core idea**: identity skip connections let very deep nets train (the residual `F(x)+x`
  eases optimisation / vanishing gradients).
- **When to use**: the **safe default** for 2-D medical classification; ResNet-50 is a
  strong, well-understood, well-calibrated baseline that plays nicely with Grad-CAM.
- **Medical-imaging use**: CXR multi-label classification, fundus grading, path-tile
  classification; a ResNet-FPN is the backbone for detection (see `segmentation.md`).
- **Reference impl**: `timm` (`resnet50`, pretrained); torchvision.
- **Validation setup**: patient-level split; report **AUROC + AUPRC with CIs** (AUPRC for
  the minority class), sensitivity/specificity, and PPV/NPV at the deployment prevalence;
  Grad-CAM sanity check that attention is on pathology, not a shortcut.
- **Scaffold**: a `timm` classifier head on a pretrained ResNet (classification template,
  forthcoming in `/model-scaffold`; today, segmentation is the shipped template).

### DenseNet (dense connectivity)
- **Paper**: Huang et al., "Densely Connected Convolutional Networks," *CVPR* 2017.
- **Core idea**: each layer receives all preceding feature maps (concatenation) — strong
  feature reuse, parameter-efficient.
- **When to use**: a strong CXR baseline (DenseNet-121 is the CheXNet backbone); good when
  parameters/compute are tight.
- **Medical-imaging use**: chest-X-ray pathology classification (CheXpert/MIMIC-CXR-style).
- **Reference impl**: `timm` (`densenet121`).
- **Validation setup**: as ResNet; multi-label → per-label AUROC/AUPRC + a macro/micro
  average, calibration per label.

### EfficientNet (compound scaling)
- **Paper**: Tan & Le, "EfficientNet," *ICML* 2019.
- **Core idea**: jointly scale depth/width/resolution by a compound coefficient — better
  accuracy/compute trade-off.
- **When to use**: when you want the **best accuracy-per-FLOP** baseline; good on
  higher-resolution images where input resolution matters (mammography, path).
- **Reference impl**: `timm` (`efficientnet_b0..b7`, `tf_efficientnetv2`).
- **Validation setup**: as ResNet; watch that the chosen resolution matches the clinically
  relevant detail; report compute if deployment-constrained.

### Inception / GoogLeNet (multi-scale)
- **Papers**: Szegedy et al., *CVPR* 2015 (Inception v1) and the BN-Inception / v3 line.
- **Core idea**: parallel multi-scale convolution branches (1×1/3×3/5×5) per block.
- **When to use**: legacy strong baseline; multi-scale lesions. Usually ResNet/EfficientNet
  preferred now.

### ConvNeXt / ConvNeXt V2 — the modern CNN (CNNs, reasserted)
- **Papers**: Liu et al., "A ConvNet for the 2020s" (ConvNeXt), *CVPR* 2022; Woo et al.,
  ConvNeXt V2 (FCMAE masked-autoencoder pretraining + Global Response Normalisation), *CVPR*
  2023.
- **Core idea**: a pure CNN modernised with transformer-era design choices (large kernels,
  LayerNorm, inverted bottlenecks) that **matches or beats ViT/Swin at equal compute** — the
  classification counterpart to the "scale the CNN, new≠better" lesson in `segmentation.md`.
- **When to use**: a strong modern backbone when you want CNN inductive bias + good transfer
  without ViT's data appetite; a sensible default alongside ResNet/EfficientNet for medical
  classification.
- **Reference impl**: `timm` (`convnext_*`, `convnextv2_*`).
- **Licence**: ConvNeXt (V1) code **and weights MIT** (commercial OK); ConvNeXt **V2 code is
  MIT but its ImageNet weights are CC-BY-NC** (non-commercial) — use V1 weights or your own
  pretraining if the model feeds a product.
- **Validation setup**: as ResNet; if you use V2's self-supervised weights, keep the
  pretraining corpus disjoint from the test patients (contamination — `/model-validation`
  MD1/MD3).

---

## Vision transformers (when data is large)

### ViT (Vision Transformer)
- **Paper**: Dosovitskiy et al., "An Image is Worth 16×16 Words," *ICLR* 2021.
- **Core idea**: split the image into patches, treat them as tokens, apply a standard
  transformer; global attention from layer 1, but **data-hungry** (needs large pretraining).
- **When to use**: large labelled sets **or** a strong pretrained ViT (ImageNet-21k, DINO,
  or a medical foundation model); underperforms CNNs on small medical sets trained from
  scratch.
- **Reference impl**: `timm` (`vit_base_patch16_224`); pretrained essential.
- **Validation setup**: as ResNet; be explicit that performance leans on the pretraining
  corpus (contamination/transfer caveat — `/model-validation` MD3/MD7).

### Swin Transformer (hierarchical, windowed attention)
- **Paper**: Liu et al., "Swin Transformer," *ICCV* 2021.
- **Core idea**: hierarchical feature maps with **shifted-window** local attention — linear
  complexity in image size; a strong **backbone for dense tasks** (detection/segmentation),
  not only classification (Swin-UNETR is its segmentation form, see `segmentation.md`).
- **When to use**: higher-resolution inputs, or when you want a transformer backbone that
  also feeds a segmentation/detection head.
- **Reference impl**: `timm` (`swin_base_patch4_window7_224`); MONAI `SwinUNETR` for 3-D seg.

### DeiT / MLP-Mixer (efficiency / token-mixing)
- **Papers**: Touvron et al., DeiT, *ICML* 2021 (data-efficient ViT via distillation);
  Tolstikhin et al., MLP-Mixer, *NeurIPS* 2021 (attention-free token mixing).
- **When to use**: DeiT when you want a ViT that trains on less data via distillation;
  MLP-Mixer mostly of conceptual interest. CNNs/Swin usually preferred for medical work.

---

## Choosing among these
Small/medium labelled data → **pretrained ResNet/DenseNet/EfficientNet**, or **ConvNeXt** for
a modern CNN backbone (mind the V2 weight licence). Large data or a strong pretrained
transformer → **ViT/Swin**. Always pretrained, always patient-level split,
always AUROC **and** AUPRC with CIs. Record the choice + paper in the decision note and hand
to `/model-scaffold`; validate with `/model-validation`, evaluate with `/model-evaluation`.

# Detection architectures (architecture-zoo)

For "find and localise lesions" questions — boxes / points, a count, and a per-lesion
hit/miss (FROC). Distinct from segmentation (a pixel mask) and classification (a per-image
label): detection localises *instances*. `/model-scaffold --task detection` emits a
torchvision Faster R-CNN repo whose FROC/mAP you compute downstream.

Each card: **paper → core idea → when to use → medical-imaging use → reference impl →
validation/experiment setup.**

---

## Self-configuring 3-D detection (the default to beat)

### nnDetection
- **Paper**: Baumgartner et al., "nnDetection: A Self-configuring Method for Medical Object
  Detection," *MICCAI* 2021.
- **Core idea**: nnU-Net's philosophy applied to **detection** — auto-configures
  preprocessing, anchors, network topology, and training from the dataset fingerprint for
  **3-D volumetric** lesion detection, with no manual tuning. (First release is 3-D only; no
  2-D / Mask R-CNN.)
- **When to use**: the **default to beat** for 3-D medical lesion detection (nodules,
  aneurysms, focal lesions on CT/MR), exactly as nnU-Net is for segmentation — start here and
  justify any custom detector against it; it removes the anchor/scale tuning a torchvision
  detector needs.
- **Medical-imaging use**: LUNA16-style nodule detection, 3-D lesion / aneurysm detection.
- **Reference impl**: `MIC-DKFZ/nnDetection` (Apache-2.0). Integrate, do not reimplement.
- **Validation setup**: report **FROC** (sensitivity per false-positive-per-scan); its
  internal CV is development-time optimism correction, not external validation
  (`/model-validation` MD3/MD6); keep the patient-level split consistent end to end.

---

## Two-stage detectors (region proposal → classify)

### R-CNN → Fast R-CNN → Faster R-CNN (+ FPN)
- **Papers**: Girshick et al., R-CNN, *CVPR* 2014; Girshick, Fast R-CNN, *ICCV* 2015; Ren
  et al., Faster R-CNN, *NeurIPS* 2015; Lin et al., **FPN**, *CVPR* 2017.
- **Core idea**: Faster R-CNN adds a learned Region Proposal Network (end-to-end); FPN adds
  a multi-scale feature pyramid so small and large lesions are both detected.
- **When to use**: the **default two-stage detector** for medical lesion detection — strong,
  well-understood, good on small objects with FPN; favour accuracy over real-time speed.
- **Medical-imaging use**: nodule / lesion / aneurysm detection on CT / MR / mammography
  (ResNet-FPN backbone).
- **Reference impl**: torchvision `fasterrcnn_resnet50_fpn`; MONAI detection (RetinaNet).
- **Validation setup**: report **FROC** (sensitivity per false-positive-per-scan) or **mAP
  with the IoU match criterion stated**; per-lesion analysis with patient-level clustering
  disclosed; not patient-level accuracy (`/model-validation` MD6).

### Mask R-CNN (detect + segment instances)
- **Paper**: He et al., Mask R-CNN, *ICCV* 2017.
- **Core idea**: a mask head on Faster R-CNN → per-instance box + class + mask.
- **When to use**: **count + localise + delineate** separate lesions (instance-level), not a
  single semantic mask (that is `segmentation.md`).
- **Reference impl**: torchvision `maskrcnn_resnet50_fpn`.
- **Validation setup**: detection metrics for the boxes + per-instance Dice for the masks.

## One-stage / query-based detectors (faster, end-to-end)

### RetinaNet (focal loss)
- **Paper**: Lin et al., "Focal Loss for Dense Object Detection," *ICCV* 2017.
- **Core idea**: a one-stage dense detector with **focal loss** to handle the extreme
  foreground/background imbalance — relevant when lesions are sparse.
- **When to use**: faster than two-stage, strong under heavy class imbalance.
- **Reference impl**: torchvision `retinanet_resnet50_fpn`; MONAI detection.

### YOLO family (incl. modern Ultralytics)
- **Papers**: Redmon et al., YOLO, *CVPR* 2016; YOLOv3+/YOLOX; **YOLOv8 / YOLOv11**
  (Ultralytics, 2023–24) are the current widely-used releases.
- **Core idea**: a single network predicts boxes + classes directly on a grid — real-time.
- **When to use**: speed-critical / interactive settings; for maximal sensitivity on small
  medical lesions, two-stage detectors or **nnDetection** (3-D) are usually preferred.
- **Licence — check before commercial use**: **Ultralytics YOLOv8/v11 are AGPL-3.0** (strong
  copyleft — a deployed derivative must itself be open-sourced, or you buy Ultralytics'
  commercial licence). If that is a problem, prefer an Apache/MIT detector — **RT-DETR**
  (real-time DETR), torchvision Faster R-CNN, or MONAI RetinaNet.

### DETR / RT-DETR (transformer, set prediction)
- **Papers**: Carion et al., "End-to-End Object Detection with Transformers," *ECCV* 2020;
  **RT-DETR** (Zhao et al., *CVPR* 2024) — a real-time, permissively licensed variant.
- **Core idea**: a transformer treats detection as direct **set prediction** (no anchors /
  NMS) via learned object queries + bipartite matching; RT-DETR makes it real-time.
- **When to use**: large datasets where an anchor-free, end-to-end pipeline is attractive;
  more data-hungry and slower to converge than CNN detectors. **RT-DETR (Apache-2.0)** is the
  permissively licensed real-time alternative to Ultralytics YOLO's AGPL.
- **Reference impl**: the official DETR repo; Deformable DETR for faster convergence; RT-DETR.

---

## Choosing among these
**3-D volumetric lesion detection → nnDetection** (self-configuring, the default to beat).
2-D lesion detection → **Faster R-CNN + FPN** (torchvision; `/model-scaffold --task
detection`). Sparse lesions / imbalance → **RetinaNet (focal loss)**. Count + delineate
instances → **Mask R-CNN**. Speed-critical → **YOLO** (mind the **AGPL-3.0** licence) or
**RT-DETR** (Apache-2.0). Large data, anchor-free → **DETR**.
Always report **FROC / mAP with the IoU criterion stated**, per-lesion with patient-level
clustering disclosed. Record the choice + paper, hand to `/model-scaffold`, validate with
`/model-validation` and `/model-evaluation`.

# Foundation models & self-supervised pretraining (architecture-zoo)

For "I have few labels / many unlabelled scans" or "adapt a released medical model" — the
label-efficient route. Two sub-families: **self-supervised pretraining** (learn features
from unlabelled data, then fine-tune) and **released foundation models** (use or prompt
existing weights). All listed weights are open / permissively licensed; verify the licence
before vendoring (the lane's `distill.py` firewall rules apply if you reuse code).

Each card: **paper → core idea → when to use → medical-imaging use → reference impl →
validation/experiment setup.**

---

## Self-supervised pretraining (label-efficient features)

### SimCLR / MoCo (contrastive)
- **Papers**: Chen et al., SimCLR, *ICML* 2020; He et al., MoCo, *CVPR* 2020.
- **Core idea**: pull augmented views of the same image together, push different images apart
  (contrastive); MoCo adds a momentum encoder + queue for many negatives.
- **When to use**: a large **unlabelled** medical pool + a small labelled set; pretrain on the
  unlabelled scans, then fine-tune the backbone on labels.
- **Medical-imaging use**: contrastive CXR pretraining (e.g. CheSS-style) before multi-label
  fine-tuning.
- **Reference impl**: `lightly`, MONAI SSL tutorials; public SimCLR/MoCo repos.
- **Validation setup**: report the **label-efficiency curve** (downstream metric vs. number
  of labels) to justify the pretraining; keep the pretraining pool disjoint from the test
  patients (contamination — `/model-validation` MD1).

### DINO / DINOv2 (self-distillation) and MAE (masked autoencoding)
- **Papers**: Caron et al., DINO, *ICCV* 2021; Oquab et al., DINOv2, 2023; He et al., MAE,
  *CVPR* 2022.
- **Core idea**: DINO self-distills (student/teacher) to learn strong ViT features without
  labels; MAE masks most patches and reconstructs them. Both yield transferable ViT backbones.
- **When to use**: pretraining ViT/Swin backbones on unlabelled medical images; DINOv2-style
  features transfer well with linear probing.
- **Medical-imaging use**: **RAD-DINO** (chest-X-ray DINOv2 backbone) and similar domain
  pretrainings.
- **Reference impl**: official DINO/DINOv2/MAE repos; `timm` for the ViT backbones.
- **Validation setup**: linear-probe + fine-tune comparison; same contamination discipline.

---

## Released foundation models (use / prompt existing weights)

### SAM → MedSAM / MedSAM2 / SAM-Med2D (promptable segmentation)
- **Papers**: Kirillov et al., Segment Anything (SAM), *ICCV* 2023; Ma et al., MedSAM,
  *Nature Communications* 2024; MedSAM2 (2025) for 3-D + video.
- **Core idea**: a promptable segmentation foundation model (point/box/text prompt → mask);
  the medical variants fine-tune SAM on large medical corpora.
- **When to use**: **few-shot / interactive** segmentation, annotation acceleration, or a
  strong zero-/low-shot baseline before training a dedicated U-Net.
- **Medical-imaging use**: prompt-driven lesion/organ masks across CT/MR/US/path/endoscopy;
  speeding up labelling for a downstream U-Net.
- **Reference impl**: `segment-anything`; OpenMedLab MedSAM / MedSAM2 (Apache-2.0).
- **Validation setup**: report performance **by prompt type** and whether prompts were
  human or automated; for a fully-automatic claim, no oracle prompts at test time.

### Interactive 3-D promptable segmentation — nnInteractive / VISTA3D / SAM-Med3D (labelling acceleration)
- **Papers/tools**: nnInteractive (Isensee et al., DKFZ, 2025); VISTA3D (NVIDIA / Project-MONAI,
  2024–25); SAM-Med3D (Wang et al., 2023); MedSAM2 / SAM2 (Meta SAM2, 2024) for 3-D + video.
- **Core idea**: **native-3-D** promptable models — a click / scribble / box / lasso on a few
  slices yields the whole volumetric mask — trained on many 3-D datasets, so they generalise
  across organs and modalities without task-specific training. (2-D SAM/MedSAM applied
  slice-by-slice loses through-plane coherence; these are built for the volume.)
- **When to use**: the **labelling-throughput lever**. When expert 3-D labels are the
  bottleneck (e.g. neuro-faculty ground truth), an interactive model turns from-scratch
  hand-segmentation into a **prompt-and-correct** pass — often several-fold faster per case.
  Also a strong zero-training 3-D baseline, or an interactive tool in the reading loop.
- **Medical-imaging use**: expert-in-the-loop CT/MR volume labelling; semi-automatic organ /
  lesion / vessel masks that seed the training set a dedicated nnU-Net then learns.
- **Reference impl**: `MIC-DKFZ/nnInteractive` (fast; point/scribble/box/lasso),
  `Project-MONAI/VISTA` (VISTA3D), `uni-medical/SAM-Med3D`. **Licence — check before
  commercial use**: nnInteractive **code is Apache-2.0 but its released weights are
  CC BY-NC-SA 4.0 (non-commercial)**; confirm the VISTA3D and SAM-Med3D *weight* licences too
  (bundled model licences often differ from the code repo). This matters when the labels feed
  a product, not only a paper.
- **Validation setup**: masks produced this way are **silver labels** — an expert must
  correct/adjudicate them, and model-derived labels must not evaluate the same or a related
  model (circularity — `/model-validation` MD8). Report the **human-correction effort**
  (edits or time per case), not only the final Dice.

### TotalSegmentator / SegVol (automatic CT organ masks)
- **Papers**: Wasserthal et al., TotalSegmentator, *Radiology: AI* 2023; SegVol (2024).
- **Core idea**: released models that segment 100+ anatomical structures from CT
  automatically (TotalSegmentator) / with semantic+spatial prompts (SegVol).
- **When to use**: you need organ/structure masks on CT and have **no training budget** —
  run it, no labels required; also a strong anatomical prior for downstream tasks.
- **Reference impl**: `TotalSegmentator` (Apache-2.0).
- **Validation setup**: if you use its masks as input or weak labels, disclose that the
  reference standard is **model-derived** (silver labels) — do not let model-derived labels
  evaluate the same model (circularity — `/model-validation` MD8).

### BiomedCLIP / PubMedCLIP (cross-modal retrieval + zero-shot)
- **Papers**: Zhang et al., BiomedCLIP, 2023 (15M biomedical image–text pairs).
- **Core idea**: a CLIP-style image–text model → zero-shot classification and image–text
  retrieval without task labels.
- **When to use**: zero-shot classification, retrieval, or as a pretrained image encoder when
  labels are scarce.
- **Reference impl**: Microsoft BiomedCLIP (Hugging Face).
- **Validation setup**: zero-shot claims need a **held-out / post-cutoff** set and a
  contamination statement (the pretraining corpus may overlap public benchmarks).

### Domain-specific medical foundation backbones (RETFound / UNI / CONCH / RAD-DINO / Merlin)
- **Papers/tools**: RETFound (Zhou et al., *Nature* 2023 — retinal SSL backbone); UNI +
  CONCH (Chen / Lu et al., *Nature Medicine* 2024 — pathology, UNI a vision backbone, CONCH a
  vision–language model); RAD-DINO (Microsoft — chest-X-ray DINOv2); Merlin (Stanford — a
  3-D abdominal-CT foundation model).
- **Core idea**: organ-/modality-specific backbones pretrained on large **domain** corpora →
  **fine-tune or linear-probe** for your task with far fewer labels than training from
  scratch or transferring from ImageNet. The domain counterpart to the general SSL backbones
  above.
- **When to use**: your task sits in one of these domains (retina, pathology WSI, chest
  X-ray, abdominal CT) and labels are scarce — start from the domain FM, not ImageNet.
- **Reference impl**: `rmaphoh/RETFound_MAE`, `mahmoodlab/UNI`, `mahmoodlab/CONCH`,
  `StanfordMIMI/Merlin` (GitHub + Hugging Face).
- **Licence — the recurring gotcha (verified)**: **most medical-FM weights are non-commercial
  / gated research licences.** RETFound, UNI and CONCH ship custom CC-BY-NC-style terms and
  gate access on Hugging Face; Merlin's *code* is MIT but confirm its *weight* terms. Verify
  per model before anything beyond a paper — a product cannot ship on CC-BY-NC weights (same
  trap as nnInteractive / ConvNeXt V2).
- **Validation setup**: keep the FM's pretraining corpus disjoint from your test patients
  (many public benchmarks sit inside these corpora — contamination, `/model-validation`
  MD1/MD3); report the **label-efficiency curve** to justify the transfer.

---

## Choosing among these
Many unlabelled scans + few labels → **SSL pretrain (DINO/MAE/SimCLR) → fine-tune**, and
report the label-efficiency curve. Need masks now, no budget → **TotalSegmentator (CT) /
MedSAM2 (interactive)**. Accelerate expert 3-D labelling → **interactive FM (nnInteractive /
VISTA3D)** — mind the non-commercial weight licence. Task in a covered domain (retina /
pathology / CXR / abdominal CT) + few labels → **domain FM transfer (RETFound / UNI / CONCH /
Merlin)** — most weights are non-commercial / gated, verify before a product. Zero-shot
classification/retrieval → **BiomedCLIP**. In every case
keep the pretraining/transfer corpus disjoint from the test patients and disclose
model-derived labels. Record the choice + paper, then hand the fine-tuning to
`/model-scaffold` and validate with `/model-validation`.

# Graph neural networks — brain connectomes & population graphs (architecture-zoo)

For when the data is a **graph, not an image grid**: a brain **connectome** (nodes = ROIs /
parcels, edges = structural connectivity from DTI tractography or functional connectivity
from fMRI correlation), or a **population graph** (nodes = subjects, edges = phenotypic /
imaging similarity). The task is usually **graph-level classification** (diagnose a subject
from their connectome), **node-level** (flag abnormal ROIs, or classify subjects on a
population graph), or **link prediction** (connectivity changes).

This is a distinct family because a CNN/U-Net assumes a regular pixel grid; a connectome has
no grid — permuting the ROI order must not change the prediction, which is exactly the
symmetry a GNN respects. Each card: **paper → core idea → when to use → medical use →
reference impl → validation/experiment setup.**

---

## The general-purpose message-passing GNNs

### GCN (graph convolutional network)
- **Paper**: Kipf & Welling, "Semi-Supervised Classification with Graph Convolutional
  Networks", *ICLR* 2017.
- **Core idea**: each layer averages a node's features with its neighbours' (a first-order
  spectral-graph approximation), stacking to widen the receptive field over the graph.
- **When to use**: the transparent baseline for any connectome / population-graph task — try
  it before anything fancier.
- **Medical use**: connectome classification; **Parisot et al.** (*Medical Image Analysis*
  2018) put subjects on a **population graph** (phenotypic-similarity edges) for autism
  (ABIDE) and Alzheimer's (ADNI) prediction — a semi-supervised node-classification framing.
- **Reference impl**: **PyTorch Geometric** (`GCNConv`) or **DGL**; do not reimplement.
- **Validation setup**: split at the **subject level** (a subject's graph — or, on a
  population graph, a subject node's label — never spans train/test); with a small cohort,
  report a **permutation test** and cross-validated CIs, not a single split.

### GraphSAGE (inductive aggregation)
- **Paper**: Hamilton, Ying & Leskovec, "Inductive Representation Learning on Large Graphs",
  *NeurIPS* 2017.
- **Core idea**: learn an **aggregator** over a sampled neighbourhood so the model generalises
  to **unseen** nodes/graphs (inductive), unlike transductive GCN.
- **When to use**: a **new subject** must be classified without retraining (the realistic
  clinical setting), or the graph is too large to process whole.
- **Medical use**: inductive connectome classification where test subjects are genuinely held
  out (the honest deployment framing for a population-graph model).
- **Reference impl**: PyTorch Geometric (`SAGEConv`) / DGL.
- **Validation setup**: exploit the inductive setup to keep the test subjects fully out of
  message passing during training (transductive leakage is a real trap on population graphs).

### GAT (graph attention)
- **Paper**: Veličković et al., "Graph Attention Networks", *ICLR* 2018.
- **Core idea**: learn **attention weights** over neighbours, so the model decides which
  connections matter instead of averaging uniformly.
- **When to use**: when *which edges/connections drive the prediction* is itself a finding
  (edge importance is a built-in interpretability signal).
- **Medical use**: connectome studies that report the most-attended edges/ROIs as candidate
  biomarkers.
- **Reference impl**: PyTorch Geometric (`GATConv` / `GATv2Conv`) / DGL.
- **Validation setup**: treat attention as a **hypothesis-generating** attribution, not proof
  — sanity-check it (does it survive label permutation?) as `/explainability` requires of any
  saliency.

### GIN (graph isomorphism network)
- **Paper**: Xu et al., "How Powerful are Graph Neural Networks?", *ICLR* 2019.
- **Core idea**: an aggregation as discriminative as the Weisfeiler-Lehman test — the most
  **expressive** simple GNN for **graph-level** classification.
- **When to use**: graph-level diagnosis where subtle topology differences separate classes
  and GCN/GAT underfit.
- **Reference impl**: PyTorch Geometric (`GINConv`).
- **Validation setup**: expressiveness raises overfitting risk on small connectome cohorts —
  pair with heavy regularisation and nested CV (as `/radiomics-ml` does for the p ≫ n regime).

---

## The brain-specific model

### BrainGNN (ROI-aware, interpretable)
- **Paper**: Li et al., "BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis",
  *Medical Image Analysis* 2021.
- **Core idea**: ROI-aware convolution + a pooling layer that scores and selects the most
  informative ROIs, so the model is **interpretable at the region level** by construction.
- **When to use**: fMRI connectome classification where you must report **which ROIs** drove
  the decision (the usual neuroimaging reviewer demand).
- **Medical use**: task-fMRI / resting-state connectome diagnosis (ASD, disorders) with a
  salient-ROI readout.
- **Reference impl**: the authors' released BrainGNN repo (on PyTorch Geometric).
- **Validation setup**: report salient ROIs with **stability across folds** (not one split);
  keep site-harmonisation (**ComBat**) fit on the **training** fold only.

---

## Connectome-specific validation traps (read before publishing)
- **Subject-level split.** A subject's connectome must not appear in more than one split; on a
  population graph, hold test-subject **labels** out of training and (ideally, inductive)
  their **nodes** out of message passing. This is `/model-validation`'s split-leakage discipline
  at the subject level.
- **Site / scanner harmonisation leakage.** Multi-site connectome data (ABIDE, ADNI) is
  harmonised with **ComBat** — fit it on the **training** fold only, never the whole cohort
  (the graph analogue of `/preprocess-imaging`'s `NORMALIZATION_LEAKAGE`).
- **p ≫ n.** A connectome has thousands of edges on tens–hundreds of subjects; treat it like
  radiomics — nested CV, regularisation, a **permutation test** for tiny cohorts, and don't
  over-read a single fold (`/radiomics-ml`).
- **Interpretability ≠ proof.** Attention / ROI-saliency is hypothesis-generating; sanity-check
  it (`/explainability`).

## Boundary — this family is not scaffolded by `/model-scaffold`
`/model-scaffold` builds **image-grid** repos (CNN / U-Net / transformer); it has **no graph
task template**. For GNNs, **integrate PyTorch Geometric / DGL directly** — they own the graph
layers, loaders, and training loop; the lane does not reimplement them. The lane's
**subject-level** gates still apply: `/model-validation` (split leakage), `/radiomics-ml`
(nested-CV rigor for p ≫ n), `/explainability` (attribution sanity), `/uncertainty-imaging`
(deployment uncertainty), and `/check-reporting` (TRIPOD+AI). Record the choice + paper in the
decision note; validate the built model with `/model-validation`.

# Architecture decision tree (architecture-zoo)

Pick an architecture from the **research question + modality + data scale + label
availability**, not from what is fashionable. Read this index, then open the matching
family card (`classification.md`, `segmentation.md`, `foundation_models.md`) for the
per-paper detail and the `/model-scaffold` template to instantiate.

## Step 1 — name the task
| The research question is about… | Task | Family card |
|---|---|---|
| "is finding X present / which class" (per image / per patient) | **classification** (binary / multi-label) | `classification.md` |
| "delineate / measure structure X" (pixel/voxel mask, volume, boundary) | **segmentation** | `segmentation.md` |
| "find and localise lesions" (boxes / points, count, FROC) | **detection** | `detection.md` |
| "I have few labels / want to pretrain on unlabelled scans" | **self-supervised pretraining → fine-tune** | `foundation_models.md` |
| "adapt a released medical foundation model" | **transfer / prompt a foundation model** | `foundation_models.md` |
| "synthesise / translate a modality" (MRI→CT, denoise) | **image-to-image / generative** | `synthesis.md` |
| "classify from a brain connectome / graph" (ROI connectivity, DTI/fMRI) | **graph neural network** | `graph.md` |
| "generate a report / answer a visual question" | **multimodal LLM** | *(use `/mllm-eval`; not a CNN choice)* |

## Step 2 — let the constraints narrow it
- **Modality / dimensionality**: 2-D (CXR, fundus, path tiles, single CT/MR slices) →
  2-D backbones / 2-D U-Net. 3-D volumes (CT, MR) → **3-D U-Net / SegResNet / nnU-Net**
  (3-D context matters; do not collapse to slices if the structure is volumetric). A
  **graph** (brain connectome, ROI-connectivity matrix — no pixel grid) → a **GNN**
  (`graph.md`); integrate PyTorch Geometric / DGL directly (`/model-scaffold` has no graph
  template).
- **Labelled data scale** (events/structures, not just images):
  - **small** (hundreds) → a **pretrained** backbone fine-tuned (ImageNet / a medical
    foundation model), strong augmentation, heavy regularisation; prefer **nnU-Net**
    for segmentation (self-configuring, hard to beat with little data).
  - **medium** (thousands) → ResNet/DenseNet/EfficientNet (classification) or
    U-Net/SegResNet (segmentation), still pretrained.
  - **large** (tens of thousands+) → ViT / Swin become competitive; consider
    self-supervised pretraining on your own unlabelled pool first.
- **Class imbalance / small structures** → segmentation: Dice/Tversky/boundary-aware
  losses + a boundary metric; classification: AUPRC alongside AUROC. (Metric choice is
  `/model-validation` / `/model-evaluation`; it constrains the loss here.)
- **Interpretability / deployment need** → simpler, well-understood backbones
  (ResNet + Grad-CAM) over a marginally better but opaque model.

## Step 3 — default picks (a safe starting point, then justify deviations)
| Task + setting | Default | Why |
|---|---|---|
| 2-D multi-label CXR classification | **ResNet-50 / EfficientNet (pretrained, `timm`)** | strong, cheap, well-calibrated baselines (**ConvNeXt** for a modern CNN — `classification.md`) |
| 3-D lesion detection (boxes, FROC) | **nnDetection** | self-configuring — the nnU-Net of detection (`detection.md`) |
| 3-D organ / lesion segmentation | **nnU-Net (v2)** | self-configuring; the standard to beat |
| 3-D segmentation, tensor-core GPU, max accuracy | **nnU-Net ResEnc (M/L/XL)** | the 2024 "Revisited" frontier; still self-configuring (`segmentation.md`) |
| 2-D segmentation, custom pipeline | **U-Net / Attention U-Net (MONAI)** | transparent, controllable |
| few labels, many unlabelled scans | **SSL pretrain (DINO/MAE) → fine-tune**, or **MedSAM/TotalSegmentator transfer** | label-efficient |
| few labels, task in a covered domain (retina / path / CXR / CT) | **domain FM transfer (RETFound / UNI / CONCH / Merlin)** | domain-pretrained; most weights non-commercial — verify (`foundation_models.md`) |
| zero-/few-shot organ masks on CT | **TotalSegmentator / MedSAM2** | released weights, no training |
| accelerate expert 3-D labelling | **interactive FM (nnInteractive / VISTA3D)** | prompt-and-correct, not from-scratch; check the weight licence (`foundation_models.md`) |

## Step 4 — write the decision note
Record the choice as a short note (`decisions/architecture_choice.md`): the **task**, the
**chosen architecture**, the **source paper** it comes from, the **reason** (against the
constraints above), the **runner-up + why not**, and the matching **`/model-scaffold`
template**. Never recommend an architecture without naming its source paper, and never
quote a benchmark number you have not cited. Then hand the choice to `/model-scaffold`.

> The zoo describes **archetypes**, not a live leaderboard. SOTA churns; the task →
> family → constraint logic does not. When a newer model claims to beat these, evaluate
> it with `/model-validation` rather than adopting it on the strength of a headline. The
> canonical warning is *nnU-Net Revisited* (Isensee et al., *MICCAI* 2024): under matched
> compute, Transformer- and Mamba-based segmentors did **not** beat a scaled CNN nnU-Net,
> and U-Mamba's Mamba layers ablated to zero contribution — the "advance" was a bigger CNN.

# Segmentation architectures (architecture-zoo)

For "delineate / measure structure X" questions — a pixel/voxel mask, a volume, a boundary.
This is the family `/model-scaffold` currently generates (a configurable U-Net), so the
scaffold notes here are concrete.

Each card: **paper → core idea → when to use → medical-imaging use → reference impl →
validation/experiment setup → `/model-scaffold` note.** For 3-D volumetric structures, prefer
a 3-D model; do not collapse to independent slices if the structure is volumetric.

---

## The U-Net lineage (encoder–decoder + skip connections)

### U-Net (2-D)
- **Paper**: Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image
  Segmentation," *MICCAI* 2015.
- **Core idea**: a contracting encoder + expanding decoder with **skip connections** that
  copy high-resolution features across, so fine boundaries survive; works with few training
  images + heavy augmentation.
- **When to use**: the **default** for 2-D medical segmentation; transparent and controllable.
- **Medical-imaging use**: CXR bone suppression / nodule masks, fundus vessel/lesion masks,
  pathology gland/nucleus masks, single-slice CT/MR masks.
- **Reference impl**: MONAI `UNet`; the shipped `/model-scaffold` `model.py` is a small
  configurable 2-D U-Net.
- **Validation setup**: patient-level split; report **Dice/IoU AND a boundary metric (HD95 /
  Normalised Surface Distance)**, per structure not only a global mean (Dice is shape- and
  size-insensitive); loss = Dice+BCE or Tversky for imbalance.
- **Scaffold**: `python3 scaffold.py --task segmentation --arch unet ...` — emits exactly
  this, with the patient-disjoint seed-locked split.

### 3-D U-Net / V-Net (volumetric)
- **Papers**: Çiçek et al., 3-D U-Net, *MICCAI* 2016; Milletari et al., V-Net, *3DV* 2016
  (3-D + a Dice loss objective).
- **Core idea**: 3-D convolutions so the model sees through-plane context; V-Net popularised
  optimising Dice directly.
- **When to use**: CT/MR **volumes** where through-plane context matters (organs, tumours,
  vessels). Use patch-based training for large volumes (memory).
- **Reference impl**: MONAI `UNet(spatial_dims=3)` / `SegResNet`; TorchIO for 3-D patches +
  augmentation.
- **Validation setup**: as U-Net, but count **structures/lesions** (not just patients) for
  per-structure Dice/HD95; report at the patient level for the clinical claim.

### Attention U-Net / Residual U-Net
- **Papers**: Oktay et al., Attention U-Net, *MIDL* 2018 (attention gates on skips);
  Zhang et al., Residual U-Net 2018 (residual blocks in the U-Net).
- **Core idea**: attention gates suppress irrelevant skip features (focus on the target);
  residual blocks ease optimisation of deeper U-Nets.
- **When to use**: small / low-contrast / variable-location targets (e.g. small vessels,
  aneurysms) where plain U-Net leaks; combine both for hard 3-D targets.
- **Medical-imaging use**: 3-D vascular / aneurysm segmentation (residual + dual-attention
  U-Net), small-lesion delineation.
- **Reference impl**: MONAI building blocks; published Attention-U-Net repos.
- **Validation setup**: as 3-D U-Net; emphasise boundary metric + small-structure stability
  (Dice is unstable on tiny structures).

### nnU-Net (v2) — the self-configuring standard
- **Paper**: Isensee et al., "nnU-Net: a self-configuring method for deep learning-based
  biomedical image segmentation," *Nature Methods* 2021.
- **Core idea**: not a new architecture but a **pipeline** that auto-configures preprocessing,
  patch size, network topology, and training from the dataset fingerprint — a U-Net done
  rigorously. Hard to beat, especially with limited data.
- **When to use**: the **default to beat** for most 3-D (and 2-D) segmentation tasks; start
  here, justify any custom architecture against it.
- **Reference impl**: `nnunetv2` (MIC-DKFZ, Apache-2.0). Integrate, do not reimplement.
- **Validation setup**: nnU-Net's own cross-validation is **development-time** optimism
  correction, not external validation (`/model-validation` MD3). Preserve the patient-level
  split: build the nnU-Net `dataset.json` folds from `/model-scaffold`'s
  `splits/split_assignment.csv` so the partition is consistent end to end.
- **Scaffold**: `/model-scaffold` emits the split + a `nnUNet`-compatible note; use its
  `split_assignment.csv` to seed nnU-Net's folds (template breadth lands in a later phase).

### Transformer-based segmentation (SegResNet / Swin-UNETR / UNETR)
- **Papers**: UNETR (Hatamizadeh et al., *WACV* 2022) and Swin-UNETR (2022) — a ViT/Swin
  encoder with a U-Net-style decoder for 3-D.
- **When to use**: large 3-D datasets where a transformer encoder helps long-range context;
  otherwise nnU-Net/CNN U-Nets remain strong with less data.
- **Reference impl**: MONAI `UNETR`, `SwinUNETR`, `SegResNet`.

---

## The 2024–2026 wave — scale the CNN (and the rigour caveat)

**Read this before adopting any "beats nnU-Net" architecture.** Isensee et al., "nnU-Net
Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation" (*MICCAI* 2024,
arXiv 2404.09556) re-ran the field under matched compute and pipeline and found the **CNN
U-Net still wins**: Transformer- and Mamba-based nets did not beat a properly scaled nnU-Net,
and most headline gains came from confounded comparisons. An ablation showed **U-Mamba's
Mamba layers contribute nothing** — its gain was the residual U-Net it was bolted onto. So
"newer" here means *a bigger CNN*, not a new paradigm. Adopt a Transformer/Mamba seg model
only after `/model-validation` reproduces its claim **on your data**, never on its headline.

### nnU-Net ResEnc presets (M / L / XL) — the current strong default
- **Paper**: Isensee et al., nnU-Net Revisited, *MICCAI* 2024 (the ResEnc presets ship in
  nnU-Net v2).
- **Core idea**: nnU-Net with a **residual encoder** and preset compute budgets (M/L/XL) that
  scale the network to modern GPUs — the accuracy frontier for 3-D segmentation as of 2026,
  still self-configuring.
- **When to use**: the **default to beat** when you have a tensor-core GPU and want maximum
  accuracy; pick the preset by VRAM (M ≈ mid-range, L/XL for larger cards). Plain nnU-Net
  stays the choice on limited compute (a Pascal-class GPU without tensor cores gains little).
- **Reference impl**: `nnunetv2` (`-p nnUNetResEncUNetMPlans` / `L` / `XL`), MIC-DKFZ,
  Apache-2.0. Integrate, do not reimplement.
- **Validation setup**: identical to nnU-Net (its CV is development-time, not external);
  **disclose the preset used** — a reduced preset chosen for compute reasons is a stated
  deviation (`/model-evaluation`).

### MedNeXt — a ConvNeXt-scaled CNN
- **Paper**: Roy et al., "MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image
  Segmentation," *MICCAI* 2023.
- **Core idea**: a fully ConvNeXt-style 3-D encoder–decoder with a scalable block design;
  often tops the "revisited" benchmarks, at a higher training cost.
- **When to use**: you want a modern CNN backbone that scales and can afford the extra
  training time; a strong nnU-Net-adjacent option on BTCV/ACDC/AMOS-type tasks.
- **Reference impl**: `MIC-DKFZ/MedNeXt` (Apache-2.0, code + weights — commercial OK).
- **Validation setup**: as nnU-Net; report **training cost alongside accuracy** (its edge is
  not free).

### STU-Net — scalable + transferable (pretrained on TotalSegmentator)
- **Paper**: Huang et al., "STU-Net: Scalable and Transferable Medical Image Segmentation
  Models," 2023.
- **Core idea**: a U-Net scaled from small to **1.4 B parameters, pretrained on the
  TotalSegmentator corpus** (104 structures) — a transfer-learning starting point, not only a
  from-scratch trainer.
- **When to use**: you want a **pretrained** 3-D seg backbone to fine-tune on a small labelled
  set (transfer), or a scalable baseline; pairs with `/model-scaffold` fine-tuning mode.
- **Reference impl**: `uni-medical/STU-Net` (Apache-2.0, code + weights S/B/L/H — commercial
  OK).
- **Validation setup**: keep the fine-tuning set disjoint from the test patients; if the
  pretraining corpus (TotalSegmentator) overlaps your task's public data, state it
  (contamination — `/model-validation` MD1).

---

## Instance / detection bridge

### Mask R-CNN (instance segmentation + detection)
- **Paper**: He et al., "Mask R-CNN," *ICCV* 2017 (mask head on Faster R-CNN); pairs with
  **FPN** (Lin et al., *CVPR* 2017) for multi-scale features.
- **Core idea**: region proposals → per-instance box + class + mask; a ResNet-FPN backbone
  gives multi-scale detection.
- **When to use**: **count + localise + delineate** separate lesions (instance-level), not a
  single semantic mask.
- **Medical-imaging use**: HCC / nodule detection-and-segmentation on multi-phase CT
  (Mask R-CNN + ResNet-FPN).
- **Reference impl**: torchvision `maskrcnn_resnet50_fpn`; MONAI detection.
- **Validation setup**: detection metrics — **FROC / sensitivity per false-positive or mAP
  with the IoU match criterion stated** — not patient-level accuracy (`/model-validation`
  MD6); per-lesion analysis with patient-level clustering disclosed.

---

## Promptable / foundation segmentation
SAM / MedSAM / MedSAM2 / SegVol / TotalSegmentator and the **native-3-D interactive** models
(nnInteractive / VISTA3D / SAM-Med3D — prompt-and-correct to accelerate expert labelling)
live in `foundation_models.md`. For organ masks on CT with no training budget, start there
(TotalSegmentator / MedSAM2); when expert 3-D labels are the bottleneck, an interactive model
turns hand-segmentation into a correction pass before you train a dedicated U-Net.

## Choosing among these
2-D, custom, transparent → **U-Net / Attention U-Net (MONAI)**. 3-D, default → **nnU-Net**
(→ **nnU-Net ResEnc M/L/XL** for maximum accuracy on a tensor-core GPU). Few labels / no
training budget → **TotalSegmentator / MedSAM2** (`foundation_models.md`); pretrained backbone
to fine-tune → **STU-Net**. Accelerate expert 3-D labelling → **interactive FM (nnInteractive /
VISTA3D)** — mind the weight licence.
Count + localise instances → **Mask R-CNN + FPN**. Always patient-level split, always Dice
**and** a boundary metric per structure. Record the choice + paper, hand to `/model-scaffold`,
validate with `/model-validation`.

# Image synthesis / translation architectures (architecture-zoo)

For "synthesise or translate a modality" questions — MRI→CT, non-contrast→contrast,
low-dose→full-dose, denoising, super-resolution, or generating training images.
`/model-scaffold --task synthesis` emits a Pix2Pix repo. **Caveat**: a synthetic image can
carry hallucinated structure, so a downstream-task or reader validation is mandatory (the
`image_synthesis.md` reviewer probe, IS1–IS4, owns this).

Each card: **paper → core idea → when to use → medical-imaging use → reference impl →
validation/experiment setup.**

---

## Conditional GANs (image-to-image)

### Pix2Pix (paired) / CycleGAN (unpaired)
- **Papers**: Isola et al., Pix2Pix, *CVPR* 2017 (paired, U-Net generator + PatchGAN);
  Zhu et al., CycleGAN, *ICCV* 2017 (unpaired, cycle-consistency).
- **Core idea**: a conditional GAN maps a source image to a target domain; Pix2Pix needs
  **paired** (registered) images, CycleGAN works **unpaired** via cycle consistency.
- **When to use**: cross-modality translation when paired data exist (Pix2Pix) or do not
  (CycleGAN — but it can hallucinate, so validate carefully).
- **Medical-imaging use**: MRI→CT for attenuation correction / planning, CBCT→CT, virtual
  contrast, stain transfer in pathology; bone suppression on CXR (paired).
- **Reference impl**: the official pytorch-CycleGAN-and-pix2pix repo; `/model-scaffold
  --task synthesis` emits a small Pix2Pix (U-Net generator + PatchGAN).
- **Validation setup**: image-fidelity metrics (SSIM / PSNR) are necessary but **not
  sufficient** — add a **downstream-task** metric (does a model / clinician perform the
  clinical task as well on synthetic as on real?) and disclose hallucination risk
  (`image_synthesis.md` IS1–IS4).

### SPADE / conditional generators
- **Paper**: Park et al., SPADE, *CVPR* 2019 (spatially-adaptive normalisation from a
  semantic map).
- **When to use**: generating images conditioned on a segmentation map (e.g. lesion
  insertion / data augmentation with controlled anatomy).
- **Medical-imaging use**: nodule / lesion synthesis for augmentation (with a perceptual
  loss; Johnson et al. 2016).

## Diffusion models (current SOTA for fidelity / diversity)

### DDPM / latent diffusion (+ conditional / 3-D medical)
- **Papers**: Ho et al., DDPM, *NeurIPS* 2020; Rombach et al., latent diffusion, *CVPR* 2022;
  Zhang et al., ControlNet, *ICCV* 2023 (spatial conditioning, e.g. on a segmentation map).
- **Core idea**: learn to reverse a gradual noising process; higher fidelity and mode
  coverage than GANs, at higher compute. **Conditioning** (class, mask, ControlNet, or a
  latent) steers what is generated — the diffusion analog of SPADE for controlled anatomy.
- **When to use**: sample quality / diversity matters and compute allows — now the default
  over GANs for medical generation, augmentation, and reconstruction. For **3-D volumetric**
  synthesis a **latent** diffusion model keeps memory tractable.
- **Reference impl**: MONAI `generative` (DiffusionModelUNet, latent diffusion) and **MAISI**
  (3-D CT latent diffusion); HuggingFace `diffusers` (+ ControlNet).
- **Validation setup**: fidelity metrics (SSIM/PSNR/FID) are necessary but **not sufficient** —
  a generative claim needs a **downstream-task efficacy** result (does a model trained/tested
  on the synthetic data do the clinical task?), not similarity alone (`/model-evaluation`),
  plus hallucination disclosure; for reconstruction, compare against the acquired ground truth.

## Reconstruction / restoration

### VAE / U-Net restoration / fastMRI baselines
- **Papers**: Kingma & Welling, VAE, *ICLR* 2014; the fastMRI benchmark (Zbontar et al.
  2018) for MRI reconstruction.
- **When to use**: denoising, artefact removal, accelerated MRI reconstruction (often a
  U-Net or unrolled model rather than a GAN).
- **Validation setup**: against the fully-sampled / full-dose reference, with a downstream
  diagnostic metric.

---

## Choosing among these
Paired translation → **Pix2Pix** (`/model-scaffold --task synthesis`). Unpaired → **CycleGAN**
(validate for hallucination). Conditioned on a map / augmentation → **SPADE**. Highest fidelity,
compute available → **diffusion** (MONAI generative). Reconstruction / denoising → **U-Net /
unrolled / fastMRI baselines**. In every case, **image-fidelity metrics are not enough** — add a
downstream-task or reader validation and disclose hallucination risk. Record the choice + paper,
hand to `/model-scaffold`, validate with `/model-validation` (and the `image_synthesis` probe).