<!-- file: composition.md — mode routing source of truth -->

# Mode composition contract

Select exactly one row. Read each named contract once, left to right; generate no unlisted section.

| Mode | Ordered contracts |
| -- | -- |
| `full` | `foundation.md(full)` → `eda.md` → `training.md` → `inference.md(attached)` → `submission.md` |
| `eda-only` | `foundation.md(eda-only)` → `eda.md` |
| `inference-only` | `foundation.md(inference-only)` → `inference.md(standalone)` → `submission.md` |

Apply `style-rules.md` to every row. Load `modality-dispatch.md` only when selected section requests modality branch.

Write `.experiments/kaggle/<competition>.py`; add the `-inference` suffix only for `inference-only`.

<!-- file: eda.md — selected by composition.md -->

# EDA section contract

Generate EDA section after foundation. Use only grounded paths and schema fields.

## Section 3: EDA

Open with a `# %% [markdown]` EDA header.

### Just-in-time configuration

Define only EDA constants, including grounded target column and sample count:

```python
# %%
SAMPLE_N = 9
TARGET_COL = "<grounded-target-column>"
```

### Dataset overview

- Load grounded training table or file index.
- Display shape, head, dtypes, missing values, and appropriate descriptive statistics.
- Confirm referenced files exist on representative sample.
- Assert non-empty data, required columns, sample availability, readable representative files immediately before using them. Never wrap overview, sample, or chart cells in `try`/`except` or conditional skips.

### Target distribution

Plot target distribution. For regression, include robust quantiles/outlier context; for segmentation/detection, summarize annotation prevalence and empty-target frequency.

### Hypothesis validation

Create markdown hypothesis cell, then executable check, for each decision-driving question. At minimum consider:

- class/target balance → loss, sampling, or stratification;
- spatial/sequence dimensions → resize, crop, padding, or batching;
- duplicates, leakage, or grouped entities → split strategy;
- missing/corrupt files → dataset guards;
- label noise or empty annotations → augmentation and evaluation behavior.

Every check ends with printed finding and explicit design implication. Never infer conclusion from plot without recording observed statistic.

### Modality display

Read `modality-dispatch.md`, select only grounded branch, define its visualization helper immediately before first use. Adapt every placeholder column/path from fact table. Show representative samples and, where applicable, width/height, volume-shape, sequence-length, or point-count distributions.

### EDA lens

Display representative records/samples; print grounded schema, target properties, missingness, duplicate/leakage checks, decisions carried into later stages. In EDA-only mode, retain these implications even though no later sections generated.

<!-- file: foundation.md — selected by composition.md -->

# Notebook foundation contract

Generate header, environment setup, imports, and path constants. Use variant selected by `composition.md`.

## Variant matrix

| Variant | Title suffix | Package setup | Scope |
| -- | -- | -- | -- |
| `full` | `⚡PTL + <ModelLibrary>` for neural training | online download by default; frozen offline packages when requested | training and inference dependencies |
| `eda-only` | `— EDA` | online download; never offline-only | exploration dependencies only |
| `inference-only` | `— Inference` | frozen offline packages | inference dependencies only |

## Section 1: Header and setup

Start with `# %% [markdown]` cell: grounded competition title, 2–3 sentences on selected scope/approach, competition URL when known. In inference-only mode, name grounded checkpoint path or Kaggle input dataset.

Follow with one setup cell:

- Online: `# ! pip download -q <library> --dest frozen_packages/`, then install from that directory with online fallback.
- Offline: `# ! cp -r ../input/python-packages/frozen_packages .`, then install with `--no-index --find-links frozen_packages/` and clearly disclosed fallback only when internet is allowed.
- Put modality-specific packages selected by `modality-dispatch.md` here, never later in notebook.
- In inference-only mode, exclude training callbacks, logger packages, and training metrics not needed to deserialize model.

## Section 2: Imports and paths

Use one `# %%` cell containing imports and global paths only:

- Standard library first (`glob`, `os`, `Path` as needed), then NumPy/pandas/plotting, then torch/model packages, then sklearn/XGBoost.
- Import `tqdm` with `from tqdm.auto import tqdm`; use for every visible notebook progress bar (data scans, training-adjacent loops, inference). Never use Rich progress bars.
- Always import `torch` for neural inference/training notebooks.
- Suppress only specific noisy warning category; never blanket-ignore exceptions.
- Define grounded `PATH_DATASET`, `PATH_OUTPUT`, and when needed `PATH_MODELS`/`PATH_CHECKPOINT` as ALL_CAPS.
- Print package and device versions immediately after imports.
- For neural training, call `pl.seed_everything(42)` and seed every non-Lightning split/sampler explicitly.

Keep stage configuration out of this cell. EDA, data, model, training, and inference constants belong immediately before their respective stage.

## Foundation lens

Print resolved paths, device, and package versions. Check required input paths without claiming Kaggle-only paths exist locally during generation.

<!-- file: inference.md — selected by composition.md -->

# Inference section contract

Use context selected by `composition.md`:

- `attached`: Section 7 of full notebook, after training.
- `standalone`: Sections 3–6 of inference-only notebook, starting from grounded checkpoint.

## Attached context

Include both paths:

1. Run inference with trained in-memory model under `torch.no_grad()`.
2. Load best saved checkpoint/model artifact in separate cell and run equivalent prediction smoke check.

Use test loader from training pipeline, move only inference outputs to CPU, retain stable sample identifiers, and verify prediction count/shape before submission.

## Standalone context

### Section 3: Load model

Ground checkpoint path, format, model class, required constructor arguments. Prefer importing `<competition>_model.py` emitted by training. If unavailable, define complete model class with verified imports rather than inventing API.

Choose loader by evidence:

- Lightning `.ckpt`: `Model.load_from_checkpoint(...)` with importable class.
- State dict `.pt`/`.pth`: construct verified architecture, load state dict, and check missing/unexpected keys.
- Serialized module: use `torch.load(..., map_location=DEVICE)` only when artifact is known to contain full trusted module.
- Custom detector/MONAI model: verify installed constructor and checkpoint contract first.

Fail clearly when no checkpoint matches; never index `sorted(...)[-1]` without empty-match guard. Set evaluation mode, move to selected device, print model type, device, parameter count.

### Section 4: Test data

Build label-free test Dataset/DataLoader or grounded modality equivalent:

- preserve sample ordering and stable IDs;
- use evaluation transforms matching training;
- set `shuffle=False`;
- assert batch shape and dtype;
- print sample and batch counts;
- assert non-empty test data before constructing or iterating loader; never use conditional empty-data branch.

Detection may use single-image iteration when required by verified predictor API. Volumetric pipelines must preserve original shape metadata for output restoration.

### Section 5: Inference loop

Run under `torch.no_grad()` and choose output activation/decoding from grounded task:

- binary classification: sigmoid only when model returns logits;
- multiclass: softmax/argmax according to submission requirements;
- regression: retain scalar/vector predictions without classification transforms;
- detection: apply verified confidence filtering and class-aware NMS when model does not already do so;
- segmentation: restore predictions to grounded original dimensions with appropriate interpolation.

Collect predictions and IDs, then assert count, shape, dtype, finiteness, and expected range before post-processing. CPU transfer is allowed here, outside training loop.

### Section 6: Post-processing

Apply only post-processing justified by metric and output contract:

- threshold calibration for classification;
- box rescaling and formatting for detection;
- morphology/component filtering for segmentation;
- inverse transforms for normalized regression targets.

Keep parameters in just-in-time config cell. Define helpers immediately before use.

## Inference lens

Assert prediction sample available, then show it; print prediction/ID counts and shapes, check NaN/Inf and range constraints, compare attached in-memory versus reloaded path when both exist. Never make required lens conditional.

<!-- file: modality-dispatch.md — consumers: eda.md, training.md, inference.md -->

## Modality-specific sample display — dispatch by `input_modality`

Use wherever samples need showing (EDA Section 3, training sanity check, inference spot check). Pick matching branch; each is self-contained cell set.

## Contents

- `image`: 2D image grids and dimension checks
- `image-3d`: volumetric loading, three-plane viewer, and statistics
- `tabular`: descriptive statistics, correlation, and target plots
- `point-cloud`: bounded 3D point display

______________________________________________________________________

### `image` — 2D images (default)

Setup: no extra installs needed.

```python
# %%
# show_images: grid of N samples with label overlay
def show_images(df, n=9, img_dir=PATH_DATASET):
    assert len(df) >= n, f"Need at least {n} images for the sample grid; found {len(df)}"
    n_cols = 3
    n_rows = (n + n_cols - 1) // n_cols
    fig, axes = plt.subplots(n_rows, n_cols, figsize=(16, 4 * n_rows))
    for ax, (_, row) in zip(axes.flat, df.sample(n).iterrows()):
        img = plt.imread(os.path.join(img_dir, row["image_id"]))  # adapt column name
        ax.imshow(img)
        ax.set_title(str(row.get("label", "")), fontsize=8)
        ax.axis("off")
    plt.tight_layout()
    plt.show()


_ = show_images(df_train)
```

Optional dimension scatter (when image sizes vary):

```python
# %%
from PIL import Image

df_train["w"], df_train["h"] = zip(
    *[Image.open(os.path.join(PATH_DATASET, r["image_id"])).size for _, r in df_train.iterrows()]
)
_ = df_train.plot.scatter("w", "h", alpha=0.3, title="Image dimensions")
```

______________________________________________________________________

### `image-3d` — volumetric / TIFF stacks / medical imaging

Setup cell (add to notebook setup `# %%`):

```python
# %%
# ! pip install -q tifffile imagecodecs ipywidgets
# ! pip list | grep -E 'tifffile|ipywidgets'
```

Imports (add to imports `# %%`):

```python
import tifffile
import ipywidgets as widgets
from ipywidgets import interact, IntSlider, FloatSlider
from matplotlib.colors import ListedColormap, BoundaryNorm
```

Load one volume:

```python
# %%
def load_volume(sample_id):
    img = tifffile.imread(os.path.join(PATH_DATASET, "train_images", f"{sample_id}.tif"))
    mask = tifffile.imread(os.path.join(PATH_DATASET, "train_labels", f"{sample_id}.tif"))
    return img, mask


sample_id = df_train.iloc[0]["id"]  # adapt column name
vol, mask_vol = load_volume(sample_id)
print(f"Volume: {vol.shape} {vol.dtype}  |  Mask: {mask_vol.shape}  labels: {np.unique(mask_vol)}")
```

Interactive 3-plane viewer + mask overlay:

```python
# %%
_MASK_COLORS = ["lightgray", "yellow", "cyan", "red"]
_MASK_CMAP = ListedColormap(_MASK_COLORS[: len(np.unique(mask_vol))])
_MASK_NORM = BoundaryNorm(np.arange(-0.5, len(np.unique(mask_vol))), _MASK_CMAP.N)


def show_volume(vol, mask_vol, z, y, x, mask_alpha):
    vZ, vY, vX = vol.shape[:3]
    fig = plt.figure(figsize=(14, 14))
    ax_xy = fig.add_subplot(2, 2, 1)
    ax_yz = fig.add_subplot(2, 2, 2)
    ax_xz = fig.add_subplot(2, 2, 3)
    ax_3d = fig.add_subplot(2, 2, 4, projection="3d")

    for ax, sl, msl, title in [
        (ax_xy, vol[z, :, :], mask_vol[z, :, :], f"Axial z={z}"),
        (ax_yz, vol[:, :, x], mask_vol[:, :, x], f"Sagittal x={x}"),
        (ax_xz, vol[:, y, :], mask_vol[:, y, :], f"Coronal y={y}"),
    ]:
        ax.imshow(sl, cmap="gray")
        ax.imshow(msl, cmap=_MASK_CMAP, norm=_MASK_NORM, alpha=mask_alpha, interpolation="nearest")
        ax.set_title(title)
        ax.axis("off")

    Yg, Xg = np.meshgrid(np.arange(vY), np.arange(vX), indexing="ij")
    Ys, Zs = np.meshgrid(np.arange(vY), np.arange(vZ), indexing="ij")
    Xc, Zc = np.meshgrid(np.arange(vX), np.arange(vZ), indexing="ij")
    ax_3d.plot_surface(Xg, Yg, np.full_like(Xg, z), color="r", alpha=0.15)
    ax_3d.plot_surface(np.full_like(Ys, x), Ys, Zs, color="b", alpha=0.10)
    ax_3d.plot_surface(Xc, np.full_like(Xc, y), Zc, color="g", alpha=0.10)
    ax_3d.set(xlabel="X", ylabel="Y", zlabel="Z")
    ax_3d.set_title("Slice planes")
    plt.tight_layout()
    plt.show()


interact(
    lambda z, y, x, mask_alpha: show_volume(vol, mask_vol, z, y, x, mask_alpha),
    z=IntSlider(min=0, max=vol.shape[0] - 1, step=1, value=vol.shape[0] // 2, description="Z-slice"),
    y=IntSlider(min=0, max=vol.shape[1] - 1, step=1, value=vol.shape[1] // 2, description="Y-slice"),
    x=IntSlider(min=0, max=vol.shape[2] - 1, step=1, value=vol.shape[2] // 2, description="X-slice"),
    mask_alpha=FloatSlider(min=0.0, max=1.0, step=0.05, value=0.3, description="Mask α"),
)
```

Volume statistics:

```python
# %%
print(f"Volume: min={vol.min()}, max={vol.max()}, mean={vol.mean():.2f}")
label_counts = dict(zip(*np.unique(mask_vol, return_counts=True)))
print(f"Mask labels: {label_counts}")
_ = pd.Series(label_counts).plot(kind="bar", title="Label voxel counts")
```

______________________________________________________________________

### `tabular` — structured / CSV only

Setup: no extra installs beyond pandas.

```python
# %%
# shape, dtypes, missing values
print(df_train.shape)
display(df_train.describe())
print(df_train.isnull().sum().sort_values(ascending=False).head(20))
```

```python
# %%
import seaborn as sns

num_cols = df_train.select_dtypes("number").columns.tolist()
assert 0 < len(num_cols) <= 20, "Feature-correlation chart requires 1–20 numeric columns"
fig, ax = plt.subplots(figsize=(len(num_cols), len(num_cols)))
sns.heatmap(df_train[num_cols].corr(), annot=True, fmt=".2f", ax=ax)
ax.set_title("Feature correlation")
plt.tight_layout()
plt.show()
```

```python
# %%
# Target distribution
_ = df_train[TARGET_COL].value_counts().plot(kind="bar", title="Target distribution")
```

______________________________________________________________________

### `point-cloud` — 3D point sets

Setup cell:

```python
# %%
# ! pip install -q open3d
# ! python -c "import open3d; print(open3d.__version__)"
```

```python
# %%
import open3d as o3d


def show_pcd(path, n_points=50_000):
    pcd = o3d.io.read_point_cloud(str(path))
    pts = np.asarray(pcd.points)[:n_points]
    fig = plt.figure(figsize=(10, 7))
    ax = fig.add_subplot(111, projection="3d")
    ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2], s=0.1, alpha=0.5)
    ax.set_title(str(path))
    plt.tight_layout()
    plt.show()


point_paths = sorted(Path(PATH_DATASET).glob("**/*.pcd"))
assert point_paths, "No point-cloud files found for the required sample display"
sample_path = point_paths[0]
show_pcd(sample_path)
```

<!-- file: style-rules.md — applied by composition.md -->

Apply ALL of these in generated script:

01. **Every section `# %% [markdown]` header: explain what, why, how it advances goal**: after `## Section Name` write 2–4 sentences: (a) what stage does, (b) why this approach chosen over alternatives, (c) how it contributes to competition objective (metric, leaderboard, submission quality). Notebook is public educational resource — write for reader seeing competition first time. Bare `## Title` headings with no explanation forbidden.
02. **Shell commands — `# ! cmd` inline, `%%bash` for multi-line blocks**: single shell command mixed into a Python cell → `# ! cmd` (valid Python syntax, visible as comment in Jupyter). Cell of 2+ consecutive shell commands (installs, `nvidia-smi` + `ls -lh` + `df -h` chains) → dedicate whole cell to `%%bash` cell magic as first line, no Python statements in that cell — simpler than repeating `# !` per line. `%matplotlib inline` verbatim — never `get_ipython().run_line_magic(...)`. Linter rejects `%`/`%%` magic → allowlist `%%bash`/`%matplotlib` in linter config; never avoid the magic
03. `# ==============================` between logical blocks within cell (not every line — only major breaks)
04. `_=` to suppress matplotlib/pandas return values: `_= df["col"].plot(...)`
05. **Every plot: axis labels + grid + legend when multiple series**: always call `plt.xlabel("...")`, `plt.ylabel("...")`, `plt.grid(True)` after any plot; chart with multiple lines/bars/hues → add `plt.legend()` or pass `legend=True`; seaborn facets use `g.set_axis_labels("x label", "y label")`
06. No `if __name__ == '__main__':` guards
07. No argparse, no dataclasses for config
08. **Blank lines — empty lines only**: in Markdown/text cells, blank lines must contain no characters. Never emit `#` alone or `# ` on blank line; Kaggle renders either as empty H1. In code cells, use real empty line rather than blank comment line.
09. **`display()` over `print()` for pandas objects**: use `display(df.head())`, `display(df.dtypes)`, `display(metrics.dropna(axis=1, how="all").head())`; `print()` for scalars and status strings only. Pattern for Markdown/text: `# Last sentence.` → empty line → `# Next paragraph.`
10. **No doctests in ipy scripts**: doctests belong in package modules, not notebook scripts — `# %% [markdown]` cell above function cell IS explanation; never duplicate as doctest
11. **Compact docstrings — never omit**: always include one-line docstring; never omit — narrative lives in `# %% [markdown]` cell immediately above function cell; full Google-style docstrings with `Args:`, `Returns:`, `Example:` blocks apply only after distillation to `src/` utils package
12. **Main path must fail fast**: never put `try`/`except`, `if`/`else`, conditional expressions, or silent fallbacks around required data loads, samples, charts, lenses, training, inference, or submission validation. Assert required preconditions directly before action; let unexpected errors halt execution. Branch only for task-specific processing selected from grounded evidence, not to make required notebook step optional.

<!-- file: submission.md — selected by composition.md -->

# Submission section contract

Generate final section from grounded sample-submission or competition output evidence.

## CSV classification/regression

- Read grounded sample submission.
- Join predictions by stable ID when ID column exists; never rely on incidental row order.
- Assign exact grounded target column(s).
- Preserve required column order and row count.
- Write `submission.csv` without index.

## Detection

- Use grounded coordinate order, scale, class mapping, score precision, and empty-detection representation.
- Format one prediction record per required sample ID.
- Validate boxes are finite, ordered, and within expected image bounds.

## Segmentation or file outputs

- Restore original spatial shape.
- Use grounded file format, dtype, naming, compression, and directory structure.
- Validate number of written files against expected sample IDs.

## Submission lens

Always verify before reporting completion:

- row/file count equals expected test count;
- columns/schema and order match grounded evidence;
- IDs are unique and cover expected set;
- predictions contain no unintended NaN/Inf;
- values, labels, boxes, and shapes satisfy grounded constraints.

End CSV workflows with `# ! head submission.csv`; use equivalent listing/schema check for non-CSV outputs. Display small sample and print final path.

<!-- file: training.md — selected by composition.md -->

# Training section contract

Generate data/features, model, and training. Derive configuration from EDA evidence.

## Section 4: Dataset/DataModule or feature engineering

Open with markdown header and just-in-time config cell for batch size, input size, validation fraction, and workers.

For neural pipelines:

- Make Dataset accept pre-split table/index plus transforms; never hide splitting in mode argument.
- Return tensors and stable identifiers needed by inference.
- Assert tensor shape and dtype at Dataset/DataLoader boundary.
- Make LightningDataModule sole owner of seeded, leakage-aware split.
- Use grouped or stratified splitting when grounded EDA requires it; never default blindly to row shuffling.
- Define train/validation/test loaders with explicit shuffle and worker behavior.

For tabular/non-neural pipelines:

- Build reproducible sklearn/XGBoost pipeline for missing values, categoricals, and feature transforms.
- Use seeded split appropriate to target and group structure.
- Keep preprocessing fitted on training data only.

Add lens cell: creates pipeline, asserts non-empty batch/sample, prints shapes and dtypes, visualizes representative batch through selected modality helper when meaningful. Never conditionally skip this required check.

## Section 5: Model

Open with markdown header and just-in-time model constants such as `MODEL_NAME`, `MAX_EPOCHS`, and `LEARNING_RATE`.

For neural training:

- Use LightningModule and current verified package APIs.
- Save hyperparameters needed for checkpoint restoration.
- Keep `forward` shallow; validate model input/output shape and dtype.
- Choose loss and TorchMetrics from grounded competition metric and EDA findings.
- Log separate train/validation metrics without manual epoch-end metric misuse.
- Use AdamW plus justified scheduler; commented alternatives are optional, not mandatory clutter.
- Write reusable model definition with `%%writefile <competition>_model.py` when companion inference notebook must import it, then import it in training notebook.

For pure tabular baselines, use verified sklearn/XGBoost API and fixed random seed; never wrap it in Lightning.

## Section 6: Training

For Lightning training:

- Use `CSVLogger`, `ModelCheckpoint`, `LearningRateMonitor`, and justified early stopping.
- Set `accelerator="auto"`, `devices="auto"`, and supported mixed-precision setting.
- Match checkpoint/early-stopping direction to grounded metric.
- Keep visible `fast_dev_run` option for debugging without claiming it was executed.
- Call `trainer.fit` after training configuration and lens checks are ready.
- Print actual best checkpoint path after training.

For non-neural training, fit pipeline and evaluate grounded metric on validation data with correct direction.

### Training lens

Read `metrics.csv` for neural runs, display metric columns, and plot train/validation curves. For non-neural runs, display validation metrics and relevant diagnostics.