# Metric guide (model-evaluation)

Load-on-demand reference for computing task-correct held-out metrics. Anchored to
**Metrics Reloaded** (Maier-Hein & Reinke et al., *Nat Methods* 2024) and its pitfalls
companion, and CLAIM 2024. The deliverable is a **per-case CSV** plus a results section;
the comparative inference is `/analyze-stats`.

## Segmentation
- **Overlap + boundary**: report Dice (or IoU) **with** a boundary metric — **HD95**
  (95th-percentile Hausdorff) or **NSD** (Normalised Surface Distance) — because Dice is
  insensitive to boundary error and unstable on small structures.
- **Per structure, not just global mean**: a global mean Dice hides per-organ/per-lesion
  failure; report per-structure with a distribution.
- **Empty references / false positives**: define behaviour for cases with no target
  (a Dice of 0/0 is undefined) — Metrics Reloaded discusses this explicitly.
- **CIs**: bootstrap over cases (resample patients, not pixels).

## Interactive / promptable segmentation
For promptable / interactive methods (SAM2, MedSAM2, nnInteractive), a single static Dice omits the
axis that makes the method interactive, and **Metrics Reloaded does not cover this regime**. Report,
in addition to the overlap + boundary metrics above:
- **Accuracy-vs-interaction trajectory**: Dice as a function of the number of corrective interactions
  (the click / interaction budget), not one operating point.
- **Interactions-to-threshold**: the number of clicks / interactions to reach a target Dice — the
  **NoC** (Number of Clicks) convention from the interactive-segmentation literature (e.g. RITM,
  SimpleClick); state the target threshold explicitly (e.g. NoC@0.80).
- **Initial vs converged / peak Dice**: the first-prompt Dice and the converged (or peak) Dice, so the
  interactive gain is visible rather than folded into one number.
- **Per-case interaction / inference time**: efficiency is a primary claim — a high-Dice method that
  needs many slow interactions may be clinically unusable.
- **Threshold-reached fraction**: the proportion of cases that reach the target Dice within the budget.
- **Subgroup robustness**: the trajectory and NoC sliced by tumor size / type / modality.

**Deterministic gate** — `check_metric_reporting.py --task interactive` still applies the
segmentation overlap + boundary checks and additionally flags an interactive claim whose report omits
the interaction axis (`INTERACTIVE_NO_INTERACTION_COUNT`, Major), the initial-vs-converged split
(`INTERACTIVE_NO_CONVERGENCE`, Minor), or per-case time (`INTERACTIVE_NO_TIME`, Minor). The **study
design** for an interactive evaluation — the human-as-operator reader arm and the two-arm
(simulated-prompting + human-operator validation) protocol-fidelity check — is covered in
`/design-study` and `/model-validation`.

## Classification
- **Discrimination with CIs**: **AUROC and AUPRC** (AUPRC tracks the minority class under
  imbalance), with bootstrap 95% CIs.
- **Operating-point metrics at the deployment prevalence**: sensitivity/specificity and
  **PPV/NPV computed at the real base rate**, not on an artificially balanced set; fix the
  threshold on the training/tuning folds, never the test set.
- **Calibration**: a reliability diagram + ECE; a discriminating model can still be
  miscalibrated.
- **Multiclass**: state the aggregation scheme (Park et al., *Radiol Med* 2024) — **one-vs-rest**
  with **macro** (unweighted) or **micro** (instance-weighted) averaging, all **pairwise** two-class
  combinations, or the prevalence-weighted **Obuchowski index**; a bare multiclass AUROC is ambiguous
  and prevalence-sensitive.

## Detection
- **FROC / mAP with the IoU match criterion stated**: report sensitivity per false-positive
  (FROC) or mAP, and **state the IoU threshold** used to match predictions to ground truth —
  the metric is undefined without it. Patient-level accuracy is not a detection metric.

## Generative / synthesis (image generation or modification)
Grounded in Park et al., *Radiol Med* 2024.
- **Full-reference** (a reference image exists): pixel/intensity similarity — **MSE / RMSE / MAE**,
  **PSNR**, **SSIM** — summarized as mean ± SD over image pairs.
- **No-reference** (no ideal replica exists): **SNR** and **CNR**, plus standardized qualitative visual
  scores (Likert) with explicit rater protocols to control inter-rater variability.
- **Downstream-task efficacy — the load-bearing check.** Image quality and clinical utility **need not
  align**: an AI-denoised CT can show higher CNR yet lower lesion sensitivity. Report the efficacy of
  the synthesized images on a **downstream task** (segmentation / detection / classification /
  quantitative measurement), not similarity alone. The gate `--task generative` flags a similarity-only
  report (`GENERATIVE_NO_DOWNSTREAM`, Major) and a synthesis report naming no quality metric
  (`GENERATIVE_NO_SIMILARITY`, Minor).

## Time-to-event (survival)
Discrimination for a time-to-event outcome uses **Harrell's C-index** and **time-dependent ROC/AUC**
(Park et al., *Radiol Med* 2024), not a static AUROC. These — with the comparative inference and
calibration-in-time — are computed and gated in **`/analyze-stats`** (survival domain); `/model-evaluation`
emits the per-case risk scores and hands them over, so the deterministic anchor for survival
discrimination lives in `/analyze-stats`, not in this reporting gate (stated per the no-prose-only rule).

## Subgroup / fairness
- Slice every headline metric by the Model Card **Factors** (scanner/vendor, site, age, sex,
  severity); enough events per subgroup to estimate it (else say so). Defer fairness depth to
  `/model-validation` + the equity probe.

## Run variance
- Report the headline as **mean ± SD over ≥ 3 seeds/runs**, or a fixed reported seed with the
  determinism caveat — a single run overstates precision.

## Hand-off
Emit `eval/per_case_metrics.csv` (one row per case, columns = the metrics) and hand to
`/analyze-stats` for DeLong/NRI/IDI/decision-curve and the publication tables; `/make-figures`
for ROC/calibration/overlay; `/model-card` for the numbers + subgroup performance.

# Metric-selection grounding and CLAIM 2024 reporting fit (model-evaluation)

The *why* behind the operational checklist in `metric_guide.md`. Where `metric_guide.md` says
**what** to compute, this doc grounds **why** that pairing is required and where the outputs land
in the manuscript. Anchored to **Metrics Reloaded** (Maier-Hein & Reinke et al., *Nature Methods*
2024) and its pitfalls companion (Reinke et al., *Nature Methods* 2024), **CLAIM 2024** (Tejani et
al., *Radiology: Artificial Intelligence* 2024), **TRIPOD+AI** (the AI extension of TRIPOD; Collins
et al., *BMJ* 2024), calibration work (Guo et al., *ICML* 2017), and the Model Card **Factors**
(Mitchell et al., *FAT\** 2019). The deliverable is still the per-case CSV; the deterministic gate
is `scripts/check_metric_reporting.py`.

> Verify exact **CLAIM 2024 item numbers** and any **NSD tolerance** against the source before
> quoting them as formal values — the mapping and tolerances below are described qualitatively by
> design. Do not hand-type a metric value: every number comes from executed code.

## The Metrics Reloaded principle: task fingerprint → metric

- **The metric is derived from the problem, not from habit.** Metrics Reloaded selects metrics from
  the problem fingerprint — task category (classification / segmentation / detection-localization),
  structure size and shape, class prevalence, and whether *where* the model is right matters. A
  metric that ignores a property that matters clinically is the wrong metric.
- **No single metric is sufficient.** Pair a counting/overlap metric with a complementary one (a
  boundary metric, a calibration summary) so a blind spot in one is covered by the other. A lone
  headline number is the recurring pitfall the companion paper warns about.
- **Report per-class / per-structure with a distribution**, not only a global mean — a mean hides
  minority-class and small-structure failure.
- **Define edge-case behaviour explicitly** (empty reference, no positive cases): several metrics
  are undefined there, and a silent convention changes the score.

## Segmentation — overlap **and** boundary, per structure

- Report an overlap metric (Dice or IoU) **with** a boundary metric (**HD95** or **NSD**). Dice is
  a volume-overlap measure: it is insensitive to boundary error and unstable on small or thin
  structures, so it can look high while the contour is clinically wrong.
- **HD95** = 95th-percentile Hausdorff distance — robust to a few outlier surface points relative to
  the raw maximum Hausdorff. **NSD** (normalised surface distance / surface Dice) = the fraction of
  the predicted surface within a **task-specific tolerance τ** of the reference surface; **τ must be
  stated** and is chosen from clinical acceptability, not invented.
- Compute **per structure** with bootstrap 95% CIs obtained by resampling **patients, not pixels**
  (Efron–Tibshirani bootstrap). State the rule for **empty-reference / false-positive-only** cases
  (a Dice of 0/0 is undefined).
- Gate: `PIXEL_ACCURACY_SEG` (pixel/voxel accuracy is dominated by background — never the headline)
  and `NO_BOUNDARY_METRIC`.

## Classification — discrimination, operating point at prevalence, then calibration

- **AUROC and AUPRC.** AUROC summarises ranking across thresholds; under class imbalance the
  precision–recall view (AUPRC) is more informative, because the ROC's false-positive rate uses the
  large negative denominator and can look optimistic when negatives dominate (Saito & Rehmsmeier,
  *PLoS ONE* 2015).
- **Operating-point metrics at the deployment prevalence.** **PPV/NPV** (and accuracy) move with the
  base rate, so values read off an artificially balanced test set mislead at deployment — report
  them at the real prevalence. Sensitivity and specificity are prevalence-independent but depend on
  the operating threshold, so **fix the threshold on the training/tuning folds, never the test set**
  (choosing it on the test set is tuning-on-test and inflates the estimate).
- Bootstrap **95% CIs at the patient level**.
- Gate: `ACCURACY_ONLY` (a bare-accuracy headline under imbalance is flagged).

## Detection / localization — FROC or mAP with the IoU criterion stated

- Report **FROC** (sensitivity versus mean false positives per image) or **mAP**, and **state the
  match criterion** — a prediction counts as a true positive only if its overlap with a
  ground-truth object meets the stated IoU (or centroid/mask) threshold. Per Metrics Reloaded's
  localization category, the metric is undefined without that criterion.
- **Patient-level accuracy is not a detection metric**, and a per-lesion result must not be reported
  as per-patient — respect the analysis unit set in Phase 1.
- Gate: `DETECTION_METRIC_MISSING`.

## Calibration — a separate axis from discrimination

- A model can **discriminate well (high AUROC) and still be miscalibrated**; report calibration
  separately (Guo et al., *ICML* 2017). Use a **reliability diagram** plus a summary (**ECE** or the
  **Brier** score).
- **ECE is binning-sensitive** — state the binning scheme (or prefer a binning-robust summary) and
  do not over-read a single ECE value.
- **TRIPOD+AI** requires reporting **both** discrimination and calibration for a clinical prediction
  model — calibration is not optional reporting.

## Subgroup slices — disaggregated reporting

- Slice every headline metric by the Model Card **Factors** (scanner/vendor, site, age, sex,
  disease severity), and report the **per-subgroup n** so the reader can see which estimates are
  thin.
- Need enough events per subgroup to estimate the metric; otherwise **say so** rather than report a
  noisy point estimate.
- This is disaggregated *reporting*. The formal fairness/equity audit lives in `/model-validation`
  plus the equity probe — cross-reference, do not duplicate it here.

## CLAIM 2024 reporting fit — where the eval outputs land

CLAIM 2024 organises items under the manuscript sections. The model-evaluation deliverable feeds the
**Methods** (metric definitions, reference standard, data partition, threshold selection) and the
**Results** (metrics with uncertainty, calibration, subgroup/failure analysis). `/check-reporting`
owns the item-by-item CLAIM 2024 / TRIPOD+AI audit; this is the routing map.

| Eval output | CLAIM 2024 area (verify item #) | Note |
|---|---|---|
| Metric definitions + how each was computed | Methods | Name the metric and its formula/library; no undefined "accuracy" headline. |
| Reference / ground-truth standard + how derived | Methods | Reader count, blinding, adjudication — state it. |
| Held-out, patient-level data partition | Methods | Cross-link `/model-validation` (split leakage). |
| Performance metrics **with uncertainty (CIs)** | Results | Bootstrap CIs at the analysis unit. |
| Calibration (reliability diagram + ECE/Brier) | Results | Required alongside discrimination (TRIPOD+AI). |
| Subgroup + failure-case analysis | Results | Per-subgroup n; flag thin slices. |
| Threshold + operating point at prevalence | Methods / Results | Threshold fixed on tuning folds, reported prevalence. |

## What the skill checks / advises

1. Run the gate — `PIXEL_ACCURACY_SEG` / `NO_BOUNDARY_METRIC` / `ACCURACY_ONLY` /
   `DETECTION_METRIC_MISSING` must all be zero.
2. Advise the author to **state τ** (NSD), **state the IoU criterion** (detection), and **state the
   deployment prevalence** (operating-point metrics); report **per-structure / per-subgroup with n**;
   give **patient-level bootstrap CIs**; and report **calibration alongside discrimination**.
3. Emit `eval/per_case_metrics.csv` for `/analyze-stats` (DeLong / NRI / IDI / decision curves /
   MRMC) — numbers are never hand-typed, and an uncertain metric or CI method is flagged `[VERIFY]`.

## Verification notes

- **Metrics Reloaded** + pitfalls companion (Maier-Hein, Reinke et al., *Nature Methods* 2024):
  named public standard — grounds the task-fingerprint principle, single-metric-insufficiency,
  per-class reporting, edge-case definition, boundary metrics, and the detection/localization
  category. Cited as a named method, not quoted.
- **CLAIM 2024** (Tejani et al., *Radiology: Artificial Intelligence* 2024): named reporting
  checklist. Exact item numbers are **not** quoted — the table maps outputs to manuscript sections;
  resolve item numbers against the source (`[VERIFY]`). `/check-reporting` owns the audit.
- **TRIPOD+AI** (Collins et al., *BMJ* 2024): named standard; grounds the calibration-and-
  discrimination requirement. Written as base TRIPOD + AI extension.
- **AUPRC under imbalance** (Saito & Rehmsmeier, *PLoS ONE* 2015, **CC-BY**): principle only (ROC vs
  PR on imbalanced data); no text copied.
- **Calibration / ECE** (Guo et al., *ICML* 2017): named methods paper; reliability diagram + ECE,
  with the binning-sensitivity caveat stated qualitatively.
- **NSD / surface Dice with tolerance**: the tolerance-based surface metric (e.g., Nikolov et al.,
  head-and-neck OAR segmentation) recommended for boundary error by Metrics Reloaded; **τ described
  qualitatively, no value invented**.
- **Bootstrap CIs** (Efron & Tibshirani): canonical resampling method; patient-level resampling
  principle.
- **Model Card Factors** (Mitchell et al., *FAT\** 2019): named documentation standard for
  disaggregated reporting axes.
- **Generative / synthesis, multiclass, and time-to-event metrics** (Park, Han & Lee, *Radiol Med*
  2024, "Conceptual review of outcome metrics and measures used in clinical evaluation of AI in
  radiology"): named review. Grounds the full-reference (MSE / RMSE / PSNR / SSIM) and no-reference
  (SNR / CNR) image-quality metrics, the **image-quality-vs-downstream-efficacy divergence** (a
  denoised CT with higher CNR but lower lesion sensitivity), multiclass aggregation (one-vs-rest /
  macro / micro / pairwise / Obuchowski index), and time-to-event discrimination (Harrell's C,
  time-dependent ROC). Concepts described qualitatively; no numeric values copied.
- No DOIs, dataset names, numeric thresholds, prevalences, NSD tolerances, or CLAIM item numbers are
  fabricated; any uncertain specific is flagged `[VERIFY]`.

#!/usr/bin/env python3
"""Task-correct metric-reporting gate for a medical-imaging model (model-evaluation).

A conservative presence linter for a metrics report / results section: it flags when
the reported metric set does not match the task and prevalence, per Metrics Reloaded
(Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024. It checks which metrics
are named (and whether confidence intervals are mentioned); it does not recompute a
number. Fires only when a required metric is clearly absent or a forbidden one is the
headline.

CHECKS (verdicts; which apply depends on --task):
  segmentation:
    NO_BOUNDARY_METRIC   (Major)  Dice/IoU named but no boundary metric (HD95 / HD /
                                  ASSD / NSD / surface distance).
    PIXEL_ACCURACY_SEG   (Major)  pixel/voxel accuracy reported for segmentation
                                  (misleading on imbalanced masks).
  classification:
    ACCURACY_ONLY        (Major)  accuracy named but no AUROC (threshold-independent
                                  discrimination).
    AUPRC_MISSING        (Minor)  AUROC named but no AUPRC (the minority-class metric
                                  under imbalance).
    MULTICLASS_NO_AVERAGING (Minor)  a multiclass claim with AUROC/accuracy but no stated
                                  aggregation scheme (one-vs-rest / macro / micro / pairwise /
                                  Obuchowski), per Park et al. (Radiol Med 2024).
  detection:
    DETECTION_METRIC_MISSING (Major)  no FROC / mAP / sensitivity-per-false-positive,
                                      or no IoU match criterion stated.
  interactive (promptable segmentation — SAM2 / MedSAM2 / nnInteractive; also runs the
  segmentation checks above, since interactive segmentation is still segmentation):
    INTERACTIVE_NO_INTERACTION_COUNT (Major)  no interaction axis (number of clicks /
                                      interactions-to-threshold / Dice-vs-interactions).
    INTERACTIVE_NO_CONVERGENCE (Minor)  no initial-prompt vs converged/peak Dice split.
    INTERACTIVE_NO_TIME        (Minor)  no per-case interaction / inference time.
  generative (image synthesis / generation — Park et al., Radiol Med 2024):
    GENERATIVE_NO_DOWNSTREAM (Major)  image-quality similarity (SSIM / PSNR / SNR / CNR)
                                  reported without a downstream-task evaluation — similarity
                                  is not clinical utility (quality and task efficacy can diverge).
    GENERATIVE_NO_SIMILARITY (Minor)  a synthesis claim with no image-quality metric named.
  all tasks:
    CI_MISSING           (Minor)  no confidence interval / uncertainty mentioned for
                                  the headline metric.

INPUTS
  --report  metrics report / results markdown (required).
  --task    segmentation | classification | detection | interactive | generative (required).

OUTPUT
  A table (stdout) and, with --out, a JSON artifact:
    {report, task, claims[{verdict, severity, detail, where}], summary}

Stdlib-only. Exit codes: 0 clean (or report-only), 1 Major claim(s) (with --strict),
2 input/usage error.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

P = {
    "dice_iou": r"\b(dice|dsc|jaccard|iou|intersection over union)\b",
    "boundary": r"\b(hd95|hd 95|hausdorff|assd|\basd\b|\bmasd\b|\bmsd\b|nsd|"
                r"normali[sz]ed surface dice|normali[sz]ed surface|surface dice similarity|"
                r"surface dsc|surface dice|surface distance|mean (?:surface|boundary) distance|"
                r"boundary)\b",
    "pixel_acc": r"\b(pixel[- ]?accuracy|voxel[- ]?accuracy|pixel-?wise accuracy)\b",
    "accuracy": r"\b(accuracy)\b",
    # 'diagnostic accuracy' / 'accuracy study' are study-type phrases, not the accuracy metric
    "accuracy_phrase": r"\b(diagnostic(?: test)? accuracy|accuracy study|accuracy studies)\b",
    "auroc": r"\b(auroc|auc[- ]?roc|\bauc\b|c[- ]?statistic|area under the (?:roc|receiver))\b",
    "auprc": r"\b(auprc|au[- ]?prc|average precision|precision[- ]?recall|pr[- ]?auc)\b",
    "sens": r"\b(sensitivit\w+|recall|true[- ]?positive rate|tpr)\b",
    "spec": r"\b(specificit\w+|true[- ]?negative rate|tnr)\b",
    "detection": r"\b(froc|map\b|mean average precision|sensitivity per (?:false positive|fp)|"
                 r"competition performance metric|cpm)\b",
    "iou_crit": r"\b(?:iou|intersection over union|overlap)\b[^.]{0,40}"
                r"(?:threshold|criterion|>=|≥|>|\bof\b|above|exceed|\d\.\d)"
                r"|match(?:ing)? criterion"
                r"|(?:cent(?:er|re|roid)|distance)[- ]?based"
                r"|hit (?:rule|criterion)|within (?:the )?lesion",
    "ci": r"confidence interval|credible interval|\b95\s*%?\s*ci|\bcis?\b|±|\+/-|\bsd\b|"
          r"standard deviation|bootstrap|interquartile|\biqr\b",
    # interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive). The
    # interaction axis: number of clicks (NoC), interactions/clicks-to-threshold, or a
    # Dice-vs-interactions trajectory — the metric a static Dice cannot express.
    "interactions": r"\b(number of clicks|#\s?clicks|clicks?[- ]?to[- ]?(?:threshold|target)|"
                    r"\bnoc\b|noc@\d+|interaction[- ]?(?:count|budget)|"
                    r"interactions?[- ]?to[- ]?(?:threshold|target)|"
                    r"number of (?:interactions|prompts|corrections|edits)|"
                    r"corrective (?:click|interaction)|"
                    r"dice[- ]?(?:vs|versus|per|over|against)[- ]?(?:click|interaction|prompt)|"
                    r"clicks required)\b",
    # separation of the initial prompt from the converged / peak operating point
    "convergence": r"\b(initial[- ]?(?:dice|prompt|click|mask|prediction)|"
                   r"first[- ]?(?:click|prompt) dice|converged? dice|convergence|peak dice|"
                   r"plateau|saturat\w+|dice after \d+|"
                   r"after (?:the )?(?:first|final|last|\d+) (?:click|prompt|interaction))\b",
    # per-case interaction / inference efficiency. Matches named time terms AND a bare
    # number-plus-time-unit ("179±114 seconds", "120-200 ms"), since real interactive papers
    # report timing that way rather than always writing "inference time" (dogfood: nnInteractive).
    "interaction_time": r"\b(interaction time|time per (?:case|click|interaction|prompt)|"
                        r"per[- ]?case (?:time|latency)|(?:inference|annotation|completion|"
                        r"segmentation|processing|response) (?:time|latency|speed)|"
                        r"seconds per (?:case|click|interaction)|runtime|wall[- ]?clock|"
                        r"time[- ]?to[- ]?(?:threshold|target)|"
                        r"\d+(?:\.\d+)?\s*(?:±|\+/-|–|-|to)?\s*\d*(?:\.\d+)?\s*"
                        r"(?:ms|msec|milliseconds?|seconds?|minutes?|hours?)\b)\b",
    # generative / synthesis image evaluation (Park et al., Radiol Med 2024): full-reference
    # pixel/intensity similarity, no-reference quality, and the downstream-task efficacy that
    # image similarity alone does not establish.
    "sim_full_ref": r"\b(mse|rmse|nrmse|\bmae\b|psnr|peak signal[- ]?to[- ]?noise|ssim|"
                    r"structural similarity|pixel[- ]?wise similarity|full[- ]?reference)\b",
    "snr_cnr": r"\b(snr|cnr|signal[- ]?to[- ]?noise|contrast[- ]?to[- ]?noise)\b",
    "downstream": r"\b(downstream[- ]?(?:task|clinical|evaluation|performance)|task[- ]?based|"
                  r"efficacy (?:in|on|for) (?:performing |executing |conducting )?(?:the )?"
                  r"(?:downstream |clinical )?task|"
                  r"(?:segmentation|detection|classification|diagnostic|lesion) "
                  r"(?:performance|sensitivity|accuracy|dice|auroc) "
                  r"(?:on|of|using|with|from) (?:the )?"
                  r"(?:synthes\w+|generat\w+|synthetic|reconstruct\w+|denoised)|"
                  r"trained on (?:the )?(?:synthes\w+|generat\w+|synthetic)|"
                  r"indirect(?:ly)? (?:evaluat|assess))\b",
    # multiclass classification aggregation scheme (Park et al., Radiol Med 2024)
    "multiclass": r"\b(multi[- ]?class|multi[- ]?categor\w+|three[- ]?class|"
                  r"(?:\d+|four|five|six|seven|eight|nine|ten)[- ]?class(?:es|\b))\b",
    "averaging": r"\b(one[- ]?vs[- ]?rest|one[- ]?versus[- ]?rest|\bovr\b|one[- ]?vs[- ]?one|\bovo\b|"
                 r"pairwise|macro[- ]?averag\w+|micro[- ]?averag\w+|weighted average|obuchowski|"
                 r"per[- ]?class (?:auroc|auc))\b",
}

# Negation BEFORE the token ('we do NOT report pixel accuracy ...').
NEG_BEFORE = re.compile(r"\b(not|never|without|neither|avoid\w*|do(?:es)?n't|did not|do not|"
                        r"instead of|rather than)\b", re.IGNORECASE)
# Negation AFTER the token tied to a result verb ('AUROC was not computed/reported').
NEG_AFTER = re.compile(
    r"\b(?:was|were|is|are|not)\s+not\s+\w+"
    r"|\bnot\s+(?:computed|reported|available|performed|calculated|presented|provided|assessed|"
    r"evaluated|measured)\b", re.IGNORECASE)


def has(text: str, key: str) -> bool:
    return re.search(P[key], text, re.IGNORECASE) is not None


def has_affirmative(text: str, key: str) -> bool:
    """has(), but a match disavowed by a nearby negation does not count — so 'we do not
    report pixel accuracy' or 'AUROC was not computed' is not treated as reporting it."""
    pat = re.compile(P[key], re.IGNORECASE)
    for m in pat.finditer(text):
        before = text[max(0, m.start() - 28): m.start()]
        after = text[m.end(): m.end() + 24]
        if NEG_BEFORE.search(before) or NEG_AFTER.search(after):
            continue
        return True
    return False


def analyze(report: str, task: str) -> dict:
    text = Path(report).read_text(encoding="utf-8")
    claims = []

    def add(v, s, d):
        claims.append({"verdict": v, "severity": s, "detail": d, "where": Path(report).name})

    if task in ("segmentation", "interactive"):
        # interactive/promptable segmentation is still segmentation: the overlap-plus-boundary
        # requirement applies to its per-structure quality regardless of the interaction axis.
        if has_affirmative(text, "pixel_acc"):
            add("PIXEL_ACCURACY_SEG", "Major",
                "pixel/voxel accuracy is reported for segmentation — misleading on imbalanced masks; "
                "report Dice/IoU with a boundary metric instead")
        if has(text, "dice_iou") and not has(text, "boundary"):
            add("NO_BOUNDARY_METRIC", "Major",
                "Dice/IoU is reported without a boundary metric (HD95 / NSD / surface distance) — "
                "overlap alone is shape- and size-insensitive; pair it with a boundary metric, "
                "per structure")
    if task == "interactive":
        # A promptable / interactive method's contribution is the accuracy-vs-interaction
        # trajectory and its efficiency, not a single operating point; a static Dice omits
        # exactly what makes it interactive (Metrics Reloaded does not cover this regime).
        if not has(text, "interactions"):
            add("INTERACTIVE_NO_INTERACTION_COUNT", "Major",
                "an interactive/promptable segmentation report does not quantify the interaction "
                "axis (number of clicks / interactions-to-threshold / Dice-vs-interactions) — a "
                "single Dice evaluates a promptable method as if it were one-shot")
        if not has(text, "convergence"):
            add("INTERACTIVE_NO_CONVERGENCE", "Minor",
                "no separation of the initial prompt from the converged/peak Dice — the interactive "
                "contribution is the improvement across interactions, not one operating point")
        if not has(text, "interaction_time"):
            add("INTERACTIVE_NO_TIME", "Minor",
                "no per-case interaction/inference time — efficiency is a primary interactive claim "
                "(a high-Dice method needing many slow interactions may not be clinically usable)")
    elif task == "classification":
        # the accuracy METRIC, excluding the study-type phrase 'diagnostic accuracy'; a
        # sensitivity+specificity pair is a threshold-pair report, not accuracy-only
        acc_stripped = re.sub(P["accuracy_phrase"], " ", text, flags=re.IGNORECASE)
        accuracy_metric = re.search(P["accuracy"], acc_stripped, re.IGNORECASE) is not None
        threshold_pair = has(text, "sens") and has(text, "spec")
        auroc_reported = has_affirmative(text, "auroc")
        if accuracy_metric and not auroc_reported and not threshold_pair:
            add("ACCURACY_ONLY", "Major",
                "accuracy is reported without AUROC — accuracy is prevalence-dependent and misleading "
                "under imbalance; report threshold-independent discrimination (AUROC)")
        if auroc_reported and not has(text, "auprc"):
            add("AUPRC_MISSING", "Minor",
                "AUROC is reported without AUPRC — AUPRC tracks the minority class and is informative "
                "under imbalance")
        if has(text, "multiclass") and (accuracy_metric or auroc_reported) and not has(text, "averaging"):
            add("MULTICLASS_NO_AVERAGING", "Minor",
                "a multiclass classification reports AUROC/accuracy without stating the aggregation "
                "scheme (one-vs-rest, macro/micro averaging, pairwise, or the prevalence-weighted "
                "Obuchowski index) — the aggregate is ambiguous and prevalence-sensitive without it")
    elif task == "detection":
        if not has(text, "detection"):
            add("DETECTION_METRIC_MISSING", "Major",
                "no detection metric (FROC / mAP / sensitivity-per-false-positive) is reported — "
                "patient-level accuracy is not a detection metric")
        elif not has(text, "iou_crit"):
            add("DETECTION_METRIC_MISSING", "Major",
                "a detection metric is reported but the IoU match criterion is not stated — mAP/FROC "
                "are undefined without the match threshold")
    elif task == "generative":
        # image synthesis / generation (Park et al., Radiol Med 2024): full-reference similarity
        # (MSE/RMSE/PSNR/SSIM) or, without a reference, no-reference quality (SNR/CNR) — but image
        # quality and downstream-task efficacy need not align, so a clinical-utility claim needs a
        # downstream-task evaluation, not similarity alone.
        has_sim = has(text, "sim_full_ref") or has(text, "snr_cnr")
        if has_sim and not has(text, "downstream"):
            add("GENERATIVE_NO_DOWNSTREAM", "Major",
                "image-quality similarity (SSIM / PSNR / SNR / CNR) is reported for a generative / "
                "synthesis model without a downstream-task evaluation — pixel/intensity similarity does "
                "not establish clinical utility (quality and task efficacy need not align, e.g. an "
                "AI-denoised CT with higher CNR but lower lesion sensitivity); evaluate a downstream "
                "task (segmentation / detection / classification) on the synthesized images")
        if not has_sim:
            add("GENERATIVE_NO_SIMILARITY", "Minor",
                "a generative / synthesis evaluation names no image-quality metric — report "
                "full-reference similarity (MSE / RMSE / PSNR / SSIM) or, when no reference exists, "
                "no-reference quality (SNR / CNR, standardized visual scores)")

    if not has(text, "ci"):
        add("CI_MISSING", "Minor",
            "no confidence interval / uncertainty is reported for the headline metric")

    n_major = sum(1 for c in claims if c["severity"] == "Major")
    return {"report": report, "task": task, "claims": claims,
            "summary": {"n_claims": len(claims), "n_major": n_major,
                        "verdict": "MAJOR_CANDIDATE" if n_major else "OK"}}


def render(result: dict) -> str:
    lines = ["| Check | Severity | Detail |", "|---|---|---|"]
    for c in result["claims"]:
        lines.append(f"| {c['verdict']} | {c['severity']} | {c['detail']} |")
    if len(lines) == 2:
        lines.append("| (none) | — | task-correct metrics with uncertainty reported |")
    return "\n".join(lines)


def main() -> int:
    ap = argparse.ArgumentParser(description="Task-correct metric-reporting gate (model-evaluation).")
    ap.add_argument("--report", required=True, help="metrics report / results markdown")
    ap.add_argument("--task", required=True,
                    choices=["segmentation", "classification", "detection", "interactive", "generative"])
    ap.add_argument("--out", help="write JSON artifact to this path")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major claim exists")
    ap.add_argument("--quiet", action="store_true", help="suppress stdout table")
    args = ap.parse_args()

    if not Path(args.report).is_file():
        sys.stderr.write(f"ERROR: --report not found: {args.report}\n")
        return 2
    result = analyze(args.report, args.task)

    if not args.quiet:
        print("=" * 41)
        print(" Metric Reporting (model-evaluation)")
        print("=" * 41)
        print(render(result))
        print()
        s = result["summary"]
        print(f"MAJOR candidate: {s['n_major']} metric-reporting issue(s)." if s["n_major"]
              else "OK: task-correct metrics with uncertainty reported.")

    if args.out:
        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
        Path(args.out).write_text(json.dumps({"detector": "check_metric_reporting", **result}, indent=2), encoding="utf-8")
        if not args.quiet:
            print(f"\nwrote {args.out}")

    return 1 if (args.strict and result["summary"]["n_major"]) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# Regression test for the metric-reporting gate (model-evaluation). Synthetic, PII-free.
set -u
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DET="$HERE/../scripts/check_metric_reporting.py"
F="$HERE/../scripts/metric_reporting_challenge/fixture"
OUT="$(mktemp -t mr_XXXX).json"; trap 'rm -f "$OUT"' EXIT
fail=0
check(){ local l="$1"; shift; if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$l"; else printf '  FAIL  %s\n' "$l"; fail=$((fail+1)); fi; }
has(){ python3 -c "import json;d=json.load(open('$OUT'));assert any(c['verdict']=='$1' for c in d['claims']),'$1'"; }
no(){ python3 -c "import json;d=json.load(open('$OUT'));assert not any(c['verdict']=='$1' for c in d['claims']),'$1'"; }
[[ -f "$DET" ]] || { echo "ENV-ERR" >&2; exit 2; }

python3 "$DET" --report "$F/seg_bad.md" --task segmentation --out "$OUT" --strict --quiet >/dev/null 2>&1
check "seg_bad exits 1" test "$?" -eq 1
check "NO_BOUNDARY_METRIC" has NO_BOUNDARY_METRIC
check "PIXEL_ACCURACY_SEG" has PIXEL_ACCURACY_SEG
python3 "$DET" --report "$F/seg_good.md" --task segmentation --out "$OUT" --strict --quiet >/dev/null 2>&1
check "seg_good exits 0" test "$?" -eq 0
check "seg_good no NO_BOUNDARY_METRIC" no NO_BOUNDARY_METRIC
python3 "$DET" --report "$F/clf_bad.md" --task classification --out "$OUT" --strict --quiet >/dev/null 2>&1
check "clf_bad exits 1 (ACCURACY_ONLY)" test "$?" -eq 1
check "ACCURACY_ONLY" has ACCURACY_ONLY
python3 "$DET" --report "$F/clf_good.md" --task classification --out "$OUT" --strict --quiet >/dev/null 2>&1
check "clf_good exits 0" test "$?" -eq 0
check "clf_good no CI_MISSING (CIs present)" no CI_MISSING

# --- false-positive robustness (folded from adversarial review): good reports must pass ---
W="$(mktemp -d)"; trap 'rm -rf "$W"; rm -f "$OUT"' EXIT
ok0() { python3 "$DET" --report "$1" --task "$2" --strict --quiet >/dev/null 2>&1; }
printf 'Detection: FROC; a hit required IoU with the lesion >= 0.3, with 95%% CI.\n' > "$W/det1.md"
check "FP: detection 'IoU ... >= 0.3' passes" ok0 "$W/det1.md" detection
printf 'mAP with overlap >= 0.5 as the match criterion, 95%% CI.\n' > "$W/det2.md"
check "FP: detection 'overlap >= 0.5' passes" ok0 "$W/det2.md" detection
printf 'Dice 0.81 and Surface DSC 0.92, with 95%% CIs.\n' > "$W/sdsc.md"
check "FP: segmentation 'Surface DSC' passes" ok0 "$W/sdsc.md" segmentation
printf 'We do NOT report pixel accuracy; instead Dice, HD95, ASSD with 95%% CIs.\n' > "$W/neg.md"
check "FP: 'do NOT report pixel accuracy' passes" ok0 "$W/neg.md" segmentation
printf 'This diagnostic accuracy study reports sensitivity 0.83 and specificity 0.90 (95%% CI).\n' > "$W/dx.md"
check "FP: 'diagnostic accuracy' study phrase + sens/spec passes" ok0 "$W/dx.md" classification
# negation: 'AUROC was not computed' -> the real ACCURACY_ONLY, no spurious AUPRC_MISSING
printf 'Classification accuracy 0.92; AUROC was not computed.\n' > "$W/aur.md"
python3 "$DET" --report "$W/aur.md" --task classification --out "$OUT" --quiet >/dev/null 2>&1
check "negation: 'AUROC was not computed' -> ACCURACY_ONLY" has ACCURACY_ONLY
check "negation: no spurious AUPRC_MISSING" no AUPRC_MISSING

# --- interactive / promptable segmentation ---
python3 "$DET" --report "$F/interactive_bad.md" --task interactive --out "$OUT" --strict --quiet >/dev/null 2>&1
check "interactive_bad exits 1" test "$?" -eq 1
check "INTERACTIVE_NO_INTERACTION_COUNT" has INTERACTIVE_NO_INTERACTION_COUNT
check "interactive_bad also flags NO_BOUNDARY_METRIC (still segmentation)" has NO_BOUNDARY_METRIC
check "interactive_bad flags INTERACTIVE_NO_TIME" has INTERACTIVE_NO_TIME
python3 "$DET" --report "$F/interactive_good.md" --task interactive --out "$OUT" --strict --quiet >/dev/null 2>&1
check "interactive_good exits 0" test "$?" -eq 0
check "interactive_good no INTERACTIVE_NO_INTERACTION_COUNT" no INTERACTIVE_NO_INTERACTION_COUNT
check "interactive_good no INTERACTIVE_NO_CONVERGENCE" no INTERACTIVE_NO_CONVERGENCE
check "interactive_good no NO_BOUNDARY_METRIC" no NO_BOUNDARY_METRIC
# FP guard: a static (non-interactive) segmentation report is NOT run under --task interactive,
# and a full interactive report must not spuriously fire the interaction verdicts.
printf 'Interactive tumor segmentation: Dice rose from an initial-click Dice 0.55 to a peak Dice 0.90 (95%% CI 0.88-0.92); HD95 5.1 mm; median 3 clicks to threshold; interaction time 30 s/case.\n' > "$W/int_ok.md"
check "FP: full interactive one-liner passes --strict" ok0 "$W/int_ok.md" interactive
# dogfood regression (nnInteractive, arXiv 2503.08373): timing written as "N seconds" / "N ms"
# — not the literal phrase "inference time" — must still satisfy the interaction-time check.
printf 'Interactive segmentation: number of clicks to threshold, converged Dice, HD95 6 mm; per case 179±114 seconds and 120-200 ms per structure.\n' > "$W/int_time.md"
python3 "$DET" --report "$W/int_time.md" --task interactive --out "$OUT" --quiet >/dev/null 2>&1
check "dogfood: 'N seconds' / 'N ms' timing -> no INTERACTIVE_NO_TIME" no INTERACTIVE_NO_TIME

# --- generative / synthesis ---
python3 "$DET" --report "$F/generative_bad.md" --task generative --out "$OUT" --strict --quiet >/dev/null 2>&1
check "generative_bad exits 1 (no downstream)" test "$?" -eq 1
check "GENERATIVE_NO_DOWNSTREAM" has GENERATIVE_NO_DOWNSTREAM
python3 "$DET" --report "$F/generative_good.md" --task generative --out "$OUT" --strict --quiet >/dev/null 2>&1
check "generative_good exits 0 (downstream task present)" test "$?" -eq 0
check "generative_good no GENERATIVE_NO_DOWNSTREAM" no GENERATIVE_NO_DOWNSTREAM
# FP guard: a synthesis report that names no similarity metric is flagged NO_SIMILARITY, not DOWNSTREAM
printf 'We generated synthetic MRI and evaluated a downstream tumor-segmentation Dice of 0.88 on the synthesized images (95%% CI 0.85-0.90).\n' > "$W/gen_down_only.md"
python3 "$DET" --report "$W/gen_down_only.md" --task generative --out "$OUT" --quiet >/dev/null 2>&1
check "generative downstream-only -> GENERATIVE_NO_SIMILARITY not DOWNSTREAM" has GENERATIVE_NO_SIMILARITY
check "generative downstream-only -> no GENERATIVE_NO_DOWNSTREAM" no GENERATIVE_NO_DOWNSTREAM

# --- multiclass classification ---
python3 "$DET" --report "$F/multiclass_bad.md" --task classification --out "$OUT" --quiet >/dev/null 2>&1
check "MULTICLASS_NO_AVERAGING" has MULTICLASS_NO_AVERAGING
# FP guard: a multiclass report that states its aggregation scheme must NOT fire the verdict
printf 'Three-class classification: one-vs-rest macro-averaged AUROC 0.90 and AUPRC 0.71 (95%% CI).\n' > "$W/mc_ok.md"
python3 "$DET" --report "$W/mc_ok.md" --task classification --out "$OUT" --quiet >/dev/null 2>&1
check "FP: multiclass with one-vs-rest macro-average -> no MULTICLASS_NO_AVERAGING" no MULTICLASS_NO_AVERAGING

echo "fail=$fail"; [[ "$fail" -eq 0 ]] && echo "ALL PASS" || echo "FAILURES: $fail"; exit "$fail"