<!-- Datasheet template (model-card skill). Question groups follow Gebru et al.,
     "Datasheets for Datasets," Communications of the ACM 2021 (arXiv:1803.09010), adapted
     for a clinical imaging dataset. Fill every [NEEDS INPUT] from user-supplied facts; never
     fabricate provenance, consent, or composition details. Keep the section headings
     (check_model_card_complete.py verifies they are present and non-empty). Put all answers
     inside the [NEEDS INPUT: ...] brackets so an unfilled field reads as unfilled. -->

# Datasheet: [NEEDS INPUT: dataset name + version]

## Motivation
- **Purpose and creators**: [NEEDS INPUT: why the dataset was created, for what task, by whom, funded by whom]

## Composition
- **Instance**: [NEEDS INPUT: what one instance represents — a patient, a study, an image, a lesion]
- **Size and split**: [NEEDS INPUT: number of instances and the patient-level train/validation/test split]
- **Label / target**: [NEEDS INPUT: the label and how it was derived — the reference standard]
- **Modalities / acquisition**: [NEEDS INPUT: scanners, vendors, protocols, sites]
- **Known imbalance / missingness / errors**: [NEEDS INPUT]

## Collection Process
- **Acquisition**: [NEEDS INPUT: consecutive, sampled, or selected on an optional modality — selection on availability is a spectrum bias to disclose]
- **Time frame**: [NEEDS INPUT: collection dates]
- **Consent / IRB / governance**: [NEEDS INPUT]

## Preprocessing / Cleaning / Labeling
- **Preprocessing and order vs split**: [NEEDS INPUT: what preprocessing was applied, and whether normalisation / harmonisation was fit on the training fold only]
- **Annotation protocol**: [NEEDS INPUT: annotators, qualifications, adjudication, blinding, inter-reader agreement]
- **De-identification**: [NEEDS INPUT]

## Uses
- **Appropriate and inappropriate uses**: [NEEDS INPUT: tasks the dataset is suitable for, and what it must NOT be used for]
- **Risks**: [NEEDS INPUT: bias, re-identification]

## Distribution
- **Availability and licence**: [NEEDS INPUT: restricted / on request / public, the confirmed licence, and access conditions]

## Maintenance
- **Maintainer and versioning**: [NEEDS INPUT: who maintains it, contact, versioning, how updates / errata are communicated]

# Data-quality audit dimensions (model-card)

A short, defensible data-quality checklist to run alongside the Model Card + Datasheet,
informed by the **METRIC framework** for assessing data quality for trustworthy medical AI
(Schwabe et al., *npj Digital Medicine* 2024) and **Datasheets for Datasets** (Gebru et al.,
*Commun. ACM* 2021). It is an **appraisal reference** (it is deliberately *not* a counted
reporting guideline). For each dimension, record a finding in the Datasheet rather than a
score; the goal is disclosure, not a number.

> Verify the exact dimension list / wording against the METRIC paper before quoting it as a
> formal instrument; the dimensions below are the well-established data-quality axes the
> framework and Datasheets converge on, phrased for medical imaging.

| Dimension | What to check | Where it surfaces |
|---|---|---|
| **Completeness** | Missing images / labels / fields; missingness pattern (MCAR/MAR/MNAR) and how handled. | Datasheet Composition / Preprocessing |
| **Correctness / plausibility** | Implausible values, mislabeled cases, label noise; reference-standard quality and inter-reader agreement. | Datasheet Preprocessing/Labeling; Model Card Training Data |
| **Consistency** | Same definition / units / protocol across sites and time; harmonisation applied **after** the split. | Datasheet Collection / Preprocessing |
| **Representativeness / spectrum** | Does the cohort match the deployment population (age, sex, severity, scanner/vendor, site)? Selection on an optional modality is a spectrum bias. | Datasheet Collection; Model Card Factors |
| **Timeliness / recency** | Acquisition time frame; drift vs. current practice/scanners; temporal split if a temporal claim is made. | Datasheet Collection; Model Card Evaluation Data |
| **Provenance / traceability** | Where each instance came from; can the dataset version be reproduced (pair with `/version-dataset`)? | Datasheet Motivation / Maintenance |
| **Label provenance** | Human vs. automated / model-derived ("silver") labels; circularity if model-derived labels evaluate the same model. | Datasheet Preprocessing/Labeling |
| **Fairness / subgroup coverage** | Are protected / clinically-relevant subgroups represented enough to estimate per-subgroup performance? | Model Card Quantitative Analyses; defer depth to `/model-validation` + equity probe |
| **Leakage safety** | Patient-level disjoint split; preprocessing fit on the training fold only; no site/scanner shortcut. | Model Card Evaluation Data; verify with `/model-validation` `check_split_leakage` |

## How to use
1. Walk the table; for each dimension, write the relevant fact into the **Datasheet** (or note
   it as a Model Card caveat). Unknown → `[NEEDS INPUT]`, never a guess.
2. Anything that affects the validity of the headline metric (leakage, representativeness,
   label provenance) is also a `/model-validation` finding — cross-check there.
3. The audit is **disclosure-oriented**: the deliverable is a complete, honest Datasheet +
   Model Card, not a quality score.

<!-- Model Card template (model-card skill). Structure follows Mitchell et al.,
     "Model Cards for Model Reporting," ACM FAccT 2019 (arXiv:1810.03993), adapted for a
     clinical medical-imaging model. Fill every [NEEDS INPUT] from user-supplied facts —
     never fabricate a number, a dataset detail, or a performance result. Keep the section
     headings (check_model_card_complete.py verifies they are present and non-empty). -->

# Model Card: [NEEDS INPUT: model name + version]

## Model Details
- **Developed by**: [NEEDS INPUT: people / organisation]
- **Model date / version**: [NEEDS INPUT]
- **Model type / architecture**: [NEEDS INPUT: e.g. 3-D U-Net (nnU-Net v2)]
- **Task**: [NEEDS INPUT: segmentation / classification / detection]
- **Provenance**: [NEEDS INPUT: in-house / vendor / open-weights + version]
- **License**: [NEEDS INPUT — state the licence the user confirmed; do not assume]
- **Citation / contact**: [NEEDS INPUT]

## Intended Use
- **Primary intended use**: [NEEDS INPUT: the clinical task + decision the model supports]
- **Intended users**: [NEEDS INPUT: e.g. radiologists as a second reader]
- **Deployment horizon**: [NEEDS INPUT: screening / triage / pre-procedure / post-hoc]

## Out-of-Scope Use
- [NEEDS INPUT: uses the model must NOT be put to; populations / modalities / scanners it
  was not validated on; autonomous use if not validated for it]

## Factors
- **Relevant factors / subgroups**: [NEEDS INPUT: age, sex, scanner/vendor, site, disease
  severity — the axes performance is expected to vary along]

## Training Data
- **Source / cohort**: [NEEDS INPUT]
- **Size**: [NEEDS INPUT: patients and images/structures]
- **Label / reference standard**: [NEEDS INPUT: who labelled, annotator count, adjudication,
  inter-reader agreement; flag any model-derived "silver" labels]
- **Preprocessing**: [NEEDS INPUT — note whether fit on the training fold only]

## Evaluation Data
- **Source / cohort**: [NEEDS INPUT]
- **Validation tier**: [NEEDS INPUT: internal split / cross-validation / temporal / external
  (site/scanner/vendor) — be honest; CV is not external validation]
- **Split**: [NEEDS INPUT: patient-level, seed-locked? reference splits/split_assignment.csv]
- **Test-set size**: [NEEDS INPUT: events / structures per class, not just patients]

## Metrics
- **Metrics reported**: [NEEDS INPUT: segmentation → Dice + HD95/NSD per structure;
  classification → AUROC + AUPRC + sensitivity/specificity, with 95% CIs; detection → FROC/mAP]
- **Operating point / threshold**: [NEEDS INPUT — fixed on train/tuning folds only]
- **Run variance**: [NEEDS INPUT: mean ± SD over >= 3 seeds, or fixed seed + determinism caveat]

## Quantitative Analyses
- **Overall performance**: [NEEDS INPUT — numbers from executed evaluation only]
- **Disaggregated / subgroup performance**: [NEEDS INPUT: performance by the Factors above —
  the fairness-relevant breakdown]

## Ethical Considerations
- [NEEDS INPUT: risks of error in this clinical context, automation bias, equity concerns,
  consent / governance of the data]

## Caveats and Recommendations
- [NEEDS INPUT: known limitations, conditions for safe use, what further validation is needed
  before deployment]

#!/usr/bin/env python3
"""Model Card / Datasheet completeness gate (model-card).

A documentation-completeness linter for an AI model's Model Card (Mitchell et al.,
ACM FAccT 2019) and, optionally, its dataset Datasheet (Gebru et al., Commun. ACM
2021). It verifies that every required section is **present** and **non-empty** — i.e.
not missing and not left as an unfilled `[NEEDS INPUT]` / TODO placeholder. It is a
presence check (the documentation analogue of check_disclosure_availability /
check_summary_box); it does NOT judge whether a stated fact is true (that is
/model-validation and the human).

CHECKS (verdicts):
  1. MISSING_SECTION        (Major)  a required section heading is absent.
  2. EMPTY_REQUIRED_SECTION (Major)  a required section is present but has no real
                                     content — empty, or every content line is an
                                     unfilled placeholder ([NEEDS INPUT] / TODO /
                                     [VERIFY] / XXXX / <...>).

Required Model Card sections: Model Details, Intended Use, Out-of-Scope Use,
Training Data, Evaluation Data, Metrics, Quantitative Analyses, Ethical
Considerations, Caveats and Recommendations.
Required Datasheet sections (with --datasheet): Motivation, Composition, Collection
Process, Preprocessing / Cleaning / Labeling, Uses, Distribution, Maintenance.

INPUTS
  --card       MODEL_CARD.md (required).
  --datasheet  DATASHEET.md (optional; if given, its sections are checked too).

OUTPUT
  A table (stdout) and, with --out, a JSON artifact:
    {card, datasheet, claims[{verdict, severity, detail, where}], summary}
  Both verdicts are Major.

Stdlib-only (re / json / argparse / pathlib). Exit codes: 0 clean (or report-only),
1 Major claim(s) found (with --strict), 2 input/usage error.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

# A full unfilled-placeholder SPAN: the entire `[NEEDS INPUT ...]` / `[VERIFY ...]` bracket
# (so the descriptive hint inside it is removed too), plus standalone TODO / TBD / <...> / XXXX.
PLACEHOLDER_SPAN = re.compile(
    r"\[(?:NEEDS INPUT|VERIFY)[^\]]*\]|<[^>]*>|\bTODO\b|\bTBD\b|X{4,}", re.IGNORECASE)

# canonical key -> (display label, [alias regexes on the normalized heading])
MODEL_CARD_REQUIRED = [
    ("model_details", "Model Details", [r"model details"]),
    ("intended_use", "Intended Use", [r"intended use"]),
    ("out_of_scope", "Out-of-Scope Use", [r"out of scope"]),
    ("training_data", "Training Data", [r"training data"]),
    ("evaluation_data", "Evaluation Data", [r"evaluation data", r"eval data", r"test data"]),
    ("metrics", "Metrics", [r"^metrics$", r"^metrics "]),
    ("quant", "Quantitative Analyses", [r"quantitative analys[ie]s"]),
    ("ethics", "Ethical Considerations", [r"ethical considerations", r"^ethics"]),
    ("caveats", "Caveats and Recommendations", [r"caveats"]),
]
DATASHEET_REQUIRED = [
    ("motivation", "Motivation", [r"^motivation"]),
    ("composition", "Composition", [r"^composition"]),
    ("collection", "Collection Process", [r"collection"]),
    ("preprocessing", "Preprocessing / Cleaning / Labeling",
     [r"preprocessing", r"cleaning", r"labeling", r"labelling"]),
    ("uses", "Uses", [r"^uses$", r"^uses "]),
    ("distribution", "Distribution", [r"^distribution"]),
    ("maintenance", "Maintenance", [r"^maintenance"]),
]


def _norm_heading(text: str) -> str:
    t = text.lower()
    t = re.sub(r"[*_`#]", "", t)
    t = re.sub(r"[^a-z0-9 /]+", " ", t)
    return re.sub(r"\s+", " ", t).strip()


def parse_sections(text: str) -> list[tuple[str, str]]:
    """Return [(normalized_heading, body_text)] for each ATX heading in order."""
    lines = text.splitlines()
    sections = []
    cur_h, cur_body = None, []
    for ln in lines:
        m = re.match(r"^(#{1,6})\s+(.*)$", ln)
        if m:
            if cur_h is not None:
                sections.append((cur_h, "\n".join(cur_body)))
            cur_h = _norm_heading(m.group(2))
            cur_body = []
        elif cur_h is not None:
            cur_body.append(ln)
    if cur_h is not None:
        sections.append((cur_h, "\n".join(cur_body)))
    return sections


def _has_real_content(body: str) -> bool:
    """True if the section carries a filled value — not empty and not only unfilled
    placeholders / field labels. The body is flattened (so a placeholder wrapped across
    lines is still removed as one span), HTML comments are dropped, then unfilled-placeholder
    spans and bold field-labels ('**Source**:') are removed; what remains is the real value."""
    body = re.sub(r"<!--.*?-->", " ", body, flags=re.DOTALL)         # drop HTML comments
    text = " ".join(body.split())                                    # flatten line wraps
    if not text:
        return False
    text = PLACEHOLDER_SPAN.sub(" ", text)                           # remove [NEEDS INPUT ...] etc.
    text = re.sub(r"\*\*[^*]+\*\*:?", " ", text)                     # remove bold field labels
    text = re.sub(r"[*_`>#]+", " ", text)                            # strip residual markdown
    low = text.lower()
    if re.search(r"\b(n/?a|none|not applicable|not collected|nil|waived)\b", low):
        return True                                                 # an explicit "empty" answer is filled
    core = re.sub(r"[^a-z0-9]+", "", low)
    return len(core) >= 3                                            # a real value remains


def _check_doc(path: Path, required, label: str) -> list[dict]:
    claims = []
    sections = parse_sections(path.read_text(encoding="utf-8"))
    for key, disp, aliases in required:
        match = None
        for norm_h, body in sections:
            if any(re.search(a, norm_h) for a in aliases):
                match = (norm_h, body)
                break
        if match is None:
            claims.append({
                "verdict": "MISSING_SECTION", "severity": "Major",
                "detail": f"{label}: required section '{disp}' is absent",
                "where": path.name,
            })
        elif not _has_real_content(match[1]):
            claims.append({
                "verdict": "EMPTY_REQUIRED_SECTION", "severity": "Major",
                "detail": f"{label}: section '{disp}' is present but empty / only an unfilled "
                          f"placeholder ([NEEDS INPUT] etc.)",
                "where": path.name,
            })
    return claims


def analyze(card: str, datasheet: str | None) -> dict:
    claims = _check_doc(Path(card), MODEL_CARD_REQUIRED, "Model Card")
    if datasheet:
        claims += _check_doc(Path(datasheet), DATASHEET_REQUIRED, "Datasheet")
    n_major = sum(1 for c in claims if c["severity"] == "Major")
    return {
        "card": card, "datasheet": datasheet, "claims": claims,
        "summary": {"n_claims": len(claims), "n_major": n_major,
                    "verdict": "MAJOR_CANDIDATE" if n_major else "OK"},
    }


def render(result: dict) -> str:
    lines = ["| Check | Severity | Detail |", "|---|---|---|"]
    for c in result["claims"]:
        lines.append(f"| {c['verdict']} | {c['severity']} | {c['detail']} |")
    if len(lines) == 2:
        lines.append("| (none) | — | all required sections present and filled |")
    return "\n".join(lines)


def main() -> int:
    ap = argparse.ArgumentParser(description="Model Card / Datasheet completeness gate.")
    ap.add_argument("--card", required=True, help="MODEL_CARD.md")
    ap.add_argument("--datasheet", help="DATASHEET.md (optional)")
    ap.add_argument("--out", help="write JSON artifact to this path")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major claim exists")
    ap.add_argument("--quiet", action="store_true", help="suppress stdout table")
    args = ap.parse_args()

    for label, p in (("--card", args.card), ("--datasheet", args.datasheet)):
        if p and not Path(p).is_file():
            sys.stderr.write(f"ERROR: {label} not found: {p}\n")
            return 2

    result = analyze(args.card, args.datasheet)

    if not args.quiet:
        print("=" * 41)
        print(" Model Card / Datasheet Completeness")
        print("=" * 41)
        print(render(result))
        print()
        s = result["summary"]
        print(f"MAJOR candidate: {s['n_major']} incomplete-documentation issue(s)." if s["n_major"]
              else "OK: all required Model Card / Datasheet sections present and filled.")

    if args.out:
        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
        Path(args.out).write_text(json.dumps({"detector": "check_model_card_complete", **result}, indent=2), encoding="utf-8")
        if not args.quiet:
            print(f"\nwrote {args.out}")

    return 1 if (args.strict and result["summary"]["n_major"]) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# Regression test for the Model Card / Datasheet completeness gate (model-card).
# Synthetic, PII-free. Stdlib-only (python3).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DET="$HERE/../scripts/check_model_card_complete.py"
CH="$HERE/../scripts/check_model_card_complete_challenge"
REF="$HERE/../references"
OUT="$(mktemp -t mcc_XXXX).json"
trap 'rm -f "$OUT"' EXIT

fail=0
check() { local label="$1"; shift
    if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$label"
    else printf '  FAIL  %s\n' "$label"; fail=$((fail+1)); fi
}
has() { python3 -c "
import json
d=json.load(open('$OUT'))
assert any(c['verdict']=='$1' for c in d['claims']), '$1 not found'"; }
count() { python3 -c "
import json
d=json.load(open('$OUT'))
assert len(d['claims'])==$1, f\"{len(d['claims'])} != $1\""; }

[[ -f "$DET" ]] || { echo "ENV-ERR: detector missing" >&2; exit 2; }

# (1) complete card + datasheet -> OK, exit 0
python3 "$DET" --card "$CH/fixture/complete/MODEL_CARD.md" --datasheet "$CH/fixture/complete/DATASHEET.md" --strict --quiet >/dev/null 2>&1
check "complete card+datasheet passes (exit 0)" test "$?" -eq 0

# (2) incomplete card -> exit 1, MISSING + EMPTY
python3 "$DET" --card "$CH/fixture/incomplete/MODEL_CARD.md" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "incomplete card exits 1" test "$?" -eq 1
check "MISSING_SECTION detected" has MISSING_SECTION
check "EMPTY_REQUIRED_SECTION detected" has EMPTY_REQUIRED_SECTION

# (3) unfilled Model Card template -> all 9 required sections empty
python3 "$DET" --card "$REF/model_card_template.md" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "unfilled card template exits 1" test "$?" -eq 1
check "9 empty sections in unfilled card template" count 9

# (4) unfilled card + datasheet templates -> 9 + 7 = 16 empty
python3 "$DET" --card "$REF/model_card_template.md" --datasheet "$REF/datasheet_template.md" --out "$OUT" --quiet >/dev/null 2>&1
check "16 empty sections across both unfilled templates" count 16

# (5) "N/A" / "None" count as a filled answer (not flagged empty)
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"; rm -f "$OUT"' EXIT
cat > "$TMP/card.md" <<'MD'
# Model Card: t
## Model Details
- **Type**: U-Net
## Intended Use
- triage
## Out-of-Scope Use
- N/A
## Training Data
- 100 patients
## Evaluation Data
- internal split
## Metrics
- Dice with CI
## Quantitative Analyses
- Dice 0.8
## Ethical Considerations
- None
## Caveats and Recommendations
- single centre
MD
python3 "$DET" --card "$TMP/card.md" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "N/A and None count as filled (exit 0)" test "$?" -eq 0

echo "fail=$fail"; [[ "$fail" -eq 0 ]] && echo "ALL PASS" || echo "FAILURES: $fail"
exit "$fail"