# Evaluation axes (mllm-eval)

Load-on-demand reference behind the ME2–ME7 axes: the clinical-efficacy metrics,
faithfulness, contamination, prompt-sensitivity, answer-matching, and reader-study
machinery an LLM/MLLM clinical evaluation must cover. Anchored to the radiology-NLP
metric literature — **RadGraph** (Jain et al., NeurIPS Datasets & Benchmarks 2021),
**CheXbert** (Smit et al., 2020), the rule-based **CheXpert** labeler (Irvin et al., 2019),
and **RadCliQ** (Yu et al., *Patterns* 2023) — and to the reporting standards CLAIM 2024,
**TRIPOD-LLM**, and **MI-CLEAR-LLM**. This skill **specifies and routes** these metrics to
their published extractors and to `/analyze-stats`; it does not run the model or compute the
scores itself. Never report n-gram overlap as clinical correctness, and never fabricate a
score.

## Clinical-efficacy metrics beyond n-gram overlap (ME2 → `NGRAM_ONLY`)

- **Why n-gram overlap fails.** BLEU / ROUGE / METEOR / CIDEr measure surface lexical
  overlap. A report can score high while inverting laterality or omitting a pneumothorax, and
  low while paraphrasing a correct finding. Yu et al. (*Patterns* 2023, RadCliQ) showed these
  metrics correlate weakly with radiologist-assessed clinical error — so an n-gram score is a
  **fluency proxy, not a correctness claim**.
- **RadGraph-F1.** Overlap of the (entity, relation) tuples the RadGraph schema (Jain et al.,
  2021) extracts from the reference and the candidate report — it rewards getting the *same
  findings and their relationships*, not the same words.
- **CheXbert-F1 / CheXpert labeler.** Agreement on the structured CheXpert observation labels
  extracted by CheXbert (Smit et al., 2020) or by the rule-based CheXpert labeler (Irvin et
  al., 2019) — a finding-label–level correctness signal.
- **RadCliQ.** A composite (Yu et al., 2023) that combines metrics to better predict the count
  of radiologist-judged errors. Report it as a composite **alongside** its components, not as a
  one-number replacement.
- **GREEN.** An LLM-based score (Ostmeier et al., EMNLP Findings 2024) that prompts a language model
  to identify and count **clinically significant errors** between the candidate and reference report
  and emits an interpretable error notation, aligning better with radiologist judgment than n-gram
  overlap. Report it as a complement to RadGraph-F1 / CheXbert-F1, not a replacement, and note its
  dependence on the grading model + prompt (a quantitative metric with a qualitative, human-readable
  error breakdown — the two halves of report-generation evaluation).
- **Advise:** report a clinical-efficacy metric (RadGraph-F1 / CheXbert-F1 / RadCliQ / GREEN)
  **with bootstrap CIs over reports** and a **per-finding / per-label breakdown**, and present
  any BLEU/ROUGE explicitly labelled as a surface-overlap measure — never as the headline.

## Faithfulness & hallucination (ME3 → `FAITHFULNESS_MISSING`)

- **Fluency is not faithfulness.** A high-overlap, well-formed report can still assert findings
  the image does not support. Measure faithfulness directly; do not infer it from an accuracy
  number.
- **Atomic-fact decomposition.** Break the generated text into atomic clinical claims and check
  each against the image / source, then report a **faithfulness (or hallucination) rate** — the
  fraction of generated claims that are grounded.
- **Direction matters.** Separate **omission** (a true finding the model missed) from
  **fabrication** (a false finding the model asserted); they carry different clinical risk and
  should be reported separately, not folded into one error count.
- **False-premise / abstention probe.** Ask about an absent finding or an unanswerable
  question; a faithful model abstains rather than confabulates. Named instruments: **MedVH**,
  **Med-HALT**.
- **Advise:** a generation or VQA claim with no faithfulness and no false-premise/abstention
  evaluation is the central MLLM gap — require both, with rates, before any clinical claim.

## Pretraining / benchmark contamination (ME4 → `CONTAMINATION_UNADDRESSED`)

- **Why public benchmarks are suspect.** VQA-RAD, SLAKE, MIMIC-CXR–derived sets, MedQA,
  PMC-VQA, PathVQA, OpenI, and PubMedQA may sit inside the model's pretraining corpus, so a
  high score can be **memorisation, not capability**. For a closed API the corpus is undisclosed,
  so contamination cannot be excluded — only **bounded** and stated.
- **Three accepted checks (any one, stated explicitly):**
  - **Cutoff vs release date** — compare the model's training cutoff against the benchmark's
    release date; a benchmark that predates the cutoff is at risk.
  - **Held-out / post-cutoff set** — evaluate on a private, institution-collected, or
    after-cutoff set the model could not have seen.
  - **Contamination probe** — canary strings, a perturbed-duplicate performance gap (score on
    verbatim items vs lightly perturbed copies), or a membership/quiz test (ask the model to
    reproduce held-out items).
- **Advise:** never write "no contamination" (or evaluate on a pre-cutoff public benchmark in
  silence) without one of the checks above; an acknowledged-but-unmitigated risk is a stated
  limitation, not a clean result.

## Prompt-sensitivity & determinism (ME5 → `PROMPT_PROVENANCE_MISSING`)

- **Outputs move with the prompt and the sampler.** Phrasing, format, the system prompt,
  temperature, top-p/top-k, and run-to-run sampling all shift results. A single-prompt
  single-run number overstates stability.
- **Closed APIs are non-deterministic even at temperature 0** — identical inputs can yield
  different outputs across calls. Treat any single-run figure as a point estimate of a
  distribution, not a fixed value.
- **Disclose (MI-CLEAR-LLM transparency):** the **exact prompt(s)** including the system prompt,
  the **decoding settings** (temperature / top-p / seed), **≥ 3 runs** with reported variance
  (e.g., mean ± SD), and a **prompt-robustness** check across **≥ 2 phrasings/formats** for the
  headline result.

## Answer-matching for VQA / classification (ME6 → `ANSWER_MATCHING_MISSING`)

- **State the matching rule.** Free-text answers must be mapped to the key by a declared rule:
  **exact** string match, **normalised** match (case/punctuation/synonym folding), or
  **LLM-as-judge**. An unspecified rule makes the accuracy unreproducible.
- **An LLM judge is itself a model.** If a model adjudicates correctness, **validate the judge
  against a human-labelled subset** and report its agreement; route judge validation to
  `/design-ai-benchmarking`. An unvalidated LLM judge can launder the system's own errors.
- **Operating discipline.** Report accuracy **at the real clinical prevalence**, not on an
  artificially balanced QA set, and state **how refusals/abstentions are scored** (counted
  wrong, excluded, or credited) — the choice can move the headline.

## Reader study for generated reports (ME7 → `READER_STUDY_MISSING`)

- **Automated metrics do not establish clinical acceptability.** Even RadGraph-F1 / CheXbert-F1
  measure agreement, not whether a clinician would act on the report safely. A deployment or
  utility claim for generated text needs a **blinded clinical reader study**.
- **Design elements:** a pre-defined **error taxonomy** (clinically significant vs insignificant;
  omission vs fabrication), a **severity scale**, and **inter-reader agreement**.
- **Route:** the rubric and the IRR design → `/design-ai-benchmarking`; the ICC/κ computation →
  `/analyze-stats`; reader and case **sizing** → `/calc-sample-size`.

## Gate mapping

The deterministic gate (`scripts/check_mllm_eval_completeness.py`) is a presence check on the
plan text, task-aware. This reference is the *why* behind each verdict:

| Axis (this doc) | Gate verdict | Severity |
|---|---|---|
| n-gram only, no clinical metric (report-gen) | `NGRAM_ONLY` | Major |
| no adjudicated reference standard (report-gen) | `REFERENCE_STANDARD_MISSING` | Major |
| no faithfulness / false-premise (report-gen, vqa) | `FAITHFULNESS_MISSING` | Major |
| public benchmark, no contamination handling | `CONTAMINATION_UNADDRESSED` | Major |
| no blinded reader study (report-gen) | `READER_STUDY_MISSING` | Major (deploy) / Minor |
| prompt / decoding / multi-run incomplete | `PROMPT_PROVENANCE_MISSING` | Minor |
| no answer-matching rule (vqa, classification) | `ANSWER_MATCHING_MISSING` | Minor |

A Major verdict is a presence gap, not proof the work is wrong — resolve it by adding the axis
to the plan (or recording, with a stated reason, why it does not apply).

## Reporting fit & hand-off

Methods / Results stub → `/write-paper`. Item-level compliance with **TRIPOD-LLM**,
**MI-CLEAR-LLM**, **CLAIM 2024** (and STARD-AI / TRIPOD+AI where a diagnostic/prognostic claim
is made) → `/check-reporting`. Reviewer-side audit of a finished manuscript uses the
`mllm_evaluation.md` (ME0–ME8) probe via `/self-review` and `/peer-review`.

## Verification notes

Each claim here is grounded in a named public method/standard or described qualitatively; no
numbers, thresholds, or dataset contents are invented.

- **n-gram metrics correlate weakly with clinical error; clinical-efficacy metrics needed** —
  Yu et al., *Patterns* 2023 (RadCliQ). Named public methods paper (matches the citation already
  vendored in the `mllm_evaluation.md` probe).
- **RadGraph-F1 (entity-relation overlap)** — Jain et al., NeurIPS Datasets & Benchmarks 2021.
  Named public methods paper.
- **GREEN (LLM-graded clinically significant errors)** — Ostmeier et al., EMNLP Findings 2024.
  Named public methods paper; described qualitatively, no scores invented.
- **CheXbert-F1 / CheXpert observation labels** — Smit et al., 2020 (CheXbert); Irvin et al.,
  2019 (CheXpert labeler). Named public methods papers; the CheXpert label set is a factual
  artifact, not invented.
- **RadCliQ as a composite** — Yu et al., *Patterns* 2023. Named public methods paper.
- **Atomic-fact faithfulness, omission-vs-fabrication, false-premise/abstention** — described as
  established evaluation principles; **MedVH** and **Med-HALT** named as instruments only (as in
  the probe), no scores invented.
- **Contamination of public benchmarks; closed-corpus unknowability; cutoff/held-out/probe
  checks (canary, perturbed-duplicate gap, membership test)** — stated as accepted principles and
  practices, qualitatively; benchmark **names** (VQA-RAD, SLAKE, MIMIC-CXR, MedQA, PMC-VQA,
  PathVQA, OpenI, PubMedQA) are factual public-dataset names, no contents reproduced.
- **Closed-API non-determinism even at temperature 0; prompt/format/sampling sensitivity** —
  described qualitatively as documented behavior; no figure attached.
- **Prompt + decoding + ≥3 runs + ≥2 phrasings disclosure** — MI-CLEAR-LLM transparency
  (named standard); the ≥3 / ≥2 conventions are this skill's own house thresholds (carried from
  SKILL.md / the ME-probe), not literature values.
- **LLM-as-judge must be validated against human labels** — stated as a methodological principle;
  judge validation routed to `/design-ai-benchmarking`.
- **Reader study for deployment/utility claims; error taxonomy, severity, IRR** — consistent with
  CLAIM 2024 / TRIPOD-LLM reporting expectations (named standards); sizing/IRR routed to
  `/calc-sample-size` and `/analyze-stats`.
- **Metrics Reloaded / CLAIM 2024 / TRIPOD-LLM / MI-CLEAR-LLM / Model Cards (Mitchell 2019) /
  Datasheets (Gebru 2021)** — named public standards, cited by name only.

#!/usr/bin/env python3
"""LLM / MLLM clinical-evaluation completeness gate (mllm-eval).

A task-aware presence linter for the methods / plan of an LLM or multimodal-LLM
clinical evaluation. It flags the axes a defensible evaluation must cover that are
**absent** from the plan text — it is a presence check on the protocol (the analogue
of check_model_card_complete for documentation), not a judge of results. Conservative:
each verdict fires only when its concept is clearly missing from the text.

CHECKS (verdicts; which apply depends on --task):
  1. NGRAM_ONLY                (Major)  report-gen names BLEU/ROUGE/METEOR but no
                                        clinical-efficacy metric (RadGraph-F1 /
                                        CheXbert / CheXpert / RadCliQ).
  2. FAITHFULNESS_MISSING      (Major)  no faithfulness / hallucination / false-premise
                                        evaluation (report-gen, vqa).
  3. REFERENCE_STANDARD_MISSING(Major)  no adjudicated reference-standard statement
                                        (report-gen).
  4. CONTAMINATION_UNADDRESSED (Major)  a public benchmark is named but no
                                        contamination / training-cutoff / held-out
                                        statement.
  5. READER_STUDY_MISSING      (Major)  report-gen with no blinded clinical reader
                                        study.
  6. PROMPT_PROVENANCE_MISSING (Minor)  no prompt + temperature/seed + multi-run
                                        disclosure.
  7. ANSWER_MATCHING_MISSING   (Minor)  vqa/classification with no answer-matching
                                        rule (exact / normalised / LLM-judge).

INPUTS
  --plan   the evaluation plan / methods markdown (required).
  --task   report_generation | vqa | classification (required).

OUTPUT
  A table (stdout) and, with --out, a JSON artifact:
    {plan, task, claims[{verdict, severity, detail, where}], summary}

Stdlib-only (re / json / argparse / pathlib). Exit codes: 0 clean (or report-only),
1 Major claim(s) found (with --strict), 2 input/usage error.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

# concept -> regex of any phrasing that SATISFIES it (searched case-insensitively)
SAT = {
    "ngram": r"\b(bleu|rouge|meteor|cider)\b",
    "clinical_metric": r"\b(radgraph|chexbert|chexpert[- ]?label|radcliq|green|f1[\w-]*chexpert|"
                       r"clinical efficacy|factual(?:ity| correctness)|entity[- ]?relation)\b",
    "faithfulness": r"\b(faithful(?:ness)?|hallucinat\w+|atomic[- ]?fact|false[- ]?premise|"
                    r"fabricat\w+|med-?hal|medvh|med-?halt|confabulat\w+|ungrounded|groundedness|"
                    r"unsupported|clinical(?:ly)?[- ]?grounded|evidence[- ]?support\w*)\b"
                    r"|not supported by the image",
    "reference_standard": r"\b(reference standard|reference reports?|adjudicat\w+|gold[- ]?standard|"
                          r"expert(?:[- ]?annotat\w+| review| radiologist)|consensus|"
                          r"ground[- ]?truth|radiolog\w+[- ]?author\w+)\b",
    "benchmark": r"\b(vqa-?rad|slake|mimic-?cxr|medqa|pmc-?vqa|path-?vqa|openi|pubmedqa|"
                 r"public benchmark)\b",
    "contamination": r"\b(contaminat\w+|(?:training|pre-?training|knowledge)[- ]cut[- ]?off|"
                     r"held[- ]?out|post[- ]?cut[- ]?off|canary|memoris\w+|memoriz\w+|"
                     r"data leakage)\b",
    "reader_study": r"reader study|blinded read\w+|clinical reader|radiologist review|"
                    r"human evaluation|expert rating|acceptability scale|likert|"
                    r"\d+\s+(?:radiologists?|readers?|clinicians?)[^.\n]{0,60}"
                    r"(?:rated|scored|graded|read|review)|"
                    r"(?:masked|blinded)[^.\n]{0,40}(?:graded|scored|rated)",
    "prompt": r"\b(prompt)\b",
    "decoding": r"\b(temperature|top-?p|top-?k|seed|greedy|sampling)\b",
    "multirun": r"\b(\d+\s*runs|repeated runs|multiple runs|across runs|run-to-run|"
                r"mean\s*(?:±|\+/-|\+-)\s|standard deviation|variance|bootstrap)\b",
    "answer_match": r"\b(exact match|normali[sz]ed match|answer matching|llm[- ]?as[- ]?judge|"
                    r"llm judge|string match|keyword match|semantic (?:equivalence|match\w*)|"
                    r"clinician[- ]?adjudicat\w+|adjudicated (?:correct|equivalent)|"
                    r"human[- ]?judg\w+)\b",
}

DEPLOY_CLAIM = re.compile(
    r"\b(deploy\w*|clinical use|ready for (?:clinical|practice)|integrat\w+ into (?:practice|workflow)|"
    r"assist\w* (?:radiologists|clinicians)|in practice)\b", re.IGNORECASE)


def has(text: str, concept: str) -> bool:
    return re.search(SAT[concept], text, re.IGNORECASE) is not None


def analyze(plan: str, task: str) -> dict:
    text = Path(plan).read_text(encoding="utf-8")
    claims = []

    def add(verdict, severity, detail):
        claims.append({"verdict": verdict, "severity": severity, "detail": detail, "where": Path(plan).name})

    is_gen = task == "report_generation"
    is_vqa = task == "vqa"

    if is_gen:
        if has(text, "ngram") and not has(text, "clinical_metric"):
            add("NGRAM_ONLY", "Major",
                "report-generation quality is reported with n-gram overlap (BLEU/ROUGE) but no "
                "clinical-efficacy metric (RadGraph-F1 / CheXbert / RadCliQ) — n-gram overlap is weakly "
                "correlated with clinical correctness")
        if not has(text, "reference_standard"):
            add("REFERENCE_STANDARD_MISSING", "Major",
                "no adjudicated expert reference-standard statement for the generated reports")
        if not has(text, "reader_study"):
            sev = "Major" if DEPLOY_CLAIM.search(text) else "Minor"
            add("READER_STUDY_MISSING", sev,
                "no blinded clinical reader study with an error taxonomy" +
                (" for a deployment/utility claim" if sev == "Major" else " (automated metrics only)"))

    if is_gen or is_vqa:
        if not has(text, "faithfulness"):
            add("FAITHFULNESS_MISSING", "Major",
                "no faithfulness / hallucination / false-premise evaluation — a fluent answer is not a "
                "faithful one")

    if has(text, "benchmark") and not has(text, "contamination"):
        add("CONTAMINATION_UNADDRESSED", "Major",
            "a public clinical benchmark is named but pretraining contamination is not addressed "
            "(training cutoff vs benchmark release, a held-out/post-cutoff set, or a contamination probe)")

    if not (has(text, "prompt") and has(text, "decoding") and has(text, "multirun")):
        missing = [m for m, c in (("prompt", "prompt"), ("temperature/seed", "decoding"),
                                  ("multi-run variance", "multirun")) if not has(text, c)]
        add("PROMPT_PROVENANCE_MISSING", "Minor",
            "prompt-sensitivity provenance incomplete — missing: " + ", ".join(missing))

    if (is_vqa or task == "classification") and not has(text, "answer_match"):
        add("ANSWER_MATCHING_MISSING", "Minor",
            "no answer-matching rule stated (exact / normalised / LLM-judge) for free-text answers")

    n_major = sum(1 for c in claims if c["severity"] == "Major")
    return {"plan": plan, "task": task, "claims": claims,
            "summary": {"n_claims": len(claims), "n_major": n_major,
                        "verdict": "MAJOR_CANDIDATE" if n_major else "OK"}}


def render(result: dict) -> str:
    lines = ["| Check | Severity | Detail |", "|---|---|---|"]
    for c in result["claims"]:
        lines.append(f"| {c['verdict']} | {c['severity']} | {c['detail']} |")
    if len(lines) == 2:
        lines.append("| (none) | — | evaluation plan covers the required MLLM axes |")
    return "\n".join(lines)


def main() -> int:
    ap = argparse.ArgumentParser(description="LLM/MLLM clinical-evaluation completeness gate.")
    ap.add_argument("--plan", required=True, help="evaluation plan / methods markdown")
    ap.add_argument("--task", required=True, choices=["report_generation", "vqa", "classification"])
    ap.add_argument("--out", help="write JSON artifact to this path")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major claim exists")
    ap.add_argument("--quiet", action="store_true", help="suppress stdout table")
    args = ap.parse_args()

    if not Path(args.plan).is_file():
        sys.stderr.write(f"ERROR: --plan not found: {args.plan}\n")
        return 2
    result = analyze(args.plan, args.task)

    if not args.quiet:
        print("=" * 41)
        print(" MLLM Evaluation Completeness")
        print("=" * 41)
        print(render(result))
        print()
        s = result["summary"]
        print(f"MAJOR candidate: {s['n_major']} evaluation-completeness gap(s)." if s["n_major"]
              else "OK: evaluation plan covers the required MLLM axes.")

    if args.out:
        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
        Path(args.out).write_text(json.dumps({"detector": "check_mllm_eval_completeness", **result}, indent=2), encoding="utf-8")
        if not args.quiet:
            print(f"\nwrote {args.out}")

    return 1 if (args.strict and result["summary"]["n_major"]) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# Regression test for the MLLM-evaluation completeness gate (mllm-eval). Synthetic, PII-free.
set -u
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DET="$HERE/../scripts/check_mllm_eval_completeness.py"
F="$HERE/../scripts/mllm_eval_completeness_challenge/fixture"
OUT="$(mktemp -t mec_XXXX).json"; trap 'rm -f "$OUT"' EXIT
fail=0
check(){ local l="$1"; shift; if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$l"; else printf '  FAIL  %s\n' "$l"; fail=$((fail+1)); fi; }
has(){ python3 -c "import json;d=json.load(open('$OUT'));assert any(c['verdict']=='$1' for c in d['claims']),'$1'"; }
no(){ python3 -c "import json;d=json.load(open('$OUT'));assert not any(c['verdict']=='$1' for c in d['claims']),'$1'"; }
[[ -f "$DET" ]] || { echo "ENV-ERR" >&2; exit 2; }

python3 "$DET" --plan "$F/plan_bad.md" --task report_generation --out "$OUT" --strict --quiet >/dev/null 2>&1
check "plan_bad exits 1" test "$?" -eq 1
check "NGRAM_ONLY" has NGRAM_ONLY
check "FAITHFULNESS_MISSING" has FAITHFULNESS_MISSING
check "REFERENCE_STANDARD_MISSING" has REFERENCE_STANDARD_MISSING
check "CONTAMINATION_UNADDRESSED" has CONTAMINATION_UNADDRESSED
python3 "$DET" --plan "$F/plan_good.md" --task report_generation --out "$OUT" --strict --quiet >/dev/null 2>&1
check "plan_good exits 0" test "$?" -eq 0
check "plan_good no NGRAM_ONLY" no NGRAM_ONLY
check "plan_good no FAITHFULNESS_MISSING" no FAITHFULNESS_MISSING

# vqa task: a bare VQA-RAD accuracy plan -> CONTAMINATION + FAITHFULNESS + ANSWER_MATCHING
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"; rm -f "$OUT"' EXIT
printf 'We report accuracy on VQA-RAD.\n' > "$TMP/vqa.md"
python3 "$DET" --plan "$TMP/vqa.md" --task vqa --out "$OUT" --strict --quiet >/dev/null 2>&1
check "vqa bare plan exits 1" test "$?" -eq 1
check "vqa FAITHFULNESS_MISSING" has FAITHFULNESS_MISSING
check "vqa CONTAMINATION_UNADDRESSED" has CONTAMINATION_UNADDRESSED
check "vqa ANSWER_MATCHING_MISSING" has ANSWER_MATCHING_MISSING

# --- false-positive robustness (folded from adversarial review): a complete plan stated in
#     non-canonical vocabulary must PASS (knowledge cutoff / gold standard / graded-on-scale /
#     groundedness / semantic match). ---
cat > "$TMP/good_vocab.md" <<'MD'
We evaluate the model on SLAKE. The reference is the original board-certified radiologists' reports
(the gold standard). Quality uses RadGraph-F1 and CheXbert-F1 with 95% CIs. Groundedness is assessed
(each statement supported vs unsupported by the image); a false-premise probe gives a hallucination rate.
Contamination is controlled against the model's knowledge cutoff versus the benchmark release. Three
radiologists graded outputs on an acceptability scale (masked). We disclose the prompt, temperature and
seed, and report mean +/- SD across 3 runs. Free-text answers were scored by clinician-adjudicated
semantic equivalence.
MD
python3 "$DET" --plan "$TMP/good_vocab.md" --task report_generation --out "$OUT" --strict --quiet >/dev/null 2>&1
check "FP: complete plan in non-canonical vocabulary passes (exit 0)" test "$?" -eq 0

echo "fail=$fail"; [[ "$fail" -eq 0 ]] && echo "ALL PASS" || echo "FAILURES: $fail"; exit "$fail"