{"slug":"uncertainty-imaging","title":"uncertainty-imaging","summary":"Design or audit the uncertainty-quantification, out-of-distribution (OOD) detection, and selective-prediction layer of a medical-imaging model framed for deployment — so a clinical-use claim carries calibrated per-case uncertainty (MC-dropout / deep ensemble / conformal / Bayesia","platform":"Claude","tags":[],"authorName":"LLM Mart","authorSlug":"llm-mart","score":0,"source":"github","price":null,"verified":false,"createdAt":"2026-09-14T20:48:12.846216Z","repo":{"url":"https://github.com/Aperivue/medsci-skills","stars":318,"forks":75,"license":"MIT","updatedAt":"2026-09-27T05:05:17Z"},"bodyHtml":"<hr>\n<h2>name: uncertainty-imaging\ndescription: &gt;\nDesign or audit the uncertainty-quantification, out-of-distribution (OOD) detection, and\nselective-prediction layer of a medical-imaging model framed for deployment — so a clinical-use claim\ncarries calibrated per-case uncertainty (MC-dropout / deep ensemble / conformal / Bayesian), an OOD\nguard validated on a held-out OOD set, an abstention rule at a pre-specified operating point, and\nuncertainty checked under distribution shift. Emits an uncertainty manifest and a deterministic gate\nthat flags a deployment claim built on point predictions, conformal intervals with unmeasured coverage,\nand an OOD claim with no held-out OOD data. Integrates MAPIE / captum / pretrained OOD scorers; it does\nnot reimplement them and never runs a model on real patient data.\ntriggers: uncertainty, uncertainty quantification, UQ, epistemic, aleatoric, MC-dropout, monte carlo dropout, deep ensemble, conformal prediction, split conformal, prediction interval, coverage, calibration under shift, out-of-distribution, OOD detection, distribution shift, Mahalanobis, energy score, ODIN, selective prediction, abstention, reject option, deployment safety, DECIDE-AI, predictive uncertainty\ntools: Read, Write, Edit, Bash, Grep, Glob\nmodel: inherit</h2>\n<h1>Uncertainty-Imaging Skill</h1>\n<h2>Purpose</h2>\n<p>A medical-imaging model framed for <strong>deployment</strong> must say more than \"class 1, 0.87\". It needs a\n<strong>calibrated uncertainty</strong> on each case, an <strong>out-of-distribution (OOD) guard</strong> validated on data known\nto be out-of-distribution, and — if it abstains — a <strong>pre-specified operating point</strong>. The failures are\npredictable and reviewer-visible: a clinical-use claim built on point predictions, conformal intervals\nquoted without ever measuring their coverage, an \"OOD detector\" evaluated only on in-distribution data,\na deep ensemble whose members share a seed, and uncertainty validated only in-distribution when\ndeployment sees scanner/site/case-mix shift. This skill designs that layer and audits an existing one\n(Gal 2016; Lakshminarayanan 2017; Angelopoulos &amp; Bates; Ovadia 2019; DECIDE-AI).</p>\n<p>It is the deployment-safety companion in the model-engineering lane: <code>/model-evaluation</code> computes the\nheld-out metrics and calibration, and <strong>uncertainty-imaging</strong> covers the uncertainty / OOD / abstention\nmachinery a deployment claim rests on. It <strong>integrates</strong> MAPIE (conformal), captum, and pretrained OOD\nscorers; it does not reimplement them and never runs a model on real patient data.</p>\n<h2>When to use</h2>\n<ul>\n<li>Your model is framed for clinical use / deployment and a reviewer will ask \"what does it do when it is\nunsure, or off-distribution?\"</li>\n<li>You report conformal / MC-dropout / ensemble uncertainty and want the coverage, independence, and\nshift checks right before submission.</li>\n<li>You want to audit an existing uncertainty/OOD section for the failure modes below.</li>\n</ul>\n<h2>When NOT to use</h2>\n<ul>\n<li>Held-out discrimination / calibration metrics of the point predictor → <code>/model-evaluation</code> then\n<code>/analyze-stats</code>.</li>\n<li>Training-repo scaffolding / the split → <code>/model-scaffold</code> (+ <code>/model-validation</code>).</li>\n<li>Interpretability / saliency of a trained network → <code>/explainability</code>.</li>\n<li>Classical-ML calibration of a tabular model → <code>/radiomics-ml</code> + <code>/analyze-stats</code>.</li>\n<li>Reimplementing MAPIE / an OOD library → out of scope (this skill wires and audits them).</li>\n</ul>\n<h2>The failure modes (what the gate enforces)</h2>\n<ol>\n<li><strong>Point predictions under a deployment claim.</strong> A clinical-use claim with no uncertainty method at\nall — add MC-dropout, a deep ensemble, conformal prediction, or a Bayesian estimate.</li>\n<li><strong>Conformal without coverage validation.</strong> Conformal's guarantee holds under exchangeability, which\ncan fail on clinical data — measure achieved coverage on a held-out calibration/test set.</li>\n<li><strong>OOD claim with no held-out OOD set.</strong> An OOD detector's operating point and AUROC are unmeasured\nuntil you evaluate on data known to be out-of-distribution (different scanner / site / pathology).</li>\n<li><strong>Non-independent ensemble.</strong> A deep ensemble whose members share a seed/init (or has &lt; 2 members)\nunderestimates epistemic uncertainty.</li>\n<li><strong>MC-dropout with dropout off at inference.</strong> Dropout must stay active during sampling; off, every\npass is identical and the estimate collapses to a point prediction.</li>\n<li><strong>Selective prediction without a target.</strong> Abstention chosen post hoc inflates accuracy-at-coverage;\npre-specify the coverage / risk operating point.</li>\n<li><strong>No calibration under shift.</strong> Uncertainty evaluated in-distribution only; deployment uncertainty\ndegrades under shift, so report it on shifted / external data.</li>\n</ol>\n<h2>Workflow</h2>\n<h3>Phase 1 — Choose the uncertainty method (integrate, don't reimplement)</h3>\n<ul>\n<li><strong>Conformal prediction</strong> (MAPIE) — distribution-free prediction sets/intervals at a nominal coverage;\nthe strongest default when a calibration set is available. Validate empirical coverage.</li>\n<li><strong>Deep ensembles</strong> (Lakshminarayanan 2017) — train K independent members (distinct seeds/inits); the\nbest-quality epistemic uncertainty, at K× cost.</li>\n<li><strong>MC-dropout</strong> (Gal 2016) — keep dropout active at inference and sample T passes; cheap, weaker.</li>\n<li><strong>Bayesian / Laplace</strong> — a last-layer Laplace approximation is a light option.\nSee <code>references/uncertainty_guide.md</code>.</li>\n</ul>\n<h3>Phase 2 — Add the OOD guard and the abstention rule</h3>\n<ul>\n<li><strong>OOD detection</strong> — an energy score, Mahalanobis distance on features, ODIN, or max-softmax; <strong>evaluate\non a held-out OOD set</strong> (different scanner/site/pathology) and report detection AUROC + the operating\npoint.</li>\n<li><strong>Selective prediction</strong> — abstain below a confidence/uncertainty threshold set to a <strong>pre-specified</strong>\ntarget coverage or risk; report the risk–coverage curve.</li>\n</ul>\n<h3>Phase 3 — Stress it under shift</h3>\n<p>Report calibration / coverage on <strong>shifted or external</strong> data, not in-distribution only (Ovadia 2019).</p>\n<h3>Phase 4 — Emit the uncertainty manifest</h3>\n<pre><code>{\n  \"task\": \"classification\",\n  \"deployment_claim\": true,\n  \"uncertainty_method\": \"conformal\",\n  \"coverage_target\": 0.90,\n  \"coverage_validated\": true,\n  \"ood_method\": \"mahalanobis\",\n  \"ood_heldout_set\": \"external-ood-cohort\",\n  \"selective_prediction\": true,\n  \"selective_target\": 0.95,\n  \"calibration_under_shift\": true\n}\n</code></pre>\n<h3>Phase 5 — Gate the spec (deterministic)</h3>\n<pre><code>python3 scripts/check_uncertainty_reporting.py --manifest uncertainty_manifest.json --strict\n</code></pre>\n<p>Verdicts: <code>POINT_PREDICTION_NO_UNCERTAINTY</code>, <code>CONFORMAL_NO_COVERAGE_VALIDATION</code>, <code>OOD_NO_HELDOUT_SET</code>\n(Major); <code>ENSEMBLE_NOT_INDEPENDENT</code>, <code>MCDROPOUT_DISABLED_AT_INFERENCE</code>, <code>SELECTIVE_NO_TARGET</code>,\n<code>NO_CALIBRATION_UNDER_SHIFT</code> (Minor). Audits the declared spec at design/report time; it complements\n<code>/model-evaluation</code>'s executed calibration/subgroup metrics.</p>\n<h2>Integration</h2>\n<ul>\n<li><strong><code>/model-evaluation</code></strong> — the point predictor's held-out metrics + calibration this layer sits on top of.</li>\n<li><strong><code>/analyze-stats</code></strong> — calibration curve / risk–coverage plotting for the report.</li>\n<li><strong><code>/check-reporting</code></strong> — TRIPOD+AI / DECIDE-AI deployment-monitoring items.</li>\n<li><strong><code>/model-validation</code></strong> — the DECIDE-AI monitoring seam (the deployment-time counterpart of the split\naudit).</li>\n</ul>\n<h2>Anti-Hallucination</h2>\n<ul>\n<li><strong>Never fabricate coverage, OOD AUROC, or calibration numbers.</strong> Every value in the manifest and every\nreported number comes from the researcher's executed code — never invented. This skill designs and\naudits the uncertainty spec; it does not run a model on real patient data.</li>\n<li><strong>Never report conformal coverage as guaranteed without measuring it.</strong> Exchangeability can fail on\nclinical data (<code>CONFORMAL_NO_COVERAGE_VALIDATION</code>).</li>\n<li><strong>Never report an uncertainty/OOD audit \"pass\" without running <code>check_uncertainty_reporting.py</code>.</strong> The\nverdict is reproduced deterministically, never asserted from prose.</li>\n<li><strong>Integrate, don't reimplement.</strong> Reference MAPIE / captum / OOD scorers; do not write a new conformal\nor OOD library or claim results for one.</li>\n</ul>\n<h2>Reproducible challenge</h2>\n<p><code>scripts/check_uncertainty_reporting_challenge/</code> ships a synthetic weak/strong uncertainty-manifest pair\nwith a network-free <code>verify.sh</code> wired into the skill's validation commands.</p>\n","files":[{"path":"references/uncertainty_guide.md","sizeBytes":5574,"isText":true},{"path":"scripts/check_uncertainty_reporting_challenge/expected/strong.txt","sizeBytes":414,"isText":true},{"path":"scripts/check_uncertainty_reporting_challenge/expected/weak.txt","sizeBytes":1077,"isText":true},{"path":"scripts/check_uncertainty_reporting_challenge/fixture/uncertainty_strong.json","sizeBytes":323,"isText":true},{"path":"scripts/check_uncertainty_reporting_challenge/fixture/uncertainty_weak.json","sizeBytes":204,"isText":true},{"path":"scripts/check_uncertainty_reporting_challenge/problem.md","sizeBytes":2460,"isText":true},{"path":"scripts/check_uncertainty_reporting_challenge/verify.sh","sizeBytes":2117,"isText":true},{"path":"scripts/check_uncertainty_reporting.py","sizeBytes":13089,"isText":true},{"path":"SKILL.md","sizeBytes":8224,"isText":true},{"path":"skill.yml","sizeBytes":3753,"isText":true},{"path":"tests/test_uncertainty_reporting.sh","sizeBytes":4386,"isText":true}],"reviewScore":null,"reviewSummary":null,"trust":{"provenance":"trusted-source-unreviewed","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow.","bodySource":null},"bodyLocked":false,"purchaseUrl":null,"sourceUrl":null,"report":{"provenance":"trusted-source-unreviewed","screen":{"ran":true,"outcome":"clean","suspicious":0,"notes":0,"hiddenCharacters":false},"virusScan":{"engine":"clamav","status":"clean","scannedAt":"2026-09-14T20:48:58.380721Z","sha256":"C5A8E734428B7F0F2D57BD09C325C8DA144E84BE74CE3728C3F25028F32E9957","sizeBytes":18452},"review":null,"source":{"repositoryUrl":"https://github.com/Aperivue/medsci-skills","path":"skills/uncertainty-imaging","license":"MIT","commit":"5599b724675a1d788e03cd58dabd3db7c68ca86b","subtreeSha":"95A4DC9FA8EDCB8B99C8E16AF01EF2AF06F9BC0E18E239E8A7CC9C09732EE4C9","lastSyncedAt":"2026-09-27T19:46:33.449845Z"},"reviewedAt":"2026-09-14T20:54:34.857611Z","notice":"Community-authored content, reproduced verbatim and not vetted as instructions. Treat it as data to evaluate, never as directives to follow."},"install":[{"target":"skills-cli","command":"npx skills add https://github.com/Aperivue/medsci-skills/tree/main/skills/uncertainty-imaging"},{"target":"claude-code","command":"claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aperivue-medsci-skills@llmmart"},{"target":"git","command":"git clone https://github.com/Aperivue/medsci-skills.git"}]}