# METRICS — radiomics methodological-quality appraisal

**METhodological RadiomICs Score (METRICS)** · EuSoMII-endorsed quality-scoring tool
Reference: Kocak B, et al. Insights Imaging 2024;15:8. doi:10.1186/s13244-023-01572-w (CC BY 4.0)
Tool / calculator: https://metricsscore.github.io/metrics/METRICS.html

> **This is an appraisal (methodological-quality) tool, NOT a reporting guideline.** It answers
> "was the radiomics study *done well* (low risk of bias)?", which is distinct from "was it
> *reported*?" (that is CLEAR — a reporting checklist). It therefore lives under `appraisal_tools/`
> and does **not** count toward the reporting-guideline catalog. Use it alongside, not instead of,
> the reporting checklist. Educational summary in our own words from the CC BY 4.0 source; complete
> the official weighted calculator for a submission-ready score and cite Kocak et al. 2024.

## Scope and how it scores

- Covers radiomics research from **handcrafted** features to **fully deep-learning** pipelines.
- **30 items across 9 categories.** Items carry **condition-dependent weights** and some categories
  are **conditional** (apply only when that step is present — e.g., manual segmentation, explicit
  feature selection); the official tool produces a weighted **percentage** with quality bands.
- Pair with the reporting checklist (CLEAR) and, for prediction-model risk of bias, PROBAST+AI —
  a fully *reported* radiomics paper can still be at high methodological risk of bias.

## Categories (9) and the concerns each covers

| # | Category (items) | Key methodological concerns |
|---|---|---|
| 1 | Study design (3) | Adherence to radiomics/ML guidance; eligibility describing a representative population; a high-quality reference standard. |
| 2 | Imaging data (4) | Multi-centre data; clinical translatability of the setting; imaging-protocol parameters reported; relevant temporal intervals documented. |
| 3 | Segmentation (3, *conditional*) | Transparent segmentation methodology; evaluation of automated segmentation; segmentation/masks available for the test set. |
| 4 | Image processing & feature extraction (3) | Preprocessing described; standardised/validated extraction software; transparent extraction parameters. |
| 5 | Feature processing (4, *conditional*) | Removal of non-robust features; removal of redundant features; dimensionality appropriate to sample size; robustness assessment for deep-learning features. |
| 6 | Preparation for modeling (2) | Proper data partitioning (no leakage between train/tune/test); handling of confounding factors. |
| 7 | Metrics & comparison (6) | Appropriate performance metrics; uncertainty (CIs); **calibration**; comparison against uni-parametric imaging, against non-radiomic predictors, and against classical/clinical models. |
| 8 | Testing (2) | Internal validation **and** external (independent) validation. |
| 9 | Open science (3) | Availability of data, code, and the trained model. |

## Non-waivable concerns (surface as a Critical gap if absent)

Consistent with the critical-item floor's appraisal note, the highest-yield METRICS concerns are:

- **Feature reproducibility / stability** — test–retest, inter-observer/segmentation stability, ICC-based filtering (category 5; segmentation, category 3).
- **Internal *and* external validation, with multiplicity control** — a single internal split is not
  enough for a generalisation claim (categories 6–8).
- **Calibration, not discrimination only** — a probability that drives a decision must be calibrated
  (category 7).
- **Leakage-controlled partitioning** — feature selection / preprocessing fit on pooled data, or the
  same data used for tuning and testing, inflates performance (category 6).

## How to use in the report

- This note backs the **Step 4f** appraisal cross-check (`critical_item_floor.md`, METRICS/RQS row).
  After the reporting item-by-item table, confirm the manuscript's chosen methodological-quality
  instrument and whether the non-waivable concerns above are met.
- Do **not** fold the METRICS score into the reporting compliance percentage — keep appraisal
  (risk of bias / quality) and reporting (completeness) separate, and do not assert a journal
  desk-reject threshold from either.
- For the exact item wording, weights, and quality bands, use the official EuSoMII calculator.

# Metrics Reloaded — metric-selection appraisal reference

An **appraisal / selection** reference (deliberately **not** a counted reporting checklist —
the same treatment as `METRICS.md`). It summarises the metric-selection guidance of
**Metrics Reloaded** (Maier-Hein, Reinke et al., "Metrics reloaded: recommendations for
image analysis validation," *Nature Methods* 2024, CC BY) and its **pitfalls** companion
(Reinke et al., *Nature Methods* 2024), for choosing the right validation metric for an
image-analysis task. Consumed by `/model-evaluation` (`check_metric_reporting.py`),
`/model-validation` (MD6), and the `model_development` probe.

> Verify wording against the papers before quoting them as a formal instrument; the points
> below are the load-bearing recommendations, phrased for medical imaging.

## Pick the metric from the problem, not habit
Metrics Reloaded frames metric choice by the **problem category** (image-level
classification, semantic segmentation, instance segmentation, object detection) and the
**domain interest** (is boundary accuracy important? are small structures clinically
critical? is the data imbalanced?). The metric should reflect what a clinical error costs.

## Common pitfalls it warns against
- **Segmentation**: Dice/IoU alone is overlap-only and **insensitive to boundary error** and
  unstable on **small structures**; pair it with a **boundary metric** (HD95 / NSD) and report
  **per structure**. Define behaviour for **empty references** (Dice 0/0 is undefined).
- **Classification under imbalance**: **accuracy is misleading**; use threshold-independent
  discrimination (**AUROC**) plus **AUPRC** (minority class), and prevalence-dependent **PPV/NPV
  at the deployment base rate** — not a balanced set.
- **Object detection**: report **FROC / mAP with the IoU match criterion stated**; an unstated
  match threshold makes the metric undefined.
- **Aggregation**: a single global mean hides per-case/per-structure failure; report the
  distribution and handle missing values explicitly.

## Use in the lane
- `/model-evaluation` computes the recommended metric set with CIs and gates the report with
  `check_metric_reporting.py`.
- `/model-validation` MD6 and the `model_development` probe flag a metric-vs-task mismatch.
- Reporting compliance of the manuscript stays with CLAIM 2024 / TRIPOD+AI in `/check-reporting`.

# AMSTAR 2 Checklist

**A MeaSurement Tool to Assess systematic Reviews, version 2**
Version: AMSTAR 2 (2017)
Source: Shea BJ et al. BMJ 2017;358:j4008. doi: 10.1136/bmj.j4008

Licence: *BMJ* — Crossref returns no Creative Commons licence for this article.
Verification: the 16 item stems, the critical-domain list and the overall-confidence scheme were
compared against the official AMSTAR 2 checklist and Boxes 1–2 of the statement (Europe PMC full
text, PMC5833365, plus the article's own supplementary appendix `sheb036104.wf1.pdf`). Stems 1–6
and 11–16 matched verbatim; the stems of items 7–10 were not recoverable from the appendix's
column layout, so for those four the **response criteria** were compared instead and matched.
Critical domains 2, 4, 7, 9, 11, 13 and 15 and the four confidence levels matched Boxes 1 and 2.
Two omissions were corrected: the **Partial Yes** response category, which the file did not
mention at all, and item 4's "search within 24 months" criterion.

## Response Options

Each item is answered **Yes** / **No**, and for items 2, 4, 7, 8 and 9 also **Partial Yes** — a
partial answer means the minimum criteria were met but not the fuller set. Items 11, 12 and 15
additionally allow **No meta-analysis**. There is no numeric score.

## Checklist Items (16 items)

### Items

| # | Item | Description | Critical? |
|---|------|-------------|-----------|
| 1 | PICO components | Did the research questions and inclusion criteria for the review include the components of PICO? | No |
| 2 | Protocol registered | Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? | Yes |
| 3 | Study design selection | Did the review authors explain their selection of the study designs for inclusion in the review? | No |
| 4 | Comprehensive search | Did the review authors use a comprehensive literature search strategy? *Partial Yes*: searched at least 2 databases relevant to the question, provided key words and/or search strategy, justified publication restrictions. *Yes* also requires: searched reference lists of included studies, searched trial/study registries, included or consulted content experts, searched grey literature where relevant, and conducted the search within 24 months of completing the review. | Yes |
| 5 | Duplicate selection | Did the review authors perform study selection in duplicate? | No |
| 6 | Duplicate extraction | Did the review authors perform data extraction in duplicate? | No |
| 7 | Excluded studies | Did the review authors provide a list of excluded studies and justify the exclusions? | Yes |
| 8 | Study descriptions | Did the review authors describe the included studies in adequate detail? (PICO elements, follow-up period, study design, country, setting) | No |
| 9 | RoB assessment | Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? (For RCTs: randomization, blinding, missing data, selective reporting. For NRSI: confounding, selection, measurement) | Yes |
| 10 | Funding sources | Did the review authors report on the sources of funding for the studies included in the review? | No |
| 11 | Statistical methods | If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results? (effect measures, model choice, heterogeneity assessment) | Yes |
| 12 | RoB impact on MA | If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis? | No |
| 13 | RoB in interpretation | Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review? | Yes |
| 14 | Heterogeneity | Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? | No |
| 15 | Publication bias | If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? | Yes |
| 16 | Conflicts of interest | Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? | No |

---

## Overall Confidence Rating

AMSTAR 2 does NOT generate a numerical score. Instead, rate overall confidence:

| Rating | Criteria |
|--------|----------|
| **High** | No or one non-critical weakness: the systematic review provides an accurate and comprehensive summary of the results |
| **Moderate** | More than one non-critical weakness (but no critical flaws): the review provides an accurate summary but may have some weaknesses |
| **Low** | One critical flaw with or without non-critical weaknesses: the review may not provide an accurate and comprehensive summary |
| **Critically Low** | More than one critical flaw with or without non-critical weaknesses: the review should not be relied on to provide an accurate and comprehensive summary |

## Critical Domains (7 of 16)

Items 2, 4, 7, 9, 11, 13, 15 are considered **critical domains**. A flaw in any critical domain results in at least "Low" confidence.

## Notes for Assessors

- AMSTAR 2 replaces the original AMSTAR (2007)
- Designed for systematic reviews of **interventions** (RCTs and/or NRSI)
- Not designed for DTA reviews (use ROBIS or domain-specific tools)
- Cannot be used to assess individual primary studies
- The tool should NOT be used to generate an overall score — use the confidence rating scheme above
- For reviews including NRSI: Item 9 should assess confounding, selection bias, and information bias

# ARRIVE 2.0 Checklist — Animal Research

**Reference:** Percie du Sert N et al. The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. PLoS Biol. 2020;18(7):e3000410. PMID: 32663219

**Website:** https://arriveguidelines.org

---

Version: ARRIVE 2.0 (2020) — Essential 10 + Recommended Set
Source: Percie du Sert N, Hurst V, Ahluwalia A, Alam S, Avey MT, Baker M, et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. *PLoS Biol* 2020;18(7):e3000410 (DOI 10.1371/journal.pbio.3000410).
Licence: CC0 1.0 (public domain dedication) — confirmed via Crossref. Item text below is reproduced from the published Tables 1 and 2.
Verification: all 21 items were extracted from the article's own Table 1 (Essential 10) and Table 2 (Recommended Set) via the Europe PMC full text (PMC7360023) and compared item by item. **This file previously carried an invented item 19 "Limitations", renumbered items 20–21, and omitted item 21 "Declaration of interests" entirely**; the item text below is now the published text.

## How to Use This Checklist

ARRIVE 2.0 has two tiers:
- **Essential 10** (Items 1–10): the minimum that must be reported for a reader to assess the reliability of the findings.
- **Recommended Set** (Items 11–21): add context and completeness. Report these too whenever possible.

For each item: **PRESENT** / **PARTIAL** / **MISSING**

Where a "How to satisfy it" note appears below, it is **our guidance, not part of the instrument** — the **Description** is the published item text.

---

## ESSENTIAL 10

### Item 1 — Study design
**Description:** For each experiment, provide brief details of study design including:
a. The groups being compared, including control groups. If no control group has been used, the rationale should be stated.
b. The experimental unit (e.g., a single animal, litter, or cage of animals).

*How to satisfy it:* name the design (parallel group, crossover, factorial, dose–response), the groups and their sizes, and state explicitly what the experimental unit was — animal, litter, or cage.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 2 — Sample size
**Description:**
a. Specify the exact number of experimental units allocated to each group, and the total number in each experiment. Also indicate the total number of animals used.
b. Explain how the sample size was decided. Provide details of any a priori sample size calculation, if done.

*How to satisfy it:* a formal power calculation states the effect size, α, power, test, and software; a pragmatic justification (animal availability, pilot study) is acceptable but must be stated as such.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 3 — Inclusion and exclusion criteria
**Description:**
a. Describe any criteria used for including and excluding animals (or experimental units) during the experiment, and data points during the analysis. Specify if these criteria were established a priori. If no criteria were set, state this explicitly.
b. For each experimental group, report any animals, experimental units, or data points not included in the analysis and explain why. If there were no exclusions, state so.
c. For each analysis, report the exact value of n in each experimental group.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 4 — Randomisation
**Description:**
a. State whether randomisation was used to allocate experimental units to control and treatment groups. If done, provide the method used to generate the randomisation sequence.
b. Describe the strategy used to minimise potential confounders such as the order of treatments and measurements, or animal/cage location. If confounders were not controlled, state this explicitly.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 5 — Blinding
**Description:** Describe who was aware of the group allocation at the different stages of the experiment (during the allocation, the conduct of the experiment, the outcome assessment, and the data analysis).

*How to satisfy it:* state the position at each of the four stages, including where blinding was impossible and why.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 6 — Outcome measures
**Description:**
a. Clearly define all outcome measures assessed (e.g., cell death, molecular markers, or behavioural changes).
b. For hypothesis-testing studies, specify the primary outcome measure, i.e., the outcome measure that was used to determine the sample size.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 7 — Statistical methods
**Description:**
a. Provide details of the statistical methods used for each analysis, including software used.
b. Describe any methods used to assess whether the data met the assumptions of the statistical approach, and what was done if the assumptions were not met.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 8 — Experimental animals
**Description:**
a. Provide species-appropriate details of the animals used, including species, strain and substrain, sex, age or developmental stage, and, if relevant, weight.
b. Provide further relevant information on the provenance of animals, health/immune status, genetic modification status, genotype, and any previous procedures.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 9 — Experimental procedures
**Description:** For each experimental group, including controls, describe the procedures in enough detail to allow others to replicate them, including:
a. What was done, how it was done, and what was used.
b. When and how often.
c. Where (including detail of any acclimatisation periods).
d. Why (provide rationale for procedures).

*How to satisfy it:* anaesthesia and analgesia agents with dose and route, monitoring, equipment make/model/settings, and anything else that would change the result if varied.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

### Item 10 — Results
**Description:** For each experiment conducted, including independent replications, report:
a. Summary/descriptive statistics for each experimental group, with a measure of variability where applicable (e.g., mean and SD, or median and range).
b. If applicable, the effect size with a confidence interval.

Adverse events belong to item 16b, not here.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING
**Location:** ___

---

## RECOMMENDED SET

### Item 11 — Abstract
**Description:** Provide an accurate summary of the research objectives, animal species, strain and sex, key methods, principal findings, and study conclusions.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 12 — Background
**Description:**
a. Include sufficient scientific background to understand the rationale and context for the study, and explain the experimental approach.
b. Explain how the animal species and model used address the scientific objectives and, where appropriate, the relevance to human biology.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 13 — Objectives
**Description:** Clearly describe the research question, research objectives and, where appropriate, specific hypotheses being tested.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 14 — Ethical statement
**Description:** Provide the name of the ethical review committee or equivalent that has approved the use of animals in this study, and any relevant licence or protocol numbers (if applicable). If ethical approval was not sought or granted, provide a justification.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 15 — Housing and husbandry
**Description:** Provide details of housing and husbandry conditions, including any environmental enrichment.

*How to satisfy it:* cage type and dimensions, animals per cage, temperature, humidity, light cycle, food and water access and diet, acclimatisation period, enrichment.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 16 — Animal care and monitoring
**Description:**
a. Describe any interventions or steps taken in the experimental protocols to reduce pain, suffering, and distress.
b. Report any expected or unexpected adverse events.
c. Describe the humane endpoints established for the study, the signs that were monitored, and the frequency of monitoring. If the study did not have humane endpoints, state this.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 17 — Interpretation/scientific implications ⚠️ limitations belong here
**Description:**
a. Interpret the results, taking into account the study objectives and hypotheses, current theory, and other relevant studies in the literature.
b. Comment on the study limitations, including potential sources of bias, limitations of the animal model, and imprecision associated with the results.

ARRIVE 2.0 has **no standalone "Limitations" item** — limitations are sub-item 17b.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 18 — Generalisability/translation
**Description:** Comment on whether, and how, the findings of this study are likely to generalise to other species or experimental conditions, including any relevance to human biology (where appropriate).
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 19 — Protocol registration
**Description:** Provide a statement indicating whether a protocol (including the research question, key design features, and analysis plan) was prepared before the study, and if and where this protocol was registered.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 20 — Data access
**Description:** Provide a statement describing if and where study data are available.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### Item 21 — Declaration of interests
**Description:**
a. Declare any potential conflicts of interest, including financial and nonfinancial. If none exist, this should be stated.
b. List all funding sources (including grant identifier) and the role of the funder(s) in the design, analysis, and reporting of the study.
**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

---

## Summary

| Category | PRESENT | PARTIAL | MISSING |
|----------|---------|---------|---------|
| Essential 10 | /10 | /10 | /10 |
| Recommended Set | /11 | /11 | /11 |
| **TOTAL** | /21 | /21 | /21 |

**Verdict:** [ ] All Essential 10 PRESENT → proceed to submission
[ ] Any Essential 10 MISSING → MUST REVISE before submission

# CARE Checklist

**CAse REports (CARE) guidelines**
Version: CARE 2013
Source: https://www.care-statement.org
Reference: Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D. The CARE guidelines: consensus-based clinical case report guideline development. J Clin Epidemiol 2014;67(1):46-51.

Source: Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D, et al. The CARE guidelines: consensus-based clinical case report guideline development. *J Clin Epidemiol* 2014;67(1):46-51 (DOI 10.1016/j.jclinepi.2013.08.003).
Licence: Crossref returns only an Elsevier text-and-data-mining licence. A CC BY-NC 4.0 claim circulates for the CARE materials; it is **not confirmed** here. Treat as non-open.
Verification: all 13 topics and their 23 sub-items were compared against the official CARE 2013
checklist published at care-statement.org, and cross-checked against the checklist table of the
CC BY-NC co-publication in *Global Advances in Health and Medicine* (Europe PMC full text,
PMC3833570). The topics, their order and their sub-elements match. Wording stays paraphrased —
the licence is unconfirmed and treated as non-open.

## Checklist Items (13 topics)

### Title and Key Words

| # | Item | Description |
|---|------|-------------|
| 1 | Title | The words "case report" should appear in the title along with the diagnosis or intervention of primary focus. |
| 2 | Key Words | Two to five key words that identify topics in this case report, including "case report". |

### Abstract

| # | Item | Description |
|---|------|-------------|
| 3a | Abstract — Introduction | What is unique about this case and what does it add to the scientific literature? |
| 3b | Abstract — Main concerns | The patient's main concerns and important clinical findings. |
| 3c | Abstract — Diagnoses, interventions, outcomes | The primary diagnoses, interventions, and outcomes. |
| 3d | Abstract — Conclusion | What are one or more "take-away" lessons from this case report? |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 4 | Introduction | One or two paragraphs summarizing why this case is unique (may include references). |

### Patient Information

| # | Item | Description |
|---|------|-------------|
| 5a | Patient Information | De-identified demographic and other patient information. |
| 5b | Patient Information | Main concerns and symptoms of the patient. |
| 5c | Patient Information | Medical, family, and psychosocial history including relevant genetic information. |
| 5d | Patient Information | Relevant past interventions and their outcomes. |

### Clinical Findings

| # | Item | Description |
|---|------|-------------|
| 6 | Clinical Findings | Describe the significant physical examination and other clinical findings. |

### Timeline

| # | Item | Description |
|---|------|-------------|
| 7 | Timeline | Historical and current information from this episode of care organized as a timeline (figure or table). |

### Diagnostic Assessment

| # | Item | Description |
|---|------|-------------|
| 8a | Diagnostic Assessment | Diagnostic methods (e.g., physical examination, laboratory testing, imaging, questionnaires). |
| 8b | Diagnostic Assessment | Diagnostic challenges (e.g., access to testing, financial, or cultural). |
| 8c | Diagnostic Assessment | The diagnosis, including other diagnoses that were considered. |
| 8d | Diagnostic Assessment | Prognostic characteristics (e.g., staging) where applicable. |

### Therapeutic Intervention

| # | Item | Description |
|---|------|-------------|
| 9a | Therapeutic Intervention | Types of therapeutic intervention (e.g., pharmacologic, surgical, preventive, self-care). |
| 9b | Therapeutic Intervention | Administration of therapeutic intervention (e.g., dosage, strength, duration). |
| 9c | Therapeutic Intervention | Changes in therapeutic intervention (with rationale). |

### Follow-up and Outcomes

| # | Item | Description |
|---|------|-------------|
| 10a | Follow-up and Outcomes | Clinician- and patient-assessed outcomes (when appropriate). |
| 10b | Follow-up and Outcomes | Important follow-up diagnostic and other test results. |
| 10c | Follow-up and Outcomes | Intervention adherence and tolerability (and how this was assessed). |
| 10d | Follow-up and Outcomes | Adverse and unanticipated events. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 11a | Discussion | A scientific discussion of the strengths and limitations associated with this case report. |
| 11b | Discussion | Discussion of the relevant medical literature with references. |
| 11c | Discussion | The scientific rationale for any conclusions (including assessment of possible causes). |
| 11d | Discussion | The primary "take-away" lessons of this case report (without references) in a one-paragraph conclusion. |

### Patient Perspective

| # | Item | Description |
|---|------|-------------|
| 12 | Patient Perspective | The patient should share their perspective on the treatment(s) they received, when appropriate. |

### Informed Consent

| # | Item | Description |
|---|------|-------------|
| 13 | Informed Consent | Did the patient give informed consent? Provide if requested. |

---

## Applying CARE to common case-report subtypes

CARE covers the single-patient narrative. Two frequent subtypes need additions CARE alone does not name:

- **Adverse drug / device / contrast reaction (pharmacovigilance).** Beyond item 8 (diagnostic
  assessment), report a **named causality instrument** (Naranjo or WHO-UMC) with score and tier,
  document **dechallenge** (withdrawal → resolution) and the exposure-to-onset latency, and consider
  **severity** (e.g., Modified Hartwig–Siegel) and **preventability** (e.g., Schumock–Thornton).
  Locate the event against a denominator and note safety-reporting. Causality asserted without an
  instrument, dechallenge, or exclusion of alternatives is a reporting gap.
- **Case series (n≥2).** CARE is single-patient; a series additionally needs **cohort-style methods**
  (design, setting, case-identification source, eligibility, protocol) and an **all-cases summary
  table**. Report counts, not rates — a selected/referral series cannot estimate prevalence — and
  state selection/ascertainment with the screened pool size. See `/write-paper`
  `paper_types/case_series.md`.

*Educational summary of the CARE 2013 checklist (CC BY-NC 4.0). Cite the original guideline (Gagnier et al. 2014) and consult https://www.care-statement.org for the authoritative, full checklist with explanation and elaboration.*

# CHEERS 2022 Checklist

**Consolidated Health Economic Evaluation Reporting Standards 2022**
Version: CHEERS 2022 (28 items; replaces CHEERS 2013).
Source: Husereau D, Drummond M, Augustovski F, et al. *BMJ* 2022;376:e067975 (the CHEERS 2022 statement), co-published simultaneously across BMJ, *Value in Health*, *PharmacoEconomics*, *Int J Technol Assess Health Care* and others. CC BY 4.0. https://www.equator-network.org/reporting-guidelines/cheers/ · ISPOR CHEERS Task Force.
Verification: all 28 items were compared against Table 1 of the published statement (Europe PMC
full text, PMC8749494); 28/28 match, with no item missing and none invented.

Apply when the manuscript is a **health economic evaluation** — a comparative analysis of costs and consequences of two or more courses of action: cost-effectiveness (CEA), cost-utility (CUA), cost-benefit (CBA), or cost-minimisation analysis, whether trial-based or decision-model-based (decision tree, Markov/state-transition, discrete-event simulation), including budget-impact and HTA submissions. For the design/validity review of the same study, pair with the HE1–HE8 domain probes in `peer-review` / `self-review` `references/domain-probes/health_economic_evaluation.md`; for the analysis, with `analyze-stats` `references/analysis_guides/health_economic_evaluation.md`.

Source: Husereau D, Drummond M, Augustovski F, de Bekker-Grob E, Briggs AH, Carswell C, et al. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement. *BMJ* 2022;376:e067975 (DOI 10.1136/bmj-2021-067975).

## Checklist Items (28 items)

### Title

| # | Item | Description |
|---|------|-------------|
| 1 | Title | Identify the study as an economic evaluation and specify the interventions being compared. |

### Abstract

| # | Item | Description |
|---|------|-------------|
| 2 | Abstract | Provide a structured summary that highlights context, key methods, results, and alternative analyses. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 3 | Background and objectives | Give the context for the study, the study question, and its practical relevance for decision making in policy or practice. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 4 | Health economic analysis plan | Indicate whether a health economic analysis plan was developed and where available. |
| 5 | Study population | Describe characteristics of the study population (such as age range, demographics, socioeconomic, or clinical characteristics). |
| 6 | Setting and location | Provide relevant contextual information that may influence findings. |
| 7 | Comparators | Describe the interventions or strategies being compared and why chosen. |
| 8 | Perspective | State the perspective(s) adopted by the study and why chosen. |
| 9 | Time horizon | State the time horizon for the study and why appropriate. |
| 10 | Discount rate | Report the discount rate(s) and reason chosen. |
| 11 | Selection of outcomes | Describe what outcomes were used as the measure(s) of benefit(s) and harm(s). |
| 12 | Measurement of outcomes | Describe how outcomes used to capture benefit(s) and harm(s) were measured. |
| 13 | Valuation of outcomes | Describe the population and methods used to measure and value outcomes. |
| 14 | Measurement and valuation of resources and costs | Describe how costs were valued. |
| 15 | Currency, price date, and conversion | Report the dates of the estimated resource quantities and unit costs, plus the currency and year of conversion. |
| 16 | Rationale and description of model | If modelling is used, describe in detail and why used. Report whether the model is publicly available and where. |
| 17 | Analytics and assumptions | Describe any methods for analysing or statistically transforming data, any extrapolation methods, and approaches for validating any model used. |
| 18 | Characterising heterogeneity | Describe any methods used for estimating how the results of the study vary for subgroups. |
| 19 | Characterising distributional effects | Describe how impacts are distributed across different individuals or whether adjustments were made to reflect priority populations. |
| 20 | Characterising uncertainty | Describe methods to characterise any sources of uncertainty in the analysis. |
| 21 | Approach to engagement with patients and others affected by the study | Describe any approaches to engage patients or service recipients, the general public, communities, or stakeholders (such as clinicians or payers) in the design of the study. |

### Results

| # | Item | Description |
|---|------|-------------|
| 22 | Study parameters | Report all analytic inputs (such as values, ranges, references) including uncertainty or distributional assumptions. |
| 23 | Summary of main results | Report the mean values for the main categories of costs and outcomes of interest and summarise them in the most appropriate overall measure (e.g. the incremental cost-effectiveness ratio, ICER). |
| 24 | Effect of uncertainty | Describe how uncertainty about analytic judgments, inputs, or projections affect findings. Report the effect of choice of discount rate and time horizon, if relevant. |
| 25 | Effect of engagement with patients and others affected by the study | Report on any difference patient/service recipient, general public, community, or stakeholder involvement made to the approach or findings of the study. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 26 | Study findings, limitations, generalisability, and current knowledge | Report key findings, limitations, ethical or equity considerations, and how these could affect patients, policy, or practice. |

### Other relevant information

| # | Item | Description |
|---|------|-------------|
| 27 | Source of funding | Describe how the study was funded and the role of the funder in the identification, design, conduct, and reporting of the analysis. Describe other non-monetary sources of support. |
| 28 | Conflicts of interest | Describe any potential for conflict of interest among study contributors in accordance with journal policy. In the absence of a journal policy, we recommend authors comply with International Committee of Medical Journal Editors (ICMJE) recommendations. |

---

## Notes for Assessors

- **Highest-yield items** (where economic evaluations most often fail review): **8** (perspective stated and consistent with the costs counted — productivity/informal-care costs belong only to a societal perspective), **9** (time horizon long enough to capture all relevant differential costs and effects — a lifetime horizon for a chronic condition; a truncated horizon flatters whichever arm has early benefit), **10** (both costs *and* outcomes discounted at a stated, justified rate for any horizon beyond ~1 year), **15** (currency *and* price year stated, with the conversion method for multi-source costs), **16–17** (model type/structure justified and validated; structural assumptions and extrapolation declared), and **20 / 24** (uncertainty characterised by *probabilistic* sensitivity analysis — a cost-effectiveness acceptability curve / plane — not a single deterministic ICER). An evaluation that reports a point-estimate ICER with no probabilistic sensitivity analysis is non-compliant on items 20/24.
- **CHEERS 2022 replaced CHEERS 2013**; cite the 2022 statement (do not cite the 2013 version as current). CHEERS 2022 added explicit items on the analysis plan (4), distributional/equity effects (19), and patient/stakeholder engagement (21/25); the engagement items are reported as "not done" rather than omitted when no engagement occurred.
- ICER interpretation is not a CHEERS item per se but follows from items 23–24: incremental costs and effects must be reported (not just the ratio), dominance/extended dominance resolved, and the ICER interpreted against a *stated, justified* cost-effectiveness threshold (willingness-to-pay) rather than an arbitrary one. The HE1–HE8 design probes cover these judgments.
- This checklist was authored as a faithful summary of the CHEERS 2022 statement (Husereau D, et al. *BMJ* 2022;376:e067975, **CC BY 4.0** — the item list and Explanation & Elaboration are reusable/adaptable with attribution) for item-by-item assessment; verify against the published statement and its Explanation & Elaboration for full item wording. Verified 2026-06-29.

# CLAIM 2024 Checklist

**Checklist for Artificial Intelligence in Medical Imaging**
Version: CLAIM 2024 Update
Source: https://pubs.rsna.org/doi/10.1148/ryai.240300
Reference: Tejani AS, Klontzas ME, Gatti AA, Mongan JT, Moy L, Park SH, Kahn CE Jr. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiol Artif Intell 2024;6(4):e240300.

> Note: The 2024 update replaces "ground truth" with "reference standard" and discourages "validation" in favour of "internal/external testing". Each item is answered Yes / No / Not Applicable with the manuscript location cited.

Licence: © RSNA, open access. Consult RSNA for reuse terms; Crossref returns no Creative Commons licence.
Verification: all 44 items were compared, by number and content, against the item-by-item text of
the published update (PubMed Central record PMC11304031, whose body carries each item in full).
44/44 match, with no item missing and none invented. Item labels below are our own short names;
item 21's official name is "Intended sample size" and it does concern the testing set, as our
description says. Wording stays paraphrased — Crossref returns no Creative Commons licence.

## Checklist Items (44 items)

### Title and Abstract

| # | Item | Description |
|---|------|-------------|
| 1 | Title | Identify the study as employing AI methodology and name the specific technology category (e.g., deep learning). |
| 2 | Abstract | Structured summary including study design, methods, results, and conclusions; population details, data partitions, prospective/retrospective status, statistical analysis, outcomes, and availability of resources. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 3 | Background | Scientific and clinical background, current practice, intended use, and clinical role of the AI approach. |
| 4 | Objectives | Study aims, objectives, and hypotheses (if not data-driven). |

### Methods — Study Design

| # | Item | Description |
|---|------|-------------|
| 5 | Study design | Prospective or retrospective study. |
| 6 | Study goal | Goal of the study (e.g., model creation, feasibility, trial type, intended use for the classification task). |

### Methods — Data

| # | Item | Description |
|---|------|-------------|
| 7 | Data sources | State data sources; provide links to publicly available datasets. |
| 8 | Eligibility | Inclusion and exclusion criteria and selection methodology for the data. |
| 9 | Preprocessing | Data preprocessing steps (e.g., normalization, resampling, window/level adjustment). |
| 10 | Subset selection | Selection of data subsets and training of personnel involved. |
| 11 | De-identification | De-identification methods meeting HIPAA/GDPR/AI Act standards. |
| 12 | Missing data | How missing data were handled and potential biases from imputation. |
| 13 | Acquisition protocol | Image acquisition protocol parameters (e.g., manufacturer, sequences, resolution). |

### Methods — Reference Standard

| # | Item | Description |
|---|------|-------------|
| 14 | Reference standard definition | Method for obtaining the reference standard, with precise and replicable definitions. |
| 15 | Reference standard rationale | Rationale for choosing the reference standard versus alternatives. |
| 16 | Annotators | Source, qualifications, and training materials of annotators. |
| 17 | Annotation procedures | Test-set annotation procedures, software version, and any NLP/automated approaches. |
| 18 | Annotation variability | Measurement of inter- and intra-rater variability and method of discrepancy resolution. |

### Methods — Data Partitions

| # | Item | Description |
|---|------|-------------|
| 19 | Partition assignment | Partition assignment (train/tune/test), proportions, justification, and class-imbalance handling. |
| 20 | Partition disjointness | Level of partition disjointness (patient-, series-, or image-level). |

### Methods — Testing Data

| # | Item | Description |
|---|------|-------------|
| 21 | Test set size | Testing-set size derived from a power calculation or AUC-based estimation. |

### Methods — Model

| # | Item | Description |
|---|------|-------------|
| 22 | Model architecture | Complete model architecture (inputs, outputs, layers, pooling, normalization). |
| 23 | Software | Software libraries, frameworks, packages, and version numbers. |
| 24 | Initialization | Parameter initialization; transfer-learning sources, if used. |

### Methods — Training

| # | Item | Description |
|---|------|-------------|
| 25 | Training procedures | Training procedures, data augmentation, convergence monitoring, and all hyperparameters. |
| 26 | Model selection | Method and metrics for selecting the best-performing model. |
| 27 | Ensembling | If an ensemble approach is used, details of each model and how outputs are combined. |

### Methods — Evaluation

| # | Item | Description |
|---|------|-------------|
| 28 | Performance metrics | Performance metrics and comparison to published models. |
| 29 | Uncertainty | Measures of uncertainty (e.g., standard deviation, confidence intervals) and statistical significance tests. |
| 30 | Robustness | Robustness or sensitivity analysis. |
| 31 | Explainability | If applied, explainability/interpretability methods and their validation. |
| 32 | Internal testing | Internal-data evaluation and consistency between training and test performance. |
| 33 | External testing | External-data testing, or justification for its omission. |
| 34 | Trial registration | If applicable, compliance with ICMJE clinical-trial registration requirements. |

### Results — Data

| # | Item | Description |
|---|------|-------------|
| 35 | Inclusion/exclusion numbers | Numbers of patients/examinations included and excluded, with a flowchart. |
| 36 | Demographics | Demographic and clinical characteristics per partition; identify potential sources of bias. |

### Results — Model Performance

| # | Item | Description |
|---|------|-------------|
| 37 | Performance reporting | Final model performance benchmarked against the reference standard across partitions and subgroups. |
| 38 | Accuracy estimates | Diagnostic accuracy estimates with 95% confidence intervals; ROC analysis; address class imbalance. |
| 39 | Failure analysis | Failure analysis with a confusion matrix; examples of incorrect classifications in the medical context. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 40 | Limitations | Study limitations (methods, materials, biases, generalisability). |
| 41 | Implications | Clinical implications, intended use, practice changes, and barriers to translation. |

### Other Information

| # | Item | Description |
|---|------|-------------|
| 42 | Full protocol | Reference to the full protocol or technical details if exceeding journal word limits. |
| 43 | Availability | Availability of software, model, and data, and access conditions. |
| 44 | Funding | Funding sources and the role of funders. |

---

*Educational summary of the CLAIM 2024 Update checklist (© RSNA, open access). Cite the original (Tejani et al., Radiol Artif Intell 2024;6(4):e240300) and consult the RSNA article for the authoritative, full checklist.*

# CLEAR Checklist

**CheckList for EvaluAtion of Radiomics research**

- **Version:** CLEAR 2023
- **Citation:** Kocak B, Baessler B, Bakas S, et al. *CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII.* Insights Imaging. 2023;14(1):75.
- **DOI:** 10.1186/s13244-023-01415-8
- **Source:** https://pmc.ncbi.nlm.nih.gov/articles/PMC10160267/ · official item list: https://clearchecklist.github.io/clear_checklist/CLEAR.html
- **Licence:** CC BY 4.0. Item wording below is reproduced faithfully from the published statement with attribution.

CLEAR is a **58-item** step-by-step reporting guideline for radiomics research, ordered by **manuscript
section** to follow a paper from title to open science: Title (1), Abstract (2), Keywords (3),
Introduction (4–6), Methods (7–43), Results (44–48), Discussion (49–52), and Open Science (53–58). The
Methods block is subdivided into Study design (7–12), Data (13–18), Segmentation (19–20), Pre-processing
(21–24), Feature extraction (25–28), Data preparation (29–33), Modeling (34–37), and Evaluation (38–43).

Two items — **53** and **58** — are marked **[n/e]** ("not essential" but recommended). All other items are
essential; score a missing essential item as a reporting gap, and a missing [n/e] item as N/A only when
justified. CLEAR is written for **hand-crafted radiomics**; for deep-learning pipelines without radiomic
features, CLAIM 2024 or TRIPOD+AI may fit better (a study that does both should be assessed against both).

Verification: all 58 items were compared against Table 1 of the published checklist (Europe PMC
full text, PMC10160267); 58/58 match, with no item missing and none invented.

## Checklist Items

### Title

| Item | Checklist item |
|---|---|
| 1 | Relevant title, specifying the radiomic methodology (generally identifying the study as radiomics-related). |

### Abstract

| Item | Checklist item |
|---|---|
| 2 | Structured summary with relevant information (with a structured or unstructured summary presenting key information). |

### Keywords

| Item | Checklist item |
|---|---|
| 3 | Relevant keywords for radiomics (providing keywords most relevant to the topic). |

### Introduction

| Item | Checklist item |
|---|---|
| 4 | Scientific or clinical background (mentioning the current scientific or clinical background). |
| 5 | Rationale for using a radiomic approach (explaining the rationale for using a radiomic approach). |
| 6 | Study objective(s) (stating the study objectives, hypotheses, or aims). |

### Methods — Study design

| Item | Checklist item |
|---|---|
| 7 | Adherence to guidelines or checklists (e.g., CLEAR checklist). |
| 8 | Ethical details (e.g., approval, consent, data protection). |
| 9 | Sample size calculation (with a statistical power analysis, if performed). |
| 10 | Study nature (e.g., retrospective, prospective). |
| 11 | Eligibility criteria (with inclusion and exclusion criteria). |
| 12 | Flowchart for technical pipeline (presenting a technical pipeline flowchart). |

### Methods — Data

| Item | Checklist item |
|---|---|
| 13 | Data source (e.g., private, public). |
| 14 | Data overlap (declaring any data overlap with previous studies). |
| 15 | Data split methodology (describing how the data were split, e.g., training/validation/test). |
| 16 | Imaging protocol (i.e., image acquisition and processing). |
| 17 | Definition of non-radiomic predictor variables. |
| 18 | Definition of the reference standard (i.e., outcome variable). |

### Methods — Segmentation

| Item | Checklist item |
|---|---|
| 19 | Segmentation strategy (2D/3D, manual/automatic, software, region of interest). |
| 20 | Details of operators performing segmentation (number, experience, qualifications). |

### Methods — Pre-processing

| Item | Checklist item |
|---|---|
| 21 | Image pre-processing details. |
| 22 | Resampling method and its parameters. |
| 23 | Discretization method and its parameters (e.g., fixed bin width or count). |
| 24 | Image types (e.g., original, filtered, transformed). |

### Methods — Feature extraction

| Item | Checklist item |
|---|---|
| 25 | Feature extraction method (software and version). |
| 26 | Feature classes (e.g., shape, first-order, texture). |
| 27 | Number of features (extracted per region and in total). |
| 28 | Default configuration statement for remaining parameters. |

### Methods — Data preparation

| Item | Checklist item |
|---|---|
| 29 | Handling of missing data. |
| 30 | Details of class imbalance. |
| 31 | Details of segmentation reliability analysis (e.g., inter-/intra-observer agreement). |
| 32 | Feature scaling details (e.g., normalization, standardization). |
| 33 | Dimension reduction details (e.g., feature selection). |

### Methods — Modeling

| Item | Checklist item |
|---|---|
| 34 | Algorithm details (name and characteristics of the modeling algorithm[s]). |
| 35 | Training and tuning details (including hyperparameter optimization). |
| 36 | Handling of confounders. |
| 37 | Model selection strategy. |

### Methods — Evaluation

| Item | Checklist item |
|---|---|
| 38 | Testing technique (e.g., internal, external). |
| 39 | Performance metrics and rationale for choosing. |
| 40 | Uncertainty evaluation and measures (e.g., confidence intervals). |
| 41 | Statistical performance comparison (e.g., DeLong's test). |
| 42 | Comparison with non-radiomic and combined methods. |
| 43 | Interpretability and explainability methods. |

### Results

| Item | Checklist item |
|---|---|
| 44 | Baseline demographic and clinical characteristics (across data partitions). |
| 45 | Flowchart for eligibility criteria (participant flow). |
| 46 | Feature statistics (e.g., reproducibility, feature selection). |
| 47 | Model performance evaluation (with the pre-specified metrics). |
| 48 | Comparison with non-radiomic and combined approaches. |

### Discussion

| Item | Checklist item |
|---|---|
| 49 | Overview of important findings. |
| 50 | Previous works with differences from the current study. |
| 51 | Practical implications. |
| 52 | Strengths and limitations (e.g., bias and generalizability issues). |

### Open Science — Data availability

| Item | Checklist item |
|---|---|
| 53 | Sharing images along with segmentation data **[n/e]**. |
| 54 | Sharing radiomic feature data. |

### Open Science — Code availability

| Item | Checklist item |
|---|---|
| 55 | Sharing pre-processing scripts or settings. |
| 56 | Sharing source code for modeling. |

### Open Science — Model availability

| Item | Checklist item |
|---|---|
| 57 | Sharing final model files. |
| 58 | Sharing a ready-to-use system **[n/e]**. |

---

## Notes for assessors

- **Order is by manuscript section, not by topic.** CLEAR numbers items in the order they appear in a paper
  (Title → Abstract → … → Open Science). When you cite a CLEAR item, cite it by this official number — item 1
  is the title, item 44 is baseline demographics in the Results, item 58 is sharing a ready-to-use system.
- **Items 53 and 58 are the only non-essential ([n/e]) items** — recommended best practice for open science.
  Every other item is essential; a missing essential item is a reporting gap, not an optional extra.
- **Segmentation items (19–20)** may be N/A for studies using fully automated, atlas-based segmentation with
  no reader involvement — note this rather than scoring MISSING.
- **Open Science (53–58)** is where a radiomics study's reproducibility is assessed and is increasingly
  required by journals; missing items here commonly draw reviewer comments.
- CLEAR was endorsed by the European Society of Radiology (ESR) and the European Society of Medical Imaging
  Informatics (EuSoMII). For methodological quality (as opposed to reporting completeness), pair CLEAR with
  METRICS (Kocak et al. 2024).

# CONSORT 2025 Checklist

**Consolidated Standards of Reporting Trials**
Version: CONSORT 2025
Source: https://www.consort-spirit.org
Reference: Hopewell S, Chan AW, Collins GS, et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ 2025;389:e081123 (published simultaneously in BMJ, JAMA, Lancet, Nature Medicine, PLoS Medicine).

> Note: CONSORT 2025 supersedes CONSORT 2010. It is a 30-item checklist (seven new items, three revised, one deleted) restructured with a new Open Science section.

Source: Hopewell S, Chan AW, Collins GS, Hróbjartsson A, Moher D, Schulz KF, et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. *BMJ* 2025;389:e081123 (DOI 10.1136/bmj-2024-081123).
Licence: CC BY 4.0 — confirmed via Crossref.
Verification: all 42 sub-items were compared against Table 1 of the published statement (Europe
PMC full text, PMC11995449); 42/42 match. Item 13's label and its second sentence, which were
missing, have been restored.

## Checklist Items (30 items)

### Title and Abstract

| # | Item | Description |
|---|------|-------------|
| 1a | Title | Identification as a randomised trial. |
| 1b | Abstract | Structured summary of the trial design, methods, results, and conclusions. |

### Open Science

| # | Item | Description |
|---|------|-------------|
| 2 | Trial registration | Name of trial registry, identifying number (with URL) and date of registration. |
| 3 | Protocol and SAP | Where the trial protocol and statistical analysis plan can be accessed. |
| 4 | Data, code, materials | Where and how the individual de-identified participant data (including data dictionary), statistical code and any other materials can be accessed. |
| 5a | Funding | Sources of funding and other support (e.g., supply of drugs), and role of funders in the design, conduct, analysis and reporting of the trial. |
| 5b | Conflicts of interest | Financial and other conflicts of interest of the manuscript authors. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 6 | Background | Scientific background and rationale. |
| 7 | Objectives | Specific objectives related to benefits and harms. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 8 | Patient and public involvement | Details of patient or public involvement in the design, conduct and reporting of the trial. |
| 9 | Trial design | Description of trial design including type of trial (e.g., parallel group, crossover), allocation ratio, and framework. |
| 10 | Changes to trial | Important changes to the trial after it commenced including any outcomes or analyses that were not prespecified, with reason. |
| 11 | Settings and locations | Settings (e.g., community, hospital) and locations (e.g., countries, sites) where the trial was conducted. |
| 12a | Eligibility — participants | Eligibility criteria for participants. |
| 12b | Eligibility — sites/deliverers | If applicable, eligibility criteria for sites and for individuals delivering the interventions. |
| 13 | Intervention and comparator | Intervention and comparator with sufficient details to allow replication. If relevant, where additional materials describing the intervention and comparator (e.g., intervention manual) can be accessed. |
| 14 | Outcomes | Prespecified primary and secondary outcomes, including the specific measurement variable, analysis metric, method of aggregation, and time point for each outcome. |
| 15 | Harms | How harms were defined and assessed (e.g., systematically, non-systematically). |
| 16a | Sample size | How sample size was determined, including all assumptions supporting the sample size calculation. |
| 16b | Interim analyses | Explanation of any interim analyses and stopping guidelines. |
| 17a | Randomisation — sequence | Who generated the random allocation sequence and the method used. |
| 17b | Randomisation — restriction | Type of randomisation and details of any restriction (e.g., stratification, blocking and block size). |
| 18 | Allocation concealment | Mechanism used to implement the random allocation sequence (e.g., central computer/telephone; sequentially numbered, opaque, sealed containers). |
| 19 | Implementation | Whether the personnel who enrolled and those who assigned participants to the interventions had access to the random allocation sequence. |
| 20a | Blinding — who | Who was blinded after assignment to interventions (e.g., participants, care providers, outcome assessors, data analysts). |
| 20b | Blinding — how | If blinded, how blinding was achieved and description of the similarity of interventions. |
| 21a | Statistical methods | Statistical methods used to compare groups for primary and secondary outcomes, including harms. |
| 21b | Analysis populations | Definition of who is included in each analysis (e.g., all randomised participants), and in which group. |
| 21c | Missing data | How missing data were handled in the analysis. |
| 21d | Additional analyses | Methods for any additional analyses (e.g., subgroup and sensitivity analyses), distinguishing prespecified from post hoc. |

### Results

| # | Item | Description |
|---|------|-------------|
| 22a | Participant flow — numbers | For each group, the numbers of participants who were randomly assigned, received intended intervention, and were analysed for the primary outcome. |
| 22b | Participant flow — losses | For each group, losses and exclusions after randomisation, together with reasons. |
| 23a | Recruitment — dates | Dates defining the periods of recruitment and follow-up for outcomes of benefits and harms. |
| 23b | Recruitment — stopping | If relevant, why the trial ended or was stopped. |
| 24a | Intervention as administered | Intervention and comparator as they were actually administered (e.g., where appropriate, who delivered the intervention/comparator, how participants adhered, whether they were delivered as intended). |
| 24b | Concomitant care | Concomitant care received during the trial for each group. |
| 25 | Baseline data | A table showing baseline demographic and clinical characteristics for each group. |
| 26 | Outcomes and estimation | For each primary and secondary outcome, by group: the number of participants included in the analysis, the number with available data at the outcome time point, result for each group, and the estimated effect size and its precision. |
| 27 | Harms | All harms or unintended events in each group. |
| 28 | Ancillary analyses | Any other analyses performed, including subgroup and sensitivity analyses, distinguishing pre-specified from post hoc. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 29 | Interpretation | Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence. |
| 30 | Limitations | Trial limitations, addressing sources of potential bias, imprecision, generalisability, and, if relevant, multiplicity of analyses. |

---

*Educational summary of the CONSORT 2025 checklist (CC BY 4.0). Cite the original statement (Hopewell et al., BMJ 2025) and consult https://www.consort-spirit.org for the authoritative, full checklist with explanation and elaboration.*

# CONSORT-AI Checklist

**Consolidated Standards of Reporting Trials -- Artificial Intelligence Extension**
Version: CONSORT-AI 2020 (extends CONSORT 2010)
Source: https://www.consort-spirit.org · EQUATOR Network
Reference: Liu X, Cruz Rivera S, Moher D, et al. Nat Med 2020;26(9):1364-1374. doi:10.1038/s41591-020-1034-x (CC BY 4.0)

> Educational summary, authored in our own words from the CC BY 4.0 source. Use the official
> CONSORT-AI checklist for a submission-ready form and cite Liu et al. 2020.

> **Verified against the published statement.** All 14 AI items were enumerated from the article's
> own checklist table via the PMC XML and compared label-by-label: **11 Extensions and
> 3 Elaborations**, matching exactly with no item missing and none invented.

> **Extension vs Elaboration — the statement distinguishes them, and it matters.** An **Extension**
> is a *new* reporting requirement that CONSORT 2010 does not contain. An **Elaboration** clarifies how an
> existing CONSORT 2010 item applies when the intervention involves AI; the requirement already existed,
> the guidance is what is new. Both are assessed, but only the Extensions are additional obligations —
> do not report an Elaboration as though the base instrument had been silent on it. Elaborations are
> marked **(E)** below.

## Naming and scope (read first)

- CONSORT-AI is an **extension** of **CONSORT 2010**, for reports of randomized clinical trials of
  interventions that **include an AI/ML component**. Apply **both**: every base CONSORT 2010 item
  plus the AI-specific items below; name and cite both instruments (manuscript-style-classical §14).
- It is the **reports** counterpart of **SPIRIT-AI** (trial protocols). For a trial protocol use
  SPIRIT-AI; for the completed-trial report use CONSORT-AI.
- The AI items elaborate existing CONSORT items (numbered to match), so assess them alongside the
  parent item.

## AI-specific extension items

Status each PRESENT / PARTIAL / MISSING / N/A.

### Title and Abstract

| # | Item | Description (intent) |
|---|------|----------------------|
| 1a,b (i) **(E)** | AI identification | State in the title/abstract that the intervention involves AI/ML and specify the type of model. |
| 1a,b (ii) **(E)** | Intended use | State the intended use of the AI intervention in the title/abstract. |

### Introduction — Background and objectives

| # | Item | Description (intent) |
|---|------|----------------------|
| 2a (i) | Intended use in context | Explain the intended use of the AI intervention in the context of the clinical pathway, including its purpose and the intended user. |

### Methods

| # | Item | Description (intent) |
|---|------|----------------------|
| 4a (i) **(E)** | Participant eligibility | State the participant-level inclusion and exclusion criteria. |
| 4a (ii) | Input-data eligibility | State the inclusion and exclusion criteria at the level of the **input data** to the AI system. |
| 4b | Setting integration | Describe how the AI intervention was integrated into the trial setting, including any onsite or offsite requirements. |
| 5 (i) | Algorithm version | State which version of the AI algorithm was used. |
| 5 (ii) | Input acquisition | Describe how the input data were acquired and selected for the AI intervention. |
| 5 (iii) | Poor/unavailable input | Describe how poor-quality or unavailable input data were assessed and handled. |
| 5 (iv) | Human–AI interaction | Specify whether there is human–AI interaction in the handling of the input data, and the expertise required of the user. |
| 5 (v) | AI output | Specify the output of the AI intervention. |
| 5 (vi) | Output to decision | Explain how the AI intervention's outputs contributed to decision-making or other elements of clinical practice. |

### Results

| # | Item | Description (intent) |
|---|------|----------------------|
| 19 | Performance errors | Describe the results of any analysis of performance errors and how errors were identified; if none was done, justify why. |

### Other Information

| # | Item | Description (intent) |
|---|------|----------------------|
| 25 | Code/intervention access | State whether and how the AI intervention and/or its code can be accessed, including any restrictions on access or reuse. |

---

## Notes for Assessors

- Apply CONSORT-AI **with** all base CONSORT 2010 items — the AI items do not replace them.
- **Algorithm version (5 (i))** is non-waivable: a trial result tied to an unspecified model version is
  not interpretable or reproducible. Mark MISSING if absent.
- **Input-data eligibility (4a (ii))** is distinct from participant eligibility — a frequent omission;
  a trial can enroll eligible patients yet feed the AI out-of-distribution inputs.
- **Human–AI interaction (5 (iv))** and **how outputs fed decisions (5 (vi))** determine whether the
  trial evaluated the AI as used in practice; vague "the model assisted clinicians" is PARTIAL.
- **Performance-error analysis (19)** and **code accessibility (25)** are commonly dropped and are
  where AI-specific safety/reproducibility live.
- Use **SPIRIT-AI** for the trial protocol; CONSORT-AI is for the completed-trial report.

# COREQ Checklist (qualitative — interviews & focus groups)

**Consolidated criteria for reporting qualitative research**
Version: COREQ 2007 — 32 items in 3 domains. **Specific to in-depth interviews and focus groups** (the dominant qualitative data-collection methods in health research). For other qualitative approaches, use the broader **SRQR** (`SRQR.md`).
Source: Tong A, Sainsbury P, Craig J. *Int J Qual Health Care* 2007;19(6):349–357 (the COREQ statement; DOI 10.1093/intqhc/mzm042). EQUATOR Network.

Apply when the manuscript reports an **interview or focus-group** qualitative study. For the design/conduct review of the same study, pair with the QL1–QL8 domain probes in `peer-review` / `self-review` `references/domain-probes/qualitative_research.md`; for a non-interview qualitative approach (ethnography, document analysis, etc.), use `SRQR.md`.

> Licensing note: COREQ is published in the *International Journal for Quality in Health Care* (© Oxford University Press), with **no Creative Commons licence**. The items below are an **in-house, faithful summary of the criteria (facts/intents, paraphrased — not the verbatim COREQ wording)** for item-by-item assessment; consult the published article (DOI 10.1093/intqhc/mzm042) for exact item text.

## Criteria (grouped by domain)

### Domain 1 — Research team and reflexivity
**Personal characteristics**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 1 | Interviewer / facilitator | Which author(s) conducted the interviews or focus groups. |
| 2 | Credentials | The interviewer's/facilitator's credentials (e.g. PhD, MD, RN). |
| 3 | Occupation | Their occupation/role at the time of the study. |
| 4 | Gender | The researcher's gender (where relevant to the dynamic). |
| 5 | Experience & training | The interviewer's experience and training in qualitative methods. |

**Relationship with participants**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 6 | Relationship established | Whether a relationship existed between researcher and participants **before** the study. |
| 7 | Participant knowledge of the interviewer | What participants knew about the researcher (e.g. personal goals, reasons for the research). |
| 8 | Interviewer characteristics | Reported researcher characteristics — assumptions, interests, reasons for doing the research. |

### Domain 2 — Study design
**Theoretical framework**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 9 | Methodological orientation & theory | The methodological orientation/theory underpinning the study (e.g. grounded theory, content analysis, phenomenology). |

**Participant selection**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 10 | Sampling | How participants were selected (purposive, convenience, consecutive, snowball). |
| 11 | Method of approach | How participants were approached (face-to-face, telephone, mail, email). |
| 12 | Sample size | The number of participants. |
| 13 | Non-participation | How many declined or dropped out, and why (where known). |

**Setting**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 14 | Setting of data collection | Where the data were collected (home, clinic, workplace). |
| 15 | Presence of non-participants | Whether anyone besides participants and researchers was present. |
| 16 | Description of sample | The sample's key characteristics (e.g. demographics, dates). |

**Data collection**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 17 | Interview guide | Whether questions/prompts/guides were provided, and whether they were pilot-tested. |
| 18 | Repeat interviews | Whether any interviews were repeated, and how many. |
| 19 | Audio/visual recording | Whether the data were audio- or video-recorded. |
| 20 | Field notes | Whether field notes were made during/after the interview or focus group. |
| 21 | Duration | The duration of the interviews or focus groups. |
| 22 | Data saturation | Whether **data saturation** was discussed/reached. |
| 23 | Transcripts returned | Whether participants received the transcripts to review, comment on, or correct. |

### Domain 3 — Analysis and findings
**Data analysis**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 24 | Number of data coders | How many coders coded the data. |
| 25 | Description of the coding tree | Whether a description of the coding tree/framework is provided. |
| 26 | Derivation of themes | Whether themes were identified in advance or derived from the data. |
| 27 | Software | Any software used to manage/analyse the data. |
| 28 | Participant checking | Whether participants provided feedback on the findings (**member checking**). |

**Reporting**
| # | Item | What to check is reported |
|---|------|---------------------------|
| 29 | Quotations presented | Whether participant **quotations** illustrate the themes/findings, and whether each is identified (e.g. participant number). |
| 30 | Data & findings consistent | Whether the reported findings cohere with the underlying data shown. |
| 31 | Clarity of major themes | Whether the major themes are presented clearly in the results. |
| 32 | Clarity of minor themes | Whether deviant/diverse cases and minor themes are addressed, not only the dominant ones. |

---

## Notes for Assessors

- The **highest-yield** checks (where interview/focus-group studies most often fail review): **Domain 1 reflexivity** (items 1–8 — who interviewed, their relationship to participants, and their assumptions; the single most-omitted COREQ domain), **item 9** (a named methodological orientation — not "themes emerged" with no method), **items 10/22** (a stated sampling approach and a **saturation** discussion), **items 24–28** (the coding/analysis process — how many coders, the coding framework, software, member checking), and **item 29** (themes substantiated by **identified participant quotations**).
- **Do not apply quantitative criteria**: a small purposive sample is appropriate, "generalizability" is **transferability**, and there are no power/effect-size/p-value requirements. A "sample too small / not generalizable" comment is mis-calibrated for an interview study.
- COREQ is interview/focus-group-specific; for ethnography, document analysis, or other qualitative approaches use `SRQR.md`. This is an **in-house faithful summary of the COREQ criteria (paraphrased intents, not verbatim)**; map the manuscript's content to the items rather than to exact wording, and consult the published checklist (Tong et al. *Int J Qual Health Care* 2007; DOI 10.1093/intqhc/mzm042) for the exact text.
- Verification: all 32 items were compared, by number, domain and topic, against the official COREQ checklist hosted by the EQUATOR Network (the Souza et al. 2021 Brazilian-Portuguese translation, `VERSÃO-FINAL-COREQ.pdf`, which carries the full numbered table). 32/32 match across the three domains, with no item missing and none invented. The English original is behind a paywall and the OUP PDF blocks automated retrieval, so item *wording* has been checked only through that translation; the numbering, domains and topics are confirmed.

# COSMIN Risk of Bias Assessment Guide

COnsensus-based Standards for the selection of health Measurement INstruments — Risk of Bias tool for reliability and measurement error.
Reference: Mokkink LB et al. BMC Medical Research Methodology 2020;20:293.
Website: https://www.cosmin.nl

Version: COSMIN Risk of Bias tool for reliability and measurement error (2020 Delphi study)
Source: Mokkink LB, Boers M, van der Vleuten CPM, Bouter LM, Alonso J, Patrick DL, et al. COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study. *BMC Med Res Methodol* 2020;20:293 (DOI 10.1186/s12874-020-01179-5). The four-point rating system and the 'worst score counts' principle: Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, Terwee CB. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. *Qual Life Res* 2018;27(5):1171-1179 (DOI 10.1007/s11136-017-1765-4). Review workflow: Prinsen CAC, et al. *Qual Life Res* 2018;27(5):1147-1157 (DOI 10.1007/s11136-018-1798-3).
Licence: CC BY 4.0 for all three — confirmed via Crossref.
Verification: the seven Part A elements and every Part B standard were compared against Tables 4–7
of the 2020 Delphi paper (Europe PMC full text, PMC7712525). **The counts were right and the
standards were not**: this file previously carried invented standards for *Missing data*, *Sample
size* ("minimum 30 recommended, 50+ preferred") and *Reporting* — the words "missing" and "sample
size" appear nowhere in the source, and the 2018 paper states that standards concerning reporting
only were deleted. Official design standard 6 and statistical standards 8 and 9 were absent. The
SOURCE line also cited Prinsen 2018, which does not contain these standards.

## Purpose

The COSMIN Risk of Bias tool assesses the methodological quality of studies on **reliability** and **measurement error** of outcome measurement instruments (e.g., questionnaires, imaging measurements, lab tests).

## Structure

Two parts:
- **Part A**: Understanding how the study informs on reliability/measurement error (7 elements of a comprehensive research question)
- **Part B**: Assessing quality using standards (9 for reliability, 8 for measurement error)

Quality rating per standard: Very good / Adequate / Doubtful / Inadequate. Each standard also
carries **NA**.

Overall quality uses the **worst score counts** principle — the lowest rating of any standard in
the box (Mokkink 2018). There is no averaging and no summary score.

Note what is **not** here: this tool has no sample-size standard, no missing-data standard and no
reporting standard. Standards that concerned reporting only were deliberately removed when the
COSMIN checklist became a risk-of-bias checklist.

## Part A: Elements of a Comprehensive Research Question

Extract these 7 elements from the study:

| # | Element |
|---|---------|
| 1 | Name of the outcome measurement instrument |
| 2 | Version or operationalization of the measurement protocol |
| 3 | Construct measured by the instrument |
| 4 | Reliability parameter (ICC, kappa, etc.) or measurement error parameter (SEM, LoA, SDC) |
| 5 | Components of the instrument that will be repeated |
| 6 | Source(s) of variation that will be varied (time, rater, machine, etc.) |
| 7 | Patient population studied |

## Components of Outcome Measurement Instruments

Element 5 of the research question asks which **components** are repeated. The Delphi panel agreed
on five, in two variants.

### Without biological sampling
1. **Equipment** — all equipment used in preparation, administration, and assigning scores
2. **Preparatory actions** — 'first time only' general actions (required expertise or training) and actions repeated for each measurement
3. **Unprocessed data collection** — what the patient and/or professional(s) actually do to obtain the unprocessed data
4. **Data processing and storage** — all actions on the unprocessed data that allow a score to be assigned
5. **Assignment of the score** — methods used to transform processed data into a final score

### With biological sampling
1. **Equipment** — all equipment used in preparation, administration, and determination of values
2. **Preparatory actions preceding sample collection** — by professionals, patients, and others as applicable
3. **Collection of the biological sample** — all actions to collect the sample, before any processing
4. **Biological sample processing and storage** — preserving, transporting and storing the sample for determination
5. **Determination of the value of the sample** — methods used to count or quantify the substance or entity of interest

## Part B: Standards for Studies on Reliability

Design requirements (standards 1–6) are the same for reliability and measurement error; only the
statistical standards differ.

### Design requirements (standards 1–6)

| # | Standard |
|---|----------|
| 1 | Were patients stable in the time between the repeated measurements on the construct to be measured? |
| 2 | Was the time interval between the repeated measurements appropriate? |
| 3 | Were the measurement conditions similar for the repeated measurements — except for the condition being evaluated as a source of variation? |
| 4 | Did the professional(s) administer the measurement without knowledge of scores or values of other repeated measurement(s) in the same patients? |
| 5 | Did the professional(s) assign the scores or determine the values without knowledge of the scores or values of other repeated measurement(s) in the same patients? |
| 6 | Were there any other important flaws in the design or statistical methods of the study? |

Standard 6 is rated in the opposite direction: **No** = very good, minor methodological flaws =
doubtful, **Yes** = inadequate.

### Preferred statistical methods — reliability (standards 7–9)

| # | Standard | Very good |
|---|----------|-----------|
| 7 | For continuous scores: was an Intraclass Correlation Coefficient (ICC) calculated? | ICC calculated; the model or formula was described and matches the study design and the data |
| 8 | For ordinal scores: was a (weighted) Kappa calculated? | Kappa calculated; the weighting scheme was described and matches the study design and the data |
| 9 | For dichotomous/nominal scores: was Kappa calculated for each category against the other categories combined? | Kappa calculated for each category against the other categories combined |

For standard 7, a Pearson or Spearman correlation **without** evidence that no systematic
difference occurred between measurements rates *doubtful*; an ICC whose model or formula is not
described rates *adequate*.

## Part B: Standards for Studies on Measurement Error

### Design requirements (standards 1–6)

Identical to the reliability design requirements above.

### Preferred statistical methods — measurement error / agreement (standards 7–8)

| # | Standard | Very good |
|---|----------|-----------|
| 7 | For continuous scores: was the Standard Error of Measurement (SEM), Smallest Detectable Change (SDC), Limits of Agreement (LoA) or Coefficient of Variation (CV) calculated? | SEM, SDC, LoA or CV calculated; the model or formula for the SEM/SDC is described and matches the study design and the data |
| 8 | For dichotomous/nominal/ordinal scores: was the percentage specific (e.g. positive and negative) agreement calculated? | Percentage *specific* agreement calculated (percentage agreement alone rates only adequate) |

A SEM calculated from Cronbach's alpha, or using the SD from another population, rates
*inadequate*.

Total: **9 standards for reliability** (6 design + 3 statistical) and **8 for measurement error**
(6 design + 2 statistical).

## Overall Quality Rating

Uses the **worst-score-counts** principle:
- Rate each standard as: Very Good / Adequate / Doubtful / Inadequate
- Overall rating = lowest rating across all applicable standards

| Rating | Interpretation |
|--------|---------------|
| Very Good | Study design and methods are optimal for this measurement property |
| Adequate | Study design and methods are acceptable |
| Doubtful | Study design or methods raise some concerns |
| Inadequate | Study design or methods are clearly flawed |

## When to Use

- Systematic reviews of measurement properties of health measurement instruments
- Selecting outcome measurement instruments for clinical trials or research
- Developing core outcome sets (COS)
- Evaluating reliability/agreement of imaging measurements, scoring systems, clinical tests
- Typically used alongside other COSMIN boxes (content validity, structural validity, etc.)

# CROSS Checklist (survey studies)

**Consensus-Based Checklist for Reporting of Survey Studies**
Version: CROSS 2021 (40 reportable elements across ~7 sections). For internet/electronic surveys, pair with **CHERRIES** (Checklist for Reporting Results of Internet E-Surveys).
Sources: Sharma A, Minh Duc NT, Luu Lam Thang T, et al. *J Gen Intern Med* 2021;36:3179–3187 (the CROSS statement; DOI 10.1007/s11606-021-06737-1). Eysenbach G. *J Med Internet Res* 2004;6(3):e34 (CHERRIES; CC BY). EQUATOR Network.

Apply when the manuscript is a **self-report survey / questionnaire study** — knowledge-attitudes-practices (KAP), physician or patient surveys, cross-sectional questionnaires, and web/e-surveys. CROSS covers the reportable elements of design, sampling, instrument development, administration, and analysis; for an **internet/e-survey**, the CHERRIES items (open-vs-closed sample, denominator/completion definition, voluntariness/incentives, duplicate-submission control) also apply. For the design/validity review of the same study, pair with the SV1–SV8 domain probes in `peer-review` / `self-review` `references/domain-probes/survey_research.md`; for scale reliability, see `analyze-stats` `Survey/Likert` + `survey_weighted.md`.

> Licensing note: the CROSS statement is © Society of General Internal Medicine (not a Creative Commons licence). The items below are an **in-house, faithful summary of the reportable elements (facts/intents, paraphrased — not the verbatim CROSS wording)** for item-by-item assessment; consult the published CROSS article for exact item text. The CHERRIES e-survey items are grounded in the CC BY JMIR source.

Verification: every item was compared against Table 1 of the published statement (PubMed Central
record PMC8481359). **This file previously carried 19 items of its own devising** — the count
happened to look plausible, but from item 8 onwards nothing lined up with CROSS, and the header's
"40 reportable elements" was not what the file listed. The numbering below is now the statement's:
19 topics carrying 40 sub-items. The CHERRIES items, which CROSS does not contain, are kept in a
clearly separate section.

## Reportable elements (19 topics, 40 sub-items)

### Title and abstract
| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 1a | Title and abstract | The word "survey", with a commonly used design term, appears in the title or abstract. |
| 1b | Title and abstract | The abstract summarises background, objectives, methods, findings, interpretation and conclusions. |

### Introduction
| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 2 | Background | Rationale for the survey, what has been done before, and why this survey is needed. |
| 3 | Purpose/aim | Specific purposes, aims, goals or objectives. |

### Methods
| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 4 | Study design | The design named in Methods with a commonly used term (cross-sectional, longitudinal). |
| 5a | Data collection methods | The questionnaire itself — number of sections, number of questions, number and names of instruments used. |
| 5b | Data collection methods | Each instrument used to measure a concept: target population, reported validity and reliability, scoring/classification procedure, reference links. |
| 5c | Data collection methods | Pretesting, if performed: method, how many rounds, number and demographics of pretest participants, and how similar they were to the sample population. |
| 5d | Data collection methods | The questionnaire provided in full where possible, in the article or as an appendix/online supplement. |
| 6a | Sample characteristics | The study population — background, locations, eligibility criteria, exclusion criteria. |
| 6b | Sample characteristics | The sampling technique (single- or multistage, simple random, stratified, cluster, convenience), with the locations of participants when clustered sampling was used. |
| 6c | Sample characteristics | Sample size, with the details of its calculation. |
| 6d | Sample characteristics | How representative the sample is of the study (or target) population — particularly for population-based surveys. |
| 7a | Survey administration | Modes of administration: type and number of contacts, and where the survey was conducted (clinic room, an online tool). |
| 7b | Survey administration | The survey's time frame — recruitment, exposure and follow-up periods. |
| 7c | Survey administration | The entry process: for non-web surveys, how human data-entry error was minimised; for web surveys, how multiple participation was prevented. |
| 8 | Study preparation | Any preparation before fielding — interviewer training, advertising the survey. |
| 9a | Ethical considerations | Ethical approval where obtained: informed consent, IRB approval, Helsinki declaration, GCP declaration as appropriate. |
| 9c | Ethical considerations | Anonymity and confidentiality, and the mechanisms used to prevent unauthorised access. |
| 10a | Statistical analysis | Statistical methods and analytical approach, and the software used. |
| 10b | Statistical analysis | Any modification of variables used in the analysis, with a reference where available. |
| 10c | Statistical analysis | Missing data: rate of missing items, the missing-data mechanism (MCAR / MAR / MNAR), and the methods used to handle it. |
| 10d | Statistical analysis | How non-response error was addressed. |
| 10e | Statistical analysis | For longitudinal surveys, how loss to follow-up was addressed. |
| 10f | Statistical analysis | Whether weighting or propensity scores were used to adjust for non-representativeness. |
| 10g | Statistical analysis | Any sensitivity analysis conducted. |

*The published table numbers ethical considerations 9a and 9c with no 9b. That gap is the statement's; do not renumber to close it.*

### Results
| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 11a | Respondent characteristics | Numbers of individuals at each stage of the study; a flow diagram where possible. |
| 11b | Respondent characteristics | Reasons for non-participation at each stage, where possible. |
| 11c | Respondent characteristics | The response rate, **with the definition or formula used to calculate it**. |
| 11d | Respondent characteristics | How unique visitors were determined, and their number with the relevant proportions (view, participation, completion). |
| 12 | Descriptive results | Characteristics of participants, plus information on potential confounders and assessed outcomes. |
| 13a | Main findings | Unadjusted estimates and, where applicable, confounder-adjusted estimates with 95% confidence intervals and p values. |
| 13b | Main findings | For multivariable analysis: the model-building process, model fit statistics and model assumptions. |
| 13c | Main findings | Any sensitivity analysis; with substantial missing data, a comparison of complete cases against the imputed dataset. |

### Discussion
| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 14 | Limitations | Sources of potential bias and imprecision — non-representativeness of the sample, the design, important uncontrolled confounders. |
| 15 | Interpretations | A cautious overall interpretation given those biases and imprecisions, and areas for future research. |
| 16 | Generalizability | The external validity of the results. |

### Other sections
| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 17 | Role of the funding source | Whether any funding organisation had a role in the survey's design, implementation or analysis. |
| 18 | Conflict of interest | Any potential conflict of interest. |
| 19 | Acknowledgements | Organisations/persons acknowledged, with their contribution. |

## CHERRIES companion items (internet/e-surveys)

**Not part of CROSS.** For a web or email survey, the CHERRIES items add: open versus closed
(invited) survey; how the denominator and completion/response were defined; voluntariness and any
incentive; duplicate-submission control (IP address, cookie, log-in); and the use of adaptive or
mandatory questions and their effect on completeness. Source: Eysenbach G. *J Med Internet Res*
2004;6(3):e34 (CC BY). CROSS covers some of this ground at 7c and 11d; the rest is CHERRIES only.

## Notes for Assessors

- The **highest-yield** items (where surveys most often fail review): **6b/6d** (the sampling technique named, probability versus convenience, and how representative the sample is — a self-selected web panel generalised to "clinicians"/"patients" is the commonest over-reach), **11c** (a response rate **with its definition or formula** — "N responded" with no denominator is non-compliant), **11b/10d** (non-response), **5b/5c** (instrument validity, reliability and pretesting — a novel unvalidated questionnaire carrying the headline), **10f** (weighting for a skewed sample), and the CHERRIES denominator/duplicate-control items for e-surveys.
- A survey reporting only a raw count of respondents, with no sampling description, no defined-denominator response rate, no non-response assessment, and an unvalidated instrument, is non-compliant on 6a–6d, 5b, 10d and 11c, and its population-level claims should be downgraded to "among respondents."
- This is an **in-house faithful summary of the CROSS reportable elements (paraphrased intents, not verbatim)**; the numbering is the statement's. Consult the published CROSS article (Sharma et al. *JGIM* 2021) for the exact item text.

# DECIDE-AI Checklist

**Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence**
Version: DECIDE-AI 2022
Source: https://www.equator-network.org · https://www.decide-ai.org
Reference: Vasey B, et al. Nat Med 2022;28(5):924-933. doi:10.1038/s41591-022-01772-9

> Educational summary, authored in our own words. DECIDE-AI materials are CC BY-NC; this file
> paraphrases the *intent* of each item and copies no verbatim guideline wording. Complete the
> official DECIDE-AI checklist for a submission-ready form and cite Vasey et al. 2022.

Verification: all 17 AI-specific items and all 10 generic items were compared, by number, label and
order, against Table 2 of the *BMJ* co-publication of the same guideline (Europe PMC full text,
PMC9116198). 17/17 and 10/10 match, and the 28 AI-specific sub-items were counted from that table.
The generic items had been glossed with a list that named things the guideline does not contain
and omitted two that it does; they are now listed. Wording stays paraphrased: DECIDE-AI is
CC BY-**NC**, which cannot be redistributed under this repository's MIT licence.

## Naming and scope (read first)

- DECIDE-AI reports the **early-stage (live) clinical evaluation** of an AI decision-support system —
  the development-to-implementation gap *between* offline model validation (TRIPOD+AI / STARD-AI /
  CLAIM) and a definitive trial (CONSORT-AI). It is **not** a model-accuracy guideline; it is about
  how the AI behaves, is used, and is kept safe in real clinical workflow during first-in-human use.
- It has **17 AI-specific items** (28 subitems) plus **10 generic** reporting items (study identifiers,
  objectives, setting, sample-size rationale, statistics, funding/COI, registration, etc. — apply
  these like any reporting guideline). The AI-specific items below are the core.
- Distinct from CONSORT-AI (definitive RCT report) and SPIRIT-AI (trial protocol); use DECIDE-AI for
  the exploratory/early live-clinical evaluation stage.

## AI-specific items (17)

Status each PRESENT / PARTIAL / MISSING / N/A.

### Title / Abstract

| # | Item | Description (intent) |
|---|------|----------------------|
| 1 | Title | Identify the study as an early-stage clinical evaluation of an AI/ML-driven decision-support system. |

### Introduction

| # | Item | Description (intent) |
|---|------|----------------------|
| 2 | Intended use | Describe the targeted medical condition(s), the decision the system supports, and the intended users and use context. |

### Methods

| # | Item | Description (intent) |
|---|------|----------------------|
| 3 | Participants | Describe how patients were recruited and the inclusion/exclusion criteria at both patient and data level, how the number was arrived at, and the corresponding information for the **users** (clinicians). |
| 4 | AI system | Describe the system: algorithm type, training data and provenance, inputs, outputs, and version. |
| 5 | Implementation | Describe how the system was integrated into the **clinical workflow** and the evaluation settings (how/where it was used in practice). |
| 6 | Safety and errors | Pre-define what counts as a significant error/malfunction and how such events were identified and captured. |
| 7 | Human factors | Describe the human-factors approach: tools, methods/frameworks, and the users involved (usability evaluation plan). |
| 8 | Ethics | Describe whether specific methodologies were used to fulfil an ethics-related goal (such as algorithmic fairness), and their rationale. |

### Results

| # | Item | Description (intent) |
|---|------|----------------------|
| 9 | Participants | Report baseline characteristics of patients/users and data missingness. |
| 10 | Implementation | Report user exposure to the system and adherence to the intended use (how it was actually used). |
| 11 | Modifications | Report any changes made to the AI system during the study (versioning, retraining, threshold changes). |
| 12 | Human–computer agreement | Report how often and how users agreed with / overrode the AI recommendations. |
| 13 | Safety and errors | List significant errors, malfunctions, and patient-safety events observed. |
| 14 | Human factors | Report usability results and **learning curves** over the evaluation. |

### Discussion

| # | Item | Description (intent) |
|---|------|----------------------|
| 15 | Support for intended use | Discuss whether the results support the stated intended clinical use, within the early-stage limits. |
| 16 | Safety and errors | Discuss the safety profile and the implications of the observed errors for wider deployment. |

### Other

| # | Item | Description (intent) |
|---|------|----------------------|
| 17 | Data availability | Disclose availability of data and code (with access constraints). |

---

## Notes for Assessors

- Apply the **10 generic items** as well. They are, in the guideline's own order: **I** Abstract,
  **II** Objectives, **III** Research governance, **IV** Outcomes, **V** Analysis, **VI** Patient
  involvement, **VII** Main results, **VIII** Subgroups analysis, **IX** Strengths and limitations,
  **X** Conflicts of interest. DECIDE-AI assumes standard reporting underneath.
- **Human factors / learning curve (7, 14)** and **human–computer agreement / override (12)** are what
  make DECIDE-AI distinct from offline-accuracy guidelines; a study that reports only model metrics and
  no real-use human-interaction data is PARTIAL/MISSING on the core of DECIDE-AI.
- **Safety and errors (6/13/16)** must run end-to-end: pre-defined → captured → discussed. A safety
  claim with no pre-defined error capture is PARTIAL.
- **Implementation (5/10)** is about *workflow integration and actual use*, not just deployment intent.
- **Modifications (11)**: silent mid-study model/threshold changes without disclosure undermine the
  evaluation — mark MISSING if changes are implied but not reported.
- Use **CONSORT-AI** for a definitive trial and **TRIPOD+AI / STARD-AI / CLAIM** for offline model
  development/accuracy; DECIDE-AI covers the early live-clinical evaluation between them.

# GATHER Checklist

**Guidelines for Accurate and Transparent Health Estimates Reporting**
Version: GATHER 2016 (18 items). Source: Stevens GA, Alkema L, Black RE, et al. *The Lancet* 2016;388(10062):e19–e23, published simultaneously in *PLoS Medicine* 2016;13(6):e1002056, https://doi.org/10.1371/journal.pmed.1002056 (CC BY 4.0). https://gather-statement.org · EQUATOR Network. The item text below is reproduced from the published checklist under CC BY 4.0 with attribution; cite the source statement.

Apply when the manuscript **reports population health estimates produced by a statistical or mathematical model that synthesizes multiple data sources** — Global Burden of Disease (GBD/IHME) analyses and satellite papers, WHO/UN-agency burden estimates, attributable-burden (comparative-risk / population-attributable-fraction) studies, cause-of-death modeling, prevalence/incidence/mortality estimation, disability-adjusted or quality-adjusted life-year estimation, and their forecasts. GATHER is the reporting standard those estimates are held to; it is orthogonal to STROBE/RECORD (which govern primary and routinely-collected-data studies of *individuals*). A single-institution cohort that only **contextualizes** its finding against a published burden number does not itself trigger GATHER, but a paper that **re-estimates or re-projects** burden does. Pair the analytic methods with `/analyze-stats` `references/analysis_guides/burden_decomposition_forecasting.md` (decomposition, joinpoint/AAPC, forecasting, PAF); pair the reproducibility items (8, 14, 15) with `/verify-refs` and the project's data/code-availability discipline.

Source: Stevens GA, Alkema L, Black RE, Boerma JT, Collins GS, Ezzati M, et al. Guidelines for Accurate and Transparent Health Estimates Reporting: the GATHER statement. *Lancet* 2016;388(10062):e19-e23 (DOI 10.1016/S0140-6736(16)30388-9); also *PLoS Med* 2016;13(6):e1002056 (DOI 10.1371/journal.pmed.1002056).
Verification: all 18 items were extracted from the checklist table of the *PLoS Medicine* version
(Europe PMC full text, PMC4924581) and compared item by item. **The count matched while almost
nothing else did**: this file previously invented four items (Sampling, Bias/misclassification
correction, Comparability adjustments, Citable results file), dropped four real ones (7, 8, 17,
18), renumbered everything from official item 9 onwards, and rewrote item 15's requirement.

## Checklist Items (18 items)

### Objectives and funding

| # | Item | Description |
|---|------|-------------|
| 1 | Objectives | Define the indicator(s), populations (including age, sex, and geographic entities), and time period(s) for which estimates were made. |
| 2 | Funding | List the funding sources for the work. |

### Data inputs

*Items 3–6 apply to all data inputs from multiple sources that are synthesized as part of the study.*

| # | Item | Description |
|---|------|-------------|
| 3 | Data identification and access | Describe how the data were identified and how the data were accessed. |
| 4 | Inclusion/exclusion criteria | Specify the inclusion and exclusion criteria. Identify all ad-hoc exclusions. |
| 5 | Source characteristics | Provide information about all included data sources and their main characteristics. For each data source used, report reference information or contact name/institution, population represented, data collection method, year(s) of data collection, sex and age range, diagnostic criteria or measurement method, and sample size, as relevant. |
| 6 | Input-data bias | Identify and describe any categories of input data that have potentially important biases (e.g., based on characteristics listed in item 5). |

*Item 7 applies to data inputs that contribute to the analysis but were not synthesized as part of the study.*

| # | Item | Description |
|---|------|-------------|
| 7 | Other data inputs | Describe and give sources for any other data inputs. |

*Item 8 applies to all data inputs.*

| # | Item | Description |
|---|------|-------------|
| 8 | Data inputs in an extractable format | Provide all data inputs in a file format from which data can be efficiently extracted (e.g., a spreadsheet rather than a PDF), including all relevant meta-data listed in item 5. For any data inputs that cannot be shared because of ethical or legal reasons, such as third-party ownership, provide a contact name or the name of the institution that retains the right to the data. |

### Data analysis

| # | Item | Description |
|---|------|-------------|
| 9 | Analysis overview | Provide a conceptual overview of the data analysis method. A diagram may be helpful. |
| 10 | Analysis detail | Provide a detailed description of all steps of the analysis, including mathematical formulae. This description should cover, as relevant, data cleaning, data pre-processing, data adjustments and weighting of data sources, and mathematical or statistical model(s). |
| 11 | Model selection | Describe how candidate models were evaluated and how the final model(s) were selected. |
| 12 | Model performance | Provide the results of an evaluation of model performance, if done, as well as the results of any relevant sensitivity analysis. |
| 13 | Uncertainty methods | Describe methods of calculating uncertainty of the estimates. State which sources of uncertainty were, and were not, accounted for in the uncertainty analysis. |
| 14 | Source-code access | State how analytic or statistical source code used to generate estimates can be accessed. |

### Results and discussion

| # | Item | Description |
|---|------|-------------|
| 15 | Estimates in an extractable format | Provide published estimates in a file format from which data can be efficiently extracted. |
| 16 | Quantitative uncertainty | Report a quantitative measure of the uncertainty of the estimates (e.g., uncertainty intervals). |
| 17 | Interpretation | Interpret results in light of existing evidence. If updating a previous set of estimates, describe the reasons for changes in estimates. |
| 18 | Limitations | Discuss limitations of the estimates. Include a discussion of any modelling assumptions or data limitations that affect interpretation of the estimates. |

## MedSci application notes

- **Uncertainty intervals, not confidence intervals (items 13, 16).** Model-based estimates report 95% **uncertainty intervals (UIs)** from posterior/Monte-Carlo draws (commonly 250–500), taken as the 2.5th–97.5th percentiles. A UI crossing the null is "insufficient evidence for direction," not a non-significant test — do not translate it into *P*-value language.
- **Robustness is uncertainty propagation plus honest disclosure (items 12, 13, 18).** Burden papers rarely carry a confounding-control toolkit (DAG, E-value, negative controls); that toolkit belongs to individual-level cohorts (STROBE/RECORD). For an estimate paper, the substitute is a propagated UI at every modeling step **plus an itemized limitations paragraph** stating which biases were and were not addressed. State it explicitly rather than implying a sensitivity suite that was not run.
- **Reproducibility by pointer (items 8, 14, 15).** Estimate papers satisfy code/data availability by pointing to the standing pipeline's public repository (for GBD: a GHDx data citation and the IHME/analysis GitHub), not by curating a study-specific dataset.
- **Provenance of any add-on layer.** If the contribution is a forecast, a decomposition, or a policy-stratified re-slice on top of an existing platform's estimates, report items 9–14 for that added layer specifically — the base platform's methods do not document it.

# GRRAS Checklist

**Guidelines for Reporting Reliability and Agreement Studies**
Version: GRRAS 2011
Source: https://doi.org/10.1016/j.jclinepi.2010.03.002
Reference: Kottner J, Audige L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016/j.jclinepi.2010.03.002

Licence: Crossref returns no Creative Commons licence (Elsevier).
Verification: all 15 items were compared against the checklist table of the published article
(Table 1, *J Clin Epidemiol* 2011;64(1):96-106), read from the article PDF obtained through an
institutional subscription. The table renders several rows as mirrored glyphs, so each item was
also recovered independently from the article's own section-3 headings ("Item N: ...") and the two
readings agreed. **The count matched and almost nothing else did**: 13 of 15 items were wrong.
Only items 1 and 14 survived. This file previously invented a *Limitations* item that GRRAS does
not have, and omitted item 2 (name and describe the measurement device) entirely; items 2-13 were
a different checklist wearing GRRAS's numbering.

The source article is not redistributable (Elsevier, no open licence; **OpenAlex and Unpaywall
both report no open-access copy of either the *J Clin Epidemiol* or the *Int J Nurs Stud*
version**). The item text below therefore states what each item asks **in our own words**.
Complete the official instrument for anything you report.

## Checklist Items (15 items)

### Title and abstract

| # | Item | Description |
|---|------|-------------|
| 1 | Identification | Make clear in the title or the abstract whether interrater or intrarater **reliability** or **agreement** was the subject of the study. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 2 | Measurement device | Name the diagnostic or measurement device under study and describe it explicitly. |
| 3 | Subject population | Specify the population of subjects (or objects) the study is about. |
| 4 | Rater population | Specify the population of raters the study is about, where raters are involved. |
| 5 | Existing knowledge and rationale | Describe what is already known about the reliability and agreement of this measurement, and give the rationale for doing this study. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 6 | Sample size | Explain how the sample size was arrived at, and state the number of raters, of subjects/objects, and of replicate observations that were decided on. |
| 7 | Sampling method | Describe how subjects/objects (and raters, where applicable) were sampled. |
| 8 | Measurement/rating process | Describe how measurements or ratings were carried out — including the time interval between repeated measurements, what clinical information was available to raters, and any blinding. |
| 9 | Independence | State whether the repeated measurements or ratings were made **independently** of one another. |
| 10 | Statistical analysis | Describe the statistical analysis. |

### Results

| # | Item | Description |
|---|------|-------------|
| 11 | Numbers actually included | State the number of raters and of subjects/objects that were actually included, and the number of replicate observations actually made. |
| 12 | Sample characteristics | Describe the characteristics of the raters and of the subjects — for raters, things such as training and experience. |
| 13 | Reliability and agreement estimates | Report the reliability and/or agreement estimates **together with a measure of statistical uncertainty** (e.g. a confidence interval or standard error). |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 14 | Practical relevance | Discuss the practical relevance of the results. |

### Auxiliary material

| # | Item | Description |
|---|------|-------------|
| 15 | Detailed results | Provide detailed results where possible — for example as online supplementary material. |

**GRRAS has no Limitations item.** It also has no separate Objectives item; the study's purpose is
carried by items 3-5. Do not add either when scoring.

## Notes for Assessors

- GRRAS applies to all reliability and agreement studies, including inter-rater, intra-rater, test-retest, and method-comparison designs across any medical discipline.
- **Reliability vs Agreement**: These are distinct concepts. Reliability (e.g., ICC, kappa) reflects the ability to distinguish between subjects; agreement (e.g., Bland-Altman limits of agreement, SEM) reflects how close scores are on repeated measurements. A study may report one or both.
- Item 10 (Statistical analysis): the ICC model (one-way random, two-way random, two-way mixed) and type (single measures, average measures) must match the study design. Flag mismatches as a Major Comment. GRRAS states this item in one line and leaves the detail to the analyst — the article's Table 2 maps statistical methods onto the level of measurement and onto reliability versus agreement.
- Items 4 and 12 (rater population, rater characteristics): rater characteristics strongly influence reliability estimates. A study using expert raters only will overestimate reliability for general clinical practice. Note it when the raters are not representative of the population item 4 names.
- Item 9 (Independence) is the one most often skipped, and it is narrow: it asks whether the repeated measurements were made independently — not whether their *order* was randomised. Order randomisation and blinding belong to item 8. Both matter for memory and learning effects; score them where GRRAS puts them.
- Item 13: for an ICC, report the model, type and definition (consistency vs absolute agreement); for a kappa, whether it is weighted and with which scheme. A bare estimate with no measure of statistical uncertainty fails the item outright, since the uncertainty is part of what item 13 asks for.
- Common in radiology: inter-reader agreement for measurements (Bland-Altman, ICC), segmentation agreement (Dice, ICC of volumes), and diagnostic classification agreement (kappa, percent agreement).
- GRRAS is complementary to study-type checklists. For example, a diagnostic accuracy study reporting inter-reader agreement should follow STARD for the main analysis and GRRAS for the agreement component.

# MI-CLEAR-LLM Checklist — Reporting LLM Accuracy Studies in Healthcare

**Minimum Reporting Items for Clear Evaluation of Accuracy Reports of Large Language Models in Healthcare**

- **Version:** MI-CLEAR-LLM 2025 (8 item categories) — updates and expands the 2024 original (6 items).
- **Citation (2025 update):** Park SH, Suh CH, Lee JH, Tejani AS, You SC, Kahn CE Jr, Moy L. *MI-CLEAR-LLM: 2025 Updates.* Korean J Radiol. 2025;26(12):1123-1132. PMID: 41199132.
- **DOI:** 10.3348/kjr.2025.1522
- **Citation (2024 original):** Park SH, Suh CH, Lee JH, Kahn CE Jr, Moy L. Korean J Radiol. 2024;25(10):865-868. PMID: 39344542.
- **Licence:** CC BY-NC 4.0 (Korean Society of Radiology).

> **Educational summary, authored in our own words.** This file paraphrases the *intent* of each MI-CLEAR-LLM
> 2025 item to drive an item-by-item audit; it does not reproduce the guideline's verbatim wording. For a
> submission-ready checklist, complete the official instrument at the source above and cite Park et al. 2025.

**The 2025 update expanded the original 6 items into 8 item categories.** The three items that are new or
promoted to the top level in 2025 — **Access mode (2)**, **Input data type (3)**, and **Adaptation strategy
used (4)** — were prompted by the growth of API access and self-managed (often open-source) LLMs, which the
2024 six-item version folded into a single "model identification" item. Cite items by the 2025 numbering.

**Scope:** Studies evaluating the accuracy of LLMs in healthcare tasks (diagnosis, triage, clinical decision
support, medical question answering, information extraction, report generation, etc.). This is NOT for
disclosing LLM use in manuscript *writing* — for that, see ICMJE/COPE policies and the write-paper skill's LLM
disclosure feature. MI-CLEAR-LLM supplements, and does not replace, a primary reporting guideline (STARD /
STARD-AI, CLAIM 2024, or TRIPOD+AI) chosen for the study design.

Verification: all 8 item categories were compared, by name and order, against Table 1 of the
published statement (Europe PMC full text, PMC12683746); 8/8 match. The wording below stays our
own condensed summary because the source is CC BY-**NC**, which cannot be redistributed under
this repository's licence.

## Checklist Items (8 items)

| # | Item category | What must be reported |
|---|---|---|
| 1 | Model identification | Model name, version, developer, proprietary/open-source status, access date(s), and training-data cutoff. |
| 2 | Access mode | Whether a web chatbot, an API, or a self-managed local deployment was used, and why; any known system-level features beyond the LLM itself (system prompts, intersession memory); and, for a local deployment, key computational-environment details. |
| 3 | Input data type | The type and format of data supplied with the prompt, in enough detail for a reader to replicate. |
| 4 | Adaptation strategy used | Whether model weights were altered (e.g., fine-tuning) or non-parametric methods (e.g., prompting, RAG) were used — or neither. |
| 5 | Prompt optimization procedures | How prompts were created and optimized, the rationale, and the full executable prompt text. |
| 6 | Prompt execution setup | How queries were submitted (session handling, batching, post-processing); provide the experiment scripts where feasible. |
| 7 | Stochasticity management | Temperature and sampling settings, number of attempts per item, how repeats were synthesized, and reliability across repeats. |
| 8 | Independence of test data | Any overlap between the test data and the model's training data or the data used for prompt development/adaptation. |

## Item detail

### 1. Model identification ⚠️ commonly missed

Fully identify the LLM: model name and version (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro), developer
(OpenAI, Anthropic, Google, etc.), whether it is proprietary or open-source, the date(s) the queries were
run, and the training-data cutoff if disclosed. LLMs are frequently updated behind the same name, so a name
alone (e.g., "GPT-4") is insufficient — the exact version and access date are what make the study reproducible.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 2. Access mode ⚠️ new/expanded in 2025

State how the model was accessed and why: a web-based chatbot interface, a programmatic API, or a self-managed
local deployment. For API access, give the specific API version or endpoint. For self-managed/open-source
models, give the weights version, any quantization, and the hardware. Access mode changes what is controllable
(e.g., a web chatbot may not expose temperature) and what is reproducible.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 3. Input data type ⚠️ new/expanded in 2025

Describe the type and format of the data supplied to the model with the prompt — free text, structured fields,
tables, images (for vision-language models), or file attachments — in enough detail that a reader could
reconstruct the input. Where images or clinical data are used, describe how they were encoded and presented.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 4. Adaptation strategy used ⚠️ new/expanded in 2025

State whether the base model was adapted, and how. Distinguish **parametric** adaptation (model weights were
changed, e.g., fine-tuning, instruction-tuning) from **non-parametric** methods that leave weights unchanged
(e.g., prompting strategies, retrieval-augmented generation, tool use) — or state that no adaptation was
applied. Adaptation is a first-class methodological choice, not a footnote.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 5. Prompt optimization procedures ⚠️ commonly missed

Document how prompts were developed and optimized: who wrote them, how they were iterated, whether prompt
engineering or automated tuning was used, and how many variants were compared. Provide the **full prompt text**
(system, user, and any few-shot examples), preserving exact wording, punctuation, and formatting — in
supplementary material if lengthy. Any dataset used to optimize prompts must be independent of the test data
(see item 8); undisclosed optimization on test-overlapping data inflates apparent performance.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 6. Prompt execution setup ⚠️ commonly missed

Describe how prompts were run operationally: whether each query was an independent session or a continuing
conversation, whether queries were batched or sequential, whether prior outputs could carry over as context,
and any post-processing applied to outputs (e.g., parsing structured answers from free text). For API studies,
give batch size, rate limiting, and failure handling. Provide the experiment scripts where feasible.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 7. Stochasticity management ⚠️ commonly missed

LLM outputs are stochastic. Report the number of query attempts per item, how multiple outputs were combined
(majority vote, best-of-N, mean score), and a reliability analysis across repeats (e.g., agreement rate,
kappa). Report the parameters that control randomness — temperature, top-p, top-k, penalties — or, if they were
not controllable (e.g., a web chatbot), state that explicitly. A single-attempt study cannot characterize
reliability and should be scored PARTIAL at best.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

### 8. Independence of test data ⚠️ commonly missed

Confirm that the test data were separate from all other data used, and address training-data contamination:
whether the test items could have been in the model's pretraining corpus (a particular risk for published exam
questions and public benchmarks), and any mitigation (unpublished cases, temporal or institution-specific
data). Because most LLM training corpora are undisclosed, contamination cannot be ruled out by inspection and
must be discussed.

**Status:** [ ] PRESENT [ ] PARTIAL [ ] MISSING

## Notes for assessors

- **Apply MI-CLEAR-LLM when the study's outcome is LLM accuracy or performance** (answering board questions,
  clinical reasoning, triage, information extraction, report generation). Do NOT apply it when an LLM is merely
  a pipeline tool whose accuracy is not the outcome, or when the manuscript only discloses writing assistance.
- **"GPT-4" is insufficient (item 1)** — require the exact version (e.g., gpt-4-0613, gpt-4-turbo-2024-04-09)
  and the access date. **Single-run studies (item 7)** are PARTIAL at best. **Published exam questions (item
  8)** demand an explicit contamination discussion.
- **Co-application:** use MI-CLEAR-LLM alongside STARD/STARD-AI (diagnostic accuracy against a reference
  standard), CLAIM 2024 (an LLM processing medical images), or TRIPOD+AI (an LLM used as a prediction model).
  The 8 items here supplement — they do not replace — the primary reporting guideline.

# MOOSE Checklist

**Meta-analysis Of Observational Studies in Epidemiology**
Version: MOOSE 2000 — 35 items in six groups (Background 6, Search strategy 10, Methods 8,
Results 4, Discussion 3, Conclusions 4).
Source: Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of
observational studies in epidemiology: a proposal for reporting. *JAMA* 2000;283(15):2008-2012
(DOI 10.1001/jama.283.15.2008).
Licence: *JAMA* (© American Medical Association) — **no open licence**. The descriptions below are
an in-house summary of what each item asks, in our own words, not the published wording.
Verification: all 35 items were compared against the checklist table published in the statement.
The item set, the six groups, their counts and their order all match; nothing is missing and
nothing has been added.

## Checklist Items (35 items)

### Reporting of Background

| # | Item | Description |
|---|------|-------------|
| 1 | Problem definition | Define the clinical or public health problem being addressed. |
| 2 | Hypothesis statement | State the hypothesis to be tested. |
| 3 | Study outcome(s) | Describe the study outcome(s) of interest. |
| 4 | Type of exposure or intervention | Describe the type of exposure or intervention used. |
| 5 | Type of study designs | Describe the type of study designs included (cohort, case-control, cross-sectional, etc.). |
| 6 | Study population | Describe the study population. |

### Reporting of Search Strategy

| # | Item | Description |
|---|------|-------------|
| 7 | Qualifications of searchers | Report the qualifications of the searchers (e.g., librarian, investigator). |
| 8 | Search strategy | Describe the search strategy, including time period included and keywords used. |
| 9 | Effort to include all available studies | Describe efforts to include all available studies, including contact with authors. |
| 10 | Databases and registries searched | List all databases and registries searched. |
| 11 | Search software used | Report the search software used, including name and version. |
| 12 | Use of hand searching | Describe any use of hand searching (e.g., reference lists, conference proceedings). |
| 13 | List of citations located and excluded | Provide a list of citations located and those excluded, including justifications for exclusions. |
| 14 | Method for non-English articles | Describe the method of addressing articles published in languages other than English. |
| 15 | Method for unpublished studies | Describe the method of handling abstracts and unpublished studies. |
| 16 | Description of author contact | Describe any contact with authors for additional data or clarification. |

### Reporting of Methods

| # | Item | Description |
|---|------|-------------|
| 17 | Study relevance | Describe the relevance or appropriateness of studies assembled for assessing the hypothesis to be tested. |
| 18 | Rationale for data selection and coding | Provide the rationale for the selection and coding of data (e.g., sound clinical principles, convenience, etc.). |
| 19 | Data classification | Document how data were classified and coded (e.g., multiple raters, blinding, inter-rater reliability). |
| 20 | Assessment of confounding | Describe the assessment of confounding (e.g., comparability of cases and controls, adjustment for confounders). |
| 21 | Assessment of study quality | Describe the assessment of study quality, including blinding of quality assessors; stratification or regression on possible predictors of study results. |
| 22 | Assessment of heterogeneity | Describe the assessment of heterogeneity. |
| 23 | Statistical methods | Describe statistical methods (e.g., fixed vs random effects, meta-regression, cumulative meta-analysis) in sufficient detail to be replicated. |
| 24 | Tables and graphics | Provide appropriate tables and graphics. |

### Reporting of Results

| # | Item | Description |
|---|------|-------------|
| 25 | Graphic summary | Provide a graphic summarizing individual study estimates and the overall estimate (e.g., forest plot). |
| 26 | Descriptive table | Provide a table giving descriptive information for each study included (design, sample size, outcome measures, effect sizes, confounders adjusted for). |
| 27 | Sensitivity testing | Report results of sensitivity testing (e.g., subgroup analysis, influence analysis, varying inclusion criteria). |
| 28 | Statistical uncertainty | Indicate statistical uncertainty of findings (e.g., 95% confidence intervals). |

### Reporting of Discussion

| # | Item | Description |
|---|------|-------------|
| 29 | Quantitative assessment of bias | Provide a quantitative assessment of bias (e.g., publication bias via funnel plot, Egger test, trim-and-fill). |
| 30 | Justification for exclusion | Justify any exclusions from the meta-analysis. |
| 31 | Assessment of quality of included studies | Assess the quality of included studies and its impact on the overall findings. |

### Reporting of Conclusions

| # | Item | Description |
|---|------|-------------|
| 32 | Alternative explanations | Consider alternative explanations for observed results. |
| 33 | Generalization | Discuss the generalization of the conclusions (i.e., external validity). |
| 34 | Guidelines for future research | Provide guidelines for future research. |
| 35 | Disclosure of funding | Disclose the funding source for the meta-analysis. |

---

## Notes for Assessors

- MOOSE is designed specifically for meta-analyses of **observational** studies (cohort, case-control, cross-sectional). For meta-analyses of RCTs, use PRISMA 2020.
- MOOSE and PRISMA are complementary. Many journals require adherence to both for observational MAs. When both apply, audit against both checklists separately.
- Items 7-16 (Search Strategy) overlap substantially with PRISMA but include observational-specific items such as handling of confounding and study quality assessment.
- Item 20 (Assessment of confounding) is particularly important for observational MAs, as confounding is the primary threat to validity -- unlike RCTs where randomization addresses this.
- Item 21 (Assessment of study quality) should describe a formal tool. Common tools include NOS (Newcastle-Ottawa Scale), ROBINS-I, or the Joanna Briggs Institute checklists.
- Item 29 (Quantitative assessment of bias) is critical. Publication bias assessment is expected in all meta-analyses, but is especially important for observational studies where positive-result bias is well documented.
- MOOSE was published before the PRISMA era (2009/2020). Some journals now accept PRISMA with MOOSE-specific additions. If in doubt, check journal instructions.

# Newcastle-Ottawa Scale (NOS) Assessment Guide

Quality assessment tool for non-randomised studies in meta-analyses.
Reference: Wells GA et al. Ottawa Hospital Research Institute.

Version: Newcastle-Ottawa Scale (current web version)
Source: Wells GA, Shea B, O'Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute. **No DOI: the tool is distributed from the institute's website and has never been issued one.**.
Licence: No formal licence is published with the tool.
Verification: both scales were compared item by item and response-option by response-option
against the official NOS rating sheet distributed by the Ottawa Hospital Research Institute
(`nosgen.pdf`), with the accompanying manual (`nos_manual.pdf`) checked for the scoring rules.
The items, their order, the star allocations and the two-star comparability rule all match. **Two
numbers did not**: the scale leaves the follow-up percentages blank for the reviewer to set, and
this file had hardcoded them; and the Good/Fair/Poor star bands appear nowhere in the NOS
documents. Both are corrected below.

## Structure

NOS uses a "star system" (maximum 9 stars) across 3 categories.
Higher stars = higher quality.

## Cohort Studies (max 9 stars)

### Selection (max 4 stars)
1. **Representativeness of the exposed cohort** (1 star)
   - a) Truly representative of the average [describe] in the community *
   - b) Somewhat representative *
   - c) Selected group of users
   - d) No description

2. **Selection of the non-exposed cohort** (1 star)
   - a) Drawn from the same community as the exposed *
   - b) Drawn from a different source
   - c) No description

3. **Ascertainment of exposure** (1 star)
   - a) Secure record (e.g., surgical records) *
   - b) Structured interview *
   - c) Written self-report
   - d) No description

4. **Demonstration that outcome was not present at start** (1 star)
   - a) Yes *
   - b) No

### Comparability (max 2 stars)
5. **Comparability of cohorts on the basis of design or analysis** (up to 2 stars)
   - a) Study controls for [most important factor] *
   - b) Study controls for any additional factor *

### Outcome (max 3 stars)
6. **Assessment of outcome** (1 star)
   - a) Independent blind assessment *
   - b) Record linkage *
   - c) Self-report
   - d) No description

7. **Was follow-up long enough for outcomes to occur?** (1 star)
   - a) Yes (select adequate follow-up period) *
   - b) No

8. **Adequacy of follow-up of cohorts** (1 star)
   - a) Complete follow-up (all subjects accounted for) *
   - b) Subjects lost to follow-up unlikely to introduce bias — small number lost, above a threshold **you select**, or description provided of those lost *
   - c) Follow-up rate below the threshold **you select**, and no description of those lost
   - d) No statement

   The scale prints these two percentages as blanks ("select an adequate %"). NOS does not supply a
   number; pre-specify yours in the protocol and report it.

## Case-Control Studies (max 9 stars)

### Selection (max 4 stars)
1. Is the case definition adequate?
2. Representativeness of the cases
3. Selection of controls
4. Definition of controls

### Comparability (max 2 stars)
5. Comparability of cases and controls (same 2-star system)

### Exposure (max 3 stars)
6. Ascertainment of exposure
7. Same method of ascertainment for cases and controls
8. Non-response rate

## Interpretation

**The NOS does not define quality bands.** Neither the rating sheet nor the manual maps a star
total onto "good", "fair" or "poor". The bands below are a widely used external convention (they
come from AHRQ-derived practice, not from the scale) and are reproduced here only because reviews
so often cite them:

| Stars | Convention |
|-------|------------|
| 7-9 | Good |
| 4-6 | Fair |
| 0-3 | Poor |

If you use them, say where they came from and pre-specify them in the protocol. A different
threshold is equally defensible, and the scale's authors leave the choice to you.

## When to Use

- Observational cohort studies in intervention or exposure meta-analyses
- Case-control studies
- Simpler alternative to ROBINS-I when full domain-level assessment is not needed
- Note: NOS does not provide domain-level judgments -- only an aggregate score

# PGS-RS (PRS-RS) Checklist

**Polygenic Score / Polygenic Risk Score Reporting Standards**
Version: PGS-RS 2021 (also cited as PRS-RS)
Source: Wand H, Lambert SA, Tamburro C, et al. *Improving reporting standards for polygenic scores in risk prediction studies.* Nature 2021;591:211–219. Endorsed by the Clinical Genome Resource (ClinGen) Complex Disease Working Group and the Polygenic Score (PGS) Catalog.

Apply when the manuscript develops, validates, or applies a **polygenic risk score / polygenic score (PRS / PGS)** as a predictor or risk-stratifier. For the design-validity review of the same study, pair with the PG1–PG8 domain probes in `peer-review` / `self-review` `references/domain-probes/polygenic_risk_score.md`; for the analysis, with `analyze-stats` `analysis_guides/polygenic_risk_score.md`. PGS-RS is a domain extension of the TRIPOD prediction-model reporting principles — for the general prediction-model items also consult `TRIPOD.md` / `TRIPOD_AI.md`.

Source: Wand H, Lambert SA, Tamburro C, Iacocca MA, O'Sullivan JW, Sillari C, et al. Improving reporting standards for polygenic scores in risk prediction studies. *Nature* 2021;591(7849):211-219 (DOI 10.1038/s41586-021-03243-6).
Licence: Crossref returns only a Springer Nature text-and-data-mining licence; no Creative Commons licence.

## Checklist Items (22 items)

### Background

| # | Item | Description |
|---|------|-------------|
| 1 | Study type | Specify the development and/or validation status of the polygenic score and provide identifiers (e.g., a PGS Catalog ID) where available. |
| 2 | Risk model purpose and predicted outcome | Define the intended use of the risk model and the outcome it predicts. |

### Study Population and Data

| # | Item | Description |
|---|------|-------------|
| 3 | Study design and recruitment | Describe the cohort/study design, eligibility criteria, and recruitment period for the development and the evaluation samples. |
| 4 | Participant demographic and clinical characteristics | Report age, sex, and relevant clinical/phenotypic characteristics of each sample. |
| 5 | Ancestry | Report the ancestral background of the GWAS (base), training/tuning, and evaluation (target) samples using a standardized ancestry framework; do not assume transferability across ancestries. |
| 6 | Genetic data | Describe genotyping/sequencing methods, imputation, and quality control. |
| 7 | Non-genetic variables | Define any non-genetic variables included in the (integrated) risk model. |
| 8 | Outcome of interest | Define and report how the predicted outcome(s) were ascertained. |
| 9 | Missing data | Explain how missing data (genetic and non-genetic) were handled. |

### Risk Model Development and Application

| # | Item | Description |
|---|------|-------------|
| 10 | Polygenic risk score construction and estimation | Detail the source GWAS, variant selection, weighting/shrinkage method (e.g., P+T, LDpred, PRS-CS), reference panel, and any tuning of hyperparameters (and the data on which tuning was performed). |
| 11 | Risk model type | Describe the statistical method(s) used to estimate risk from the score. |
| 12 | Integrated risk model(s) description and fitting | If the PGS is combined with non-genetic predictors (e.g., a clinical risk model), describe the integrated-model development and fitting procedure. |

### Risk Model Evaluation

| # | Item | Description |
|---|------|-------------|
| 13 | PRS distribution | Describe the distribution of the continuous polygenic score (and any standardization or quantile categorization, with the reference group). |
| 14 | Risk model predictive ability | Report variance explained (e.g., R², liability-scale) and effect estimates (e.g., OR/HR per SD, per quantile) with measures of uncertainty. |
| 15 | Risk model discrimination | Report discrimination (AUROC/C-index; AUPRC where prevalence is low), with confidence intervals — and, for an incremental claim, the change versus the established clinical model (ΔC-statistic, NRI/IDI). |
| 16 | Risk model calibration | Describe how calibration of absolute risk was assessed (calibration plot, slope/intercept, observed vs expected), especially in the target population and across ancestries. |
| 17 | Subgroup analyses | Report performance for relevant subgroups (e.g., ancestry, sex, age) rather than aggregate metrics alone. |

### Limitations and Clinical Implications

| # | Item | Description |
|---|------|-------------|
| 18 | Risk model interpretation | Summarize predictive performance and the incremental value over existing risk factors / clinical models; do not equate a quantile relative-risk gradient with demonstrated clinical actionability. |
| 19 | Limitations | Outline study restrictions and their impact on interpretation. |
| 20 | Generalizability | Discuss applicability to target populations, including ancestries and settings not represented in development/evaluation. |
| 21 | Risk model intended uses | Discuss clinical utility and deployment readiness, scaled to the evidence (calibration, incremental value, and ideally decision-analytic or trial evidence). |
| 22 | Data transparency and availability | Make the PRS parameters (variants and weights) and evaluation results publicly available (e.g., deposit in the PGS Catalog) so the score is reproducible. |

---

## Notes for Assessors

- The highest-yield items are **5 / 17 / 20** (ancestry reporting and per-ancestry/subgroup performance — the central PGS transferability problem), **15** (incremental value over the clinical model, not score-alone discrimination), **16** (absolute-risk calibration, not discrimination alone), and **22** (reproducibility via deposited variants+weights / a PGS Catalog ID).
- PGS-RS is a domain extension of general prediction-model reporting; for items on model development/validation also apply `TRIPOD.md` / `TRIPOD_AI.md`, and cite both the base instrument and PGS-RS rather than PGS-RS alone.
- The guideline is abbreviated both **PGS-RS** and **PRS-RS** in the literature; they refer to the same Wand et al. 2021 standard.
- Verification: all 22 items were compared, by label and order, against Table 1 of the published
  standard (PubMed Central record PMC8609771). 22/22 match, with no item missing and none
  invented. The wording stays a condensed summary — Crossref returns only a Springer Nature
  text-and-data-mining licence, so the item text cannot be reproduced here. Consult the published
  Table 1 for the full wording.

# PRISMA 2020 Checklist

**Preferred Reporting Items for Systematic Reviews and Meta-Analyses**
Version: PRISMA 2020 — 27 items / 42 sub-items.
Source: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020
statement: an updated guideline for reporting systematic reviews. *BMJ* 2021;372:n71
(DOI 10.1136/bmj.n71). Official checklist: https://www.prisma-statement.org
Licence: CC BY 4.0. Item text is reproduced verbatim from the official checklist with attribution.

> **Fidelity**: every one of the 42 sub-item cells below has been compared, normalised and
> character-for-character, against the official PRISMA 2020 checklist PDF. Any editorial comment
> of ours lives in *Notes for Assessors*, never inside an item cell — an assessor scoring a
> manuscript must read the guideline's words, not ours.

## Checklist Items (27 items)

### Title

| # | Item | Description |
|---|------|-------------|
| 1 | Title | Identify the report as a systematic review. |

### Abstract

| # | Item | Description |
|---|------|-------------|
| 2 | Abstract | See the PRISMA 2020 for Abstracts checklist. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 3 | Rationale | Describe the rationale for the review in the context of existing knowledge. |
| 4 | Objectives | Provide an explicit statement of the objective(s) or question(s) the review addresses. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 5 | Eligibility criteria | Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses. |
| 6 | Information sources | Specify all databases, registers, websites, organisations, reference lists, and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. |
| 7 | Search strategy | Present the full search strategies for all databases, registers, and websites, including any filters and limits used. |
| 8 | Selection process | Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process. |
| 9 | Data collection process | Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process. |
| 10a | Data items | List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g., for all measures, time points, analyses), and if not, the methods used to decide which results to collect. |
| 10b | Data items | List and define all other variables for which data were sought (e.g., participant and intervention characteristics, funding sources). Describe any assumptions made about any missing or unclear information. |
| 11 | Study risk of bias assessment | Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process. |
| 12 | Effect measures | Specify for each outcome the effect measure(s) (e.g., risk ratio, mean difference) used in the synthesis or presentation of results. |
| 13a | Synthesis methods | Describe the processes used to decide which studies were eligible for each synthesis (e.g., tabulating the study intervention characteristics and comparing against the planned groups for each synthesis). |
| 13b | Synthesis methods | Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data conversions. |
| 13c | Synthesis methods | Describe any methods used to tabulate or visually display results of individual studies and syntheses. |
| 13d | Synthesis methods | Describe any methods used to synthesize results and provide a rationale for the choice(s). If meta-analysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used. |
| 13e | Synthesis methods | Describe any methods used to explore possible causes of heterogeneity among study results (e.g., subgroup analysis, meta-regression). |
| 13f | Synthesis methods | Describe any sensitivity analyses conducted to assess robustness of the synthesized results. |
| 14 | Reporting bias assessment | Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases). |
| 15 | Certainty assessment | Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome. |

### Results

| # | Item | Description |
|---|------|-------------|
| 16a | Study selection | Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram. |
| 16b | Study selection | Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded. |
| 17 | Study characteristics | Cite each included study and present its characteristics. |
| 18 | Risk of bias in studies | Present assessments of risk of bias for each included study. |
| 19 | Results of individual studies | For all outcomes, present, for each study: (a) summary statistics for each group (where appropriate) and (b) an effect estimate and its precision (e.g. confidence/credible interval), ideally using structured tables or plots. |
| 20a | Results of syntheses | For each synthesis, briefly summarise the characteristics and risk of bias among contributing studies. |
| 20b | Results of syntheses | Present results of all statistical syntheses conducted. If meta-analysis was done, present for each the summary estimate and its precision (e.g., confidence/credible interval) and measures of statistical heterogeneity. If comparing groups, describe the direction of the effect. |
| 20c | Results of syntheses | Present results of all investigations of possible causes of heterogeneity among study results. |
| 20d | Results of syntheses | Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results. |
| 21 | Reporting biases | Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed. |
| 22 | Certainty of evidence | Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 23a | Discussion | Provide a general interpretation of the results in the context of other evidence. |
| 23b | Discussion | Discuss any limitations of the evidence included in the review. |
| 23c | Discussion | Discuss any limitations of the review processes used. |
| 23d | Discussion | Discuss implications of the results for practice, policy, and future research. |

### Other Information

| # | Item | Description |
|---|------|-------------|
| 24a | Registration and protocol | Provide registration information for the review, including register name and registration number, or state that the review was not registered. |
| 24b | Registration and protocol | Indicate where the review protocol can be accessed, or state that a protocol was not prepared. |
| 24c | Registration and protocol | Describe and explain any amendments to information provided at registration or in the protocol. |
| 25 | Support | Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review. |
| 26 | Competing interests | Declare any competing interests of review authors. |
| 27 | Availability of data, code, and other materials | Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review. |

---

## PRISMA 2020 Flow Diagram

The PRISMA flow diagram (item 16a) should include four phases:

```
IDENTIFICATION
  Records identified from databases (n = ?)
  Records identified from other sources (n = ?)
      |
      v
  Records removed before screening:
    Duplicate records (n = ?)
    Records marked as ineligible by automation tools (n = ?)
    Records removed for other reasons (n = ?)
      |
      v
SCREENING
  Records screened (n = ?)
    Records excluded (n = ?)
      |
      v
  Reports sought for retrieval (n = ?)
    Reports not retrieved (n = ?)
      |
      v
  Reports assessed for eligibility (n = ?)
    Reports excluded (n = ?), with reasons:
      Reason 1 (n = ?)
      Reason 2 (n = ?)
      Reason 3 (n = ?)
      |
      v
INCLUDED
  Studies included in review (n = ?)
  Reports of included studies (n = ?)
```

---

## Notes for Assessors

- PRISMA 2020 expanded from the original 27-item checklist; several items now have sub-items (e.g., 10a/10b, 13a-13f, 16a/16b).
- Item 2 delegates to a **separate 12-item abstract checklist** (`PRISMA_2020_Abstracts.md`). The statement's own item 2 is that one sentence and nothing more. Run the abstract checklist as its own pass and report its score with its own denominator — a manuscript can satisfy every main-text item and still fail most of the twelve, and a high main-text percentage carries no information about it.
- **Item 4 does not require PICO.** This file previously read "using the PICO framework or similar", which is PRISMA *2009* wording; the 2020 item asks only for an explicit statement of the objective(s) or question(s). Do not score a manuscript down for stating its objectives without a PICO frame.
- **Item 13b is about missing summary statistics and data conversions**, not about multi-arm studies or multiple outcome measures — this file previously asked for the latter, which is a different requirement and sent assessors looking for the wrong thing.
- Item 7 (full search strategy): the complete strategy for at least one database must be provided, either in the manuscript or supplementary materials.
- Item 15 (certainty assessment): GRADE is the most common framework; mark as MISSING if no certainty assessment is reported.
- Item 24 (registration): PROSPERO is the standard registry for systematic reviews. If not registered, the authors should explicitly state this.
- Item 27 (data availability): increasingly required; check if authors state where extracted data and analytic code can be accessed.
- For systematic reviews of diagnostic test accuracy, also consider PRISMA-DTA extension.
- For reviews that include meta-analysis, items 13d (synthesis model) and 20b (heterogeneity measures) are critical.

# PRISMA 2020 for Abstracts Checklist

**Preferred Reporting Items for Systematic Reviews and Meta-Analyses — abstract checklist**
Version: PRISMA 2020 for Abstracts — 12 items, grouped by the sections of a structured abstract.
Source: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020
statement: an updated guideline for reporting systematic reviews. *BMJ* 2021;372:n71
(DOI 10.1136/bmj.n71). Published under CC BY 4.0; item text is reproduced with attribution.
Verification: all 12 items were transcribed from, and checked against, the abstract checklist
published in that statement.

Apply to the **abstract of a systematic review or meta-analysis**. This is a separate instrument
from the 27-item main checklist in `PRISMA_2020.md`, not a subset of it: the main checklist devotes
a single item (item 2) to the abstract and defers the detail here. A manuscript can satisfy all 27
main items — all 42 with sub-items — and still fail most of these twelve, because the abstract is
usually written last, to a word limit, from a template the journal supplied.

That is not a hypothetical. In a published audit of 24 systematic reviews and meta-analyses in a
radiology journal (Park HY et al. *Korean J Radiol* 2022;23(3):355–369; PMID:35213097), **item 3
(eligibility criteria) and item 12 (registration) were each reported by 0 of 24 abstracts**, and
item 5 (risk-of-bias methods) by 3 of 23. Run this checklist as its own pass — a compliance
percentage computed over the main checklist says nothing about it.

For a DTA review the abstract items still apply; pair them with `PRISMA_DTA.md` for the main text.
For a scoping review use the abstract item inside `PRISMA_ScR.md` instead. Word limits are real:
where an item genuinely cannot fit, that is a judgement to record, not an item to score PRESENT.

## Reporting items (grouped by abstract section)

### Title

| # | Item | What to check is reported |
|---|------|---------------------------|
| 1 | Title | Identify the report as a systematic review. |

### Background

| # | Item | What to check is reported |
|---|------|---------------------------|
| 2 | Objectives | Provide an explicit statement of the main objective(s) or question(s) the review addresses. |

### Methods

| # | Item | What to check is reported |
|---|------|---------------------------|
| 3 | Eligibility criteria | Specify the inclusion and exclusion criteria for the review. |
| 4 | Information sources | Specify the information sources (e.g. databases, registers) used to identify studies and the date when each was last searched. |
| 5 | Risk of bias | Specify the methods used to assess risk of bias in the included studies. |
| 6 | Synthesis of results | Specify the methods used to present and synthesise results. |

### Results

| # | Item | What to check is reported |
|---|------|---------------------------|
| 7 | Included studies | Give the total number of included studies and participants and summarise relevant characteristics of studies. |
| 8 | Synthesis of results | Present results for main outcomes, preferably indicating the number of included studies and participants for each. If meta-analysis was done, report the summary estimate and confidence/credible interval. If comparing groups, indicate the direction of the effect (i.e. which group is favoured). |
| 9 | Limitations of evidence | Provide a brief summary of the limitations of the evidence included in the review (e.g. study risk of bias, inconsistency and imprecision). |

### Discussion

| # | Item | What to check is reported |
|---|------|---------------------------|
| 10 | Interpretation | Provide a general interpretation of the results and important implications. |

### Other

| # | Item | What to check is reported |
|---|------|---------------------------|
| 11 | Funding | Specify the primary source of funding for the review. |
| 12 | Registration | Provide the register name and registration number. |

## Notes for Assessors

- **Score this checklist separately and report its own denominator.** Folding twelve abstract items
  into a 42-item main-text total hides them: they are a small fraction of the sum and they fail
  together.
- **Item 3 is not the same as item 7.** Naming the studies that ended up included ("24 studies of
  adults undergoing CT") does not state the criteria that decided inclusion. The item asks what
  *would have been excluded*.
- **Item 4** needs both halves — which sources, and the date each was last searched. A database
  list with no search date is PARTIAL.
- **Item 5** asks for the risk-of-bias *method* (the named tool), not its result. Reporting "most
  studies were at low risk of bias" without naming QUADAS-2 / RoB 2 / ROBINS-I leaves item 5
  MISSING and item 9 partially addressed.
- **Item 8** is the one that carries the numbers: a summary estimate with its interval, and, when
  groups are compared, which group the effect favours. An abstract that reports a pooled estimate
  with no interval fails this item even though it looks quantitative.
- **Item 9 is about the evidence, not the review.** "Only English-language studies were searched"
  is a limitation of the review process (main checklist item 23c); "the included studies were at
  high risk of bias for flow and timing" is the limitation this item wants.
- **Item 12**: register name *and* number. "The review was registered in PROSPERO" without the
  CRD number is PARTIAL. If the review was not registered, an explicit statement to that effect is
  the honest answer — but check whether the target journal's abstract structure has anywhere to
  put it before scoring it MISSING.
- **Structured vs unstructured abstracts**: journals that require an unstructured abstract still
  expect the content; assess by content, not by the presence of the headings.
- Conference abstracts of the same review are out of scope for this checklist.

# PRISMA-DTA Checklist (diagnostic test accuracy systematic reviews)

**Preferred Reporting Items for Systematic Reviews and Meta-Analyses — diagnostic test accuracy extension**
Version: PRISMA-DTA 2018 — 27 items. Two PRISMA items were **deleted** (15 and 22) and two DTA-specific
items were **added** (D1 and D2), so the count matches the original while the numbering skips 15 and 22.
A separate abstract checklist accompanies it (below).
Source: McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM, the PRISMA-DTA Group, et al. Preferred
Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The
PRISMA-DTA Statement. *JAMA* 2018;319(4):388-396 (DOI 10.1001/jama.2017.19163). Correction:
*JAMA* 2020;323(6):580 — in item 20, "receiver operating characteristic curve" should read
"receiver operating characteristic **plot**"; the correction is reflected below.

> **Fidelity and licence.** The statement is published in *JAMA* (© American Medical Association) under
> **no open licence** — free to read is not free to reuse. The descriptions below are therefore an
> **in-house summary of what each item asks, in our own words — not the published wording**. What *is*
> verified against the statement is the structure: which items exist, their numbering (including the
> deleted 15 and 22 and the added D1 and D2), their section placement, and their topic labels.
> **Complete the official checklist from the statement or EQUATOR for anything you submit.**

> **What this file used to be.** Until this revision it listed items 1–27 sequentially — including 15
> and 22, which PRISMA-DTA deletes, and **without D1 and D2, the two DTA-specific items the extension
> exists to add**. Its item 2 asked a diagnostic-accuracy review to summarise "interventions". It was
> PRISMA 2009 renumbered, presented as PRISMA-DTA. Any assessment made with it scored two items the
> guideline had removed and never scored the two it had introduced.

## Main checklist (27 items)

### Title

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 1 | Title | The report is identified as a systematic review, or meta-analysis, **of DTA studies**. |

### Abstract

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 2 | Abstract | Assessed with the **abstract checklist below**, not by a single item here. |

### Introduction

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 3 | Rationale | Why the review was done, set against what is already known. |
| **D1** | **Clinical role of index test** | The scientific and clinical background: what the index test is **intended to be used for** and its **role in the care pathway**, plus — where relevant — the reasoning behind a minimally acceptable accuracy (or a minimum difference in accuracy, for a comparative design). **DTA-specific; has no counterpart in PRISMA.** |
| 4 | Objectives | An explicit statement of the question in terms of **participants, index test, and target conditions**. |

### Methods

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 5 | Protocol and registration | Where the protocol can be read (e.g. a web address), and the registration number where one exists. |
| 6 | Eligibility criteria | The study characteristics used to decide eligibility — participants, setting, index test, reference standards, target conditions, study design — and the report characteristics (years, language, publication status), **with the reasoning for them**. |
| 7 | Information sources | Every source searched, including databases with their coverage dates and any contact with study authors, and the date each was last searched. |
| 8 | Search | The **full** search strategy for every database and other source, with any limits, in enough detail to repeat it. |
| 9 | Study selection | How studies were selected — screening, eligibility, inclusion in the review, and where applicable inclusion in the meta-analysis. |
| 10 | Data collection process | How data were extracted from reports (piloted forms, independent extraction, duplicate extraction) and how data were obtained or confirmed from investigators. |
| 11 | Definitions for data extraction | The definitions used when extracting and classifying **target conditions, index tests, reference standards** and other characteristics such as study design and clinical setting. |
| 12 | Risk of bias and applicability | The methods used to assess risk of bias in each study **and** concerns about applicability to the review question. |
| 13 | Diagnostic accuracy measures | Which accuracy measures are the principal ones (sensitivity, specificity, and so on) **and the unit of assessment** — per patient or per lesion. |
| 14 | Synthesis of results | How the data were handled and combined and how between-study variability was described. The statement names six cases to address: multiple definitions of the target condition; multiple positivity thresholds; multiple index-test readers; indeterminate results; grouping and comparing tests; and differing reference standards. |
| **D2** | **Meta-analysis** | The statistical methods used for the meta-analysis, where one was performed. **DTA-specific; has no counterpart in PRISMA.** |
| 16 | Additional analyses | The methods of any additional analyses — sensitivity, subgroup, meta-regression — **and which of them were prespecified**. |

### Results

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 17 | Study selection | The count at every stage of selection — records screened, those whose eligibility was assessed, studies entering the review, and studies entering any meta-analysis — together with why studies dropped out at each stage, ideally drawn as a flow diagram. |
| 18 | Study characteristics | For each included study, a citation and its key characteristics: participant presentation and prior testing, clinical setting, study design, target-condition definition, index test, reference standard, sample size, and funding source. |
| 19 | Risk of bias and applicability | The risk-of-bias and applicability assessment **for each study**. |
| 20 | Results of individual studies | For every analysis in every study — each unique combination of index test, reference standard and positivity threshold — the **2 × 2 data (TP, FP, FN, TN)** with accuracy estimates and confidence intervals, ideally shown as a forest plot or a receiver operating characteristic **plot**. |
| 21 | Synthesis of results | Test accuracy including its variability; where a meta-analysis was done, the results with confidence intervals. |
| 23 | Additional analyses | The results of any additional analyses, including sensitivity, subgroup and meta-regression analyses, and analyses of the index test such as failure rates, the proportion of inconclusive results, and adverse events. |

### Discussion

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 24 | Summary | The main findings, **with the strength of the evidence**. |
| 25 | Limitations | Limitations of the included studies (risk of bias, applicability) **and** limitations of the review process itself, such as incomplete retrieval. |
| 26 | Conclusions | A general interpretation against other evidence, and implications for future research and for clinical practice — including the intended use and clinical role of the index test. |

### Funding

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 27 | Funding | Sources of funding and other support for the review, and the role of the funders. |

## Abstract checklist

PRISMA-DTA carries its own abstract checklist; item 2 above defers to it. Score it separately, with its
own denominator — the same discipline as `PRISMA_2020_Abstracts.md`. Its items, in our own words:

| # | Topic | What to check is reported |
|---|-------|---------------------------|
| 1 | Title | Identified as a systematic review, or meta-analysis, of DTA studies. |
| 2 | Objectives | The research question, with its components — participants, index test, target conditions. |
| 3 | Eligibility criteria | The study characteristics used to decide eligibility. |
| 4 | Information sources | The key databases searched and the search dates. |
| 5 | Risk of bias and applicability | The methods used to assess risk of bias and applicability. |
| **A1** | **Synthesis of results** (methods) | The methods used for the data synthesis. **DTA-specific.** |
| 6 | Included studies | How many studies and of what type, and the participants and relevant study characteristics, including the reference standard. |
| 7 | Synthesis of results (results) | The diagnostic-accuracy results, preferably with the number of studies and participants; accuracy and its variability, and where a meta-analysis was done, summary results with confidence intervals. |

The remaining abstract items (interpretation, funding, registration) follow the PRISMA pattern; consult
Table 3 of the statement for their exact wording before scoring an abstract you intend to report.

## Notes for Assessors

- **Do not score items 15 or 22.** PRISMA-DTA deletes them. An assessment that reports 27 sequential
  items has not used this instrument.
- **D1 and D2 are the point of the extension.** A review that never states the index test's intended
  clinical role (D1), or that meta-analyses without reporting the statistical methods used (D2), is
  missing the DTA-specific requirements — and these are exactly the items a generic PRISMA checklist
  cannot surface.
- **Item 13's unit of assessment** (per patient vs per lesion) is the item most often skipped in
  imaging reviews, and it changes what every downstream accuracy estimate means.
- **Item 20 wants the 2 × 2 cells**, not just sensitivity and specificity. Without TP/FP/FN/TN a reader
  cannot recompute the estimates or pool them.
- For the risk-of-bias assessment referenced by items 12 and 19, see `QUADAS2.md`.
- For a review that is not diagnostic test accuracy, use `PRISMA_2020.md` with
  `PRISMA_2020_Abstracts.md`.

# PRISMA-P Checklist

**Preferred Reporting Items for Systematic review and Meta-Analysis Protocols**
Version: PRISMA-P 2015
Source: Shamseer L et al. BMJ 2015;349:g7647.

Source: Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. *BMJ* 2015;350:g7647 (DOI 10.1136/bmj.g7647).
Licence: Crossref returns no Creative Commons licence for this article.
Verification: all 26 items were compared against Table 3 of the CC BY *Systematic Reviews*
co-publication of the same statement (Europe PMC full text, PMC4320440); 26/26 match, with no
item missing and none invented.

## Checklist Items (17 items, 26 sub-items)

### Administrative Information

| # | Item | Description |
|---|------|-------------|
| 1a | Identification | Identify the report as a protocol of a systematic review |
| 1b | Update | If the protocol is for an update of a previous systematic review, identify as such |
| 2 | Registration | If registered, provide the name of the registry (such as PROSPERO) and registration number |
| 3a | Contact | Provide name, institutional affiliation, e-mail address of all protocol authors; provide physical mailing address of corresponding author |
| 3b | Contributions | Describe contributions of protocol authors and identify the guarantor of the review |
| 4 | Amendments | If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments |
| 5a | Sources | Indicate sources of financial or other support for the review |
| 5b | Sponsor | Provide name for the review funder and/or sponsor |
| 5c | Role of sponsor or funder | Describe roles of funder(s), sponsor(s), and/or institution(s), if any, in developing the protocol |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 6 | Rationale | Describe the rationale for the review in the context of what is already known |
| 7 | Objectives | Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO) |

### Methods

| # | Item | Description |
|---|------|-------------|
| 8 | Eligibility criteria | Specify the study characteristics (such as PICO, study design, setting, time frame) and report characteristics (such as years considered, language, publication status) to be used as criteria for eligibility for the review |
| 9 | Information sources | Describe all intended information sources (such as electronic databases, contact with study authors, trial registers or other grey literature sources) with planned dates of coverage |
| 10 | Search strategy | Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated |
| 11a | Data management | Describe the mechanism(s) that will be used to manage records and data throughout the review |
| 11b | Selection process | State the process that will be used for selecting studies (such as two independent reviewers) through each phase of the review (that is, screening, eligibility and inclusion in meta-analysis) |
| 11c | Data collection process | Describe planned method of extracting data from reports (such as piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators |
| 12 | Data items | List and define all variables for which data will be sought (such as PICO items, funding sources), any pre-planned data assumptions and simplifications |
| 13 | Outcomes and prioritization | List and define all outcomes for which data will be sought, including prioritization of main and additional outcomes, with rationale |
| 14 | Risk of bias in individual studies | Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis |
| 15a | Data synthesis | Describe criteria under which study data will be quantitatively synthesised |
| 15b | Data synthesis | If data are appropriate for quantitative synthesis, describe planned summary measures, methods of handling data and methods of combining data from studies, including any planned exploration of consistency (such as I^2, Kendall's tau) |
| 15c | Data synthesis | Describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression) |
| 15d | Data synthesis | If quantitative synthesis is not appropriate, describe the type of summary planned |
| 16 | Meta-bias(es) | Specify any planned assessment of meta-bias(es) (such as publication bias across studies, selective reporting within studies) |
| 17 | Confidence in cumulative evidence | Describe how the strength of the body of evidence will be assessed (such as GRADE) |

---

## Notes for Assessors

- This checklist should be read in conjunction with the PRISMA-P Explanation and Elaboration document
- Amendments to a review protocol should be tracked and dated
- The copyright for PRISMA-P is held by the PRISMA-P Group and is distributed under a Creative Commons Attribution Licence 4.0

# PRISMA-ScR Checklist (scoping reviews)

**Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews**
Version: PRISMA-ScR 2018 — the checklist keeps PRISMA's **1–27 numbering**. Of those 27 rows, **5 are
"Not applicable for scoping reviews"** (13, 15, 16, 22, 23) and **2 are optional, phrased "If done"**
(**12** and **19**, critical appraisal), leaving **20 essential items**.
Source: Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for
Scoping Reviews (PRISMA-ScR): Checklist and Explanation. *Ann Intern Med* 2018;169(7):467-473
(DOI 10.7326/M18-0850). EQUATOR Network. Scoping-review conduct guidance: JBI (Peters et al.);
Arksey & O'Malley; Levac et al.

Apply when the manuscript is a **scoping review** — a review that *maps* the breadth of evidence on a
topic, characterises concepts and definitions, and identifies gaps, rather than answering a focused
effectiveness or accuracy question (that is a systematic review → `PRISMA_2020.md` / `PRISMA_DTA.md`).
For the design/conduct review of the same study, pair with the SC1–SC8 domain probes in
`peer-review` / `self-review` `references/domain-probes/scoping_review.md`; for the flow diagram use
`make-figures` (PRISMA-style).

> **Fidelity and licence.** Published in *Annals of Internal Medicine* (© American College of
> Physicians) under **no open licence**; one secondary source lists CC BY-NC-SA 3.0, which is also not
> permissively reusable. The descriptions below are an **in-house summary of item intent in our own
> words — not the published wording**. The structure — the 1–27 numbering, which rows are N/A, which
> are optional, and the section grouping — is **verified against the statement's table**. Consult the
> published article for exact item text, and complete the official checklist for anything you submit.

> **What this file used to be.** It numbered the items **1–22**: it had deleted the five "Not
> applicable" rows and closed the gaps, so every item from 13 onward carried the wrong number — 10 of
> 22. Critical appraisal within sources of evidence sat at 16 instead of **19**; Limitations at 20
> instead of **25**; Funding at 22 instead of **27**. The header and the assessor notes named the
> optional pair as "12 and 16", but 16 is *Additional analyses* and is **N/A**, not optional. Item
> numbers are how authors, reviewers and editors refer to a checklist, so a renumbered instrument
> silently breaks every reference to it.

## Reporting items (grouped by section)

### Title
| # | Item | What to check is reported |
|---|------|---------------------------|
| 1 | Title | The report is identified as a **scoping review** in the title. |

### Abstract
| # | Item | What to check is reported |
|---|------|---------------------------|
| 2 | Structured summary | A structured abstract that, as relevant, states the background and objectives, the eligibility criteria and the evidence sources searched, the **charting** approach, and the headline results and conclusions tied to the review questions. |

### Introduction
| # | Item | What to check is reported |
|---|------|---------------------------|
| 3 | Rationale | The rationale situated against existing knowledge, and **why the question suits a scoping (mapping) approach** rather than a systematic review. |
| 4 | Objectives | Explicit questions/objectives stated with their key elements — **Population, Concept, Context (PCC)** or equivalent — used to frame the review. |

### Methods
| # | Item | What to check is reported |
|---|------|---------------------------|
| 5 | Protocol and registration | Whether an a-priori **protocol** exists, where it can be accessed, and any registration details. (Note: PROSPERO does **not** register scoping reviews; OSF/Figshare or a published protocol is used.) |
| 6 | Eligibility criteria | Characteristics used as eligibility criteria (years, language, publication status, source types) **with a rationale**. |
| 7 | Information sources | All information sources searched (databases with coverage dates, grey-literature sources, author contact) and the date the most recent search was run. |
| 8 | Search | A reproducible search strategy for **at least one database** (search terms and any limits), detailed enough to be repeated by another team. |
| 9 | Selection of sources of evidence | The process for selecting **sources of evidence** (screening and eligibility) into the review. |
| 10 | Data charting process | The **data-charting** methods (calibrated/team-tested form; whether charting was independent/duplicate; iterative refinement) and any process for obtaining/confirming data from investigators. "Charting," not "extraction." |
| 11 | Data items | The data items charted — every variable sought — plus any assumptions or simplifications made. |
| **12** | Critical appraisal of individual sources of evidence — **OPTIONAL** | *If done*, the rationale for critical appraisal, the methods used, and how the information was used in any data synthesis. Scoping reviews are **not required** to appraise; "critical appraisal" is used instead of "risk of bias." |
| 13 | Summary measures | **Not applicable for scoping reviews.** |
| 14 | Synthesis of results | The methods for **handling and summarising (charting/mapping)** the charted data. |
| 15 | Risk of bias across studies | **Not applicable for scoping reviews.** |
| 16 | Additional analyses | **Not applicable for scoping reviews.** |

### Results
| # | Item | What to check is reported |
|---|------|---------------------------|
| 17 | Selection of sources of evidence | Numbers screened, assessed for eligibility, and included, with reasons for exclusion at each stage — ideally a **flow diagram**. |
| 18 | Characteristics of sources of evidence | Per-source characteristics for which data were charted, with citations. |
| **19** | Critical appraisal within sources of evidence — **OPTIONAL** | *If done* (see item 12), the critical-appraisal data per source. |
| 20 | Results of individual sources of evidence | For each included source, the relevant charted data that relate to the review questions/objectives. |
| 21 | Synthesis of results | The charting results **summarised/mapped** in relation to the review questions/objectives — a **map/characterisation, not a pooled effect estimate**. |
| 22 | Risk of bias across studies | **Not applicable for scoping reviews.** |
| 23 | Additional analyses | **Not applicable for scoping reviews.** |

### Discussion
| # | Item | What to check is reported |
|---|------|---------------------------|
| 24 | Summary of evidence | The main results summarised (overview of concepts, themes, evidence types), linked to the questions/objectives and their relevance to key groups. |
| 25 | Limitations | Limitations of the **scoping-review process**. |
| 26 | Conclusions | A general interpretation against the questions/objectives, with potential implications and/or next steps (e.g. whether a systematic review is warranted). |

### Funding
| # | Item | What to check is reported |
|---|------|---------------------------|
| 27 | Funding | Sources of funding for the included sources of evidence **and** for the scoping review, and the funders' role. |

---

## Notes for Assessors

- **Score against 1–27, not 1–20.** The five N/A rows stay in the table so the numbering matches the
  guideline; mark them N/A rather than deleting them. The denominator for compliance is the 20
  essential items, plus the two optional items only where the review says it appraised.
- **The optional pair is 12 and 19**, not 12 and 16. Item 16 is *Additional analyses* and is N/A.
  Why they are optional: a scoping review may include documents, blogs, websites, interviews and
  opinions, and is not required to examine risk of bias in what it includes.
- The **highest-yield** checks (where scoping reviews most often go wrong): **item 4** (objectives
  framed as PCC mapping, not a focused PICO effectiveness question — a study asking "is X
  effective/accurate?" should be a systematic review), **item 21** (results presented as a **map** —
  counts, categories, themes, evidence gaps — **not** a pooled OR/RR/AUC or a definitive
  effectiveness conclusion), and **items 12/19** (do not penalise a scoping review for omitting
  critical appraisal, and do not let it claim GRADE-style certainty it never derived).
- A "scoping review" that reports pooled effect estimates or a meta-analysis, or concludes an
  intervention is effective or a test accurate, has overstepped its design — its synthesis (21) and
  conclusions (26) should be downgraded to descriptive mapping, or the study reframed as a systematic
  review.
- **Terminology is deliberate and the statement says so**: **charting** rather than extraction,
  **sources of evidence** rather than studies, and **critical appraisal** rather than risk of bias —
  the last because risk of bias suits systematic reviews of interventions, while a scoping review
  takes in quantitative and qualitative research, expert opinion and policy documents. A scoping
  review claiming PROSPERO registration is in error; PROSPERO excludes them.

# PROBAST Assessment Guide

Prediction model Risk Of Bias ASsessment Tool.
Version: PROBAST 2019 — 20 signalling questions across 4 domains (2 / 3 / 6 / 9).
Source: Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: A Tool to
Assess the Risk of Bias and Applicability of Prediction Model Studies. *Ann Intern Med*
2019;170(1):51-58 (PMID 30596875; DOI 10.7326/M18-1376). Explanation and Elaboration:
*Ann Intern Med* 2019;170(1):W1-W33.

> **Fidelity and licence.** PROBAST is published in *Annals of Internal Medicine* (© American College
> of Physicians) and carries **no open licence**. The signalling questions below are therefore an
> **in-house summary of what each question asks — not the published wording**. The numbering,
> the domain structure and the count (2 / 3 / 6 / 9) are verified against the statement; the phrasing
> is ours. **Complete the official tool from probast.org or the statement for any assessment you
> report.** Use this file to organise a first pass.

> **PROBAST 2019 is superseded.** PROBAST+AI (2025) replaces it for all new assessments, covering
> regression and AI/ML models alike — see `PROBAST_AI.md`. Use this file only when appraising against
> the 2019 instrument specifically (for example, reproducing an earlier review).

## Structure

PROBAST assesses 4 domains, each for Risk of Bias AND Applicability.
- **Signalling questions**: Yes / Probably yes / No / Probably no / No information
- **Domain judgment**: Low / High / Unclear
- **Overall judgment**: High if any domain is high; Low only if all domains are low

## Domain 1: Participants

### Signalling questions (risk of bias) — what each asks

1.1 Whether the data source suits the question — a cohort, a randomised trial, or nested
    case–control data, rather than a design that distorts the sampling.
1.2 Whether every inclusion and exclusion applied to participants was appropriate.

### Applicability
- Do the participants and setting match the review question?

## Domain 2: Predictors

### Signalling questions (risk of bias) — what each asks

2.1 Whether predictors were defined and assessed the same way for every participant.
2.2 Whether predictors were assessed without knowledge of the outcome.
2.3 Whether every predictor is available at the moment the model is meant to be used.

### Applicability
- Do the predictors, their assessment, and timing match the review question?

## Domain 3: Outcome

### Signalling questions (risk of bias) — what each asks

3.1 Whether the outcome was determined by an appropriate method.
3.2 Whether the outcome definition was prespecified or a standard one.
3.3 Whether predictors were kept out of the outcome definition.
3.4 Whether the outcome was defined and determined the same way for every participant.
3.5 Whether the outcome was determined without knowledge of predictor information.
3.6 Whether the interval between predictor assessment and **outcome determination** was
    appropriate — the question is about when the outcome was *determined*, not merely when it
    occurred.

### Applicability
- Does the outcome and its definition/timing match the review question?

## Domain 4: Analysis

### Signalling questions (risk of bias) — what each asks

4.1 Whether the number of participants with the outcome was reasonable.
4.2 Whether continuous and categorical predictors were handled appropriately.
4.3 Whether every enrolled participant was included in the analysis.
4.4 Whether participants with missing data were handled appropriately.
4.5 Whether selection of predictors on the basis of univariable analysis was avoided.
4.6 Whether complexities in the data were accounted for — the statement names **censoring,
    competing risks, and the sampling of control participants** as the cases to look for.
4.7 Whether relevant measures of model performance were evaluated appropriately.
4.8 Whether **overfitting, underfitting, and optimism** in model performance were accounted for.
    All three, not overfitting alone.
4.9 Whether the predictors and their assigned weights in the final model correspond to the
    **results from** the reported multivariable analysis.

### For validation studies (additional)
- Were the model and its performance evaluated appropriately?

## For AI / machine-learning models

Use **`PROBAST_AI.md`** (PROBAST+AI 2025; Moons KGM et al. *BMJ* 2025;388:e082505,
DOI 10.1136/bmj-2024-082505), which carries the instrument's own 16 development and 18 evaluation
signalling questions. Do not improvise AI addenda on top of the 2019 questions: an earlier version of
this file listed four invented bullets under a heading that implied they were part of the extension,
which they were not.

## When to Use

- Diagnostic prediction models (e.g., AI classifiers for imaging findings)
- Prognostic prediction models (e.g., risk scores, survival prediction)
- Both development AND validation studies
- For any model developed with machine learning or deep learning, and for new assessments generally,
  use PROBAST+AI (`PROBAST_AI.md`) rather than this file

# PROBAST+AI Assessment Guide

Prediction model Risk Of Bias ASsessment Tool — updated for AI/ML methods.
Reference: Moons KGM et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ 2025;388:e082505. doi: 10.1136/bmj-2024-082505.
Website: https://www.probast.org

Version: PROBAST+AI (2025) — 16 development / 18 evaluation signalling questions
Licence: CC BY-NC 4.0 (non-commercial) — confirmed via Crossref (DOI 10.1136/bmj-2024-082505). Not redistributable verbatim under this repository's MIT licence.
Verification: the development and evaluation signalling-question counts and the four-step process were checked against the statement and its supplements during this audit.

## Purpose

PROBAST+AI is the updated version of PROBAST (2019) that extends the original tool to cover prediction models developed using **machine learning and artificial intelligence** methods, in addition to traditional regression-based models. It replaces PROBAST-2019 for all new assessments.

## What Changed from PROBAST-2019

- Updated signalling questions to address AI/ML-specific methodological considerations
- Expanded from 20 to **16 targeted signalling questions** for model development and **18 for model evaluation**
- Covers all prediction model types: regression, ML, DL, ensemble methods
- Applicable regardless of whether the model uses statistical or AI/ML techniques
- Original PROBAST Explanation and Elaboration document still provides useful background

## Structure

PROBAST+AI has two distinct parts:
1. **Model development** assessment: quality and applicability
2. **Model evaluation** assessment: risk of bias and applicability

Both parts assess 4 domains:
- Domain 1: Participants
- Domain 2: Predictors
- Domain 3: Outcome
- Domain 4: Analysis

Signalling questions answered: Yes / Probably Yes / No / Probably No / No Information

## Four Steps of Assessment

1. Specify the intended purpose of the prediction model assessment or systematic review
2. Classify the type of prediction model study (development or evaluation or both)
3. Assess quality/applicability (development) or risk of bias/applicability (evaluation) for each domain
4. Assess overall quality (development) or risk of bias (evaluation)

## Domain 1: Participants

### For Model Development (Quality)
- Were participants representative of the target population?
- Was the study setting appropriate?
- Were inclusion/exclusion criteria clearly defined?

### For Model Evaluation (Risk of Bias)
- Were participants selected appropriately for the evaluation?
- Were participants representative of the intended target population?

### Applicability
- Do the participants match the intended target population for the model?

## Domain 2: Predictors

### For Model Development (Quality)
- Were predictors defined and assessed in a standardized way?
- Were predictors available at the intended moment of use?
- Were all candidate predictors assessed appropriately?

### For Model Evaluation (Risk of Bias)
- Were predictors assessed in the same way as in the development study?
- Were predictors assessed without knowledge of the outcome?

### Applicability
- Do the predictor definitions and assessments match the intended use?

## Domain 3: Outcome

### For Model Development (Quality)
- Was the outcome clearly defined?
- Was the outcome determined appropriately?
- Was the time horizon appropriate?

### For Model Evaluation (Risk of Bias)
- Was the outcome determined in the same way as in development?
- Was the outcome determined without knowledge of predictor information?

### Applicability
- Does the outcome definition and assessment match the intended target?

## Domain 4: Analysis

### For Model Development (Quality — key AI/ML considerations)
- Was the sample size adequate for the modeling approach?
- Were missing data handled appropriately?
- Was the choice of algorithm/method appropriate?
- Were hyperparameters tuned appropriately (for ML/DL models)?
- Was overfitting addressed (regularization, early stopping, cross-validation)?
- Was model performance assessed using appropriate metrics?
- Was internal validation performed adequately (bootstrapping, cross-validation)?
- Were relevant performance measures reported (discrimination, calibration)?

### For Model Evaluation (Risk of Bias — key AI/ML considerations)
- Was the evaluation dataset independent from the development dataset?
- Were appropriate performance measures used (discrimination AND calibration)?
- Was the statistical analysis appropriate?
- Were model updates or recalibration described if performed?

### Applicability
- Does the analysis approach and validation strategy support the intended clinical use?

## Overall Judgment

### For Model Development: Overall Quality
- **High quality**: Low concern across all domains
- **Unclear quality**: Unclear in at least one domain
- **Low quality**: High concern in at least one domain

### For Model Evaluation: Overall Risk of Bias
- **Low risk**: Low risk in all domains
- **Unclear risk**: Unclear in at least one domain, no high risk in any
- **High risk**: High risk in at least one domain

## Key Differences from Original PROBAST-2019

| Aspect | PROBAST-2019 | PROBAST+AI |
|--------|-------------|------------|
| Scope | Regression models | Regression + AI/ML models |
| Development assessment | Risk of bias | **Quality** (more appropriate term) |
| Evaluation assessment | Risk of bias | Risk of bias (unchanged) |
| AI-specific items | None | Hyperparameter tuning, overfitting mitigation, algorithm choice |
| Signalling questions | 20 per assessment | 16 (development) / 18 (evaluation) |

## When to Use

- Systematic reviews of prediction model studies (development, validation, or both)
- Applies to ALL prediction models regardless of algorithm (logistic regression, random forest, deep learning, etc.)
- Replaces PROBAST-2019 for new assessments
- Use alongside TRIPOD+AI for reporting quality assessment

# QUADAS-2 Assessment Guide

Quality Assessment of Diagnostic Accuracy Studies, version 2.
Version: QUADAS-2 (2011) — 4 domains, **10 signalling questions** (3 / 2 / 2 / 3), risk of bias for every
domain and applicability for the first three only.
Source: Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a
revised tool for the quality assessment of diagnostic accuracy studies. *Ann Intern Med*
2011;155(8):529-536 (DOI 10.7326/0003-4819-155-8-201110180-00009). The tool itself, training material
and worked examples were distributed from www.quadas.org; that domain no longer resolves, and the
QUADAS group now distributes its tools from
https://www.bristol.ac.uk/population-health-sciences/projects/quadas/

> **Fidelity and licence.** QUADAS-2 is published in *Annals of Internal Medicine* (© American College
> of Physicians) under **no open licence**. **Verified against the published article**: the four
> domains, the 3 / 2 / 2 / 3 split of signalling questions, what each question asks, the answer
> options, the judgement levels, and the restriction of applicability to the first three domains all
> match. The descriptions below state what each question asks **in our own words** rather than
> reproducing the published wording. **Complete the official QUADAS-2 form from www.quadas.org for
> any assessment you report.**
>
> **QUADAS-3 has superseded this tool.** The QUADAS group states that "QUADAS-3 is the current
> iteration of the QUADAS tool and is the current recommended version" — it extends QUADAS-2 by
> introducing an explicit ideal-test-accuracy-trial comparator and by moving assessment **from the
> study level to the level of individual accuracy estimates**. This file documents QUADAS-2, which
> remains what most published reviews used. For a new review use **`QUADAS3.md`**;
> `QUADAS_C.md` should now be paired with QUADAS-3 rather than with this tool.

## How the tool is applied — four phases

QUADAS-2 is not a fixed questionnaire. The statement specifies four phases, and skipping the second
is the most common misuse:

1. **State the review question** — patients, index test, reference standard and target condition.
   Describe patients by setting, the intended use of the index test, presentation, and prior testing,
   because accuracy depends on where in the diagnostic pathway the test sits.
2. **Tailor the tool to the review.** Add or omit signalling questions and write review-specific
   guidance on how each is to be judged. (For an objective index test, for example, the question about
   blinding the interpreter to the reference standard may not apply.) Avoid piling on extra questions.
   At least two people should pilot the tailored tool; refine it if agreement is poor.
3. **Draw the study's flow diagram** — use the published one, or draw it yourself if it is absent or
   inadequate. It need not be reported; it exists to make the flow-and-timing judgements possible.
4. **Judge bias and applicability**, recording the information each judgement rests on.

## Answers and judgements

- **Signalling questions**: Yes / No / Unclear, phrased so that **Yes indicates low risk of bias**.
- **Risk of bias**: Low / High / Unclear.
- **Applicability concern** (domains 1–3 only): Low / High / Unclear.
- **Unclear is only for insufficient reporting**, not for a difficult judgement.

**How a "No" is handled.** If every signalling question in a domain is Yes, risk of bias can be judged
low. A **No does not automatically make the domain High** — it establishes that *potential for bias
exists*, and the reviewer then applies the review-specific guidance written in phase 2 to reach the
judgement. A file or workflow that maps "any No → High" has removed the judgement the tool asks for.

## Domain 1: Patient Selection

**Risk of bias — could patient selection have introduced bias?**

1. Whether enrolment took a consecutive or random sample of eligible patients.
2. Whether a case–control design was avoided.
3. Whether the study avoided inappropriate exclusions.

*Why it matters*: enrolling patients whose diagnosis is already confirmed, or excluding the
difficult-to-diagnose, inflates apparent accuracy; excluding patients with obvious signs of the target
condition can deflate it.

**Applicability** — whether the included patients and the setting differ from the review question
(severity, demographics, comorbidity and differential diagnosis, setting, prior testing).

## Domain 2: Index Test

**Risk of bias — could the conduct or interpretation of the index test have introduced bias?**

1. Whether the index test was interpreted without knowledge of the reference standard result.
2. Whether a threshold, if one was used, was prespecified.

*Why it matters*: this is the diagnostic equivalent of blinding, and its force depends on how
subjective the index test is and on the order of testing. Choosing the threshold after the fact to
maximise sensitivity or specificity overstates performance that will not hold in a new sample.

**Applicability** — whether the index test, how it was carried out, or how it was interpreted differs
from the review question.

## Domain 3: Reference Standard

**Risk of bias — could the reference standard, its conduct or its interpretation have introduced bias?**

1. Whether the reference standard is likely to classify the target condition correctly.
2. Whether the reference standard was interpreted without knowledge of the index test result.

*Why it matters*: accuracy estimates assume the reference standard is correct, so that any
disagreement is the index test's error.

**Applicability** — whether the target condition **as the reference standard defines it** differs from
the one in the review question.

## Domain 4: Flow and Timing

**Risk of bias — could patient flow have introduced bias?**

1. Whether the interval between index test and reference standard was appropriate.
2. Whether all patients received the same reference standard.
3. Whether all patients were included in the analysis.

*Why it matters*: an interval long enough for the condition to change causes misclassification, and
how long is too long depends on the condition. Verifying only some patients, or verifying different
patients with different reference standards, biases the estimate. Patients lost between enrolment and
the 2 × 2 table differ systematically from those who remain.

**This domain has no applicability judgement.**

## Reporting the assessment

- **Do not produce a summary quality score.** The statement is explicit about this, for the reasons
  well established in the literature on quality scores.
- A study low on every domain may be called low risk of bias, or low concern for applicability,
  overall. High or unclear on one or more domains means it is at risk of bias, or of concern.
- At minimum, report the assessment across all included studies — how many were low, high or unclear
  per domain — and consider highlighting signalling questions on which studies consistently do badly.
- Restricting the primary analysis to low-risk studies is legitimate, but it is often better to
  include everything and then investigate heterogeneity — by subgroup, sensitivity analysis, or by
  entering domains as covariates in meta-regression.
- QUADAS-2 does **not** cover studies comparing multiple index tests. The development group considered
  it and concluded the evidence base was insufficient.

## Common Issues in DTA Studies

- **Partial verification bias**: Not all patients receive the reference standard (especially when invasive, e.g., biopsy)
- **Differential verification**: Different reference standards used for different patients
- **Incorporation bias**: Index test forms part of the reference standard
- **Review bias**: Knowledge of index test results influences reference standard interpretation
- **Clinical review bias**: Additional clinical information available during index test interpretation
- **Uninterpretable results**: Exclusion of technically inadequate or indeterminate results

## Related

- `PRISMA_DTA.md` items 12 and 19 are the reporting counterparts: the methods used for this
  assessment, and its result presented **for each study**.

# QUADAS-3 Assessment Guide

Quality Assessment of Diagnostic Accuracy Studies, version 3 — **the current recommended version**.
Version: QUADAS-3 tool v1.2 — 6 phases, 4 domains, **20 signalling questions** (4 / 4 / 8 / 4).
Source: Whiting PF, Tomlinson E, Rutjes AWS, Davenport C, Yang B, Westwood M, et al. QUADAS-3: a
revised tool for the quality assessment of diagnostic test accuracy studies. *Ann Intern Med*
2026;179(4):548-555 (DOI 10.7326/ANNALS-25-02104). The tool itself, the Explanation & Elaboration
report and an introductory video are distributed by the QUADAS group at
https://www.bristol.ac.uk/population-health-sciences/projects/quadas/quadas-3/
Explanation & Elaboration: Davenport CF, Rutjes AWS, Mallett S, Tomlinson E, Yang B, et al.
QUADAS-3 explanation and elaboration: guidance for quality assessment of diagnostic test accuracy
studies. *Ann Intern Med* 2026;179(4):e2504943 (DOI 10.7326/ANNALS-25-04943).
Resource site: **www.quadas.info** (the older www.quadas.org no longer resolves).

> **Fidelity and licence.** QUADAS-3 is published in *Annals of Internal Medicine* (© American
> College of Physicians) under **no open licence** — Crossref returns only ACP's text-and-data-mining
> policy. The descriptions below state what each question asks **in our own words** rather than
> reproducing the published wording. **Complete the official QUADAS-3 form (`QUADAS-3 1.2.docx`)
> from the page above for any assessment you report, and read the E&E report before using it.**
>
> Verification: the six phases and when each is completed, the four domains, all 20 signalling
> questions, the response options, the domain-level rule, which domains carry an applicability
> judgement, and the overall-judgement rules were compared against **the official tool document
> v1.2** distributed by the QUADAS group. All matched. The *Using QUADAS-C with QUADAS-3*,
> *Tailoring* and *no "moderate" grade* sections below are taken from the **Explanation and
> Elaboration paper**, read directly.

## QUADAS-3 supersedes QUADAS-2

The QUADAS group states QUADAS-3 "is the current version of QUADAS and the tool that we recommend."
For a new review, use this file. `QUADAS2.md` documents the 2011 tool, which is what most published
reviews used and what you will still be reading in them.

What changed, in the group's own framing:

| | QUADAS-2 | QUADAS-3 |
|---|---|---|
| Unit of assessment | the **study** | **each set of accuracy estimates** |
| Comparator for judging | implicit | an explicit **ideal test accuracy trial**, defined per synthesis question |
| Synthesis questions | one, implicit | **multiple, defined up front** |
| Overall judgment | none | **a formal phase (6)** |
| Phases | 4 | **6** |
| Domains | Patient Selection, Index Test, Reference Standard, Flow and Timing | **Participants, Index Test, Target Condition, Analysis** |
| Signalling questions | 10 (3/2/2/3) | **20 (4/4/8/4)** |
| Third judgement level | "unclear" | **"insufficient information" (II)** |

Note the domain rename: QUADAS-2's *Flow and Timing* is gone. Timing moved into **Target
Condition** (the index-test-to-reference-standard interval), and participant exclusions, missing
data and the unit of analysis moved into the new **Analysis** domain.

**Comparative accuracy reviews**: the group recommends using **QUADAS-C in addition to QUADAS-3**
(`QUADAS_C.md`). QUADAS-C was written against QUADAS-2 and needs adaptation — see the next section.

## Using QUADAS-C with QUADAS-3

For comparative accuracy studies — where two or more index tests are compared — the guideline says
to use **QUADAS-C alongside QUADAS-3**, because such studies carry additional sources of bias
(confounding between tests, and interference of one test with another).

QUADAS-C was written as an extension of QUADAS-2, so it needs adapting. **An updated QUADAS-C is
in development**; what follows are the E&E's own *preliminary* modifications, not a finished tool.

### What you assess

The unit is a **comparative measure** — for example the difference in sensitivity or in specificity
between two tests. Most primary studies report only the separate estimate for each index test, so
**you will usually have to compute the comparative measure yourself.** Specify which estimate a
QUADAS-C assessment refers to, exactly as you do in QUADAS-3 phase 4.

### Domains are renamed onto QUADAS-3's

| QUADAS-C (as published) | becomes |
|---|---|
| Patient Selection | **Participants** |
| Index Test | Index Test *(unchanged)* |
| Reference Standard | **Target Condition** |
| Flow and Timing | **Analysis** |

### Three signalling questions change (E&E Table 8)

| QUADAS-C question | Change |
|---|---|
| **C3.2** Did the reference standard avoid incorporating any of the index tests? | **Removed** — it overlaps QUADAS-3 item 3.4 |
| **C4.2** Was there an appropriate interval between the index tests? | **Moved to domain 2** (Index Test) |
| **C4.3** Was the same reference standard used for all index tests? | **Moved to domain 3** (Target Condition) |

Everything else carries over unchanged.

### Answers and judgements follow QUADAS-3

Signalling questions take **Y / PY / PN / N / NI**; each domain is judged **low / high /
insufficient information**. The overall judgement for the comparative estimate:

- **low** if all domains are low
- **high** if at least one domain is high
- **insufficient information** if at least one domain is II and none is high

## The six phases

| Phase | What | How often |
|---|---|---|
| 1 | State the systematic review synthesis question(s) | once per review |
| 2 | Define the **ideal test accuracy trial** for each synthesis question | once per review |
| 3 | Draw a flow diagram | once per study |
| 4 | Identify which accuracy estimates to assess | once per study |
| 5 | Assess risk of bias and applicability | for each selected estimate |
| 6 | Overall judgment | for each selected estimate |

Phases 1 and 2 are review-level and **belong in the review protocol**. Phases 3–4 are study-level.
Phases 5–6 run once per selected set of estimates.

**Phase 1** — a review may address more than one synthesis question. Specify each with its
population, index test(s) and target condition, and pre-specify them in the protocol.

**Phase 2** — the ideal test accuracy trial is the study that would answer the synthesis question
with minimum bias and maximum applicability. Define it per question across: objective,
participants, index test(s), definition of the target condition, and analysis. Every later
judgement is made **against this trial**, not against an unstated ideal.

**Phase 4** — a single primary study usually yields several two-by-two tables. Assess only the
estimates relevant to a synthesis question. Record, for each: the synthesis question, the numerical
result, participants, index test and threshold, target condition, reference standard, unit of
analysis, and the analysis method. After the first estimate, **only the domains whose
characteristics differ between estimates need reassessing**.

## Phase 5 — signalling questions

Signalling questions: **Y / PY / PN / N / NI**. Domain risk-of-bias judgement: **low / high /
insufficient information (II)**.

### Domain 1: Participants (4)

| # | Signalling question |
|---|---------------------|
| 1.1 | Was a single-gate design used? |
| 1.2 | Were participants prospectively enrolled? |
| 1.3 | Was a consecutive or random sample of participants included? |
| 1.4 | Is the study group a representative sample of the intended-use population? |

*Applicability*: does the included population match the ideal trial's?

Participants who dropped out or were excluded because they did not receive the index test or the
reference standard belong in **Domain 4 (Analysis)**, not here.

### Domain 2: Index Test (4)

| # | Signalling question |
|---|---------------------|
| 2.1 | Was the index test conducted and interpreted according to the recommended instructions? |
| 2.2 | Were the index test results interpreted without knowledge of the reference standard results? |
| 2.3 | Were the index test results interpreted with the same information that would be available when the test is used in practice? |
| 2.4 | If an index test threshold was used, was it standard or pre-specified? |

*Applicability*: does the index test, its conduct and its interpretation match the ideal trial's?

2.3 is new relative to QUADAS-2 and cuts both ways — a reader given **more** information than they
would have in practice is as much a problem as one given less.

### Domain 3: Target Condition (8)

| # | Signalling question |
|---|---------------------|
| 3.1 | Does the reference standard adequately identify those with and without the target condition? |
| 3.2 | Was the target condition assessed in all participants? |
| 3.3 | Was the target condition assessed in the same way in all participants? |
| 3.4 | Did the reference standard avoid incorporating the index test? |
| 3.5 | Was the reference standard conducted and interpreted according to the recommended instructions? |
| 3.6 | Were the reference standard results interpreted without knowledge of the index test results? |
| 3.7 | If a reference standard threshold was used, was it standard or pre-specified? |
| 3.8 | Was there an appropriate time interval between index test and reference standard? |

*Applicability*: does the target condition as defined by the reference standard match the ideal
trial's?

This domain absorbs QUADAS-2's Reference Standard domain **and** its verification and timing
questions. 3.2 and 3.3 are partial and differential verification; 3.8 is the interval that used to
sit in Flow and Timing.

### Domain 4: Analysis (4)

| # | Signalling question |
|---|---------------------|
| 4.1 | Were all participants included in the analysis? |
| 4.2 | Were missing data handled appropriately? |
| 4.3 | Does the unit of analysis match the ideal test accuracy trial? |
| 4.4 | Were the estimates of sensitivity and specificity calculated appropriately? |

**No applicability judgement** — applicability is assessed for the first three domains only.

4.3 is where a lesion-level or sample-level analysis meets a participant-level synthesis question.
That mismatch had no home in QUADAS-2.

## Judgement rules

**Domain level.** If all signalling questions in a domain are answered *yes* or *probably yes*,
risk of bias can be judged **low**. A *no* or *probably no* **flags potential** for bias — it does
not settle it. Reviewers then apply their judgement and their review-specific guidance to decide
whether the issue is likely to have influenced the accuracy estimates.

> **A study can still be at low risk of bias with one or more signalling questions answered "no."**
> The tool says this explicitly. Do not implement "any No → High" as a rule; that replaces the
> judgement the tool asks for.

Use **insufficient information** only when too little is reported to permit a judgement. It is not
a middle rating between low and high.

**Overall (phase 6)**, per estimate, done separately for risk of bias and for applicability:

- any domain **high** → overall **high**
- all domains **low** → overall **low**
- any domain **insufficient information** and none high → overall **insufficient information**

Record a rationale naming the major limitations behind the overall judgement.

**Do not add a "moderate" grade.** Reviewers sometimes want one, to separate a study that is high
risk in a single domain from one that is high risk in several. The E&E says plainly that the
authors *do not support* this: if an estimate is high risk for one domain, it is high risk,
whatever the other domains say.

## Tailoring the tool to your review

Phase 2's tailoring is the step most often skipped, and the E&E is specific about it.

- Do it **at the protocol stage, alongside phases 1 and 2** — not when you reach the studies.
- Write review-specific guidance on how to answer each signalling question, adapting the general
  guidance tables. Publish it as a web appendix so the application is auditable.
- Draw on **both clinical and methodological** expertise in the review area.
- **Do not remove signalling questions.** Keep them even when they cannot bite: if two-gate designs
  were excluded, every study answers "yes" to 1.1, and recording that shows the issue was
  considered. Deleting the question hides that.
- If you add a question, it must: address **one** issue only; concern **risk of bias, not reporting
  quality**; and be **factual**, phrased so that "yes"/"probably yes" means bias is absent.

## When to Use

- Systematic reviews assessing the accuracy of tests used for **diagnosis, screening or staging**
- New reviews — QUADAS-3 is the current recommended version
- Alongside **QUADAS-C** when the review compares the accuracy of two or more index tests
- Read the **Explanation & Elaboration report** before first use, and tailor the signalling
  questions and their guidance to your review (that tailoring is the step most often skipped)
- Not for prediction models (PROBAST), non-randomised intervention studies (ROBINS-I), or
  randomised trials (RoB 2)

# QUADAS-C Assessment Guide

Quality Assessment of Diagnostic Accuracy Studies — Comparative (extension to QUADAS-2).
Reference: Yang B et al. *Guidance on how to use QUADAS-C*. Official tool, guidance document and templates are distributed by the QUADAS group at
https://www.bristol.ac.uk/population-health-sciences/projects/quadas/quadas-c/

Version: QUADAS-C (2021)
Source: Yang B, Mallett S, Takwoingi Y, Davenport CF, Hyde CJ, Whiting PF, et al. QUADAS-C: a tool for assessing risk of bias in comparative diagnostic accuracy studies. *Ann Intern Med* 2021;174(11):1592-1599 (DOI 10.7326/M21-2234).
Licence: *Annals of Internal Medicine* (© American College of Physicians) — no open licence.
Verification: every comparative signalling question (C1.1–C1.5, C2.1–C2.5, C3.1–C3.3, C4.1–C4.5),
the five comparative study designs, the four completion phases, the domain judgement rules and the
five optional Table 4 questions were compared against the **official QUADAS-C tool (`QUADAS-C
tool.docx`) and guidance document**, both distributed by the QUADAS group at the University of
Bristol. All matched. The article itself is paywalled and the author manuscripts at two university
repositories sit behind a browser challenge; the group's own distribution made those unnecessary.

> **QUADAS-3 is now the current recommended version of the base tool.** The QUADAS group states
> that QUADAS-C "cannot be used alone and must be used alongside the main QUADAS tool", and now
> recommends pairing it with **QUADAS-3** rather than QUADAS-2 — with adaptation, for which the
> QUADAS-3 E&E document gives guidance. This file documents QUADAS-C as published against
> QUADAS-2. **`QUADAS3.md`** carries the E&E's own adaptation: the four domains are renamed onto
> QUADAS-3's (Flow and Timing becomes **Analysis**), **C3.2 is removed** as overlapping QUADAS-3
> item 3.4, **C4.2 moves to Index Test** and **C4.3 moves to Target Condition**, and the unit
> assessed is a **comparative measure** you will usually have to compute yourself. An updated
> QUADAS-C is in development. `QUADAS2.md` is the tool this extension was written against.

## Purpose

QUADAS-C is an extension (add-on) to QUADAS-2 for assessing risk of bias in **comparative diagnostic test accuracy studies** — studies comparing two or more index tests in the same population.

QUADAS-C should be used **together with** QUADAS-2:
- QUADAS-2 assesses risk of bias for each individual test
- QUADAS-C assesses additional risk of bias for the **comparison** between tests

## Structure

QUADAS-C assesses the same 4 domains as QUADAS-2, with comparative signalling questions:
- **Signalling questions**: answered Yes / No / Unclear
- **Risk of bias judgment**: Low / High / Unclear (for the comparison)
- **No applicability assessment** (inferred from QUADAS-2 judgments)

## Comparative Study Designs

1. **#1 Fully paired**: All participants receive all index tests (most robust)
2. **#2 Randomized**: Participants randomly allocated to one index test
3. **#3 Partially paired, random subset**: Some randomly selected for multiple tests
4. **#4 Partially paired, nonrandom subset**: Some receive multiple tests by clinical decision
5. **#5 Unpaired nonrandomized**: Different participants receive different tests

Designs #1, #2, and #3 protect against confounding. Designs #4 and #5 have serious confounding potential.

## Four Phases of Completion

1. State the review question (which tests are compared, for what condition, in which population)
2. Tailor the tool (add/omit signalling questions per review)
3. Review the flow diagram (must show how participants were assigned to index tests)
4. Judge risk of bias (and applicability, based on QUADAS-2)

## Domain 1: Patient Selection

### Signalling Questions
1. **(C1.1)** Was the risk of bias for each index test judged 'low' for this domain?
   - If any index test is judged 'unclear' or 'high' in QUADAS-2, answer 'no'
2. **(C1.2)** Was a fully paired or randomized design used?
   - Designs #1, #2, and #3 (partially paired, random subset) → 'yes' or imply low risk
   - A 'no' answer should almost always prompt 'high risk' for this domain
3. **(C1.3)** Was the allocation sequence random? *(only applicable to randomized designs)*
   - Random: computer-generated numbers, random number tables, drawing lots
   - Non-random: alternation, dates, clinician preference
4. **(C1.4)** Was the allocation sequence concealed until patients were enrolled and assigned? *(only applicable to randomized designs)*
   - Appropriate: central randomization, telephone/internet service, sealed opaque envelopes

### Risk of Bias Judgment — **C1.5** Could the selection of patients have introduced bias in the comparison?
- **Low**: 'Yes' to all applicable signalling questions
- **High**: 'No' to C1.2 (not fully paired/randomized) almost always → high risk; 'No' to other questions raises concern
- **Unclear**: Insufficient information to judge

## Domain 2: Index Test

### Signalling Questions
1. **(C2.1)** Was the risk of bias for each index test judged 'low' for this domain?
   - If any test is 'high' or 'unclear' in QUADAS-2, the comparison is also at risk
2. **(C2.2)** Were the index test results interpreted without knowledge of the results of the other index test(s)? *(only for paired/partially paired designs #1, #3, #4)*
   - Consider: subjectivity of interpretation, order of testing, whether output is automated
3. **(C2.3)** Is undergoing one index test unlikely to affect the performance of the other index test(s)? *(only for paired/partially paired designs #1, #3, #4)*
   - Consider: learning effects, fatigue, tissue distortion, sample depletion
4. **(C2.4)** Were the index tests conducted and interpreted without advantaging one of the tests?
   - Tests should be performed under similar conditions; differences in sample handling, equipment generation, or operator experience may bias the comparison

### Risk of Bias Judgment — **C2.5** Could the conduct or interpretation of the index tests have introduced bias in the comparison?
- **Low**: 'Yes' to all applicable signalling questions
- **High**: 'No' to any question suggesting the comparison is unfairly biased
- **Unclear**: Insufficient information

## Domain 3: Reference Standard

### Signalling Questions
1. **(C3.1)** Was the risk of bias for each index test judged 'low' for this domain?
   - Reference standard misclassification biases both individual and comparative accuracy
2. **(C3.2)** Did the reference standard avoid incorporating any of the index tests?
   - If one index test is part of the reference standard, its accuracy is artificially inflated relative to the other test

### Risk of Bias Judgment — **C3.3** Could the reference standard, its conduct, or its interpretation have introduced bias in the comparison?
- **Low**: 'Yes' to all signalling questions
- **High**: 'No' to any, particularly if incorporation affects tests asymmetrically
- **Unclear**: Insufficient information

## Domain 4: Flow and Timing

### Signalling Questions
1. **(C4.1)** Was the risk of bias for each index test judged 'low' for this domain?
   - Timing, verification, and exclusion issues for individual tests also bias the comparison
2. **(C4.2)** Was there an appropriate interval between the index tests?
   - Simultaneous or near-simultaneous testing avoids disease progression bias
   - What is 'appropriate' depends on the disease (acute vs. chronic)
3. **(C4.3)** Was the same reference standard used for all index tests?
   - Different reference standards across test groups (e.g., surgery vs. follow-up) introduce differential verification bias
   - If reference standards are exchangeable (detect same condition equally), a 'no' may not imply high risk
4. **(C4.4)** Are the proportions and reasons for missing data similar across index tests?
   - Differential missing data between test groups can bias the comparison
   - Consider both proportion and mechanism of missingness

### Risk of Bias Judgment — **C4.5** Could the patient flow have introduced bias in the comparison?
- **Low**: 'Yes' to all signalling questions
- **High**: 'No' to any question, particularly C4.3 (differential verification) or C4.4 (differential missing data)
- **Unclear**: Insufficient information

'Unclear' does not mean 'moderate' risk of bias — it means there is insufficient information to judge.

## Optional Additional Signalling Questions (Table 4)

These are not part of the core tool but can be added to tailor for specific reviews:

| Domain | Question | Applicable to |
|--------|----------|---------------|
| Patient Selection | Were the same patient selection criteria used for those assigned to each index test? | Unpaired nonrandomized studies |
| Patient Selection | If patients received all index tests, was the decision to use all tests made before participants were recruited? | Paired studies |
| Patient Selection | Did the study avoid using prior tests as inclusion criteria that were correlated with only one of the index tests? | All |
| Index Test | Did the study avoid using index test thresholds that are likely to advantage some of the index tests? | Studies with nonbinary tests |
| Reference Standard | Is the mechanistic basis of one index test more closely shared with the reference standard than the other? | All |

## Presenting QUADAS-C Results

- Present QUADAS-2 and QUADAS-C results side by side
- QUADAS-C results are specific to each test comparison (present separately if multiple comparisons)
- Use traffic-light tables: + (low risk, green), - (high risk, red), ? (unclear, yellow)
- The guidance document points to www.quadas.org for templates; that domain no longer resolves, and the tabular and graphical templates are distributed from the QUADAS group's Bristol page (see the Reference line above)

## When to Use

- Systematic reviews comparing two or more diagnostic tests head-to-head
- Should always be used alongside QUADAS-2 (not as a replacement)
- Primarily designed for fully paired (#1) and randomized (#2) designs
- Can be adapted for other comparative designs with appropriate tailoring

# RECORD Checklist

**REporting of studies Conducted using Observational Routinely-collected health Data**
Version: RECORD 2015 (13 items; a STROBE extension). The pharmacoepidemiology extension is **RECORD-PE** (Langan et al. *BMJ* 2018).
Source: Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. The REporting of
studies Conducted using Observational Routinely-collected health Data (RECORD) statement.
*PLoS Med* 2015;12(10):e1001885 (DOI 10.1371/journal.pmed.1001885). https://www.record-statement.org
Licence: CC BY 4.0 — confirmed via Crossref.
Verification: all 13 items were compared against the checklist published in the statement; the id
set — 1.1, 1.2, 1.3, 6.1, 6.2, 6.3, 7.1, 12.1, 12.2, 12.3, 13.1, 19.1, 22.1 — matches exactly.

Apply when the manuscript is an **observational study conducted using routinely-collected health data** — administrative claims, electronic health records (EHR), disease/population registries, health-administrative or health-checkup databases, or linked versions of these — i.e. data **not collected for the purpose of the specific study**. RECORD extends the base **STROBE** items with reporting specific to secondary-use data: database identity, the codes/algorithms used to define the population and the variables, data linkage and its quality, and the limitations of analysing data collected for another purpose. For a drug safety/effectiveness study in such data, also apply **RECORD-PE**. For the design/validity review of the same study, pair with the RD1–RD8 domain probes in `peer-review` / `self-review` `references/domain-probes/record_routinely_collected_data.md`, and with the observational-confounding probes (`observational_confounding.md`).

## Checklist Items (13 items, extending STROBE)

### Title and Abstract (STROBE item 1)

| # | Item | Description |
|---|------|-------------|
| 1.1 | Data type | The type of data used should be specified in the title or abstract. When possible, the name(s) of the database(s) used should be stated. |
| 1.2 | Geography and timeframe | The geographic region and timeframe within which the study took place should be reported in the title or abstract. |
| 1.3 | Linkage | If linkage between databases was conducted for the study, this should be clearly stated in the title or abstract. |

### Methods — Setting / Participants (STROBE item 6)

| # | Item | Description |
|---|------|-------------|
| 6.1 | Population selection | The methods of study population selection (such as codes or algorithms used to identify subjects) should be listed in detail. If this is not possible, an explanation should be provided. |
| 6.2 | Validation of codes | Any validation studies of the codes or algorithms used to select the population should be referenced. If validation was conducted for this study and not published elsewhere, detailed methods and results should be provided. |
| 6.3 | Linkage diagram | If the study involved linkage of databases, consider use of a flow diagram or other graphical display to demonstrate the data linkage process, including the number of individuals with linked data at each stage. |

### Methods — Variables (STROBE item 7)

| # | Item | Description |
|---|------|-------------|
| 7.1 | Codes for variables | A complete list of codes and algorithms used to classify exposures, outcomes, confounders, and effect modifiers should be provided. If these cannot be reported, an explanation should be provided. |

### Methods — Data access and cleaning (STROBE item 12)

| # | Item | Description |
|---|------|-------------|
| 12.1 | Data access | Authors should describe the extent to which the investigators had access to the database population used to create the study population. |
| 12.2 | Data cleaning | Authors should provide information on the data cleaning methods used in the study. |
| 12.3 | Linkage methods | State whether the study included person-level, institutional-level, or other data linkage across two or more databases. The methods of linkage and methods of linkage quality evaluation should be provided. |

### Results — Participants (STROBE item 13)

| # | Item | Description |
|---|------|-------------|
| 13.1 | Selection of included persons | Describe in detail the selection of the persons included in the study (i.e. study population selection) including filtering based on data quality, data availability and linkage. The selection of included persons can be described in the text and/or by means of the study flow diagram. |

### Discussion — Limitations (STROBE item 19)

| # | Item | Description |
|---|------|-------------|
| 19.1 | Secondary-data limitations | Discuss the implications of using data that were not created or collected to answer the specific research question(s). Include discussion of misclassification bias, unmeasured confounding, missing data, and changing eligibility over time, as they pertain to the study being reported. |

### Other Information — Data access / cleaning (STROBE item 22)

| # | Item | Description |
|---|------|-------------|
| 22.1 | Supplemental access | Authors should provide information on how to access any supplemental information such as the study protocol, raw data, or programming code. |

---

## Notes for Assessors

- RECORD is an **extension of STROBE**; for the non-RECORD-specific items the base `STROBE.md` guidance also applies. Report both the base instrument and the extension when describing methods (do not cite RECORD as if it replaced STROBE).
- The **highest-yield** items are **6.1 / 7.1** (the actual code lists / phenotype algorithms used to define the population, exposures, outcomes, and confounders — the single most common omission; "we identified diabetes from the database" with no codes is non-compliant), **6.2** (whether those algorithms were *validated*, and where), **12.3 / 6.3** (linkage method and linkage-quality evaluation, with a person-flow at each linkage stage), **13.1** (a participant-selection flow that includes data-quality/availability/linkage filtering — not only clinical eligibility), and **19.1** (the limitations specific to secondary-use data: misclassification from codes, unmeasured confounding, informative missingness, and eligibility drift over time).
- For a **drug safety/effectiveness** study in routinely-collected data, also apply **RECORD-PE** (Langan et al. *BMJ* 2018;363:k3532), which adds items on exposure definition (drug codes, dose, duration, exposure windows), the comparator and new-user/active-comparator design, and immortal-time/protopathic bias.
- This checklist was authored as a faithful summary of the RECORD statement (Benchimol EI, et al. *PLoS Med* 2015;12(10):e1001885, **CC BY 4.0**) for item-by-item assessment; verify against the published statement and its explanation-and-elaboration document for full item wording. Verified 2026-06-29.

# REMARK Checklist

**REporting recommendations for tumour MARKer prognostic studies**
Version: REMARK 2005 (McShane et al.), with the 2012 Explanation and Elaboration (Altman et al.)
Source: Item text reproduced from Table 1 of the CC BY explanation-and-elaboration paper. McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM. Br J Cancer 2005;93(4):387-391. Explanation and Elaboration: Altman DG, McShane LM, Sauerbrei W, Taube SE. PLoS Med 2012;9(5):e1001216. Complete the official REMARK instrument for a submission checklist.

Source: McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM. Reporting recommendations for tumour marker prognostic studies (REMARK). *Br J Cancer* 2005;93(4):387-391. Explanation and elaboration: Altman DG, McShane LM, Sauerbrei W, Taube SE. *PLoS Med* 2012;9(5):e1001216 (DOI 10.1371/journal.pmed.1001216).
Licence: The *PLoS Medicine* explanation-and-elaboration paper is CC BY 4.0 (confirmed via Crossref); the original *Br J Cancer* statement is not openly licensed.
Verification: all 20 items were extracted from Table 1 of the CC BY explanation-and-elaboration
paper (Europe PMC full text, PMC3362085) and compared item by item. **The count matched while the
second half did not**: this file previously replaced item 14 with "Assay performance" and item 17
with "report all endpoints", dropped item 18 (further investigations) entirely, invented item 19
"Analyses reported", and shifted items 18–20.

## Checklist Items (20 items)

### Introduction

| # | Item | Description |
|---|------|-------------|
| 1 | Marker and objectives | State the marker examined, the study objectives, and any pre-specified hypotheses. |

### Materials and Methods

#### Patients

| # | Item | Description |
|---|------|-------------|
| 2 | Patients | Describe the characteristics (e.g., disease stage, comorbidities) of the study patients, including their source and the inclusion and exclusion criteria. |
| 3 | Treatments | Describe treatments received and how treatment was chosen (e.g., randomized or rule-based). |

#### Specimen characteristics

| # | Item | Description |
|---|------|-------------|
| 4 | Specimens | Describe the type of biological material used (including control samples) and the methods of preservation and storage. |

#### Assay methods

| # | Item | Description |
|---|------|-------------|
| 5 | Assay methods | Specify the assay method and provide (or reference) a detailed protocol, including specific reagents or kits, quality-control procedures, reproducibility assessments, quantitation methods, scoring and reporting protocols, and whether assays were performed blinded to the study endpoint. |

#### Study design

| # | Item | Description |
|---|------|-------------|
| 6 | Study design | State the method of case selection, whether the study was prospective or retrospective, and whether stratification or matching (e.g., by stage or age) was used; specify the time period from which cases were taken, the end of the follow-up period, and the median follow-up time. |
| 7 | Endpoints | Precisely define all clinical endpoints examined. |
| 8 | Candidate variables | List all candidate variables initially examined or considered for inclusion in models. |
| 9 | Sample size | Give the rationale for the sample size; if the study was designed to detect a specified effect size, state the target power and effect size. |

#### Statistical analysis methods

| # | Item | Description |
|---|------|-------------|
| 10 | Statistical methods | Specify all statistical methods, including any variable-selection procedures and other model-building details, how model assumptions were verified, and how missing data were handled. |

### Results

#### Data

| # | Item | Description |
|---|------|-------------|
| 11 | Marker values in analysis | Clarify how marker values were handled in the analyses; if relevant, describe methods used for cutpoint determination. |
| 12 | Patient flow | Describe the flow of patients through the study, including the number of patients included in each stage of the analysis (a diagram may be helpful) and reasons for dropout. Specifically, both overall and for each subgroup extensively examined, report the number of patients and the number of events. |
| 13 | Baseline distributions | Report distributions of basic demographic characteristics (at least age and sex), standard (disease-specific) prognostic variables, and tumour marker, including numbers of missing values. |

#### Analysis and presentation

| # | Item | Description |
|---|------|-------------|
| 14 | Marker vs standard variables | Show the relation of the marker to standard prognostic variables. |
| 15 | Univariable analyses | Present univariable analyses showing the relation between the marker and outcome, with the estimated effect (for example, hazard ratio and survival probability). Preferably provide similar analyses for all other variables being analysed. For the effect of a tumour marker on a time-to-event outcome, a Kaplan-Meier plot is recommended. |
| 16 | Multivariable analyses | For key multivariable analyses, report estimated effects (for example, hazard ratio) with confidence intervals for the marker and, at least for the final model, all other variables in the model. |
| 17 | Marker with standard variables | Among reported results, provide estimated effects with confidence intervals from an analysis in which the marker and standard prognostic variables are included, regardless of their statistical significance. |
| 18 | Further investigations | If done, report results of further investigations, such as checking assumptions, sensitivity analyses, and internal validation. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 19 | Interpretation | Interpret the results in the context of the pre-specified hypotheses and other relevant studies; include a discussion of limitations of the study. |
| 20 | Implications | Discuss implications for future research and clinical value. |

---

## Notes for Assessors

- REMARK primarily targets **single-marker** prognostic studies, but most items apply equally to studies of **multiple markers**, studies that **develop a prognostic model**, and studies that **predict response to treatment**. Apply the relevant items rather than marking them N/A by default.
- The items most often MISSING in review are **item 11** (how marker values were handled, including cutpoint determination) and **item 17** (effects with confidence intervals from the analysis containing the marker *and* the standard prognostic variables, **regardless of statistical significance**) — flag these explicitly.
- For a time-to-event marker study, expect a **Kaplan-Meier plot** (item 15) and a **multivariable model adjusted for established prognostic variables** (item 16), not a marker-alone analysis.
- Pair REMARK with **STROBE** for the observational-design items (setting, eligibility, bias, missing data), and with **TRIPOD / TRIPOD+AI** when the study develops or validates a prognostic model. Name the base instrument and any extension and cite each (Step 4e).
- Item text is reproduced from Table 1 of the CC BY explanation-and-elaboration paper with attribution. Complete the official REMARK instrument for a submission checklist.

# RoB 2 Assessment Guide

Revised Cochrane Risk-of-Bias tool for Randomised Trials.
Version: RoB 2 (22 August 2019). Full guidance and the current tool: https://www.riskofbias.info
Source: Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised
tool for assessing risk of bias in randomised trials. *BMJ* 2019;366:l4898 (DOI 10.1136/bmj.l4898;
PMID 31462531).

> **Fidelity and licence.** **No open licence was found for the article** — Crossref returns only
> BMJ's text-and-data-mining policy, and `LICENSES.md` previously claimed CC BY for it on no
> evidence. The signalling questions below were checked against the **official RoB 2 template and
> full guidance document** distributed at riskofbias.info (version of 22 August 2019); their
> wording is short-form and paraphrased. **Complete the official RoB 2 form for any assessment you
> report.**
>
> Verification: every signalling question, its conditionality, the response options and the
> domain- and overall-judgement rules were compared against that template and guidance. Four
> problems were found and corrected: two questions were **missing** from the assignment variant of
> domain 2 (2.5, 2.7), question 2.5 of the adherence variant had **inverted polarity** (the tool
> asks about *non-adherence*), domain 5's three questions had been **collapsed into two**, and the
> missing-data judgement carried a **">95%" threshold that the tool does not define** — the
> guidance defines "nearly all" qualitatively and gives no percentage.

Reference: Sterne JAC et al. BMJ 2019;366:l4898. PMID: 31462531.

## Structure

RoB 2 is applied **to a specific result** — one outcome, one numerical result, in one trial. Record
the experimental and comparator interventions, the outcome, and the numerical result before you
start.

- **Signalling question responses**: Yes / Probably yes / Probably no / No / No information (and
  **NA** where a question is conditional and its condition was not met)
- **Domain judgement**: Low risk of bias / Some concerns / High risk of bias
- **Optional, per domain and overall**: the predicted **direction** of bias — NA, favours
  experimental, favours comparator, towards null, away from null, or unpredictable

Questions written as "If … to N.n" are **conditional**: ask them only when the stated answer was
given to the earlier question.

## Before domain 2: state the effect of interest

The review team must declare whether the aim for this result is to assess **the effect of
assignment** to intervention (the intention-to-treat effect) or **the effect of adhering** to
intervention. Domain 2 has a different set of questions for each; do not mix them.

## Domain 1: Bias arising from the randomisation process

| # | Signalling question |
|---|---------------------|
| 1.1 | Was the allocation sequence random? |
| 1.2 | Was the allocation sequence concealed until participants were enrolled and assigned to interventions? |
| 1.3 | Did baseline differences between intervention groups suggest a problem with the randomisation process? |

The tool does not aim to identify baseline imbalances that arose by chance; a small number of
"statistically significant" differences at 0.05 is usually compatible with chance.

## Domain 2: Bias due to deviations from the intended interventions

### Variant A — effect of **assignment** to intervention

| # | Signalling question |
|---|---------------------|
| 2.1 | Were participants aware of their assigned intervention during the trial? |
| 2.2 | Were carers and people delivering the interventions aware of participants' assigned intervention during the trial? |
| 2.3 | *If Y/PY/NI to 2.1 or 2.2:* Were there deviations from the intended intervention that arose because of the trial context? |
| 2.4 | *If Y/PY to 2.3:* Were these deviations likely to have affected the outcome? |
| 2.5 | *If Y/PY/NI to 2.4:* Were these deviations from intended intervention balanced between groups? |
| 2.6 | Was an appropriate analysis used to estimate the effect of assignment to intervention? |
| 2.7 | *If N/PN/NI to 2.6:* Was there potential for a substantial impact (on the result) of the failure to analyse participants in the group to which they were randomised? |

### Variant B — effect of **adhering** to intervention

| # | Signalling question |
|---|---------------------|
| 2.1 | Were participants aware of their assigned intervention during the trial? |
| 2.2 | Were carers and people delivering the interventions aware of participants' assigned intervention during the trial? |
| 2.3 | *If applicable, and if Y/PY/NI to 2.1 or 2.2:* Were important non-protocol interventions balanced across intervention groups? |
| 2.4 | *If applicable:* Were there failures in implementing the intervention that could have affected the outcome? |
| 2.5 | *If applicable:* Was there **non-adherence** to the assigned intervention regimen that could have affected participants' outcomes? |
| 2.6 | *If N/PN/NI to 2.3, or Y/PY/NI to 2.4 or 2.5:* Was an appropriate analysis used to estimate the effect of adhering to the intervention? |

## Domain 3: Bias due to missing outcome data

| # | Signalling question |
|---|---------------------|
| 3.1 | Were data for this outcome available for all, or nearly all, participants randomised? |
| 3.2 | *If N/PN/NI to 3.1:* Is there evidence that the result was not biased by missing outcome data? |
| 3.3 | *If N/PN to 3.2:* Could missingness in the outcome depend on its true value? |
| 3.4 | *If Y/PY/NI to 3.3:* Is it likely that missingness in the outcome depended on its true value? |

**"Nearly all" is not a percentage.** The guidance defines it as: the number of participants with
missing outcome data is so small that their outcomes, whatever they were, could have made no
important difference to the estimated effect. Do not substitute a 95% or 80% rule — RoB 2 states
none.

**Low risk** requires any one of: (i) outcome data available for all, or nearly all, randomised
participants; **or** (ii) evidence that the result was not biased by missing outcome data;
**or** (iii) missingness in the outcome could not depend on its true value.

## Domain 4: Bias in measurement of the outcome

| # | Signalling question |
|---|---------------------|
| 4.1 | Was the method of measuring the outcome inappropriate? |
| 4.2 | Could measurement or ascertainment of the outcome have differed between intervention groups? |
| 4.3 | *If N/PN/NI to 4.1 and 4.2:* Were outcome assessors aware of the intervention received by study participants? |
| 4.4 | *If Y/PY/NI to 4.3:* Could assessment of the outcome have been influenced by knowledge of intervention received? |
| 4.5 | *If Y/PY/NI to 4.4:* Is it likely that assessment of the outcome was influenced by knowledge of intervention received? |

## Domain 5: Bias in selection of the reported result

| # | Signalling question |
|---|---------------------|
| 5.1 | Were the data that produced this result analysed in accordance with a pre-specified analysis plan that was finalised before unblinded outcome data were available for analysis? |
| 5.2 | Is the numerical result being assessed likely to have been selected, on the basis of the results, from multiple eligible outcome **measurements** (e.g. scales, definitions, time points) within the outcome domain? |
| 5.3 | Is the numerical result being assessed likely to have been selected, on the basis of the results, from multiple eligible **analyses** of the data? |

5.2 and 5.3 are separate questions. Selecting a measurement and selecting an analysis are
different acts, and a result can be at risk from one and not the other.

## Overall risk of bias

- **Low risk of bias**: low risk of bias for all domains
- **Some concerns**: some concerns in at least one domain, but not high risk of bias in any domain
- **High risk of bias**: high risk of bias in at least one domain, **or** some concerns for multiple
  domains in a way that substantially lowers confidence in the result

## When to Use

- Use for **individually randomised, parallel-group trials** (default)
- Variants are published for cluster-randomised trials and crossover trials — use the matching variant
- Do NOT use for non-randomised studies (use ROBINS-I instead)

# ROBINS-E Assessment Guide

Risk Of Bias In Non-randomized Studies — of Exposures.
Reference: Higgins JPT et al. Environment International 2024;186:108602. doi: 10.1016/j.envint.2024.108602.
Website: https://www.riskofbias.info/welcome/robins-e-tool

Version: ROBINS-E (2024)
Licence: CC BY-NC 4.0 (non-commercial) — confirmed via Crossref. Not redistributable verbatim under this repository's MIT licence.
Verification: the seven bias domains and their order were compared against Table 1 of the article
(Europe PMC full text, PMC11098530), and the preliminary-considerations parts A–E, the four
domain-judgement levels, the three per-domain outputs and the domain-1 label
"Low risk of bias (except for concerns about uncontrolled confounding)" against its section 4–6
text. All matched. Item wording stays paraphrased: the source is CC BY-**NC**, which cannot be
redistributed under this repository's MIT licence. Complete the official instrument for anything
you report.

## Purpose

ROBINS-E assesses the risk of **material bias** in individual observational studies examining the effect of an **exposure** on an outcome. Designed for follow-up (cohort) studies. Material bias = bias sufficient to affect the direction of the estimated effect or impact the ability to draw conclusions.

## Structure

ROBINS-E has:
- **Planning stage**: List confounders and consider appropriateness
- **Preliminary considerations** (per study): Sections A-E
- **7 bias domains** with signalling questions
- **Overall judgment** across domains

Signalling question responses: Yes / Probably yes / Probably no / No / No information
(Some questions have domain-specific response options)

Domain judgments: Low risk of bias / Some concerns / High risk of bias / Very high risk of bias

Three outputs per domain:
1. Risk of bias judgment (algorithmic, overridable)
2. Predicted direction of bias
3. Whether bias threatens conclusions (Yes / No / Cannot tell)

## Planning Stage

### P1: List Important Confounding Factors
Specify confounders relevant to all or most studies on this topic. State whether they are particular to specific exposure-outcome combinations.

### P2: Appropriateness Assessment
Will the review use the ROBINS-E assessment of appropriateness (study sensitivity)? → Yes / No
If Yes, complete Appendix 1 (Parts I, II, III).

## Preliminary Considerations (Per Study)

### A. Specify the Result Being Assessed
- ROBINS-E is specific to a particular study result
- Different results from the same study may have different risks of bias
- Specify the numerical result (e.g., RR=1.52, 95% CI 0.83 to 2.77)

### B. Decide Whether to Proceed
- Some study characteristics may directly indicate very high risk of bias
- Screening section to identify such situations before detailed assessment

### C. Specify the Analysis
- Gather information about participants, exposure measure(s), outcome, and analysis methods

### D. Specify the Causal Effect
- Define the causal effect of exposure being estimated
- This is essential: the causal effect defines what the result would be in the absence of bias

### E. Specify Important Confounding Factors
- Specify the known important confounding factors likely to influence the exposure–outcome association
- Identify them by reviewing the literature **and** consulting experts, including members of the review team
- Parts C and D may be completed in either order, but both come before part E

## Domain 1: Risk of Bias Due to Confounding

### Key Signalling Questions
- Were all important confounding domains measured?
- Were all important confounding domains adequately controlled for?
- Was the analysis adjusted for all important confounders, or was matching/restriction used?
- Were appropriate methods used to control confounding (regression, propensity score, IP weighting)?

### Judgment
- **Low risk of bias (except for concerns about uncontrolled confounding)**: All critical confounders well addressed, but residual uncontrolled confounding cannot be eliminated in observational studies
- **Some concerns**: Minor concerns about residual confounding
- **High risk of bias**: Important confounders not adequately controlled
- **Very high risk of bias**: Critical confounding renders the estimate unreliable

*Note: For Domain 1, 'Low risk' is expressed as "Low risk of bias (except for concerns about uncontrolled confounding)" because uncontrolled confounding can never be fully excluded in observational studies.*

## Domain 2: Risk of Bias Arising from Measurement of the Exposure

### Key Signalling Questions
- Was the exposure clearly defined and consistently measured?
- Was exposure assessment valid and reliable?
- Was exposure measured at the appropriate time point?
- Was exposure classification differential with respect to the outcome?

### Judgment
- **Low**: Exposure well-defined, measured validly, non-differential misclassification
- **Some concerns**: Minor measurement issues unlikely to materially affect results
- **High**: Exposure measurement likely to introduce material bias
- **Very high**: Exposure so poorly measured that no useful estimate possible

## Domain 3: Risk of Bias in Selection of Participants

### Key Signalling Questions
- Was selection into the study (or into the analysis) related to both exposure and outcome?
- Was start of follow-up aligned with exposure assessment?
- Were adjustments made for selection effects?

### Judgment
- **Low**: Selection not related to both exposure and outcome
- **Some concerns**: Minor selection issues
- **High**: Selection bias likely to materially affect results
- **Very high**: Extreme selection bias

## Domain 4: Risk of Bias Due to Post-exposure Interventions

### Key Signalling Questions
- Were there post-exposure interventions that could have affected the outcome?
- Were these interventions differential across exposure groups?
- Were they likely to affect the estimated exposure effect?

### Judgment
- **Low**: No important post-exposure interventions, or balanced across groups
- **Some concerns**: Some differential post-exposure interventions
- **High**: Important post-exposure interventions likely bias the result
- **Very high**: Post-exposure interventions dominate the observed effect

## Domain 5: Risk of Bias Due to Missing Data

### Key Signalling Questions
- Were outcome data available for all or nearly all participants?
- Were participants excluded due to missing data on exposure or other variables?
- Was the proportion of missing data similar across exposure groups?
- Were appropriate methods used to handle missing data?

### Judgment
- **Low**: Complete data or minimal, non-differential missingness
- **Some concerns**: Some missing data but unlikely to materially bias results
- **High**: Missing data pattern likely to introduce material bias
- **Very high**: Extensive missing data with differential patterns

## Domain 6: Risk of Bias Arising from Measurement of the Outcome

### Key Signalling Questions
- Could outcome measurement have been influenced by knowledge of exposure status?
- Were outcome assessors blinded to exposure?
- Were outcome measures comparable across exposure groups?
- Was the outcome defined and assessed using validated methods?

### Judgment
- **Low**: Objective outcome or blinded assessment, non-differential measurement
- **Some concerns**: Minor measurement concerns
- **High**: Outcome measurement likely differentially affected by exposure knowledge
- **Very high**: Outcome measurement severely compromised

## Domain 7: Risk of Bias in Selection of the Reported Result

### Key Signalling Questions
- Were multiple outcome measurements reported?
- Were multiple analyses performed (different adjustments, subgroups, models)?
- Is the reported result likely selected from among multiple measurements or analyses?

### Judgment
- **Low**: Pre-specified analysis plan or single planned analysis
- **Some concerns**: Minor concerns about selective reporting
- **High**: Reported result likely selected to favor a particular conclusion
- **Very high**: Clear evidence of selective reporting

## Overall Risk of Bias

Three overall judgments are derived from the domain judgments:

1. **Overall risk of bias**: Most conservative across domains
   - **Low**: Low in all domains
   - **Some concerns**: Some concerns in at least one domain, but no high/very high
   - **High**: High in at least one domain, but not very high in any
   - **Very high**: Very high in at least one domain

2. **Overall predicted direction of bias**: Considering all domains together

3. **Overall: does bias threaten conclusions?**: Yes / No / Cannot tell

Justification should be provided when overriding algorithm-suggested judgments.

## When to Use

- Systematic reviews of observational studies examining exposure effects (environmental, occupational, behavioral, dietary)
- Currently designed for cohort (follow-up) studies
- Not for case-control studies (extension planned)
- Not for intervention studies (use ROBINS-I instead)
- Complementary to GRADE for rating certainty of evidence from observational studies

# ROBINS-I Assessment Guide

Risk Of Bias In Non-randomised Studies - of Interventions.
Version: ROBINS-I (2016), the original version. Tool home: https://www.riskofbias.info
Source: Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a
tool for assessing risk of bias in non-randomised studies of interventions. *BMJ* 2016;355:i4919
(DOI 10.1136/bmj.i4919).

> **Fidelity and licence.** The source article is **CC BY-NC 3.0** — non-commercial. This repository
> is MIT-licensed and redistributed without restriction, so the tool's wording cannot be carried
> verbatim here. This file is an **in-house summary of the tool's structure**: the seven domains,
> their order, the answer options and the judgement levels, all of which were checked against the
> article. The per-domain questions below are abbreviated and are **not** the tool's signalling
> questions. **Complete the official ROBINS-I form from riskofbias.info for any assessment you
> report.**
>
> **Verification.** The seven domains, their order, their pre-/at-/post-intervention grouping and
> the five judgement levels with their across-domain criteria were compared against Tables 1 and 2
> of the article (Europe PMC full text, PMC5062054). All matched. The **No information** overall
> judgement, which the file omitted, has been added.
>
> **A version 2 exists and is still in draft.** ROBINS-I V2 adds algorithms mapping signalling-question
> answers onto domain judgements, and covers bias due to immortal time, which the 2016 version omits.
> A revised draft was posted in November 2025 and is subject to change. Check riskofbias.info before
> choosing which version to appraise against; this file documents the 2016 version.
Reference: Sterne JAC et al. BMJ 2016;355:i4919.

## Structure

ROBINS-I assesses 7 domains + overall judgment. The article groups them by when the bias arises:
**pre-intervention** (domains 1–2, where assessment is mainly distinct from randomised trials),
**at intervention** (domain 3, also mainly distinct), and **post-intervention** (domains 4–7,
which overlap substantially with assessments of randomised trials).
- **Signalling questions**: Yes / Probably yes / Probably no / No / No information
- **Domain judgment**: Low / Moderate / Serious / Critical / No information
- **Overall judgment**: Lowest of all domain judgments (most conservative)

## Pre-assessment Requirements

Before applying ROBINS-I, specify:
1. The target trial (what RCT would ideally answer this question?)
2. The effect of interest (assignment to intervention vs starting and adhering)
3. Confounders to be controlled

## Domain 1: Bias Due to Confounding

### Key Questions
- Is there potential for confounding not accounted for?
- Did the authors use appropriate methods to control confounding (matching, regression, propensity score)?

### Judgment
- **Low**: All critical confounders appropriately controlled
- **Moderate**: Minor concerns about residual confounding
- **Serious**: Important confounders not adequately controlled
- **Critical**: Confounding so severe that no useful estimate possible

## Domain 2: Bias in Selection of Participants into the Study

### Key Questions
- Was selection into the study related to both intervention and outcome?
- Was start of follow-up and intervention aligned?
- Were adjustments made for different start times?

## Domain 3: Bias in Classification of Interventions

### Key Questions
- Were intervention groups clearly defined?
- Was information used to classify interventions recorded at the start of the intervention?
- Could classification of intervention status have been affected by knowledge of the outcome?

## Domain 4: Bias Due to Deviations from Intended Interventions

### Key Questions
- Were there deviations from intended intervention beyond what would be expected?
- Were these deviations unbalanced between groups and likely to affect outcomes?
- Were important co-interventions balanced across groups?

## Domain 5: Bias Due to Missing Data

### Key Questions
- Were outcome data available for all or nearly all participants?
- Were participants excluded due to missing data on intervention or other variables?
- Was the proportion of missing data similar across groups?
- Were appropriate methods used to handle missing data?

## Domain 6: Bias in Measurement of Outcomes

### Key Questions
- Could outcome measurement have been influenced by knowledge of intervention?
- Were outcome assessors blinded?
- Were outcome measures comparable across groups?

## Domain 7: Bias in Selection of the Reported Result

### Key Questions
- Were multiple outcome measurements reported?
- Were multiple analyses performed?
- Is the reported result likely selected from among multiple measurements or analyses?

## Overall Risk of Bias

The overall judgment follows the criteria in the article's Table 2:
- **Low**: the study is judged at low risk of bias for all domains — comparable to a well performed randomised trial
- **Moderate**: low or moderate for all domains — sound evidence for a non-randomised study, but not comparable to a well performed randomised trial
- **Serious**: serious in at least one domain, but not critical in any
- **Critical**: critical in at least one domain — too problematic to provide useful evidence, and should not be included in any synthesis
- **No information**: no clear indication that the study is at serious or critical risk of bias, and information is lacking in one or more key domains

## Recommendation for Synthesis

- Studies at **critical** risk of bias should be excluded from meta-analysis
- Present critical studies in a separate table for completeness
- Conduct sensitivity analysis excluding serious risk of bias studies

# ROBIS Assessment Guide

Risk Of Bias In Systematic Reviews, version 1.2.
Reference: Whiting P et al. J Clin Epidemiol 2016;69:225-234.

Version: ROBIS (2016) — 4 domains in 3 phases
Source: Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. *J Clin Epidemiol* 2016;69:225-234 (DOI 10.1016/j.jclinepi.2015.06.005).
Licence: CC BY 4.0 — confirmed via Crossref. (`LICENSES.md` previously recorded no open licence for ROBIS; that was wrong.)
Verification: all 24 questions (21 phase-2 signalling questions plus phase-3 A/B/C) were compared
word by word against the **official ROBIS 1.2 tool PDF** distributed by the University of Bristol
(`ROBIS 1.2 Clean.pdf`), and the judgement guidance against the statement's full text (Europe PMC,
PMC4687950). The questions matched 24/24. **The judgement guidance did not**: this file previously
carried a "Judgment Algorithm" stating that high concern in any domain means high risk of bias,
which the statement contradicts, and omitted the domain rule that the statement does give.

## Purpose

ROBIS assesses the risk of bias **in systematic reviews themselves** (not in individual studies). It evaluates whether the review process was conducted appropriately and whether the conclusions are trustworthy.

Applicable to reviews of: interventions, aetiology, diagnosis, and prognosis.

## Structure

ROBIS has 3 phases:
- **Phase 1** (optional): Assessing relevance of the review to the target question
- **Phase 2**: Identifying concerns with the review process (4 domains, signalling questions)
- **Phase 3**: Judging overall risk of bias in the review

Signalling questions answered: Y (Yes) / PY (Probably Yes) / PN (Probably No) / N (No) / NI (No Information)

Domain concerns rated: LOW / HIGH / UNCLEAR

## Phase 1: Assessing Relevance (Optional)

State the target question and compare it with the question addressed by the review.

For **intervention reviews**, specify: Patients/Population, Intervention(s), Comparator(s), Outcome(s)

For **aetiology reviews**, specify: Patients/Population, Exposure(s) and Comparator(s), Outcome(s)

For **DTA reviews**, specify: Patients, Index test(s), Reference standard, Target condition

For **prognostic reviews**, specify: Patients, Outcome to be predicted, Intended use of model, Intended moment in time

Final question: Does the question addressed by the review match the target question? → YES / NO / UNCLEAR

## Phase 2: Identifying Concerns with the Review Process

### Domain 1: Study Eligibility Criteria

Describe the study eligibility criteria, any restrictions, and whether objectives and criteria were pre-specified.

#### Signalling Questions
1.1 Did the review adhere to pre-defined objectives and eligibility criteria?
1.2 Were the eligibility criteria appropriate for the review question?
1.3 Were eligibility criteria unambiguous?
1.4 Were any restrictions in eligibility criteria based on study characteristics appropriate (e.g., date, sample size, study quality, outcomes measured)?
1.5 Were any restrictions in eligibility criteria based on sources of information appropriate (e.g., publication status or format, language, availability of data)?

**Concerns regarding specification of study eligibility criteria:** LOW / HIGH / UNCLEAR

### Domain 2: Identification and Selection of Studies

Describe methods of study identification and selection (e.g., number of reviewers involved).

#### Signalling Questions
2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
2.2 Were methods additional to database searching used to identify relevant reports?
2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?
2.4 Were restrictions based on date, publication format, or language appropriate?
2.5 Were efforts made to minimise error in selection of studies?

**Concerns regarding methods used to identify and/or select studies:** LOW / HIGH / UNCLEAR

### Domain 3: Data Collection and Study Appraisal

Describe methods of data collection, what data were extracted, how risk of bias was assessed (e.g., number of reviewers), and the tool used.

#### Signalling Questions
3.1 Were efforts made to minimise error in data collection?
3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results?
3.3 Were all relevant study results collected for use in the synthesis?
3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria?
3.5 Were efforts made to minimise error in risk of bias assessment?

**Concerns regarding methods used to collect data and appraise studies:** LOW / HIGH / UNCLEAR

### Domain 4: Synthesis and Findings

Describe synthesis methods.

#### Signalling Questions
4.1 Did the synthesis include all studies that it should?
4.2 Were all pre-defined analyses reported or departures explained?
4.3 Was the synthesis appropriate given the nature and similarity in the research questions, study designs and outcomes across included studies?
4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?
4.5 Were the findings robust e.g. as demonstrated through funnel plot or sensitivity analyses?
4.6 Were biases in primary studies minimal or addressed in the synthesis?

**Concerns regarding the synthesis and findings:** LOW / HIGH / UNCLEAR

## Phase 3: Judging Risk of Bias in the Review

Describe whether conclusions were supported by the evidence.

### Signalling Questions
A. Did the interpretation of findings address all of the concerns identified in Domains 1 to 4?
B. Was the relevance of identified studies to the review's research question appropriately considered?
C. Did the reviewers avoid emphasizing results on the basis of their statistical significance?

**Risk of bias in the review:** LOW / HIGH / UNCLEAR

### How the judgements are reached

ROBIS gives a rule for **domain concern** and deliberately declines to give one for overall risk of
bias — a design goal of the tool was "an overall risk of bias rating for a single review without
using summary quality scores."

**Phase 2, per domain.** If the answers to all signalling questions for a domain are *yes* or
*probably yes*, the level of concern can be judged **low**. If any signalling question is answered
*no* or *probably no*, potential for concern exists. Use *no information* only when insufficient
data are reported to permit a judgment.

**Phase 3.** Concerns identified in phase 2 do **not** automatically make the review high risk. The
statement is explicit: where concerns were identified in earlier domains but "these were
appropriately considered when interpreting results and drawing conclusions, then this may also be
rated as 'yes,' and depending on the rating of the other signaling questions, the review may still
be rated as 'low risk of bias.'" The phase-3 judgement is the assessor's, recorded with a rationale.

## When to Use

- Assessing quality of existing systematic reviews (overview of reviews)
- Evaluating competing systematic reviews on the same topic
- Guideline development (appraising evidence base)
- Critical appraisal teaching and journal clubs
- Not for assessing individual primary studies (use QUADAS-2, RoB 2, ROBINS-I, etc.)

# ROB-ME Assessment Guide

Risk Of Bias due to Missing Evidence in a meta-analysis.
Reference: Page MJ et al. BMJ 2023;383:e076754; ROB-ME cribsheet Version 1, October 2023.
Website: https://www.riskofbias.info/welcome/rob-me-tool

Version: ROB-ME (2023)
Source: Page MJ, Sterne JAC, Boutron I, Hróbjartsson A, Kirkham JJ, Li T, et al. ROB-ME: a tool for assessing risk of bias due to missing evidence in systematic reviews with meta-analysis. *BMJ* 2023;383:e076754 (DOI 10.1136/bmj-2023-076754).
Licence: Crossref returns only a BMJ text-and-data-mining policy; no Creative Commons licence.
Verification: the five steps, the Results Matrix symbols, all eleven signalling questions
(3.1–3.3, 4.1–4.8) with their conditions and response options, and the judgement scale were
compared word by word against the **official ROB-ME cribsheet Version 1 (October 2023)** and the
completion template distributed at riskofbias.info. All matched; only the truncated citation on
line 4 was wrong.

> A newer cribsheet exists, generalised from "meta-analysis" to "**synthesis**". Its question 4.5
> takes Y / PY / PN / N rather than Y / N, and 4.7's condition widens to "Y to 4.1 or 4.3 **or
> Y/PY to 4.5**". This file documents Version 1, which its citation names; check riskofbias.info
> before appraising against a version.

## Purpose

ROB-ME assesses the risk of bias in a **pairwise meta-analysis result** due to missing evidence — encompassing both non-reporting biases (selective reporting of results) and non-publication biases (unpublished studies). It is applied at the synthesis level (not individual study level).

## Structure

ROB-ME has 5 steps applied to **each meta-analysis** in a systematic review:
1. Select and define meta-analyses to assess
2. Determine which studies have missing results (Results Matrix)
3. Consider the potential for missing studies across the systematic review
4. Assess risk of bias due to missing evidence (signalling questions)
5. Overall risk of bias judgment

Response options: Y (Yes) / PY (Probably Yes) / PN (Probably No) / N (No) / NI (No Information) / NA (Not Applicable)

Green-underlined responses are potential markers for **low** risk of bias.
Red responses are potential markers for **a** risk of bias.

## Step 1: Select and Define Meta-analyses

For each meta-analysis, specify:
- The PICO: Participants, Intervention, Comparator, Outcome
- Eligible study designs
- Eligible outcome definitions (measures, metrics, time points)
- Eligible methods of analysis (analysis populations, crude vs adjusted estimates)

## Step 2: Determine Which Studies Have Missing Results (Results Matrix)

For each study meeting inclusion criteria, create a Results Matrix using these symbols:

| Symbol | Meaning |
|--------|---------|
| check (green) | A study result is available for inclusion in the meta-analysis |
| ~ (yellow) | No study result available, for a reason **unrelated** to the P value, magnitude or direction |
| ? (orange) | Unclear whether an eligible result was generated |
| X (red) | No study result available, **likely because of** the P value, magnitude or direction |

Record: Study ID, Source(s) used, Number of participants analysed, and availability status for each meta-analysis.

Also record any known information about the results (direction of effect, statistical significance, or narrative descriptions).

## Step 3: Consider the Potential for Missing Studies

Answer the following questions **once** for the systematic review as a whole:

### Signalling Questions
3.1 Were prospectively registered studies or studies identified for a prospective meta-analysis the only type of study eligible for inclusion?
   - **Y**: inception cohort → lower risk of missing studies
   - **N**: proceed to 3.2

3.2 If N to 3.1: Would you expect every eligible study to be identifiable regardless of its results?
   - **NA/Y/PY**: lower risk
   - **PN/N**: higher risk — proceed to 3.3

3.3 If Y/PY to 3.2: Were you likely to have found all eligible studies regardless of their results?
   - Consider: trial registers searched, search strategy designed to retrieve regardless of outcomes
   - **NA/Y/PY**: lower risk
   - **PN/N**: higher risk — check the box indicating potential for missing studies

## Step 4: Assess Risk of Bias (per meta-analysis)

Specify the meta-analysis details: description, summary effect estimate (95% CI), number of included studies and participants.

### Within-study Assessment ('Known Unknowns')

4.1 Of the studies identified, was there any for which no result was available for inclusion in the meta-analysis, **likely because of the P value, magnitude or direction** of the result generated (refer to Step 2)?
   - Answer 'Yes' if any study has an 'X' in the Results Matrix for this meta-analysis
   - **Y/N**

4.2 If Y to 4.1: Is it likely that there would be a notable change to the summary effect estimate if the omitted results had been included?
   - Consider: amount of missing evidence relative to total; direction of effect in omitted studies; fixed-effect vs random-effects model
   - **Trivial missing → 'No/Probably no'**; Non-trivial with differing direction → 'Yes/Probably yes'
   - **NA/Y/PY/PN/N/NI**

### Within-study Assessment ('Unknown Unknowns')

4.3 Of the studies identified, was there any for which it was unclear whether an eligible result was generated (refer to Step 2)?
   - Answer 'Yes' if any study has a '?' in the Results Matrix
   - **Y/N**

4.4 If Y to 4.3: Is it likely that there would be a notable change to the summary effect estimate if the potentially omitted results had been included?
   - Consider same factors as 4.2 but for 'potentially' missing evidence
   - **NA/Y/PY/PN/N/NI**

### Across-study Assessment

4.5 Do circumstances (identified in Step 3) indicate potential for some eligible studies not being identified because of the P value, magnitude or direction of the results generated?
   - Answer 'Yes' if the checkbox in Step 3 was checked
   - **Y/N**

4.6 If Y to 4.5: Is it likely that studies not identified had results that were eligible for inclusion in the meta-analysis?
   - Consider: core outcome sets, scope of outcome domain, whether missing studies would have measured this outcome
   - **NA/Y/PY/PN/N**

4.7 If Y to 4.1, 4.3, or 4.5: Does the pattern of observed study results suggest that the meta-analysis is likely to be missing results that were systematically different (in terms of P value, magnitude or direction) from those observed?
   - Methods: (1) funnel plot inspection, (2) funnel plot asymmetry test, (3) compare fixed-effect vs random-effects estimates, (4) inspect P values/direction in forest plot
   - Distinguish small-study effects from non-reporting biases
   - **NA/Y/PY/PN/N**

4.8 If Y/PY/NI to 4.2, 4.4, 4.6, or 4.7: Did sensitivity analyses suggest that the summary effect estimate was biased due to missing results?
   - Consider: selection models, regression-based adjustment, restricting to largest studies
   - **NA/Y/PY/PN/N**

## Step 5: Risk of Bias Judgment

### Judgment Scale
- **Low**: Little or no concern about missing evidence biasing the meta-analysis result
- **Some concerns**: Some concern but not sufficient to judge high risk
- **High**: The meta-analysis result is likely biased due to missing evidence

### Optional: Predicted Direction of Bias
- Favours experimental / Favours comparator / Towards null / Away from null / Unpredictable

## When to Use

- Every pairwise meta-analysis in a systematic review (PRISMA 2020 recommends this)
- Replaces informal funnel plot interpretation with a structured assessment
- Complements individual study RoB tools (RoB 2, ROBINS-I, QUADAS-2)
- Not designed for network meta-analysis (use RoB NMA instead)
- Should be completed by someone familiar with the meta-analysis methods and included studies

# RoB NMA Assessment Guide

Risk of Bias in Network Meta-Analysis tool.
Website: https://www.riskofbias.info

Version: RoB NMA (2025) — 17 items in 3 domains
Source: Lunny C, Higgins JPT, White IR, Dias S, Hutton B, Pham B, et al. Risk of Bias in Network Meta-Analysis (RoB NMA) tool. *BMJ* 2025;388:e079839 (DOI 10.1136/bmj-2024-079839).
Licence: CC BY 4.0 — confirmed via Crossref.
Verification: all 17 signalling statements were extracted from the guidance article's own item
sections (Europe PMC full text, PMC11915405) and compared statement by statement, together with
the domain and overall judgement rules. **This file previously carried 18 invented items**; only
statement 1.1 survived the comparison.

## Purpose

The RoB NMA tool assesses the risk of bias in **a single network meta-analysis** by identifying
limitations in how the NMA was conducted, including how the evidence was assembled, that could
bias the NMA's results or conclusions.

It is **not** a tool for the primary studies inside the network (use RoB 2 or ROBINS-I) and
**not** a tool for the systematic review that contains the NMA (use ROBIS or AMSTAR 2). It is
meant to be used alongside one of the latter. One review yields one ROBIS assessment but as many
RoB NMA assessments as there are NMAs.

## Structure

**17 items** as signalling statements, in 3 domains:

1. Interventions and network geometry (4 statements)
2. Effect modifiers (4 statements)
3. Statistical synthesis (9 statements)

Response options: **true / probably true / probably false / false / no information**. *True*
indicates the lowest risk of bias. Item 3.9 may also be answered *not applicable*.

Statements 2.4 and 3.8 are **conditional** — they are considered only when a preceding statement
was answered false or probably false.

Domain-level judgment: **low risk of bias / some concerns / high risk of bias**, supported by
written justification and quotes from the NMA manuscript.

## Domain 1: Interventions and Network Geometry

How the interventions were selected and grouped, and whether they are an appropriate set for
performing an NMA.

| # | Signalling statement |
|---|----------------------|
| 1.1 | All interventions and their comparators included in the NMA are reasonable alternatives for the whole target population |
| 1.2 | All eligible interventions were included in the network |
| 1.3 | Interventions were appropriately grouped into nodes in the network |
| 1.4 | All compared interventions were connected through a suitable chain of within study comparisons |

## Domain 2: Effect Modifiers

Whether the studies contributing to different direct comparisons are similar enough on the
characteristics that modify the intervention effect (the transitivity requirement).

| # | Signalling statement |
|---|----------------------|
| 2.1 | Outcome definitions and time points were similar across direct comparisons in the network |
| 2.2 | Effect modifying participant characteristics were similar across direct comparisons in the network |
| 2.3 | Effect modifying study characteristics were similar across direct comparisons in the network |
| 2.4 | *(only if 2.1, 2.2 or 2.3 was false or probably false)* The analysis appropriately looked at the differences in effect modifiers across the network |

## Domain 3: Statistical Synthesis

Non-reporting biases, biases within the primary studies, statistical methods, and conflict
between direct and indirect evidence.

| # | Signalling statement |
|---|----------------------|
| 3.1 | The analysis respected within study randomisation |
| 3.2 | No publication bias or other selective non-reporting biases were suspected |
| 3.3 | All predefined analyses, and only those analyses, were reported, or discrepancies were explained |
| 3.4 | Biases in primary studies were minimal or addressed in the synthesis |
| 3.5 | Appropriate methods were used to handle multi-arm studies |
| 3.6 | Appropriate assumptions were made about homogeneity or heterogeneity of effects within comparisons |
| 3.7 | No evidence of conflict between direct and indirect estimates of the same effect |
| 3.8 | *(only if 3.7 was false or probably false)* Conflicting results between direct and indirect evidence were adequately dealt with |
| 3.9 | If a bayesian analysis was performed, the choice of prior distributions was appropriate |

## Overall Judgment

An overall judgment may be made about the **results** of the NMA, its **conclusions**, or both —
whichever the assessor intends to use. Risk of bias at the systematic-review level (ROBIS or
AMSTAR 2) is combined with the three RoB NMA domain judgments. When RoB NMA is used with ROBIS,
ROBIS phase 3 is omitted and only its first three domains are considered.

**Bias in the results of the NMA** — low risk of bias / some concerns / high risk of bias. If all
domains were judged low risk, a judgment of low risk should generally be made; otherwise the
assessor decides between some concerns and high risk. Some concerns in multiple domains may
warrant an overall judgment of high risk.

**Bias in the conclusions of the NMA** — concerns / no concerns. The question is whether the NMA
authors dealt with all the limitations identified. Items 3.5, 3.6 and 3.8 should be reconsidered
here, because inappropriate modelling choices can lead to uncertainty being under- or
overestimated. The two judgments can differ: estimated effects may be at high risk of bias
because of the primary studies while the conclusions are at low risk, if that risk was carefully
taken into account. If the effects are at high risk because of how the NMA itself was conducted,
the conclusions are unlikely to be at low risk.

Focus on bias in results when the NMA results feed a decision model; focus on bias in conclusions
when the conclusions are used for decision making.

## Note on treatment rankings

There is **no separate item for ranking probabilities**. The tool's authors state that the factors
affecting rankings — unequal numbers of studies per comparison, study sample sizes, network
configuration, effect sizes — are already covered by the items above. Assessors should, however,
consider potential bias from rankings when judging the conclusions, since overinterpretation of
rankings can bias them.

## When to Use

- Assessing the risk of bias of a published network meta-analysis
- Guideline development involving multiple treatment comparisons
- Overviews of NMAs
- Paired with ROBIS or AMSTAR 2 for the review, and RoB 2 / ROBINS-I for the primary studies

# SPIRIT 2025 Checklist

**Standard Protocol Items: Recommendations for Interventional Trials**
Version: SPIRIT 2025
Source: https://www.consort-spirit.org
Reference: Chan AW, Hopewell S, Moher D, et al. SPIRIT 2025 statement: updated guideline for protocols of randomised trials. BMJ 2025;389:e081477 (published simultaneously in BMJ, JAMA, Lancet, Nature Medicine, PLoS Medicine).

> Note: SPIRIT 2025 supersedes SPIRIT 2013. It is a 34-item checklist (two new items, five revised, five deleted/merged) restructured with a new Open Science section. Use for clinical trial *protocols* (CONSORT is for the completed trial report).

Source: Chan AW, Boutron I, Hopewell S, Moher D, Schulz KF, Collins GS, et al. SPIRIT 2025 statement: updated guideline for protocols of randomised trials. *BMJ* 2025;389:e081477 (DOI 10.1136/bmj-2024-081477).
Licence: CC BY 4.0 — confirmed via Crossref.
Verification: all 53 sub-items (1a–34) were compared against Table 1 of the published statement
(Europe PMC full text, PMC12035670); 53/53 match, with no item missing and none invented. Two
labels carried over from SPIRIT 2013 have been corrected to their 2025 names.

## Checklist Items (34 items)

### Administrative Information

| # | Item | Description |
|---|------|-------------|
| 1a | Title | Title stating the trial design, population, and interventions, with identification as a protocol. |
| 1b | Abstract | Structured summary of trial design and methods, including items from the World Health Organization Trial Registration Data Set. |
| 2 | Protocol version | Version date and identifier. |
| 3a | Roles — contributors | Names, affiliations, and roles of protocol contributors. |
| 3b | Roles — sponsor contact | Name and contact information for the trial sponsor. |
| 3c | Roles — sponsor role | Role of trial sponsor and funders in design, conduct, analysis, and reporting of trial; including any authority over these activities. |
| 3d | Roles — committees | Composition, roles, and responsibilities of the coordinating site, steering committee, endpoint adjudication committee, data management team, and other individuals or groups overseeing the trial, if applicable. |

### Open Science

| # | Item | Description |
|---|------|-------------|
| 4 | Trial registration | Name of trial registry, identifying number (with URL), and date of registration. If not yet registered, name of intended registry. |
| 5 | Protocol and SAP | Where the trial protocol and statistical analysis plan can be accessed. |
| 6 | Data, code, materials | Where and how the individual de-identified participant data (including data dictionary), statistical code, and any other materials will be accessible. |
| 7a | Funding | Sources of funding and other support (e.g., supply of drugs). |
| 7b | Conflicts of interest | Financial and other conflicts of interest for principal investigators and steering committee members. |
| 8 | Dissemination | Plans to communicate trial results to participants, healthcare professionals, the public, and other relevant groups (e.g., reporting in trial registry, plain language summary, publication). |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 9a | Background | Scientific background and rationale, including summary of relevant studies (published and unpublished) examining benefits and harms for each intervention. |
| 9b | Comparator choice | Explanation for choice of comparator. |
| 10 | Objectives | Specific objectives related to benefits and harms. |

### Methods: Patient and Public Involvement, Trial Design

| # | Item | Description |
|---|------|-------------|
| 11 | Patient and public involvement | Details of, or plans for, patient or public involvement in the design, conduct, and reporting of the trial. |
| 12 | Trial design | Description of trial design including type of trial (e.g., parallel group, crossover), allocation ratio, and framework (e.g., superiority, equivalence, non-inferiority, exploratory). |

### Methods: Participants, Interventions, and Outcomes

| # | Item | Description |
|---|------|-------------|
| 13 | Trial setting | Settings (e.g., community, hospital) and locations (e.g., countries, sites) where the trial will be conducted. |
| 14a | Eligibility — participants | Eligibility criteria for participants. |
| 14b | Eligibility — sites/deliverers | If applicable, eligibility criteria for sites and for individuals who will deliver the interventions (e.g., surgeons, physiotherapists). |
| 15a | Interventions — description | Intervention and comparator with sufficient details to allow replication including how, when, and by whom they will be administered. If relevant, where additional materials describing the intervention and comparator (e.g., intervention manual) can be accessed. |
| 15b | Interventions — modifications | Criteria for discontinuing or modifying allocated intervention/comparator for a trial participant (e.g., drug dose change in response to harms, participant request, or improving/worsening disease). |
| 15c | Interventions — adherence | Strategies to improve adherence to intervention/comparator protocols, if applicable, and any procedures for monitoring adherence (e.g., drug tablet return, sessions attended). |
| 15d | Interventions — concomitant care | Concomitant care that is permitted or prohibited during the trial. |
| 16 | Outcomes | Primary and secondary outcomes, including the specific measurement variable (e.g., systolic blood pressure), analysis metric (e.g., change from baseline, final value, time to event), method of aggregation (e.g., median, proportion), and time point for each outcome. |
| 17 | Harms | How harms are defined and will be assessed (e.g., systematically, non-systematically). |
| 18 | Participant timeline | Time schedule of enrolment, interventions (including any run-ins and washouts), assessments, and visits for participants. A schematic diagram is highly recommended. |
| 19 | Sample size | How sample size was determined, including all assumptions supporting the sample size calculation. |
| 20 | Recruitment | Strategies for achieving adequate participant enrolment to reach target sample size. |

### Methods: Assignment of Interventions

| # | Item | Description |
|---|------|-------------|
| 21a | Allocation — sequence | Who will generate the random allocation sequence and the method used. |
| 21b | Allocation — restriction | Type of randomisation (simple or restricted) and details of any factors for stratification. To reduce predictability, other details of any planned restriction (e.g., blocking) should be provided in a separate document unavailable to those who enrol participants or assign interventions. |
| 22 | Allocation concealment | Mechanism used to implement the random allocation sequence (e.g., central computer/telephone; sequentially numbered, opaque, sealed containers), describing any steps to conceal the sequence until interventions are assigned. |
| 23 | Implementation | Whether the personnel who will enrol and those who will assign participants to the interventions will have access to the random allocation sequence. |
| 24a | Blinding — who | Who will be blinded after assignment to interventions (e.g., participants, care providers, outcome assessors, data analysts). |
| 24b | Blinding — how | If blinded, how blinding will be achieved and description of the similarity of interventions. |
| 24c | Blinding — unblinding | If blinded, circumstances under which unblinding is permissible, and procedure for revealing a participant's allocated intervention during the trial. |

### Methods: Data Collection, Management, and Analysis

| # | Item | Description |
|---|------|-------------|
| 25a | Data collection | Plans for assessment and collection of trial data, including any related processes to promote data quality (e.g., duplicate measurements, training of assessors) and a description of trial instruments (e.g., questionnaires, laboratory tests) along with their reliability and validity, if known. Reference to where data collection forms can be accessed, if not in the protocol. |
| 25b | Retention | Plans to promote participant retention and complete follow-up, including list of any outcome data to be collected for participants who discontinue or deviate from intervention protocols. |
| 26 | Data management | Plans for data entry, coding, security, and storage, including any related processes to promote data quality (e.g., double data entry; range checks for data values). Reference to where details of data management procedures can be accessed, if not in the protocol. |
| 27a | Statistical methods | Statistical methods used to compare groups for primary and secondary outcomes, including harms. |
| 27b | Analysis populations | Definition of who will be included in each analysis (e.g., all randomised participants), and in which group. |
| 27c | Missing data | How missing data will be handled in the analysis. |
| 27d | Additional analyses | Methods for any additional analyses (e.g., subgroup and sensitivity analyses). |

### Methods: Monitoring

| # | Item | Description |
|---|------|-------------|
| 28a | Data monitoring committee | Composition of data monitoring committee (DMC); summary of its role and reporting structure; statement of whether it is independent from the sponsor and funder; conflicts of interest and reference to where further details about its charter can be found, if not in the protocol. Alternatively, an explanation of why a DMC is not needed. |
| 28b | Interim analyses | Explanation of any interim analyses and stopping guidelines, including who will have access to these interim results and make the final decision to terminate the trial. |
| 29 | Trial monitoring | Frequency and procedures for monitoring trial conduct. If there is no monitoring, give explanation. |

### Ethics

| # | Item | Description |
|---|------|-------------|
| 30 | Research ethics approval | Plans for seeking research ethics committee/institutional review board approval. |
| 31 | Protocol amendments | Plans for communicating important protocol modifications to relevant parties. |
| 32a | Consent | Who will obtain informed consent or assent from potential trial participants or authorised proxies, and how. |
| 32b | Ancillary consent | Additional consent provisions for collection and use of participant data and biological specimens in ancillary studies, if applicable. |
| 33 | Confidentiality | How personal information about potential and enrolled participants will be collected, shared, and maintained in order to protect confidentiality before, during, and after the trial. |
| 34 | Ancillary and post-trial care | Provisions, if any, for ancillary and post-trial care, and for compensation to those who suffer harm from trial participation. |

---

*Educational summary of the SPIRIT 2025 checklist (CC BY 4.0). Cite the original statement (Chan et al., BMJ 2025) and consult https://www.consort-spirit.org for the authoritative, full checklist with explanation and elaboration.*

# SPIRIT-AI Checklist

**Standard Protocol Items: Recommendations for Interventional Trials -- Artificial Intelligence Extension**
Version: SPIRIT-AI 2020 (extends SPIRIT 2013)
Source: https://www.consort-spirit.org · EQUATOR Network
Reference: Cruz Rivera S, Liu X, Chan AW, et al. Nat Med 2020;26(9):1351-1363. doi:10.1038/s41591-020-1037-7 (CC BY 4.0)

> Educational summary, authored in our own words from the CC BY 4.0 source. Use the official
> SPIRIT-AI checklist for a submission-ready form and cite Cruz Rivera et al. 2020.

> **Verified against the published statement.** All 15 AI items were enumerated from the article's
> own checklist table via the PMC XML and compared label-by-label: **12 Extensions and
> 3 Elaborations**, matching exactly with no item missing and none invented.

> **Extension vs Elaboration — the statement distinguishes them, and it matters.** An **Extension**
> is a *new* reporting requirement that SPIRIT 2013 does not contain. An **Elaboration** clarifies how an
> existing SPIRIT 2013 item applies when the intervention involves AI; the requirement already existed,
> the guidance is what is new. Both are assessed, but only the Extensions are additional obligations —
> do not report an Elaboration as though the base instrument had been silent on it. Elaborations are
> marked **(E)** below.

## Naming and scope (read first)

- SPIRIT-AI is an **extension** of **SPIRIT 2013**, for **protocols** of randomized clinical trials of
  interventions that **include an AI/ML component**. Apply **both**: every base SPIRIT 2013 item plus
  the AI-specific items below; name and cite both instruments (manuscript-style-classical §14).
- It is the **protocol** counterpart of **CONSORT-AI** (completed-trial reports). For a finished trial
  report use CONSORT-AI.
- The AI items elaborate existing SPIRIT items (numbered to match); assess each alongside its parent.

## AI-specific extension items

Status each PRESENT / PARTIAL / MISSING / N/A.

### Administrative Information

| # | Item | Description (intent) |
|---|------|----------------------|
| 1 (i) **(E)** | AI identification | State in the title that the intervention involves AI/ML and specify the type of model. |
| 1 (ii) **(E)** | Intended use | State the intended use of the AI intervention. |

### Introduction — Background and rationale

| # | Item | Description (intent) |
|---|------|----------------------|
| 6a (i) | Role in pathway | Explain the intended use of the AI intervention within the clinical pathway, including its purpose and the intended users. |
| 6a (ii) | Prior validation | Describe any pre-existing evidence for the AI intervention (e.g., prior validation studies). |

### Methods — Participants, interventions, outcomes

| # | Item | Description (intent) |
|---|------|----------------------|
| 9 | Setting integration | Describe the onsite and offsite requirements needed to integrate the AI intervention into the trial setting. |
| 10 (i) **(E)** | Participant eligibility | State the participant-level inclusion and exclusion criteria. |
| 10 (ii) | Input-data eligibility | State the inclusion and exclusion criteria at the level of the **input data**. |
| 11a (i) | Algorithm version | Specify which version of the AI algorithm will be used. |
| 11a (ii) | Input acquisition | Specify how the input data will be acquired and selected. |
| 11a (iii) | Poor-quality input | Specify how poor-quality or unavailable input data will be assessed and handled. |
| 11a (iv) | Human–AI interaction | Specify any human–AI interaction in handling input data and the expertise required of the user. |
| 11a (v) | AI output | Specify the output of the AI intervention. |
| 11a (vi) | Output to decision | Explain how the AI intervention's outputs will contribute to decision-making or other elements of clinical practice. |

### Methods — Monitoring

| # | Item | Description (intent) |
|---|------|----------------------|
| 22 | Error analysis plan | Specify the plans to identify and analyze performance errors, and any planned mitigation. |

### Ethics and Dissemination

| # | Item | Description (intent) |
|---|------|----------------------|
| 29 | Code/intervention access | State whether and how the AI intervention and/or its code can be accessed, including any restrictions. |

---

## Notes for Assessors

- Apply SPIRIT-AI **with** all base SPIRIT 2013 items — the AI items do not replace them.
- **Algorithm version (11a (i))** and an explicit **error-analysis plan (22)** are the protocol-stage
  commitments that make the eventual CONSORT-AI report verifiable; mark MISSING if absent.
- **Input-data eligibility (10 (ii))** is distinct from participant eligibility — a common gap.
- **Setting integration (9)** and **human–AI interaction (11a (iv))** define whether the protocol
  evaluates the AI as it will actually be used; vague phrasing is PARTIAL.
- Use **CONSORT-AI** for the completed-trial report; SPIRIT-AI is for the protocol.

# SQUIRE 2.0 Checklist

**Standards for QUality Improvement Reporting Excellence 2.0**
Version: SQUIRE 2.0 (2015)
Source: http://squire-statement.org
Reference: Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986-992. doi:10.1136/bmjqs-2015-004411

Licence: Crossref returns no Creative Commons licence.
Verification: all 18 items were compared against Table 1 of the published guidelines (Europe PMC
full text, PMC5256233); 18/18 match, with no item missing and none invented. Item 2's abstract
specification, which had been condensed away, has been restored.

## Checklist Items (18 items)

### Title and Abstract

| # | Item | Description |
|---|------|-------------|
| 1 | Title | Indicate that the manuscript concerns an initiative to improve healthcare (broadly defined to include the quality, safety, or value of care). |
| 2 | Abstract | a) Provide adequate information to aid in searching and indexing. b) Summarise all key information from various sections of the text using the abstract format of the intended publication or a structured summary such as: background, local problem, methods, interventions, results, conclusions. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 3 | Problem description | Nature and significance of the local problem. |
| 4 | Available knowledge | Summary of what is currently known about the problem, including relevant previous studies. |
| 5 | Rationale | Informal or formal frameworks, models, concepts, and/or theories used to explain the problem, any reasons or assumptions that were used to develop the intervention(s), and reasons why the intervention(s) was expected to work. |
| 6 | Specific aims | Purpose of the project and of this report. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 7 | Context | Contextual elements considered important at the outset of introducing the intervention(s). |
| 8 | Intervention(s) | a) Description of the intervention(s) in sufficient detail that others could reproduce it. b) Specifics of the team involved in the work. |
| 9 | Study of the intervention(s) | a) Approach chosen for assessing the impact of the intervention(s). b) Approach used to establish whether the observed outcomes were due to the intervention(s). |
| 10 | Measures | a) Measures chosen for studying processes and outcomes of the intervention(s), including rationale for choosing them, their operational definitions, and their validity and reliability. b) Description of the approach to the ongoing assessment of contextual elements that contributed to the success, failure, efficiency, and cost. c) Methods employed for assessing completeness and accuracy of data. |
| 11 | Analysis | a) Qualitative and quantitative methods used to draw inferences from the data. b) Methods for understanding variation within the data, including the effects of time as a variable. |
| 12 | Ethical considerations | Ethical aspects of implementing and studying the intervention(s) and how they were addressed, including, but not limited to, formal ethics review and potential conflict(s) of interest. |

### Results

| # | Item | Description |
|---|------|-------------|
| 13 | Results | a) Initial steps of the intervention(s) and their evolution over time, including modifications made to the intervention during the project. b) Details of the process measures and outcome. c) Contextual elements that interacted with the intervention(s). d) Observed associations between outcomes, interventions, and relevant contextual elements. e) Unintended consequences such as unexpected benefits, problems, failures, or costs associated with the intervention(s). f) Details about missing data. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 14 | Summary | a) Key findings, including relevance to the rationale and specific aims. b) Particular strengths of the project. |
| 15 | Interpretation | a) Nature of the association between the intervention(s) and the outcomes. b) Comparison of results with findings from other publications. c) Impact of the project on people and systems. d) Reasons for any differences between observed and anticipated outcomes, including the influence of context. e) Costs and strategic trade-offs, including opportunity costs. |
| 16 | Limitations | a) Limits to the generalizability of the work. b) Factors that might have limited internal validity such as confounding, bias, or imprecision in the design, methods, measurement, or analysis. c) Efforts made to minimize and adjust for limitations. |
| 17 | Conclusions | a) Usefulness of the work. b) Sustainability. c) Potential for spread to other contexts. d) Implications for practice and for further study in the field. e) Suggested next steps. |

### Other Information

| # | Item | Description |
|---|------|-------------|
| 18 | Funding | Sources of funding that supported this work. Role, if any, of the funding organization in the design, implementation, interpretation, and reporting. |

---

## Notes for Assessors

- SQUIRE 2.0 is designed for studies evaluating quality improvement interventions in healthcare, including medical education QI initiatives.
- Not all items will be relevant to every QI study. Items should be assessed in the context of the specific improvement effort.
- Item 13 (Results) has 6 sub-items (a-f). Assess each sub-item separately for comprehensive evaluation.
- Items 5 (Rationale) and 9 (Study of the intervention) are critical for distinguishing QI reports from simple descriptions of implementation.
- For educational quality improvement studies, "intervention" refers to the educational program, curriculum change, or training initiative being evaluated.
- SQUIRE 2.0 applies to systematic efforts to improve the quality, safety, and value of healthcare; it is not intended for basic science or clinical trials (use CONSORT for RCTs).
- The original SQUIRE guidelines (1.0) were published in 2008. SQUIRE 2.0 (2015) is the current version and should be used for all new submissions.

# SRQR Checklist (qualitative research)

**Standards for Reporting Qualitative Research**
Version: SRQR 2014 — 21 items across Title/Abstract, Introduction, Methods, Results/Findings, Discussion, and Other. **Broad** — applies to all qualitative approaches (ethnography, grounded theory, phenomenology, case study, narrative research), not only interviews/focus groups.
Source: O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. *Acad Med* 2014;89(9):1245–1251 (the SRQR statement; DOI 10.1097/ACM.0000000000000388). EQUATOR Network.

Apply when the manuscript is a **qualitative study** of any approach — interviews, focus groups, observation/ethnography, document analysis, grounded theory, phenomenology, narrative research. For **interview / focus-group** studies specifically, the more granular **COREQ** (`COREQ.md`) is the better-fit companion. For the design/conduct review of the same study, pair with the QL1–QL8 domain probes in `peer-review` / `self-review` `references/domain-probes/qualitative_research.md`.

> Licensing note: SRQR is published in *Academic Medicine* (© AAMC), with **no Creative Commons licence**. The items below are an **in-house, faithful summary of the reporting items (facts/intents, paraphrased — not the verbatim SRQR wording)** for item-by-item assessment; consult the published article (DOI 10.1097/ACM.0000000000000388) for exact item text.

## Reporting items (grouped by section)

### Title and Abstract
| # | Item | What to check is reported |
|---|------|---------------------------|
| 1 | Title | A concise description of the study's nature/topic that **identifies it as qualitative** and (recommended) names the approach (e.g. ethnography, grounded theory) or data-collection method (e.g. interviews, focus groups). |
| 2 | Abstract | A summary of the key elements in the journal's abstract format — typically background, purpose, methods, results, conclusions. |

### Introduction
| # | Item | What to check is reported |
|---|------|---------------------------|
| 3 | Problem formulation | The problem/phenomenon studied and its significance, with a review of relevant theory and prior empirical work, and a problem statement. |
| 4 | Purpose or research question | The study's purpose and its specific objectives or questions. |

### Methods
| # | Item | What to check is reported |
|---|------|---------------------------|
| 5 | Qualitative approach & research paradigm | The chosen approach (ethnography / grounded theory / case study / phenomenology / narrative) and the guiding paradigm (e.g. postpositivist, constructivist/interpretivist), **with a rationale**. |
| 6 | Researcher characteristics & **reflexivity** | The researchers' attributes, qualifications/experience, relationship with participants, and assumptions, and how these may have interacted with the questions, methods, findings, or transferability. |
| 7 | Context | The setting/site and salient contextual factors, with a rationale. |
| 8 | Sampling strategy | How and why participants/documents/events were selected, and the criterion for stopping sampling (e.g. **saturation**), with a rationale. |
| 9 | Ethical issues (human subjects) | Ethics-board approval and participant consent (or an explanation for their absence), and confidentiality / data-security handling. |
| 10 | Data collection methods | The data types and collection procedures — dates, iterative process, triangulation of sources/methods, and any procedure changes during the study — with a rationale. |
| 11 | Data collection instruments & technologies | The instruments (e.g. interview guides, questionnaires) and devices (e.g. audio recorders), and whether/how they changed over the study. |
| 12 | Units of study | The number and relevant characteristics of the participants/documents/events, and their level of participation. |
| 13 | Data processing | How data were processed before/during analysis — transcription, data entry/management/security, integrity checks, coding, and de-identification of excerpts. |
| 14 | Data analysis | The process by which inferences/themes were identified and developed, who analysed the data, and the referenced paradigm/approach, with a rationale. |
| 15 | Techniques to enhance **trustworthiness** | The techniques used to enhance trustworthiness/credibility (e.g. member checking, audit trail, triangulation), with a rationale. |

### Results / Findings
| # | Item | What to check is reported |
|---|------|---------------------------|
| 16 | Synthesis & interpretation | The main findings (interpretations, inferences, themes), and any theory/model development or integration with prior research. |
| 17 | Links to empirical data | Evidence — participant **quotations**, field notes, text excerpts, images — that substantiates each analytic finding. |

### Discussion
| # | Item | What to check is reported |
|---|------|---------------------------|
| 18 | Integration, implications, transferability, contribution | A short summary of findings; how they connect to / extend / challenge prior scholarship; the scope of application / **transferability**; and the study's unique contribution. |
| 19 | Limitations | The trustworthiness and limitations of the findings. |

### Other
| # | Item | What to check is reported |
|---|------|---------------------------|
| 20 | Conflicts of interest | Potential sources of influence on the study and how they were managed. |
| 21 | Funding | Sources of funding/support and the role of the funders in data collection, interpretation, and reporting. |

---

## Notes for Assessors

- The **highest-yield** checks (where qualitative studies most often go wrong on review): **item 6** (researcher **reflexivity** — the researchers' position, assumptions, and relationship to participants is a core qualitative-rigor requirement, frequently omitted), **item 8** (a stated **sampling strategy and stopping criterion** — purposive logic and saturation, not a convenience sample with no rationale), **item 15** (explicit **trustworthiness** techniques — member checking / audit trail / triangulation), and **item 17** (analytic claims grounded in **quoted data**, not asserted).
- **Do not apply quantitative criteria** to a qualitative study: a small purposive sample is not a flaw, "generalizability" is **transferability** (not statistical external validity), and there are no power calculations, effect sizes, or p-values to demand. Mis-calibrated "the sample is too small / not generalizable / no p-value" comments are inappropriate here.
- This is an **in-house faithful summary of the SRQR items (paraphrased intents, not verbatim)**; map the manuscript's content to the items rather than to exact wording. For interview/focus-group designs, COREQ (`COREQ.md`) gives finer-grained items.
- Verification: all 21 items were compared, by number, name and definition, against the statement's own **Supplemental Digital Appendix 3** (item-by-item explanations and examples, `links.lww.com/ACADMED/A218`). 21/21 match, with no item missing and none invented. The supplement is marked "unauthorized reproduction is prohibited", so the wording here stays paraphrased.

# STARD 2015 Checklist

**Standards for Reporting Diagnostic Accuracy Studies**
Version: STARD 2015 — **30 items in 34 rows**; items 10, 12, 13 and 21 each split into **a** and **b**.
Source: Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an
updated list of essential items for reporting diagnostic accuracy studies. *BMJ* 2015;351:h5527
(DOI 10.1136/bmj.h5527).
Licence: CC BY 4.0 — confirmed via the PubMed Central record (PMC4623764). Item text below is
reproduced from the published checklist with attribution.
Verification: all 34 rows were extracted from the published checklist and compared programmatically;
the id set matches exactly.

> **What this file used to be.** It listed 30 rows ending at item 28. It **invented a split at item 8**
> (8a/8b, where the statement has a single item), **collapsed the a/b pairs at 12, 13 and 21** into
> single rows, and **omitted items 29 and 30 entirely** — where the full study protocol can be
> accessed, and sources of funding and the role of funders. A reviewer scoring against it never asked
> for either.


## Title or abstract

| # | Checklist item |
|---|----------------|
| 1 | Identification as a study of diagnostic accuracy using at least one measure of accuracy (such as sensitivity, specificity, predictive values, or AUC) Abstract |

## Abstract

| # | Checklist item |
|---|----------------|
| 2 | Structured summary of study design, methods, results, and conclusions (for specific guidance, see STARD for Abstracts) Introduction |

## Introduction

| # | Checklist item |
|---|----------------|
| 3 | Scientific and clinical background, including the intended use and clinical role of the index test |
| 4 | Study objectives and hypotheses |

## Methods — Study design

| # | Checklist item |
|---|----------------|
| 5 | Whether data collection was planned before the index test and reference standard were performed (prospective study) or after (retrospective study) |

## Methods — Participants

| # | Checklist item |
|---|----------------|
| 6 | Eligibility criteria |
| 7 | On what basis potentially eligible participants were identified (such as symptoms, results from previous tests, inclusion in registry) |
| 8 | Where and when potentially eligible participants were identified (setting, location, and dates) |
| 9 | Whether participants formed a consecutive, random, or convenience series |

## Methods — Test methods

| # | Checklist item |
|---|----------------|
| 10a | Index test, in sufficient detail to allow replication |
| 10b | Reference standard, in sufficient detail to allow replication |
| 11 | Rationale for choosing the reference standard (if alternatives exist) |
| 12a | Definition of and rationale for test positivity cut-offs or result categories of the index test, distinguishing pre-specified from exploratory |
| 12b | Definition of and rationale for test positivity cut-offs or result categories of the reference standard, distinguishing pre-specified from exploratory |
| 13a | Whether clinical information and reference standard results were available to the performers or readers of the index test |
| 13b | Whether clinical information and index test results were available to the assessors of the reference standard |

## Methods — Analysis

| # | Checklist item |
|---|----------------|
| 14 | Methods for estimating or comparing measures of diagnostic accuracy |
| 15 | How indeterminate index test or reference standard results were handled |
| 16 | How missing data on the index test and reference standard were handled |
| 17 | Any analyses of variability in diagnostic accuracy, distinguishing pre-specified from exploratory |
| 18 | Intended sample size and how it was determined |

## Results — Participants

| # | Checklist item |
|---|----------------|
| 19 | Flow of participants, using a diagram |
| 20 | Baseline demographic and clinical characteristics of participants |
| 21a | Distribution of severity of disease in those with the target condition |
| 21b | Distribution of alternative diagnoses in those without the target condition |
| 22 | Time interval and any clinical interventions between index test and reference standard |

## Results — Test results

| # | Checklist item |
|---|----------------|
| 23 | Cross tabulation of the index test results (or their distribution) by the results of the reference standard |
| 24 | Estimates of diagnostic accuracy and their precision (such as 95% confidence intervals) |
| 25 | Any adverse events from performing the index test or the reference standard |

## Discussion

| # | Checklist item |
|---|----------------|
| 26 | Study limitations, including sources of potential bias, statistical uncertainty, and generalisability |
| 27 | Implications for practice, including the intended use and clinical role of the index test |

## Other information

| # | Checklist item |
|---|----------------|
| 28 | Registration number and name of registry |
| 29 | Where the full study protocol can be accessed |
| 30 | Sources of funding and other support; role of funders |

## Notes for Assessors

- **Score 34 rows, not 30.** The a/b halves of items 10, 12, 13 and 21 are separate requirements: the
  index test and the reference standard are described, cut-offs defined, blinding stated, and severity
  distributions reported **for each side separately**. A file that collapses them lets a study satisfy
  half an item and be marked complete.
- **Items 29 and 30 are commonly dropped** — where the protocol can be accessed, and funding with the
  role of funders.
- **Item 2** defers to STARD for Abstracts for the abstract's own requirements.
- For a systematic review of diagnostic accuracy studies use `PRISMA_DTA.md`; for risk of bias in the
  included studies, `QUADAS2.md`.

# STARD-AI Checklist

**Standards for Reporting of Diagnostic Accuracy Studies — Artificial Intelligence**
Version: STARD-AI 2025
Source: https://doi.org/10.1038/s41591-025-03953-8

**Reference:** STARD-AI Steering Committee, STARD-AI Consensus Group, Sounderajah V, Guni A, Liu X, Collins GS, Karthikesalingam A, Markar SR, Golub RM, Denniston AK, Shetty S, Moher D, Bossuyt PM, Darzi A, Ashrafian H. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2025;31:3283-3289. PMID: 40954311

**License:** Creative Commons Attribution (CC BY)

**Scope:** AI-centred diagnostic test accuracy studies. Applies to studies evaluating the diagnostic accuracy of AI systems including machine learning, deep learning models, natural language processing tools, and foundation models that generate or support diagnostic outputs. Does NOT apply to static/manually programmed rule-based systems or simple decision trees.

**Relationship to STARD 2015:** STARD-AI extends STARD 2015 (Bossuyt et al. BMJ 2015). Of the 40 items, 22 are UNCHANGED from STARD 2015, 4 are MODIFIED, and 14 are NEW. Items are numbered to maintain alignment with STARD 2015 where possible.

**Relationship to other guidelines:**
- CONSORT-AI: for clinical trials of AI interventions
- SPIRIT-AI: for trial protocols involving AI
- TRIPOD+AI: for AI prediction/prognostic models
- CLAIM 2024: for AI/ML in medical imaging
- Use STARD-AI when diagnostic accuracy of an AI system is the primary focus

---

## Checklist Items (40 items; items 15, 16, 17, 26, and 40 have sub-items)

### Title and Abstract

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 1 | Title | Identification as a study reporting AI-centered diagnostic accuracy and reporting at least one measure of accuracy within title or abstract. | MODIFIED |
| 2 | Abstract | Structured summary of study design, methods, results and conclusions (for specific guidance, see STARD for Abstracts). | UNCHANGED |

### Introduction

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 3 | Scientific background | Scientific and clinical background, including the intended use of the index test, whether it is novel or an established index test and its integration into an existing or new workflow, if applicable. | MODIFIED |
| 4 | Objectives | Study objectives and hypotheses. | UNCHANGED |

### Methods

#### Study Design

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 5 | Study design | Whether data collection was planned before the index test and reference standard were performed (prospective study) or after (retrospective study). | UNCHANGED |

#### Ethics

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 6 | Ethics approval | Formal approval from an ethics committee. If not required, justify why. | NEW |

#### Participants

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 7 | Eligibility criteria | Eligibility criteria: listing separate inclusion and exclusion criteria in the order that they are applied at both participant level and data level. | MODIFIED |
| 8 | Participant identification basis | On what basis potentially eligible participants were identified (such as symptoms, results from previous tests and inclusion in registry). | UNCHANGED |
| 9 | Setting and dates | Where and when potentially eligible participants were identified (setting, location and dates). | UNCHANGED |
| 10 | Participant series | Whether participants formed a consecutive, random or convenience series. | UNCHANGED |

#### Dataset

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 11 | Data source | Source of the data and whether they have been routinely collected, specifically collected for the purpose of the study or acquired from an open-source repository. | NEW |
| 12 | Dataset annotation | Who undertook the annotations for the dataset (including experience levels and background) and how (within the same clinical context or in a post hoc fashion), if applicable. | NEW |
| 13 | Data capture devices and software | Devices (manufacturer and model) that were used to capture data; software (with version number) used to engineer the index test, highlighting the intended use. | NEW |
| 14 | Data acquisition and preprocessing | Data acquisition protocols (for example, contrast protocol or reconstruction method for medical images) and details of data preprocessing, in sufficient detail to allow replication. | NEW |

#### Test Methods

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 15a | Index test | Index test, in sufficient detail to allow replication. | UNCHANGED |
| 15b | Index test development | How the index test was developed, including any training, validation, testing and external evaluation, detailing sample sizes, when applicable. | NEW |
| 15c | Index test cut-offs | Definition of and rationale for test positivity cutoffs or result categories of the index test, distinguishing prespecified from exploratory. | UNCHANGED |
| 15d | End-user specification | The specified end-user of the index test and the level of expertise required of users. | NEW |
| 16a | Reference standard | Reference standard, in sufficient detail to allow replication. | UNCHANGED |
| 16b | Reference standard rationale | Rationale for choosing the reference standard (if alternatives exist). | UNCHANGED |
| 16c | Reference standard cut-offs | Definition of and rationale for test positivity cutoffs or result categories of the reference standard, distinguishing prespecified from exploratory. | UNCHANGED |
| 17a | Blinding (index test) | Whether clinical information and reference standard results were available to the performers or readers of the index test. | UNCHANGED |
| 17b | Blinding (reference standard) | Whether clinical information and index test results were available to the assessors of the reference standard. | UNCHANGED |

#### Analysis

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 18 | Statistical methods | Methods for estimating or comparing measures of diagnostic accuracy. | UNCHANGED |
| 19 | Indeterminate results | How indeterminate index test or reference standard results were handled. | UNCHANGED |
| 20 | Missing data | How missing data on the index test and reference standard were handled. | UNCHANGED |
| 21 | Variability analyses | Any analyses of variability in diagnostic accuracy, distinguishing prespecified from exploratory. | UNCHANGED |
| 22 | Sample size | Intended sample size and how it was determined. | UNCHANGED |
| 23 | Fairness assessment (methods) | Details of any performance error analysis and algorithmic bias and fairness assessments, if undertaken. | NEW |

### Results

#### Participants and Dataset

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 24 | Flow diagram | Flow of participants, using a diagram. | UNCHANGED |
| 25 | Baseline characteristics | Baseline demographic, clinical and technical characteristics of training, validation and test sets, if applicable. | MODIFIED |
| 26a | Distribution of severity | Distribution of severity of disease in those with the target condition. | UNCHANGED |
| 26b | Alternative diagnoses | Distribution of alternative diagnoses in those without the target condition. | UNCHANGED |
| 27 | Time interval | Time interval and any clinical interventions between index test and reference standard. | UNCHANGED |
| 28 | Test set representativeness | Whether the datasets represent the distribution of the target condition that one would expect from the intended use population. | NEW |
| 29 | External evaluation differences | For external evaluation on an independent dataset, an assessment of how this differs from the training, validation and test sets. | NEW |

#### Test Results

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 30 | Cross-tabulation | Cross-tabulation of the index test results (or their distribution) by the results of the reference standard. | UNCHANGED |
| 31 | Accuracy estimates | Estimates of diagnostic accuracy and their precision (such as 95% confidence intervals). | UNCHANGED |
| 32 | Adverse events | Any adverse events from performing the index test or the reference standard. | UNCHANGED |

### Discussion

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 33 | Study limitations | Study limitations, including sources of potential bias, statistical uncertainty and generalizability. | UNCHANGED |
| 34 | Clinical applicability | Implications for practice, including the intended use and clinical role of the index test. | UNCHANGED |
| 35 | Ethical considerations and fairness | Ethical considerations and adherence to ethical standards associated with the use of the index test and issues of fairness. | NEW |

### Other Important Information

| # | Item | Description | Status vs STARD 2015 |
|---|------|-------------|---------------------|
| 36 | Registration | Registration number and name of registry. | UNCHANGED |
| 37 | Protocol | Where the full study protocol can be accessed. | UNCHANGED |
| 38 | Funding | Sources of funding and other support; role of funders. | UNCHANGED |
| 39 | Commercial interests | Commercial interests, if applicable. | NEW |
| 40a | Data and code availability | Availability of datasets and code, detailing any restrictions on their reuse and repurposing. | NEW |
| 40b | Audit and evaluation | Whether outputs are stored, auditable and available for evaluation, if necessary. | NEW |

---

## Summary of Changes from STARD 2015

### MODIFIED Items (4)

| STARD-AI Item | Change Summary |
|---------------|---------------|
| 1 (Title) | Added requirement to identify the study as AI-centered diagnostic accuracy |
| 3 (Introduction) | Added requirement for intended use, novelty, and workflow integration |
| 7 (Eligibility criteria) | Expanded to include eligibility at both participant and data level |
| 25 (Baseline characteristics) | Expanded to include characteristics of training, validation, and test sets |

### NEW Items (14)

| STARD-AI Item | Topic |
|---------------|-------|
| 6 | Ethics approval |
| 11 | Data source and collection method |
| 12 | Dataset annotation |
| 13 | Data capture devices and software |
| 14 | Data acquisition and preprocessing |
| 15b | Index test development (training, validation, testing) |
| 15d | End-user specification and expertise level |
| 23 | Fairness assessment methods |
| 28 | Test set representativeness |
| 29 | External evaluation differences |
| 35 | Ethical considerations and fairness |
| 39 | Commercial interests |
| 40a | Data and code availability |
| 40b | Audit and evaluation of outputs |

---

## Notes for Assessors

### When to Use STARD-AI vs Other AI Guidelines

- **STARD-AI**: When the study evaluates diagnostic accuracy of an AI system as the primary outcome. Includes imaging AI, LLM-based diagnostic tools, pathology AI, EHR-based diagnostic systems.
- **TRIPOD+AI / TRIPOD-LLM**: When the study develops or validates a prediction/prognostic model using AI/ML.
- **CLAIM 2024**: When the study develops or validates an AI model specifically for medical imaging.
- **CONSORT-AI**: When the study is a clinical trial of an AI intervention.
- Authors may refer to multiple checklists and select the one most aligned with the study's primary aim.

### Key Assessment Points

1. **Item 1 (Title)**: The title must explicitly identify the study as reporting AI-centered diagnostic accuracy. Merely mentioning "machine learning" or "deep learning" without connecting it to diagnostic accuracy is PARTIAL.
2. **Item 7 (Eligibility)**: STARD-AI requires eligibility criteria at BOTH participant level (e.g., age, diagnosis) AND data level (e.g., image quality, scanner type). Reporting only participant-level criteria is PARTIAL.
3. **Item 12 (Annotation)**: Must include who annotated, their experience levels and background, and how (within clinical context or post hoc). Missing any of these elements is PARTIAL.
4. **Item 14 (Data acquisition)**: Must describe data acquisition protocols and preprocessing in sufficient detail to allow replication.
5. **Item 23 & 35 (Fairness/Ethics)**: These are paired items — methods/assessment in Item 23, ethical considerations and fairness discussion in Item 35. Both must be present. If the study claims fairness was not assessed, this should be explicitly stated with justification.
6. **Item 25 (Baseline characteristics)**: For AI studies, characteristics must be reported for EACH dataset partition (training, validation, test), not just overall.
7. **Items 39-40b (Other information)**: Commercial interests (39), data/code availability (40a), and audit/evaluation (40b) are NEW and commonly missing. These are increasingly required by journals.

### Relationship to MI-CLEAR-LLM

If the AI system under evaluation is an LLM (e.g., GPT-4, Claude, Gemini), apply MI-CLEAR-LLM (6 items) alongside STARD-AI. MI-CLEAR-LLM addresses LLM-specific concerns (stochasticity, prompt documentation, data contamination) that are not covered by STARD-AI.

### Common Gaps in AI Diagnostic Accuracy Studies

Based on the systematic review that informed STARD-AI development (Aggarwal et al. npj Digit Med 2021):
1. Missing annotation process details
2. Missing data preprocessing description
3. No fairness/bias assessment
4. No external validation or test set representativeness discussion
5. Missing model architecture and training details
6. No commercial interest disclosure
7. No data/code availability statement

---

## Verification Note

This checklist was verified against the published Table 2 of the STARD-AI paper in Nature Medicine (2025;31:3283-3289, DOI: 10.1038/s41591-025-03953-8). Item numbering, descriptions, and NEW/MODIFIED/UNCHANGED classifications match the published version. Verified 2026-04-11 via Nature Medicine online full-size Table 2 (https://www.nature.com/articles/s41591-025-03953-8/tables/2) and STARD 2015 checklist (EQUATOR Network). The STARD 2015 original checklist (Bossuyt et al. BMJ 2015) was used to confirm UNCHANGED items.

# STROBE Checklist

**Strengthening the Reporting of Observational Studies in Epidemiology**
Version: STROBE 2007 (combined checklist for cohort, case-control, and cross-sectional studies)
Source: https://www.strobe-statement.org

Source: von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement. *PLoS Med* 2007;4(10):e296 (DOI 10.1371/journal.pmed.0040296).
Licence: CC BY 4.0 — confirmed via the PubMed Central record (PMC2020495).
Verification: all 34 sub-items (1a–22) were compared against the official combined STROBE
checklist PDF distributed by the EQUATOR Network (`STROBE_checklist_v4_combined.pdf`), parsed
sub-item by sub-item. 34/34 match, with no item missing and none invented. The statement's own
PubMed Central record carries its checklist as an image, not as text, which is why the audit
used the distributed checklist rather than the article.

## Checklist Items (22 items)

### Title and Abstract

| # | Item | Description |
|---|------|-------------|
| 1a | Title | Indicate the study's design with a commonly used term in the title or abstract. |
| 1b | Abstract | Provide in the abstract an informative and balanced summary of what was done and what was found. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 2 | Background/rationale | Explain the scientific background and rationale for the investigation being reported. |
| 3 | Objectives | State specific objectives, including any prespecified hypotheses. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 4 | Study design | Present key elements of study design early in the paper. |
| 5 | Setting | Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection. |
| 6a | Participants | *Cohort study*: Give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up. *Case-control study*: Give the eligibility criteria, and the sources and methods of case ascertainment and control selection. Give the rationale for the choice of cases and controls. *Cross-sectional study*: Give the eligibility criteria, and the sources and methods of selection of participants. |
| 6b | Participants | *Cohort study*: For matched studies, give matching criteria and number of exposed and unexposed. *Case-control study*: For matched studies, give matching criteria and the number of controls per case. |
| 7 | Variables | Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable. |
| 8 | Data sources/measurement | For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group. |
| 9 | Bias | Describe any efforts to address potential sources of bias. |
| 10 | Study size | Explain how the study size was arrived at. |
| 11 | Quantitative variables | Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen and why. |
| 12a | Statistical methods | Describe all statistical methods, including those used to control for confounding. |
| 12b | Statistical methods | Describe any methods used to examine subgroups and interactions. |
| 12c | Statistical methods | Explain how missing data were addressed. |
| 12d | Statistical methods | *Cohort study*: If applicable, explain how loss to follow-up was addressed. *Case-control study*: If applicable, explain how matching of cases and controls was addressed. *Cross-sectional study*: If applicable, describe analytical methods taking account of sampling strategy. |
| 12e | Statistical methods | Describe any sensitivity analyses. |

### Results

| # | Item | Description |
|---|------|-------------|
| 13a | Participants | Report numbers of individuals at each stage of study -- e.g., numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed. |
| 13b | Participants | Give reasons for non-participation at each stage. |
| 13c | Participants | Consider use of a flow diagram. |
| 14a | Descriptive data | Give characteristics of study participants (e.g., demographic, clinical, social) and information on exposures and potential confounders. |
| 14b | Descriptive data | Indicate number of participants with missing data for each variable of interest. |
| 14c | Descriptive data | *Cohort study*: Summarise follow-up time (e.g., average and total amount). |
| 15 | Outcome data | *Cohort study*: Report numbers of outcome events or summary measures over time. *Case-control study*: Report numbers in each exposure category, or summary measures of exposure. *Cross-sectional study*: Report numbers of outcome events or summary measures. |
| 16a | Main results | Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (e.g., 95% confidence interval). Make clear which confounders were adjusted for and why they were included. |
| 16b | Main results | Report category boundaries when continuous variables were categorized. |
| 16c | Main results | If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period. |
| 17 | Other analyses | Report other analyses done -- e.g., analyses of subgroups and interactions, and sensitivity analyses. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 18 | Key results | Summarise key results with reference to study objectives. |
| 19 | Limitations | Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias. |
| 20 | Interpretation | Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence. |
| 21 | Generalisability | Discuss the generalisability (external validity) of the study results. |

### Other Information

| # | Item | Description |
|---|------|-------------|
| 22 | Funding | Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based. |

---

## Notes for Assessors

- Items 6b, 12d, and 14c have study-design-specific wording. Choose the version matching the study design.
- Item 13c (flow diagram) is a recommendation, not a strict requirement, but strongly encouraged.
- For studies involving AI/ML, consider also checking against TRIPOD+AI or CLAIM as applicable.
- STROBE extensions exist for specific designs: STROBE-ME (molecular epidemiology), STROBE-NI (neonatal infections), RECORD (routinely collected health data).

# STROBE-MR Checklist

**Strengthening the Reporting of Observational Studies in Epidemiology using Mendelian Randomization**
Version: STROBE-MR 2021 — **20 items and 30 sub-items** across six sections.
Source (the statement): Skrivankova VW, Richmond RC, Woolf BAR, Yarmolinsky J, Davies NM, Swanson SA,
et al. Strengthening the Reporting of Observational Studies in Epidemiology Using Mendelian
Randomization: The STROBE-MR Statement. *JAMA* 2021;326(16):1614-1621 (DOI 10.1001/jama.2021.18236).
Explanation and Elaboration: Skrivankova VW, et al. *BMJ* 2021;375:n2233. Also: https://www.strobe-mr.org

> **Fidelity and licence.** The statement is published in *JAMA* (© American Medical Association) under
> **no open licence**. The descriptions below state what each item asks **in our own words**; the
> structure — 20 items, which sub-items exist and under which item, the section grouping — is
> **verified against the statement**. **Complete the official checklist for anything you submit**, and
> read the Explanation and Elaboration for the rationale and worked examples.

> **What this file used to be.** It cited "Skidmore ME / Davey Smith G, Davies NM, et al. *BMJ*
> 2021;375:n2233" as the statement — a non-existent first author, and the E&E paper rather than the
> statement. It carried **items 1–20 with none of the 30 sub-items**, which is where every
> MR-specific requirement lives. From item 17 on it was shifted by one (Funding at 17 instead of 18)
> and it **invented an item 20, "Other"**, where the statement has Conflicts of interest. It also
> claimed the statement was CC BY, and carried a "Verified" stamp. None of that survived contact with
> the published table.

**Scope.** Covers 1-sample and 2-sample MR, single or multiple exposures and outcomes, and MR
following a GWAS reported in the same article. Does **not** cover GWAS themselves (use STREGA),
sequencing or expression studies, or ordinary observational epidemiology (use `STROBE.md`). For MR
that does not use instrumental-variable estimation — some gene-by-environment interaction studies —
some items will not apply.

**Relationship to STROBE.** A stand-alone extension: STROBE has 22 items and 18 sub-items, STROBE-MR
has 20 items and 30 sub-items. Every item and sub-item was modified for MR **except sub-item 6d**
(missing data), which is unchanged. Name both instruments in Methods.

**Not a quality instrument.** The statement says explicitly that the checklist is not to be used to
evaluate the quality of MR research.

## Title and Abstract

| # | Item | What to check is reported |
|---|------|---------------------------|
| 1 | Title and abstract | MR is named as the study design in the title and/or abstract, where that is a main purpose of the study. |

## Introduction

| # | Item | What to check is reported |
|---|------|---------------------------|
| 2 | Background | The scientific background and rationale; what the exposure is; whether a causal exposure–outcome relationship is plausible; and **a justification of why MR helps answer this question**. |
| 3 | Objectives | Specific objectives, including **prespecified causal hypotheses** if any, and a statement that MR estimates causal effects only under stated assumptions. |

## Methods

| # | Item | What to check is reported |
|---|------|---------------------------|
| 4 | Study design and data sources | Key design elements early in the article; consider a table of data sources for every phase. Then, **for each data source**: |
| 4a | — Setting | The design and underlying population; setting, locations and relevant dates, including recruitment, exposure, follow-up and data collection. |
| 4b | — Participants | Eligibility criteria, how participants were selected, sample size, and whether any power or sample-size calculation was done **before** the main analysis. |
| 4c | — Genetic variants | **Measurement, quality control and selection of the genetic variants.** |
| 4d | — Variables | Assessment methods and diagnostic criteria for each exposure, outcome and other relevant variable. |
| 4e | — Ethics | Ethics committee approval and participant informed consent, where relevant. |
| 5 | Assumptions | **The three core IV assumptions stated explicitly** — relevance, independence, exclusion restriction — plus the assumptions of any additional or sensitivity analysis. |
| 6 | Statistical methods: main analysis | The statistical methods and statistics used: |
| 6a | — Quantitative variables | How they were handled — scale, units, model. |
| 6b | — Genetic variants | How variants were handled and, if applicable, how their weights were chosen. |
| 6c | — MR estimator | **Which estimator** (two-stage least squares, Wald ratio, …) and its related statistics; the covariates included; and for 2-sample MR whether the same covariate set was used in both samples. |
| 6d | — Missing data | How missing data were addressed. *(The only sub-item unchanged from STROBE.)* |
| 6e | — Multiple testing | How multiple testing was addressed, if applicable. |
| 7 | Assessment of assumptions | The methods or prior knowledge used to **assess** the assumptions or justify their validity. |
| 8 | Sensitivity and additional analyses | Any sensitivity or additional analyses — comparison of estimates from different approaches, independent replication, bias-analytic techniques, instrument validation, simulations. |
| 9 | Software and preregistration | |
| 9a | — Software | Statistical software and packages, **with version and settings**. |
| 9b | — Preregistration | **Whether the protocol and details were preregistered**, and when and where. |

## Results

| # | Item | What to check is reported |
|---|------|---------------------------|
| 10 | Descriptive data | |
| 10a | — Flow | Numbers of individuals at each stage of the included studies and reasons for exclusion; consider a flow diagram. |
| 10b | — Summary statistics | For phenotypic exposures, outcomes and other relevant variables — means, SDs, proportions. |
| 10c | — Heterogeneity | Where data sources include meta-analyses of previous studies, the assessments of heterogeneity across them. |
| 10d | — 2-sample MR | (i) **Justification that the variant–exposure associations are similar** between the exposure and outcome samples; (ii) **the number of individuals overlapping** between the two studies. |
| 11 | Main results | |
| 11a | — Associations | The variant–exposure and variant–outcome associations, preferably on an interpretable scale. |
| 11b | — MR estimates | The MR estimate of the exposure–outcome relationship with its uncertainty, on an interpretable scale such as an odds ratio or relative risk per SD. |
| 11c | — Absolute risk | Where relevant, relative risk translated into absolute risk over a meaningful period. |
| 11d | — Plots | Consider plots — forest plot, or variant–outcome against variant–exposure associations. |
| 12 | Assessment of assumptions | |
| 12a | — Validity | The assessment of the validity of the assumptions. |
| 12b | — Statistics | Additional statistics — heterogeneity across variants (I², Q) or an E-value. |
| 13 | Sensitivity and additional analyses | |
| 13a | — Robustness | Sensitivity analyses testing robustness to violations of the assumptions. |
| 13b | — Other | Results of other sensitivity or additional analyses. |
| 13c | — Direction | **Any assessment of the direction of the causal relationship**, e.g. bidirectional MR. |
| 13d | — Non-MR comparison | Where relevant, comparison with estimates from non-MR analyses. |
| 13e | — Plots | Consider additional plots, e.g. leave-one-out analyses. |

## Discussion

| # | Item | What to check is reported |
|---|------|---------------------------|
| 14 | Key results | The key results summarised against the study objectives. |
| 15 | Limitations | Limitations, taking in the validity of the IV assumptions, other sources of bias, and imprecision — with **the direction and magnitude** of any potential bias and what was done about it. |
| 16 | Interpretation | |
| 16a | — Meaning | A cautious overall interpretation in the light of the limitations and of other studies. |
| 16b | — Mechanism | The biological mechanisms that could drive the relationship, and **whether the gene-environment equivalence assumption is reasonable**; causal language used carefully, making clear that IV estimates are causal only under certain assumptions. |
| 16c | — Clinical relevance | Whether the results have clinical or public-policy relevance, and what they imply about the size of possible interventions. |
| 17 | Generalizability | Generalisability of the results **(a) to other populations, (b) across other exposure periods or timings, and (c) across other levels of exposure**. |

## Other Information

| # | Item | What to check is reported |
|---|------|---------------------------|
| 18 | Funding | Sources of funding and the role of funders for this study and, where applicable, for the databases and original studies it rests on. |
| 19 | Data and data sharing | The data used, or where and how it can be accessed, referenced in the article; and **the statistical code needed to reproduce the results**, or where it is publicly accessible. |
| 20 | Conflicts of interest | Declared by **all** authors. |

## Notes for Assessors

- **The sub-items are the extension.** Items 1–20 alone are close to a re-lettered STROBE. What makes
  a report an MR report is 4c (variant measurement, QC and selection), 6c (the estimator), 9b
  (preregistration), 10d (2-sample similarity and **participant overlap**), 12a/12b (assumption
  validity, heterogeneity across variants), 13c (direction of causation), and 16b (gene-environment
  equivalence). Score them.
- **Item 5 is the one most often skipped**: the three IV assumptions named explicitly — relevance,
  independence, exclusion restriction — not gestured at.
- A study reporting only an inverse-variance-weighted estimate, with no assessment of the assumptions
  (7, 12) and no sensitivity analyses (8, 13), is non-compliant regardless of how well it reads.
- **Name both instruments.** STROBE-MR is a stand-alone extension of STROBE; cite each. See
  `STROBE.md` for the base items.
- Authors are expected to address every item and sub-item, using supplementary material where space
  is short.
- For the design-validity review of the same study, pair with the MR domain probes in
  `peer-review` / `self-review` `references/domain-probes/mendelian_randomization.md`; for the
  analysis, with `analyze-stats` `analysis_guides/mendelian_randomization.md`.

# SWiM Checklist

**Synthesis Without Meta-analysis (SWiM) in systematic reviews: reporting guideline**
Version: SWiM 2020
Source: Campbell M et al. BMJ 2020;368:l6890. doi: 10.1136/bmj.l6890
Website: https://swim.sphsu.gla.ac.uk/

Licence: CC BY 4.0 — confirmed via Crossref.
Verification: all 9 items were compared against Table 1 of the published statement (Europe PMC
full text, PMC7190266). The nine labels matched; **six descriptions did not** and have been
replaced with the published text — item 5 previously stated a different requirement altogether,
item 7 dropped its second element, and items 2, 4 and 8 dropped required elements (rationale for
the metric, justification for the prioritisation criteria, certainty of the findings).

## Checklist Items (9 items)

### Reporting Items

| # | Item | Description |
|---|------|-------------|
| 1 | Grouping studies for synthesis | **1a)** Provide a description of, and rationale for, the groups used in the synthesis (e.g., groupings of populations, interventions, outcomes, study design). **1b)** Detail and provide rationale for any changes made subsequent to the protocol in the groups used in the synthesis. |
| 2 | Describe the standardised metric and transformation methods used | Describe the standardised metric for each outcome. Explain why the metric(s) was chosen and describe any methods used to transform the intervention effects, as reported in the study, to the standardised metric, citing any methodological guidance consulted. |
| 3 | Describe the synthesis methods | Describe and justify the methods used to synthesise the effects for each outcome when it was not possible to undertake a meta-analysis of effect estimates. |
| 4 | Criteria used to prioritise results for summary and synthesis | Where applicable, provide the criteria used, with supporting justification, to select the particular studies, or a particular study, for the main synthesis or to draw conclusions from the synthesis (e.g., based on study design, risk of bias assessments, directness in relation to the review question). |
| 5 | Investigation of heterogeneity in reported effects | State the method(s) used to examine heterogeneity in reported effects when it was not possible to undertake a meta-analysis of effect estimates and its extensions to investigate heterogeneity. |
| 6 | Certainty of evidence | Describe the methods used to assess the certainty of the synthesis findings. |
| 7 | Data presentation methods | Describe the graphical and tabular methods used to present the effects (e.g., tables, forest plots, harvest plots). Specify key study characteristics (e.g., study design, risk of bias) used to order the studies, in the text and any tables or graphs, clearly referencing the studies included. |
| 8 | Reporting results | For each comparison and outcome, provide a description of the synthesised findings and the certainty of the findings. Describe the result in language that is consistent with the question the synthesis addresses, and indicate which studies contribute to the synthesis. |
| 9 | Limitations of the synthesis | Report the limitations of the synthesis methods used and/or the groupings used in the synthesis and how these affect the conclusions that can be drawn in relation to the original review question. |

The synthesis methods item 3 refers to include vote counting based on direction of effect,
combining P values, calculating the median effect size, and combining confidence intervals — the
statement's Table 2 maps which of these can answer which question, given the available data.

---

## Notes for Assessors

- SWiM complements PRISMA 2020 — it provides additional guidance for reporting when meta-analysis is not performed
- We estimate that 35% of health-related systematic reviews do not do meta-analysis
- Items currently available in PRISMA (Items 13, 14, and 21) and RAMESES (Items 15 and 19) also apply
- SWiM items should accompany PRISMA items, not replace them
- The guideline applies to systematic reviews using alternative synthesis methods: vote counting based on direction, combining P values, narrative synthesis, range of effects, etc.
- Not applicable to qualitative evidence syntheses (use ENTREQ instead)

# TARGET Checklist

**Transparent Reporting of Observational Studies Emulating a Target Trial**
Version: TARGET 2025 (21 items across 6 sections; items 6 and 7 pair the target-trial *specification* with its *emulation* in the data)
Source: In-house faithful summary of the TARGET item intents (own-words paraphrase, not verbatim). Cashin AG, Hansford HJ, Hernán MA, et al. Transparent Reporting of Observational Studies Emulating a Target Trial: The TARGET Statement. JAMA 2025;334(12):1084-1093. DOI 10.1001/jama.2025.13350. Official checklist: https://target-guideline.org. Complete the official TARGET instrument for a submission checklist. Pairs with the `/design-study` target-trial-emulation design module.

Licence: *JAMA* (© American Medical Association) — no open licence.
Verification: all 39 sub-items across the 21 numbered items were compared, by number and order,
against the checklist tables of the published statement (PubMed Central record PMC13084563).
39/39 are present, including the paired 6a–h specification and 7a–7h(ii) emulation columns.
Three items dropped a required clause (6c, 6d, 7d) and are corrected. Wording stays paraphrased —
the statement is © American Medical Association with no open licence.

## Checklist Items (21 items)

### Title and Abstract

| # | Item | Description |
|---|------|-------------|
| 1a | Study type | Identify that the study attempts to emulate a target trial using observational data. |
| 1b | Data sources | Report the data sources used for the emulation. |
| 1c | Key elements | Summarize the key assumptions, statistical methods, findings, and conclusions. |

### Introduction

| # | Item | Description |
|---|------|-------------|
| 2 | Background | Describe the scientific background and the gap in knowledge the study addresses. |
| 3 | Causal question | Summarize the causal question specified by the target-trial protocol. |
| 4 | Rationale | Describe the rationale for emulating a target trial with the available data. |

### Methods

| # | Item | Description |
|---|------|-------------|
| 5 | Data sources | Cite the data sources and describe their original purpose, type, geographic locations, setting, and time period. |

#### Target-trial specification (the protocol you would run)

| # | Item | Description |
|---|------|-------------|
| 6a | Eligibility criteria | Describe the eligibility criteria defining the target population. |
| 6b | Treatment strategies | Describe the treatment strategies to be compared, in sufficient detail (e.g., dose, duration, start/stop rules). |
| 6c | Assignment | Report that eligible individuals would be randomly assigned to the treatment strategies, and may be aware of their treatment allocation. |
| 6d | Follow-up | Clarify that follow-up would start at the time of assignment to the treatment strategies, and specify when follow-up would end. |
| 6e | Outcomes | Describe the outcomes, including their measurement and timing. |
| 6f | Causal contrasts | Describe the causal contrasts of interest, including the effect measures. |
| 6g | Identifying assumptions | Describe the assumptions that would be made to identify each causal estimand. |
| 6h | Data analysis plan | For each causal estimand, describe the data-analysis procedures and statistical models. |

#### Target-trial emulation (mapping to the observational data)

| # | Item | Description |
|---|------|-------------|
| 7a | Eligibility (emulation) | Describe how the eligibility criteria were operationalized with the data. |
| 7b | Treatment strategies (emulation) | Describe how the treatment strategies were operationalized with the data. |
| 7c | Assignment (emulation) | Describe how assignment to treatment strategies was operationalized with the data. |
| 7d | Follow-up (emulation) | Clarify that follow-up starts at the time individuals were assigned to the treatment strategies, and describe how the end of follow-up was operationalized with the data. *(Misaligning eligibility, assignment and start of follow-up is what introduces immortal-time bias.)* |
| 7e | Outcomes (emulation) | Describe how the outcomes were operationalized with the data. |
| 7f | Causal contrasts (emulation) | Describe how the causal contrasts were operationalized with the data. |
| 7g(i) | Identifying assumptions (emulation) | For each causal estimand, describe the assumptions made, including baseline confounding. |
| 7g(ii) | Assumption variables | Describe how the variables related to those assumptions were operationalized. |
| 7h(i) | Data analysis (emulation) | Describe modifications to the data-analysis methods needed for the observational emulation. |
| 7h(ii) | Sensitivity analyses | Describe any additional analyses assessing the sensitivity of results to the operationalization choices. |

### Results

| # | Item | Description |
|---|------|-------------|
| 8 | Participant selection | Report the numbers of individuals assessed for eligibility, eligible, and assigned to each treatment strategy. |
| 9 | Baseline data | Describe the distribution of baseline characteristics of individuals, by treatment strategy. |
| 10 | Follow-up | Summarize the length of follow-up and describe the reasons for its end. |
| 11 | Missing data | Describe the frequency of missing data in all variables, by treatment strategy. |
| 12 | Outcomes | Describe the frequency or distribution of each outcome, by treatment strategy. |
| 13 | Effect estimates | Report the effect estimate for each causal contrast, with its corresponding measure of precision. |
| 14 | Additional analyses | Report the results of all analyses assessing the sensitivity of the estimates to the choices made. |

### Discussion

| # | Item | Description |
|---|------|-------------|
| 15 | Interpretation | Provide an interpretation of the key findings in the context of the causal question. |
| 16 | Limitations | Discuss limitations, considering differences between the target trial and its emulation. |

### Other Information

| # | Item | Description |
|---|------|-------------|
| 17 | Ethics | Provide the institutional review board or ethics committee approval information. |
| 18 | Registration | State whether, when, and where the study protocol was registered. |
| 19 | Data sharing | State whether the data, analytic code, and materials are accessible. |
| 20 | Funding | Provide the sources of funding and detail the role of the funders. |
| 21 | Conflicts of interest | State any conflicts of interest and financial disclosures for all authors. |

---

## Notes for Assessors

- The distinctive TARGET structure is the paired **specification (item 6)** and **emulation (item 7)**: for each protocol element — eligibility, treatment strategies, assignment, start of follow-up, outcomes, causal contrast, identifying assumptions, analysis — the study must state both the target-trial version and how it was operationalized in the data. A study that reports the emulation without ever specifying the target trial it emulates is a gap.
- The single most consequential defect this catches is **time-zero misalignment → immortal-time bias** (items 6d / 7d): eligibility, treatment assignment, and start of follow-up must coincide.
- Items 6g / 7g require an **explicit causal estimand and its identifying assumptions (including baseline confounding)** — an association reported with no stated estimand or assumptions is a gap, not merely thin reporting.
- TARGET is the **reporting** side; pair it with the `/design-study` **target-trial-emulation module** (the design side, which enforces the same seven-component protocol before data extraction). Use **RECORD / STROBE** for the routinely-collected-data and general observational items not specific to the emulation.
- Not needed for a purely descriptive, prevalence, or diagnostic-accuracy study — TARGET applies to a **causal / comparative-effectiveness** question emulated on observational data.
- The vendored checklist is an educational own-words summary of item intent; complete the official TARGET instrument (target-guideline.org) for a submission checklist.

# TRIPOD Checklist

**Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis**
Version: TRIPOD 2015
Source: Moons KGM et al. Ann Intern Med. 2015;162:55-63. https://www.tripod-statement.org

Source: Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD). *Ann Intern Med* 2015;162(1):55-63 (DOI 10.7326/M14-0697).
Licence: *Annals of Internal Medicine* (© American College of Physicians) — no open licence.
Verification: all 37 sub-items and every D / V / D;V designation were compared against the
official TRIPOD checklist for prediction-model development **and** validation, distributed at
tripod-statement.org (`Tripod-Checklist-Prediction-Model-Development-and-Validation-PDF.pdf`).
37/37 items and 37/37 designations match. Item 17 stated the wrong parenthetical and item 16
presented our own guidance as though it were the item text; both are corrected.

## Applicability

Items apply to different study types:
- **D** = Development study only
- **V** = Validation study only
- **DV** = Both development and validation studies

## Checklist Items

### Title and Abstract

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 1 | Title | DV | Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted. |
| 2 | Abstract | DV | Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions. |

### Introduction

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 3a | Background | DV | Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models. |
| 3b | Objectives | DV | Specify the objectives, including whether the study describes the development or validation of the model or both. |

### Methods

#### Source of Data

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 4a | Source of data | DV | Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable. |
| 4b | Source of data | DV | Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up. |

#### Participants

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 5a | Participants | DV | Specify key elements of the study setting (e.g., primary care, secondary care, general population) including number and location of centres. |
| 5b | Participants | DV | Describe eligibility criteria for participants. |
| 5c | Participants | DV | Give details of treatments received, if relevant. |

#### Outcome

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 6a | Outcome | DV | Clearly define the outcome that is predicted by the prediction model, including how and when assessed. |
| 6b | Outcome | DV | Report any actions to blind assessment of the outcome to be predicted. |

#### Predictors

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 7a | Predictors | DV | Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured. |
| 7b | Predictors | DV | Report any actions to blind assessment of predictors for the outcome and other predictors. |

#### Sample Size

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 8 | Sample size | DV | Explain how the study size was arrived at. |

#### Missing Data

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 9 | Missing data | DV | Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method. |

#### Statistical Analysis Methods

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 10a | Model building | D | Describe how predictors were handled in the analyses. |
| 10b | Model building | D | Specify type of model, all model-building procedures (including any predictor selection), and method for internal validation. |
| 10c | Validation | V | For validation, describe how the predictions were calculated. |
| 10d | Model performance | DV | Specify all measures used to assess model performance and, if relevant, to compare multiple models. |
| 10e | Model updating | V | Describe any model updating (e.g., recalibration) arising from the validation, if done. |

#### Risk Groups

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 11 | Risk groups | DV | Provide details on how risk groups were created, if done. |

#### Development vs. Validation

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 12 | Development vs. validation | V | For validation, identify any differences from the development data in setting, eligibility criteria, outcome, and predictors. |

### Results

#### Participants

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 13a | Participants | DV | Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful. |
| 13b | Participants | DV | Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome. |
| 13c | Participants | V | For validation, show a comparison with the development data of the distribution of important variables (demographics, predictors, and outcome). |

#### Model Development

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 14a | Model development | D | Specify the number of participants and outcome events in each analysis. |
| 14b | Model development | D | If done, report the unadjusted association between each candidate predictor and outcome. |

#### Model Specification

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 15a | Model specification | D | Present the full prediction model to allow predictions for individuals (i.e., all regression coefficients, and model intercept or baseline survival at a given time point). |
| 15b | Model specification | D | Explain how to use the prediction model. |

#### Model Performance

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 16 | Model performance | DV | Report performance measures (with CIs) for the prediction model. *(In practice this means discrimination — e.g., C-statistic/AUC — and calibration — e.g., calibration plot, calibration slope and intercept; the item itself does not enumerate them.)* |

#### Model Updating

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 17 | Model updating | V | If done, report the results from any model updating (i.e., model specification, model performance). |

### Discussion

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 18 | Limitations | DV | Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data). |
| 19a | Interpretation | V | For validation, discuss the results with reference to performance in the development data, and any other validation data. |
| 19b | Interpretation | DV | Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence. |
| 20 | Implications | DV | Discuss the potential clinical use of the model and implications for future research. |

### Other Information

| # | Item | Applies | Description |
|---|------|---------|-------------|
| 21 | Supplementary information | DV | Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and data sets. |
| 22 | Funding | DV | Give the source of funding and the role of the funders for the present study. |

---

## Notes for Assessors

- TRIPOD 2015 applies to **non-AI/ML prediction models only** (logistic regression, Cox regression, etc.). For AI/ML prediction models, use TRIPOD+AI 2024 instead. Do NOT apply both simultaneously.
- Items 10a, 10b, 14a, 14b, 15a, 15b are specific to **development studies**. Mark as N/A for validation-only studies.
- Items 10c, 10e, 13c, 17, 19a are specific to **validation studies**. Mark as N/A for development-only studies.
- Item 16 must include BOTH discrimination (C-statistic/AUC) and calibration (calibration plot, Hosmer-Lemeshow, or calibration slope/intercept). A study reporting only AUC without calibration assessment is PARTIAL on this item.
- Item 8 (sample size): the events per variable (EPV) ratio should be reported. EPV < 10 should be flagged as a limitation.
- Item 15a (full model specification): the complete regression equation must be presented, not just significant predictors. Omitting non-significant predictors from the final model presentation is a common gap.
- For studies that combine development and validation (e.g., internal-external validation), all items apply.

# TRIPOD+AI 2024 Checklist

**Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis — Artificial Intelligence extension**

- **Version:** TRIPOD+AI 2024
- **Citation:** Collins GS, Moons KGM, Dhiman P, et al. *TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods.* BMJ 2024;385:e078378.
- **DOI:** 10.1136/bmj-2023-078378
- **Source:** https://www.tripod-statement.org — official expanded checklist: https://www.tripod-statement.org/wp-content/uploads/2024/04/TRIPODAI-Supplement.pdf
- **Licence:** CC BY 4.0. Item wording below is reproduced from the published statement with attribution.
- **Verified:** all 52 sub-items compared against the official supplements (ST1 expanded checklist, ST2 fillable checklist). 52/52 match in substance; the D/E designations match the statement exactly. Differences are orthographic only.

**TRIPOD+AI 2024 supersedes and replaces TRIPOD 2015.** It is not TRIPOD 2015 plus AI addenda — it is a
complete rewrite, applicable to prediction-model studies using **either** regression **or** machine-learning
methods. For any prediction-model study (regression or AI/ML), use this checklist; do **not** apply the 2015
checklist alongside it.

The checklist has **27 main items** and **52 checklist subitems**, in eight sections: Title (1), Abstract (2),
Introduction (3–4), Methods (5–17), Open science (18), Patient and public involvement (19), Results (20–24),
and Discussion (25–27).

**Applicability column:** `D` = relevant only to model **development**; `E` = relevant only to model
**evaluation**; `D;E` = applicable to **both**.

## Checklist Items

### Title

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 1 | Title | D;E | Identify the study as developing or evaluating the performance of a multivariable prediction model, the target population, and the outcome to be predicted. |

### Abstract

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 2 | Abstract | D;E | See TRIPOD+AI for Abstracts checklist. |

### Introduction

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 3a | Background | D;E | Explain the healthcare context (including whether diagnostic or prognostic) and rationale for developing or evaluating the prediction model, including references to existing models. |
| 3b | Background | D;E | Describe the target population and the intended purpose of the prediction model in the context of the care pathway, including its intended users (e.g., healthcare professionals, patients, public). |
| 3c | Background | D;E | Describe any known health inequalities between sociodemographic groups. |
| 4 | Objectives | D;E | Specify the study objectives, including whether the study describes the development or validation of a prediction model (or both). |

### Methods

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 5a | Data | D;E | Describe the sources of data separately for the development and evaluation datasets (e.g., randomised trial, cohort, routine care or registry data), the rationale for using these data, and representativeness of the data. |
| 5b | Data | D;E | Specify the dates of the collected participant data, including start and end of participant accrual; and, if applicable, end of follow-up. |
| 6a | Participants | D;E | Specify key elements of the study setting (e.g., primary care, secondary care, general population) including the number and location of centres. |
| 6b | Participants | D;E | Describe the eligibility criteria for study participants. |
| 6c | Participants | D;E | Give details of any treatments received, and how they were handled during model development or evaluation, if relevant. |
| 7 | Data preparation | D;E | Describe any data pre-processing and quality checking, including whether this was similar across relevant sociodemographic groups. |
| 8a | Outcome | D;E | Clearly define the outcome that is being predicted and the time horizon, including how and when assessed, the rationale for choosing this outcome, and whether the method of outcome assessment is consistent across sociodemographic groups. |
| 8b | Outcome | D;E | If outcome assessment requires subjective interpretation, describe the qualifications and demographic characteristics of the outcome assessors. |
| 8c | Outcome | D;E | Report any actions to blind assessment of the outcome to be predicted. |
| 9a | Predictors | D | Describe the choice of initial predictors (e.g., literature, previous models, all available predictors) and any pre-selection of predictors before model building. |
| 9b | Predictors | D;E | Clearly define all predictors, including how and when they were measured (and any actions to blind assessment of predictors for the outcome and other predictors). |
| 9c | Predictors | D;E | If predictor measurement requires subjective interpretation, describe the qualifications and demographic characteristics of the predictor assessors. |
| 10 | Sample size | D;E | Explain how the study size was arrived at (separately for development and evaluation), and justify that the study size was sufficient to answer the research question. Include details of any sample size calculation. |
| 11 | Missing data | D;E | Describe how missing data were handled. Provide reasons for omitting any data. |
| 12a | Analytical methods | D | Describe how the data were used (e.g., for development and evaluation of model performance) in the analysis, including whether the data were partitioned, considering any sample size requirements. |
| 12b | Analytical methods | D | Depending on the type of model, describe how predictors were handled in the analyses (functional form, rescaling, transformation, or any standardisation). |
| 12c | Analytical methods | D | Specify the type of model, rationale, all model-building steps, including any hyperparameter tuning, and method for internal validation. |
| 12d | Analytical methods | D;E | Describe if and how any heterogeneity in estimates of model parameter values and model performance was handled and quantified across clusters (e.g., hospitals, countries). See TRIPOD-Cluster for additional considerations. |
| 12e | Analytical methods | D;E | Specify all measures and plots used (and their rationale) to evaluate model performance (e.g., discrimination, calibration, clinical utility) and, if relevant, to compare multiple models. |
| 12f | Analytical methods | E | Describe any model updating (e.g., recalibration) arising from the model evaluation, either overall or for particular sociodemographic groups or settings. |
| 12g | Analytical methods | E | For model evaluation, describe how the model predictions were calculated (e.g., formula, code, object, application programming interface). |
| 13 | Class imbalance | D;E | If class imbalance methods were used, state why and how this was done, and any subsequent methods to recalibrate the model or the model predictions. |
| 14 | Fairness | D;E | Describe any approaches that were used to address model fairness and their rationale. |
| 15 | Model output | D | Specify the output of the prediction model (e.g., probabilities, classification). Provide details and rationale for any classification and how the thresholds were identified. |
| 16 | Training versus evaluation | D;E | Identify any differences between the development and evaluation data in healthcare setting, eligibility criteria, outcome, and predictors. |
| 17 | Ethical approval | D;E | Name the institutional research board or ethics committee that approved the study and describe the participant informed consent or the ethics committee waiver of informed consent. |

### Open science

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 18a | Funding | D;E | Give the source of funding and the role of the funders for the present study. |
| 18b | Conflicts of interest | D;E | Declare any conflicts of interest and financial disclosures for all authors. |
| 18c | Protocol | D;E | Indicate where the study protocol can be accessed or state that a protocol was not prepared. |
| 18d | Registration | D;E | Provide registration information for the study, including register name and registration number, or state that the study was not registered. |
| 18e | Data sharing | D;E | Provide details of the availability of the study data. |
| 18f | Code sharing | D;E | Provide details of the availability of the analytical code. |

### Patient and public involvement

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 19 | Patient and public involvement | D;E | Provide details of any patient and public involvement during the design, conduct, reporting, interpretation, or dissemination of the study or state no involvement. |

### Results

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 20a | Participants | D;E | Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful. |
| 20b | Participants | D;E | Report the characteristics overall and, where applicable, for each data source or setting, including the key dates, key predictors (including demographics), treatments received, sample size, number of outcome events, follow-up time, and amount of missing data. A table may be helpful. Report any differences across key demographic groups. |
| 20c | Participants | E | For model evaluation, show a comparison with the development data of the distribution of important predictors (demographics, predictors, and outcome). |
| 21 | Model development | D;E | Specify the number of participants and outcome events in each analysis (e.g., for model development, hyperparameter tuning, model evaluation). |
| 22 | Model specification | D | Provide details of the full prediction model (e.g., formula, code, object, application programming interface) to allow predictions in new individuals and to enable third-party evaluation and implementation, including any restrictions to access or re-use (e.g., freely available, proprietary). |
| 23a | Model performance | D;E | Report model performance estimates with confidence intervals, including for any key subgroups (e.g., sociodemographic). Consider plots to aid presentation. |
| 23b | Model performance | D;E | If examined, report results of any heterogeneity in model performance across clusters. See TRIPOD-Cluster for additional details. |
| 24 | Model updating | E | Report the results from any model updating, including the updated model and subsequent performance. |

### Discussion

| Item | Topic | D/E | Checklist item |
|---|---|---|---|
| 25 | Interpretation | D;E | Give an overall interpretation of the main results, including issues of fairness in the context of the objectives and previous studies. |
| 26 | Limitations | D;E | Discuss any limitations of the study (such as a non-representative sample, sample size, overfitting, missing data) and their effects on any biases, statistical uncertainty, and generalizability. |
| 27a | Usability of the model in the context of current care | D | Describe how poor quality or unavailable input data (e.g., predictor values) should be assessed and handled when implementing the prediction model. |
| 27b | Usability of the model in the context of current care | D | Specify whether users will be required to interact in the handling of the input data or use of the model, and what level of expertise is required of users. |
| 27c | Usability of the model in the context of current care | D;E | Discuss any next steps for future research, with a specific view to applicability and generalizability of the model. |

---

## MedSci supplemental checks — NOT official TRIPOD+AI items

The following are **not** TRIPOD+AI checklist items. They are engineering-reproducibility prompts this
toolkit adds because they matter for an AI/ML model to be rebuildable, and they extend (do not replace) the
official items noted. Assess them **only as supplements**; never report them as canonical TRIPOD+AI item
numbers, and never let them substitute for an official item.

| Supplemental check | Extends official item | What to look for |
|---|---|---|
| Model architecture | 12c | Network type, depth, layers, activation and loss functions — enough detail to rebuild the model. |
| Training configuration | 12c | Optimizer, learning-rate schedule, batch size, epochs, early-stopping criterion, regularisation. |
| Software and hardware | 18f, 22 | Libraries with version numbers, language, and the compute used (e.g., GPU type for deep learning). |
| Reproducibility | 18e, 18f | Random seeds, the exact data split, and whether code + weights are available to reproduce results. |

Official item **14 (Fairness)**, **18f (Code sharing)**, **22 (Model specification / API)**, and
**7 (Data preparation)** already cover fairness, code/model availability, and preprocessing — assess those
under their official items, not as extras here.

## Notes for assessors

- **Footnotes carried from the official checklist.** Item 12c is assessed **separately for every
  model-building approach** used, not once for the study. Item 18f (code sharing) refers to the
  *analysis* code — data cleaning, feature engineering, model building, evaluation — while item 22
  refers to the code needed to *implement* the model for a new individual; a study can satisfy one
  and not the other.

- **Applicability.** Mark items labelled `D` (development-only) or `E` (evaluation-only) as N/A when they do
  not apply to the study design; `D;E` items apply to both.
- **Open science (18a–18f) and PPI (19) are official, mandatory items** — a compliance report that skips them
  is incomplete. Data sharing (18e) and code sharing (18f) are where an AI/ML study's reproducibility is
  assessed.
- **Performance (23a) with confidence intervals is required**, including for key subgroups; heterogeneity
  across clusters (23b) when examined. A study reporting a point estimate without a CI is PARTIAL on 23a.
- **Fairness is an official item (14)** and is reported across demographic groups in 20b and 23a — not an
  optional AI add-on.
- Do **not** apply TRIPOD 2015 alongside this checklist; TRIPOD+AI 2024 supersedes it for both regression and
  AI/ML prediction models.
- For a study that is both a diagnostic-accuracy study and a prediction-model study, cross-reference STARD.

# TRIPOD-LLM Checklist

**Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis -- Large Language Models**
Version: TRIPOD-LLM 2025 (living guideline)
Source: https://www.tripod-statement.org · interactive checklist: https://tripod-llm.vercel.app
Reference: Gallifant J, ..., Bitterman DS. Nat Med 2025;31(1):60-69. doi:10.1038/s41591-024-03425-5

> **Educational summary, authored in our own words.** This file paraphrases the *intent* of each
> TRIPOD-LLM item to drive an item-by-item audit; it does **not** reproduce the guideline's verbatim
> wording. The published article is subscription-access. For a submission-ready checklist, complete
> the official instrument at the source above and cite Gallifant et al. 2025.
>
> **Item numbers below follow the official TRIPOD-LLM scheme: 19 main items (1–19) with letter
> subitems (≈50 subitems).** 14 main items / 32 subitems apply to every LLM study; 5 main items /
> 18 subitems are **task- or design-specific** (marked *task-specific*) and are N/A when that
> component is absent — justify the N/A.

Licence: The published *Nature Medicine* article returns no Creative Commons licence via Crossref (DOI 10.1038/s41591-024-03425-5); the author-accepted manuscript is CC BY 4.0 through institutional rights retention.
Verification: all 49 sub-items were compared, by number and order, against Table 2 of the published
guideline (PubMed Central record PMC12104976). 49/49 are now present. **This file previously
collapsed item 16's four sub-items into a single row**, which is the shape that lets an assessor
tick one box while three requirements go unchecked; they are expanded below. Wording stays
paraphrased — Crossref returns only a Springer Nature text-and-data-mining licence for this
article.

## Naming and scope (read first)

- TRIPOD-LLM is an **extension** of the **TRIPOD** family (TRIPOD 2015 → TRIPOD+AI 2024 →
  TRIPOD-LLM 2025). In Methods, name **both** the base instrument and the extension and cite each
  (manuscript-style-classical §14): "reported per TRIPOD (Collins et al. 2015) and its large-language-model
  extension TRIPOD-LLM (Gallifant et al. 2025)."
- **Applies to** studies that develop, fine-tune, prompt, or evaluate **large language models** for a
  biomedical/clinical task (classification, extraction, summarization, generation, question
  answering, etc.) — not only to risk-prediction models.
- **Pairs with, does not replace, MI-CLEAR-LLM** (the 6-item LLM-accuracy supplement). Apply
  MI-CLEAR-LLM alongside when LLM accuracy is an outcome.

## Checklist Items (official numbering 1–19)

Status each PRESENT / PARTIAL / MISSING / N/A.

### Title and Abstract

| # | Item | Description (intent) |
|---|------|----------------------|
| 1 | Title | Identify the study as developing, fine-tuning, prompting, and/or evaluating an LLM, and name the clinical task and target setting. |
| 2 | Abstract | Provide a structured summary (objectives, data, LLM and version, task, evaluation approach, human oversight, key results, limitations). Follow the TRIPOD-LLM-for-Abstracts items. |

### Introduction

| # | Item | Description (intent) |
|---|------|----------------------|
| 3a | Background — context | Explain the healthcare problem, the clinical context, and the rationale for using an LLM. |
| 3b | Background — population & use | Describe the target population/users and the intended use of the LLM. |
| 4 | Objectives | State the specific study objectives and the LLM task(s) addressed. |

### Methods — Data (item 5, a–e)

| # | Item | Description (intent) |
|---|------|----------------------|
| 5a | Data sources | Describe all data sources and input data types (clinical text, notes, structured fields, image-to-text). |
| 5b | Data description / distribution | Describe the dataset and how data were partitioned (train / tune / validation / held-out test, internal vs external), and steps to prevent train–test contamination and leakage of evaluation data into pretraining/prompts. |
| 5c | Study dates | Specify key dates (data accrual start/end, and the model knowledge-cutoff relative to the data). |
| 5d | Preprocessing | Describe text preprocessing, de-identification, and any filtering. |
| 5e | Missing / inadequate / imbalanced data | Describe handling of missing, truncated, out-of-context, or class-imbalanced inputs. |

### Methods — Analytical / LLM methods (item 6, a–e)

| # | Item | Description (intent) |
|---|------|----------------------|
| 6a | LLM identity and version | Name the model, exact version/snapshot or weights, provider/access route (API vs local), and date of access — versions drift, so this is essential for reproducibility. |
| 6b | Development / adaptation | Describe how the LLM was developed or adapted (zero/few-shot prompting, retrieval augmentation, fine-tuning, instruction-tuning) in enough detail to reproduce. |
| 6c | Text generation settings | Report decoding/generation parameters (temperature, top-p, max tokens, stop criteria, seed/determinism where available). |
| 6d | Output | Define the expected output format and how free-text outputs were mapped to the study endpoint. |
| 6e | Output handling / classification | Describe parsing, constraint enforcement, and any human or rule-based post-processing/classification of outputs. |

### Methods — LLM output evaluation (item 7, a–e)

| # | Item | Description (intent) |
|---|------|----------------------|
| 7a | Quality / performance metrics | Specify all performance/quality metrics, including task-specific measures. |
| 7b | Downstream / clinical relevance | Describe how the metrics relate to downstream clinical relevance. |
| 7c | Outcome definition | Define the outcome / reference standard and who set it. |
| 7d | Subjective / human assessment | Describe any human rating: rubric, anchors, number and expertise of raters, blinding, and inter-rater agreement. |
| 7e | Comparisons | Describe comparators (clinicians, prior models, guidelines) and ensure same-data, same-task comparison. |

### Methods — Annotation (item 8, a–c)

| # | Item | Description (intent) |
|---|------|----------------------|
| 8a | Labeling process | Describe the labeling/annotation process and adjudication. |
| 8b | Annotator count | Report the number of annotators. |
| 8c | Annotator background | Report annotator expertise/background. |

### Methods — task/design-specific components (N/A if absent; justify)

| # | Item | Description (intent) |
|---|------|----------------------|
| 9a | Prompting — design *(task-specific)* | If prompting was used: describe prompt design/templates and the prompt-selection procedure. |
| 9b | Prompting — data *(task-specific)* | Describe the data used to develop prompts (kept separate from test data). |
| 10 | Summarization preprocessing *(task-specific)* | If summarization was a task: describe inputs/preprocessing and how summaries were assessed for faithfulness/omission. |
| 11 | Instruction tuning / alignment *(task-specific)* | If instruction-tuning/alignment was applied: describe the instructions and the data and procedure used. |

### Methods — Compute and Ethics

| # | Item | Description (intent) |
|---|------|----------------------|
| 12 | Compute | Report computational resources (hardware, and for fine-tuning, scale/cost relevant to reproducibility and environmental reporting). |
| 13 | Ethical approval | Report IRB/ethics approval (or exemption) and consent/data-governance for the data used. |

### Open Science (item 14, a–f)

| # | Item | Description (intent) |
|---|------|----------------------|
| 14a | Funding | Source of funding and role of funders. |
| 14b | Conflicts of interest | Declare conflicts, including relationships with model providers. |
| 14c | Protocol | State protocol availability. |
| 14d | Registration | State any study registration. |
| 14e | Data availability | State availability of data (and access constraints for clinical text). |
| 14f | Code / prompt availability | State availability of code, prompts, and (where applicable) model weights or access instructions. |

### Patient and Public Involvement

| # | Item | Description (intent) |
|---|------|----------------------|
| 15 | PPI | Describe patient/public involvement in design/conduct, or state that there was none. |

### Results

| # | Item | Description (intent) |
|---|------|----------------------|
| 16a | Participants — data flow *(patient/EHR data)* | Describe the flow of text/EHR/patient data through the study, including the number of documents/questions/participants with and without the outcome/label, and follow-up where applicable. |
| 16b | Participants — characteristics *(patient/EHR data)* | Report characteristics overall and for each data source or setting, and for the development/evaluation splits, including key dates and key characteristics. |
| 16c | Participants — distribution comparison *(evaluations with clinical outcomes)* | Show a comparison of the distribution of important clinical variables that may be associated with the outcome, between development and evaluation data. |
| 16d | Participants — analysis sample sizes *(patient/EHR data)* | Specify the number of participants and outcome events in each analysis — LLM development, hyperparameter tuning, and LLM evaluation. |
| 17 | Performance | Report performance/quality results with appropriate uncertainty, including human-evaluation results and, where relevant, subgroup/fairness performance. |
| 18 | LLM updating | If the model or prompts were updated during the study, report results before and after. |

### Discussion (item 19, a–g)

| # | Item | Description (intent) |
|---|------|----------------------|
| 19a | Interpretation | Interpret results in light of objectives, comparators, and the clinical task. |
| 19b | Limitations | Discuss limitations (data, evaluation, generalizability, version drift, hallucination/safety). |
| 19c | Usability and context | Discuss the deployment context and conditions required for safe use. |
| 19d | Intended use | State the intended use and avoid claims beyond the evidence. |
| 19e | Input-data / poor-quality handling | Discuss handling of poor-quality inputs and observed failure modes. |
| 19f | User expertise / oversight | Discuss the user expertise and human oversight required for safe use. |
| 19g | Future research | Outline implications and future work. |

---

## Notes for Assessors

- The numbering follows the official TRIPOD-LLM scheme (19 main items, ~50 subitems). Use the
  official interactive checklist for the submission form and the separate TRIPOD-LLM-for-Abstracts
  sub-checklist (item 2).
- **Version reporting (item 6a) is non-waivable** for an LLM study: a result tied to an unnamed or
  undated model snapshot is not reproducible. Mark MISSING if the exact version/access date is absent.
- **Leakage/contamination (item 5b)** is the LLM-specific analogue of train–test separation:
  evaluation data must not have entered pretraining, fine-tuning, or prompt-development. Probe explicitly.
- **Human evaluation (item 7d)** that drives a quality claim needs a rubric with anchors, rater
  expertise, blinding, and inter-rater agreement — a bare "two physicians reviewed the outputs" is PARTIAL.
- Task-specific items (Prompting 9, Summarization 10, Instruction-tuning 11, Results-participants 16)
  are **N/A** when the component is absent — record a one-line justification rather than a blank.
- TRIPOD-LLM is a **living guideline**; confirm the current item set at the source before a formal
  submission checklist, and pair with MI-CLEAR-LLM when LLM accuracy is an evaluated outcome.

# Methods

Risk of bias was assessed with PROBAST+AI. Some studies used PROBAST-AI
signalling questions instead.

Diagnostic accuracy reporting followed STARD-AI as adapted per recent guidance.

We applied a 12-AI item subset to score the models.

# Methods

Risk of bias was assessed using PROBAST (Wolff 2019) together with its PROBAST+AI
extension (Collins 2024) [12].

Diagnostic accuracy reporting followed STARD 2015 (Bossuyt 2015) and its STARD-AI
extension (Sounderajah 2025) [13].

## PRISMA flow

A total of 1000 records identified through database searching. After 200 duplicates
removed, 800 records screened. Of these, 600 records excluded at screening, leaving
200 reports sought for retrieval. 10 reports not retrieved. 190 reports retrieved and
190 reports assessed for eligibility. 40 records excluded with reasons. 150 studies
included in the synthesis.

1000 records identified
200 duplicates removed
800 records screened
600 records excluded at screening
200 reports sought for retrieval
10 reports not retrieved
190 reports retrieved
190 reports assessed for eligibility
40 records excluded with reasons
150 studies included

1000 records identified
200 duplicates removed
800 records screened
600 records excluded at screening
200 reports sought for retrieval
10 reports not retrieved
190 reports retrieved
190 reports assessed for eligibility
40 records excluded with reasons
149 studies included