---
name: design-ai-benchmarking
description: >
  Design and validity review for studies that benchmark one or more AI systems against a human-expert
  panel as the reference. Covers the evaluation question and arm definition, decoupled multi-dimensional
  rubrics with anchors, planted calibration probes, reviewer-panel construction, inter-rater reliability
  targets, LLM-as-judge versus human-as-judge adjudication, construct-independence guards, and a
  structured rating-export schema. Use before data collection on an AI-vs-expert evaluation.
triggers: AI benchmarking, AI vs human expert, reader study design, expert panel evaluation, LLM-as-judge, AI evaluation rubric, model benchmark design, human baseline comparison, AI-output rating, evaluation rubric design
tools: Read, Write, Edit, Bash, Grep, Glob
model: inherit
---

# Design-AI-Benchmarking Skill

## Purpose

This skill pressure-tests an AI-vs-human-expert benchmark **before any ratings are collected**, so that
the comparison is fair, the rubric measures distinct constructs, the scale is calibrated, and the
reported reliability is interpretable. It is the AI-evaluation specialization of `/design-study`: where
`/design-study` reviews a study in general, this skill owns the specific machinery of comparing AI
system(s) to a panel of human experts (or to each other) on rated outputs.

Use it when:
- one or more AI systems will be scored against a human-expert reference (reader study, annotation
  panel, AI-output evaluation, model-vs-model bench)
- a rubric and rating protocol must be locked before reviewers begin
- a benchmark feels vulnerable to "the highest score is just the most tautological item" or
  "low agreement, but we cannot tell why" criticism
- a reviewer or editor asks how the evaluation controlled for rater drift, leakage, or judge bias

Do **not** use it for: general study/validity review (use `/design-study`); statistical execution such
as ICC or DeLong (use `/analyze-stats`); reporting-guideline item audits (use `/check-reporting`);
or reviewing an already-written manuscript (use `/peer-review` or `/self-review`).

---

## Communication Rules

- Communicate with the user in their preferred language.
- Use English for statistical, machine-learning, and reporting-guideline terminology.
- Be direct about evaluation-validity risks, but always propose the smallest feasible fix first.
- Never invent reviewer ratings, reference labels, or agreement statistics; those come from collected
  data only.

---

## Standard Output

```text
## AI-Benchmark Design Review
Evaluation question: ...
Arms / systems compared: ...
Reference (human-expert panel): ...
Unit of rating: (item / case / output)

### Rubric (decoupled dimensions)
- dimension -> construct -> anchors (1..k)

### Calibration probes (blinded, randomized)
- positive-control / known-bad / instability / mechanism-contradiction

### Reviewer panel
- n reviewers, metadata captured, per-reviewer randomized order

### Reliability plan
- overall IRR target + control-item IRR (reported separately)

### Judge strategy
- human-as-judge / LLM-as-judge / both + adjudication rule

### Validity risks
1. ...

### Minimal fixes
- ...

### Decision
- Ready to collect / Needs rubric revision / Needs arm or judge redesign
```

---

## Workflow

### Phase 1: Define the evaluation question and arms

Pin down, in writing:
- the exact claim the benchmark must support (e.g., "system A's outputs are perceptually
  indistinguishable from expert outputs", not "system A is deployment-ready")
- every arm/system being compared, and what each arm receives as input (same items, same information
  access, same output format) so no arm has a hidden advantage
- the human-expert reference: who they are, and whether they set ground truth, provide a comparison
  arm, or both
- the unit of rating (item, case, output) and how many units each reviewer sees

**Gate:** Present the reconstructed evaluation question, arms, and reference to the user and confirm
before designing the rubric. A wrong reconstruction misdirects the entire benchmark.

### Phase 2: Design a decoupled multi-dimensional rubric

- **Decouple the axes.** Each rated dimension measures one construct. Keep "is the output valid/correct"
  separate from "is it novel", "is it feasible/measurable", "does it add value over current tools", and
  "would it change action". A candidate can be high-validity yet low-added-value ("real but redundant");
  a single blended score hides this divergence.
- **Anchor every scale point** with a short verbal descriptor; pilot the anchors with at least one
  reviewer before locking.
- **Pre-specify discriminant validity**: hypothesize which dimensions should correlate vs be orthogonal,
  then report the full inter-dimension correlation matrix to confirm the rubric measures distinct
  constructs.
- A worked rubric template lives in `${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md`.

### Phase 3: Insert and randomize calibration probes

Plant a small number of deliberate control items, blinded and randomized across raters (record who
received which via a `probe_arm` flag), to (i) anchor the scale, (ii) measure rater drift/fatigue, and
(iii) audit the rubric and pipeline itself. Four useful flavors:
- **Positive control / "too-good" item** — a known-strong or near-tautological item; tests whether
  raters equate "largest effect" with "best", and whether the construct-independence gate (Phase 7) works.
- **Known-bad negative control** — an engineered defect (fabricated reference, missing key statistic);
  expected to score low.
- **Instability item** — an estimate that reverses or fails to replicate on a holdout; tests
  caveat-handling.
- **Mechanism-contradiction item** — an empirical direction that opposes the proposed mechanism.

Probes are *planted or adjudicated*, never fabricated to fit a hypothesis.

### Phase 4: Construct the reviewer panel

- Recruit reviewers spanning the intended expertise gradient; pre-specify any expertise stratification.
- Capture reviewer metadata (years of experience, prior AI-evaluation experience, subspecialty) for
  descriptive reporting and stratified analysis.
- Randomize item order **per reviewer** (not one global seed) and record the order; plan to analyze
  order and fatigue effects.
- Require each item to be judged standalone; discourage cross-item references in free-text, which signal
  non-independent rating.

**Gate:** Present the panel composition, stratification, and randomization plan for user review before
recruitment is finalized.

### Phase 5: Set inter-rater reliability targets

- Pre-specify the agreement statistic (e.g., ICC for continuous ratings, weighted kappa for ordinal)
  and a target with justification.
- **Report reliability on the planted control items separately** as primary evidence of rubric and
  scale validity. A low overall ICC is interpretable only if raters at least converge on the controls;
  surfacing both numbers prevents "low agreement => bad rubric" or "bad raters" misreads.
- Plan the minimum ratings-per-item needed for a stable agreement estimate (delegate the math to
  `/analyze-stats`).

### Phase 5b: Reader allocation under burden constraints (anchor-and-rotate)

When the item pool is larger than one reader can rate in a session, do **not** force every reader
to rate every item (that caps the total pool at the per-reader limit and discards coverage). Use an
**anchor-and-rotate** (balanced-incomplete-block) layout: all readers rate a shared **anchor set**
(which carries the inter-rater ICC/kappa, alongside the planted controls), and each reader
additionally rates a **rotating unique block**, so total coverage grows independently of the
per-reader cap. The usually-binding constraint is the **number of available expert readers**, not the
item count — solve the reverse problem (`max_pool = anchor + R*(cap-anchor)//m`) to size the must-rate
set to a realistic panel. Pre-specify anchor membership, raters-per-item, and the rotation seed before
rating. Formulas, trade-offs, and a stdlib reference implementation are in
`${CLAUDE_SKILL_DIR}/references/anchor_rotate_reader_allocation.md`.

### Phase 6: Choose the judge strategy and adjudication

- Decide human-as-judge, LLM-as-judge, or both. If an LLM is used as a judge, treat it as one more arm
  whose ratings must themselves be validated against the human panel on the control items.
- Pre-specify the **adjudication rule** for disagreement (e.g., majority, a third senior reviewer,
  consensus discussion) and who adjudicates.
- Blind judges to arm identity wherever feasible; record any unavoidable unblinding.

### Phase 7: Construct-independence and leakage guards

- Exclude any predictor or input that is a definitional component of the outcome (mathematical
  definition), and flag near-tautological composites built from the outcome's defining components — they
  produce an inflated, near-circular result and belong as labeled probes, not discoveries.
- Verify no arm sees post-decision or outcome-derived information the others do not.
- Confirm the reference labels were not derived from the same model output being evaluated.

### Phase 8: Lock a structured export schema

Define the machine-readable rating record up front: per-item ratings across every rubric dimension,
free-text justifications, follow-up flags, the `probe_arm` flag, reviewer id and metadata, item order,
and timing. A synthetic schema lives in `${CLAUDE_SKILL_DIR}/references/benchmark_export_schema.json`.

**Gate:** Present the final rubric, probe set, panel plan, judge strategy, and export schema together;
collect explicit user approval before any rating begins. Locking these before data collection is the
whole point — changes afterward compromise the comparison.

---

## Handoff Rules

- route to `/analyze-stats` for ICC / weighted kappa / DeLong, agreement sample size, and effect-size
  real-world translation of the benchmark results
- route to `/check-reporting` for STARD-AI, CLAIM, or TRIPOD+AI item-level reporting once the design is locked
- route to `/design-study` when the broader study around the benchmark (cohort logic, analysis unit,
  comparator) also needs review
- route to `/peer-review` or `/self-review` only after ratings exist and a manuscript is being assessed

---

## What This Skill Does NOT Do

- It does not compute agreement statistics or run analyses directly (that is `/analyze-stats`).
- It does not collect or fabricate ratings, reference labels, or probe outcomes.
- It does not draft manuscript prose or run a reporting-guideline audit.
- It does not replace a full peer review of a finished manuscript.

## Anti-Hallucination

- **Never fabricate references.** All citations must be verified via `/search-lit` with a confirmed DOI
  or PMID. Mark unverified references as `[UNVERIFIED - NEEDS MANUAL CHECK]`.
- **Never invent reviewer ratings, agreement statistics, reference labels, or probe outcomes** — these
  come from collected data only. A reported ICC, kappa, or score with no underlying rating record is the
  failure mode this skill exists to prevent.
- **Never invent clinical definitions, diagnostic criteria, or guideline recommendations.** If uncertain,
  flag with `[VERIFY]` and ask the user.
- If a reporting-guideline item, journal policy, or evaluation standard is uncertain, state the
  uncertainty rather than guessing.

## Reference Files

- `${CLAUDE_SKILL_DIR}/references/elicitation_rubric_template.md` -- a synthetic, decoupled
  multi-dimension rating rubric with anchors and a planted-probe column.
- `${CLAUDE_SKILL_DIR}/references/benchmark_export_schema.json` -- a synthetic JSON schema for the
  per-item rating export (ratings, justifications, probe_arm, reviewer metadata, order, timing).
- `${CLAUDE_SKILL_DIR}/references/anchor_rotate_reader_allocation.md` -- anchor-and-rotate
  (balanced-incomplete-block) reader allocation: formulas, the reverse "max pool for R readers"
  ceiling, trade-offs, and a stdlib reference implementation (Phase 5b).

schema_version: 2
name: design-ai-benchmarking
layer: D
owner_domain: study_design
maturity: official

when_to_use: "Design or review a study that benchmarks one or more AI systems against a human-expert panel: arm definition, decoupled rubric, calibration probes, reviewer panel, IRR targets, judge adjudication, and a structured export schema, before ratings are collected."
when_NOT_to_use: "General study/validity review (use design-study); statistical execution such as ICC/DeLong (use analyze-stats); reporting-guideline item audits (use check-reporting); reviewing a finished manuscript (use peer-review or self-review)."

inputs:
  - "evaluation question and the AI system(s) / arms to compare"
  - "candidate outputs or items to be rated"
  - "available human-expert reviewers and their metadata"
outputs:
  - "benchmark design and validity review (decision notes)"
  - "decoupled rating rubric with anchors and planted calibration probes"
  - "structured rating-export schema (JSON)"
side_effects:
  - writes_decision_notes
downstream_consumers:
  - analyze-stats
  - check-reporting
  - design-study
forbidden_actions:
  - fabricate_reviewer_ratings_or_reference_labels
  - approve_benchmark_with_unblinded_or_leaky_rubric
  - report_irr_without_separate_control_item_reliability

# v2.1 quality card
purpose: "Surface design and validity risks specific to AI-vs-human-expert benchmarks (rubric coupling, scale calibration, rater independence, judge choice) before data collection begins."
safety_boundaries:
  - "Advisory only: writes decision notes plus rubric/schema artifacts, never ratings or analysis results."
  - "Calibration probes and reference labels are planted or adjudicated, never fabricated to fit a hypothesis."
known_limitations:
  - "A design review reduces but cannot eliminate evaluation bias; it does not guarantee a valid benchmark."
  - "No standalone demo; rubric and panel decisions require domain-expert judgement."
validation_commands:
  - "carry the rubric and export schema into analyze-stats for ICC/agreement, then check-reporting (STARD-AI / CLAIM / TRIPOD+AI)"
evidence_surface: manual_workflow