Faithful implementation of the four LLM-as-judge autoraters used in\nPaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).
\nThese are the metrics the paper uses to demonstrate that PaperOrchestra\nbeats single-agent and AI-Scientist-v2 baselines. Use them to:
\n| Autorater | \nWhat it does | \nInputs | \nOutput | \n
|---|---|---|---|
| Citation F1 — P0/P1 partition | \nPartitions reference list into P0 (must-cite) and P1 (good-to-cite) given the paper text | \none paper text + its references list | \nJSON {ref_num: \"P0\"\\|\"P1\"} | \n
| Literature Review Quality | \n6-axis 0-100 score for Intro+Related Work, with anti-inflation hard caps | \none paper PDF/text + reference avg citation count | \nJSON with axis_scores, penalties, summary, overall_score | \n
| SxS Overall Paper Quality | \nHolistic side-by-side preference judgment | \ntwo papers (PDF or text) | \nJSON with winner ∈ {paper_1, paper_2, tie} | \n
| SxS Literature Review Quality | \nSide-by-side preference, Intro+Related Work only | \ntwo papers | \nJSON with winner ∈ {paper_1, paper_2, tie} | \n
The paper uses Gemini-3.1-Pro and GPT-5 as judges, set to temperature 0.0\n(Gemini) or default 1.0 (GPT-5, which doesn't allow temperature\nadjustment). Use whatever your host LLM is.
\nThis is a two-step procedure:
\nFor both the ground-truth paper AND the generated paper, run the LLM with\nreferences/citation-f1-prompt.md:
inputs:\n paper_text: full paper LaTeX or markdown\n references_str: numbered reference list (e.g., \"1. Vaswani et al. (2017)\n Attention Is All You Need. NeurIPS. 2. He et al. (2016)\n Deep Residual Learning for Image Recognition. CVPR. ...\")\n\noutput: JSON {\"1\": \"P0\", \"2\": \"P1\", \"3\": \"P0\", ...}\nSave both partitions:
\nbench/<paper_id>/gt_partition.jsonbench/<paper_id>/gen_partition.jsonThe paper uses Semantic Scholar paper IDs to match references between the\ntwo lists. The compute_f1.py script does this deterministically given\ntwo input lists:
python skills/paper-autoraters/scripts/compute_f1.py \\\n --gt-partition gt_partition.json \\\n --gt-refs gt_refs.json \\\n --gen-partition gen_partition.json \\\n --gen-refs gen_refs.json \\\n --out f1_report.json\nWhere gt_refs.json and gen_refs.json are lists of {ref_num, paper_id, title} produced by your host's S2-resolution pass (the same\nfuzzy match + S2 verification used by literature-review-agent/scripts/).
Output JSON contains P0 / P1 / overall Precision, Recall, F1.
\nLoad references/litreview-quality-prompt.md. Inputs:
avg_citation_count for the venue/field (used as the baseline for\ncitation count anchoring, e.g., 58.52 for CVPR 2025, 59.18 for ICLR 2025\nper the paper)The prompt instructs the model to evaluate ONLY the literature-review\nfunction of the paper (Introduction + Related Work / Background sections).\nIt produces a strict JSON output with per-axis scores and justifications.
\nCritical anti-inflation rules baked into the prompt:
\n| Rule | \nCap | \n
|---|---|
| Default expectation | \noverall 45-70 | \n
| > 85 requires strong evidence on ALL axes | \n— | \n
| > 90 extremely rare (near-survey-level mastery) | \n— | \n
| Any axis < 50 → overall rarely > 75 | \n— | \n
| Mostly descriptive review | \nCritical Analysis ≤ 60 | \n
| Novelty asserted without comparison | \nPositioning ≤ 60 | \n
| Sparse/inconsistent citations | \nCitation Rigor ≤ 60 | \n
| Citation count < 50% of avg | \nCoverage ≤ 55 | \n
| Citation count > 120% of avg | \nCoverage = \"strong\" | \n
Plus penalty table:
\n| Penalty | \nRange | \n
|---|---|
| Overclaiming novelty | \n-5 to -15 | \n
| Missing key recent work | \n-5 to -15 | \n
| Mostly descriptive review | \n-5 to -10 | \n
| Weak gap statements | \n-5 to -10 | \n
| Citation dumping | \n-5 to -10 | \n
Save the output to litreview_quality_score.json. The score JSON is the\nsame shape used by content-refinement-agent/scripts/score_delta.py, so\nyou can re-use the halt-rule logic to compare iterations.
Load references/sxs-paper-quality-prompt.md. Inputs:
paper_1 and paper_2)The prompt produces a JSON with paper_1_holistic_analysis,\npaper_2_holistic_analysis, comparison_justification, and\nwinner ∈ {paper_1, paper_2, tie}.
To mitigate LLM positional bias (the paper notes this in §5.4), run the\ncomparison twice with the order swapped:
\ncall_1: paper_A → paper_1, paper_B → paper_2 → winner1\ncall_2: paper_B → paper_1, paper_A → paper_2 → winner2\nFinal outcome: a win (both calls agree on paper A), tie (one win + one\ntie, or two ties), or loss (both agree on paper B). The paper uses this\nexact ordering protocol.
Load references/sxs-litreview-prompt.md. Same input/output shape as the\nSxS paper quality autorater, but the model is instructed to evaluate\nonly the Introduction and Related Work / Background sections of each\npaper. Same positional-bias mitigation: run twice, swap order.
references/citation-f1-prompt.md — verbatim P0/P1 partition prompt from App. F.3references/litreview-quality-prompt.md — verbatim 6-axis litreview rubric from App. F.3references/sxs-paper-quality-prompt.md — verbatim SxS paper-quality prompt from App. F.3references/sxs-litreview-prompt.md — verbatim SxS litreview prompt from App. F.3scripts/compute_f1.py — Precision / Recall / F1 from two partition JSONs