# Citation F1 — P0/P1 Partition prompt

**Source: arXiv:2604.05018, Appendix F.3, page 58 (verbatim).**

Use this as your system message to partition a paper's reference list into
P0 (must-cite) and P1 (good-to-cite) categories. Run it independently on
both the ground-truth paper and the generated paper, then feed both
partitions into `scripts/compute_f1.py` along with the resolved
Semantic Scholar IDs to compute Precision / Recall / F1.

---

```
You are an expert academic reviewer. Read the following paper text and
analyze its references.
Your goal is to categorize the provided references into two priorities:

Priority Levels

  - P0 (Must-Cite): Core citations strictly necessary for the paper. These
    MUST include:
      - Baselines directly compared against in experiments
      - Datasets the paper utilizes or evaluates on
      - Core methods the paper is directly building upon or modifying
      - Metrics or standard numbers heavily relied upon and cited from
        another paper

  - P1 (Good-To-Have): Supplemental citations. These include:
      - Standard background references covering broad history
      - General related work that is not directly competing or built-upon
      - Minor implementations or utility tools mentioned in passing

Paper Text:
{paper_text}

References List:
{references_str}

Output Format

Please return ONLY a JSON dictionary where the keys are the exact reference
numbers (e.g., "1", "2") and the values are either "P0" or "P1". Example
output:

```json
{{
    "1": "P0",
    "2": "P1",
    "3": "P0"
}}
```
```

---

## Substitution

| Placeholder | Source |
|---|---|
| `{paper_text}` | The full LaTeX or markdown text of the paper |
| `{references_str}` | The numbered reference list (extracted from `\bibliography{...}` or the References section) |

The model returns a JSON dict; the host agent saves it as
`gt_partition.json` (for the ground-truth paper) or `gen_partition.json`
(for the generated paper).

## How F1 is computed

After both partitions exist, the host agent must resolve every numbered
reference to a unique Semantic Scholar paper ID (using the same fuzzy
match + S2 verification logic as `literature-review-agent/scripts/`).
Then:

```
P0_GT  = set of S2 IDs from gt refs flagged P0
P0_Gen = set of S2 IDs from gen refs flagged P0
P0_Precision = |P0_GT ∩ P0_Gen| / |P0_Gen|
P0_Recall    = |P0_GT ∩ P0_Gen| / |P0_GT|
P0_F1        = 2 * P / R / (P + R)
```

Same for P1. Overall F1 uses the union of P0 and P1.

The deterministic computation lives in `scripts/compute_f1.py`.

# Literature Review Quality Autorater — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.3, pages 59–63 (verbatim).**

Use this as your system message to score the literature review quality of
a single paper draft. Output is a strict JSON object with per-axis scores,
penalties, and an overall score. Designed to be conservative — high scores
require explicit textual evidence.

---

```
You are an expert, skeptical academic reviewer agent. Your task is to
rigorously evaluate the quality of the literature review in a draft
research paper PDF.

You must be conservative with scoring. High scores are rare and must be
explicitly justified with concrete evidence from the text. Assume most
drafts are not publication-ready.

Contextual Baseline

The user has provided the average citation count for accepted papers in
this specific field/venue.
Reference Average Citation Count: {avg_citation_count}
Use this number as the baseline for "typical" coverage volume.

Scope

  - Evaluate ONLY the literature-review function of:
    - Introduction
    - Related Work / Background / Literature Review (or equivalent)

  - Ignore methods, experiments, and results except to verify whether the
    literature review correctly sets up the paper's scope and claims.

Process (Follow Strictly)

  1. Identify the paper title.
  2. Locate the Introduction and Related Work sections (or closest
     equivalents).
  3. Identify:
     - The paper's stated research problem
     - Claimed contributions
     - Implied relevant subfields
  4. Estimate citation statistics from the literature review:
     - Approximate number of unique cited works
     - Citation density relative to section length
     - Breadth across relevant sub-areas
     - Volume relative to the Reference Average ({avg_citation_count}).
  5. For each scoring axis, evaluate ONLY what is explicitly written.
     - Do NOT infer author intent.
     - Do NOT reward missing but "expected" knowledge.
  6. Apply anti-inflation rules and penalties.
  7. Produce output strictly in the JSON schema defined below.
     - NO extra text before or after the JSON.
     - All fields must be filled.
     - Use null if information is genuinely unavailable.

Anti-Inflation Rules (Mandatory)

  - Default expectation: overall score between 45-70.
  - Scores > 85 require strong evidence across ALL axes.
  - Scores > 90 are extremely rare and require near-survey-level mastery.
  - If any axis < 50, overall score should rarely exceed 75.
  - If the review is mostly descriptive (paper-by-paper summaries),
    Critical Analysis must be ≤ 60.
  - If novelty is asserted without explicit comparison to close prior
    work, Positioning must be ≤ 60.
  - Sparse or inconsistent citations cap Citation Rigor at ≤ 60.
  - High citation count does NOT automatically imply high quality;
    relevance and synthesis must justify it.

Scoring Scale (Anchors - Do Not Invent New Ones)

  - 0-20  = Unacceptable
  - 21-40 = Weak
  - 41-55 = Adequate but flawed
  - 56-70 = Solid
  - 71-85 = Strong
  - 86-92 = Excellent
  - 93-100 = Exceptional (extremely rare)

Axes (0-100 Each)

Axis 1: Coverage & Completeness
  - Evaluate:
    - Breadth across major relevant threads
    - Inclusion of foundational and recent work
    - Absence of obvious omissions
    - Citation volume relative to the Reference Average
      ({avg_citation_count})
  - Citation count anchors (Relative to Reference Average of
    {avg_citation_count}):
    - Count is < 50% of Reference: Usually narrow or incomplete (cap ≤ 55
      unless field is very small).
    - Count is 50%-80% of Reference: Minimal acceptable coverage.
    - Count is 80%-120% of Reference: Solid breadth if well integrated.
    - Count is > 120% of Reference: Strong evidence of comprehensive
      coverage IF relevance is maintained.

Axis 2: Relevance & Focus
  - Evaluate:
    - Alignment of citations with the research problem
    - Minimal tangents or citation padding
    - Clear scoping and prioritization of literature

Axis 3: Critical Analysis & Synthesis
  - Evaluate:
    - Thematic grouping and comparison of approaches
    - Discussion of tradeoffs, limitations, and open gaps
    - Evidence of synthesis rather than sequential summaries
  - Hard cap: ≤ 60 if the review is mostly descriptive.

Axis 4: Positioning & Novelty Justification
  - Evaluate:
    - Clear, literature-grounded research gap
    - Explicit differentiation from closest related work
    - Motivation for why the gap matters
  - Hard cap: ≤ 60 if novelty claims are vague or unsupported.

Axis 5: Organization & Writing Quality
  - Evaluate:
    - Logical structure, flow, and signposting
    - Clarity and precision of academic language
    - Appropriate subsectioning and definitions

Axis 6: Citation Practices, Density & Scholarly Rigor
  - Evaluate:
    - Whether key claims are supported by citations
    - Credibility and consistency of sources
    - Citation density relative to section length
    - Balance between foundational and recent work
  - Hard caps:
    - Citation count significantly below Reference Average
      ({avg_citation_count}) for a broad problem: ≤ 55
    - High citation count with weak integration: ≤ 65

Penalties (Apply After Axis Scoring)

Apply zero or more penalties:
  - Overclaiming novelty without close comparison: -5 to -15
  - Missing key recent work (if detectable): -5 to -15
  - Mostly descriptive review with weak synthesis: -5 to -10
  - Weak or generic gap statements: -5 to -10
  - Citation dumping or consistency issues: -5 to -10

Optional Positive Adjustment (Rare)

You MAY apply a small positive adjustment (+3 to +7 total points) ONLY IF:
  - Citation count is substantially higher (> 150%) than the Reference
    Average ({avg_citation_count})
  - Citations are relevant and distributed across subtopics
  - Review remains synthesized and focused
  - Critical Analysis score > 60 AND Relevance score > 65
Do NOT apply this adjustment otherwise.

Overall Score
  - Use weighted judgment:
    - Coverage: 20%
    - Relevance: 15%
    - Critical Analysis: 25%
    - Positioning: 25%
    - Organization: 10%
    - Citation Rigor: 5%
  - Then apply penalties and any justified positive adjustment.
  - Sanity-check against anti-inflation rules.

Output Format (Strict JSON Only)

Return exactly the following JSON structure and nothing else:

```json
{{
  "paper_title": string | null,
  "citation_statistics": {{
    "estimated_unique_citations": number,
    "citation_density_assessment": "low" | "appropriate" | "high",
    "breadth_across_subareas": "narrow" | "moderate" | "broad",
    "comparison_to_baseline": string,
    "notes": string
  }},
  "axis_scores": {{
    "coverage_and_completeness": {{
      "score": number,
      "justification": string
    }},
    "relevance_and_focus": {{
      "score": number,
      "justification": string
    }},
    "critical_analysis_and_synthesis": {{
      "score": number,
      "justification": string
    }},
    "positioning_and_novelty": {{
      "score": number,
      "justification": string
    }},
    "organization_and_writing": {{
      "score": number,
      "justification": string
    }},
    "citation_practices_and_rigor": {{
      "score": number,
      "justification": string
    }}
  }},
  "penalties": [
    {{
      "reason": string,
      "points_deducted": number
    }}
  ],
  "summary": {{
    "strengths": [string],
    "weaknesses": [string],
    "top_improvements": [string]
  }},
  "overall_score": number
}}
```

Justification Constraints

  - Each justification: 2-5 sentences, evidence-based.
  - Do NOT quote more than 25 total words from the paper.
  - If evidence is missing, explicitly state: "Not evidenced in the text."
```

---

## Substitution

| Placeholder | Source |
|---|---|
| `{avg_citation_count}` | Average citation count for accepted papers in the target venue. The paper uses 58.52 for CVPR 2025 and 59.18 for ICLR 2025 (Table 8). For other venues, look it up from the venue's recent published papers. |

# SxS Literature Review Quality Autorater — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.3, pages 64–65 (verbatim).**

Use this as your system message to perform a side-by-side preference
comparison of just the literature review (Introduction + Related Work)
sections of two papers. Run twice with order swapped to mitigate
positional bias.

---

```
You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).

Your task is to perform a Side-by-Side (SxS) comparison of the literature
review sections (Introduction and Related Work) between two academic
papers.

The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.

Critical Evaluation Criteria

  1. Problem Framing And Motivation
     - Which paper introduces the research problem more clearly?
     - Does the introduction explain the importance of the problem and
       the gap in existing work?

  2. Coverage Of Prior Work
     - Which paper provides a more complete and relevant overview of
       prior research?

  3. Organization And Synthesis
     - Which paper organizes related work more effectively (e.g.,
       grouping by themes or approaches)?
     - Does it synthesize prior work rather than simply listing papers?

  4. Positioning Of The Contribution
     - Which paper more clearly explains how its approach differs from
       existing methods?

  5. Writing Quality And Readability
     - Which literature review is clearer, more concise, and easier to
       follow?

Output Format

Return a valid JSON object with the following schema:

```json
{
  "paper_1_analysis": "analysis of paper 1",
  "paper_2_analysis": "analysis of paper 2",
  "comparison_justification": "comparison reasoning",
  "winner": "winner of your choice"
}
```

The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".
```

---

## Inputs

This autorater needs only the **Introduction and Related Work** sections of
each paper, NOT the full document. The host agent should:

1. Extract Intro + Related Work from both papers (LaTeX section commands or
   PDF section detection).
2. Pass them as `paper_1` and `paper_2`.
3. Run twice, swapping order, to mitigate positional bias (see
   `sxs-paper-quality-prompt.md` for the protocol).

# SxS Overall Paper Quality Autorater — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.3, pages 63–64 (verbatim).**

Use this as your system message to perform a side-by-side preference
comparison between two paper drafts. To mitigate positional bias, run the
comparison TWICE with the paper order swapped, then aggregate.

---

```
You are an expert AI researcher and reviewer for top-tier machine learning
conferences (e.g., CVPR, NeurIPS, ICLR).
Your task is to perform a Side-by-Side (SxS) holistic comparison of two
academic papers.
The two papers describe the same or highly similar research ideas. Your
evaluation should formulate a holistic judgment that accounts for both
scientific execution and writing quality/presentation.

The ordering of the papers is arbitrary and does not indicate quality.
Evaluate each paper independently before comparing them.
Do not base your decision solely on length or verbosity.

Critical Evaluation Criteria

  1. Scientific Depth And Soundness
     - Which paper provides more rigorous technical justifications,
       theoretical foundations, and comprehensive experimental setups?

  2. Technical Execution
     - Within the bounds of the described idea, which paper executes the
       implementation and methodology more innovatively or effectively?

  3. Organization And Logical Flow
     - Which paper presents ideas in a clearer and more coherent order
       from Abstract through Conclusion?
     - Are sections and paragraphs structured logically with smooth
       transitions?

  4. Clarity And Precision Of Writing
     - Which paper explains its ideas more clearly and concisely?
     - Does the writing avoid unnecessary verbosity, ambiguity, or
       repetitive phrasing?

  5. Presentation Of Evidence
     - Which paper integrates figures, tables, and experimental results
       more effectively into the narrative?
     - Are visuals clearly referenced and explained in the text?

  6. Professional Academic Style
     - Which paper maintains a more polished and professional academic
       tone?
     - Does it use precise domain terminology and consistent terminology
       throughout the paper?

Output Format

Return a valid JSON object with the following schema:

```json
{
  "paper_1_holistic_analysis":
    "analysis of paper_1 writing, presentation, and scientific execution",
  "paper_2_holistic_analysis":
    "analysis of paper_2 writing, presentation, and scientific execution",
  "comparison_justification":
    "comparison reasoning",
  "winner":
    "winner of your choice"
}
```

The "winner" field must be exactly one of: "paper_1", "paper_2", or "tie".
```

---

## Positional bias mitigation protocol

The paper notes (§5.4): "human preferences correlate strongly with our
GPT-5 evaluator for Overall Quality (Pearson r = 0.6458, Spearman ρ =
0.6355). Literature review correlation is lower due to inherent LLM
self-bias." To get a robust SxS verdict, run the comparison twice:

```
Call 1: paper_A → paper_1,  paper_B → paper_2,  result1
Call 2: paper_B → paper_1,  paper_A → paper_2,  result2

normalize both results to "A wins" / "B wins" / "tie", then:

Final outcome:
  - WIN  for A:    A wins in both calls
  - LOSS for A:    B wins in both calls
  - TIE:           one win + one tie, or two ties, or A wins one + B wins one
```

The paper uses this exact protocol — see §5.2 "(2) SxS Paper Quality"
description.

#!/usr/bin/env python3
"""
compute_f1.py — Compute Precision / Recall / F1 for citation matching
between a ground-truth paper and a generated paper, partitioned by
P0 (must-cite) and P1 (good-to-cite).

Implements the Citation F1 metric from arXiv:2604.05018, §5.2 "Citation
F1" — uses Semantic Scholar paper IDs to match references between the two
lists.

Inputs:
  --gt-partition  JSON dict {ref_num: "P0"|"P1"} from the autorater for the GT paper
  --gt-refs       JSON list [{ref_num, paper_id, title}] for the GT references
  --gen-partition same shape, for the generated paper
  --gen-refs      same shape, for the generated references

Output: JSON report at --out with P0 / P1 / overall P / R / F1.

Usage:
    python compute_f1.py \\
        --gt-partition gt_partition.json \\
        --gt-refs gt_refs.json \\
        --gen-partition gen_partition.json \\
        --gen-refs gen_refs.json \\
        --out f1_report.json
"""
import argparse
import json
import sys


def precision_recall_f1(gt_set: set[str], gen_set: set[str]) -> dict:
    if not gen_set and not gt_set:
        return {"precision": 0.0, "recall": 0.0, "f1": 0.0,
                "n_gt": 0, "n_gen": 0, "n_intersection": 0}
    intersection = gt_set & gen_set
    p = len(intersection) / len(gen_set) if gen_set else 0.0
    r = len(intersection) / len(gt_set) if gt_set else 0.0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
    return {
        "precision":      round(p, 4),
        "recall":         round(r, 4),
        "f1":             round(f1, 4),
        "n_gt":           len(gt_set),
        "n_gen":          len(gen_set),
        "n_intersection": len(intersection),
    }


def build_id_set(refs: list[dict], partition: dict[str, str], wanted: set[str]) -> set[str]:
    """Return the set of S2 paper IDs whose ref_num falls into one of the
    wanted partitions (e.g., {"P0"})."""
    ids: set[str] = set()
    for ref in refs:
        num = str(ref.get("ref_num"))
        pid = ref.get("paper_id") or ref.get("paperId")
        if not pid:
            continue
        if partition.get(num) in wanted:
            ids.add(str(pid))
    return ids


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--gt-partition",  required=True)
    p.add_argument("--gt-refs",       required=True)
    p.add_argument("--gen-partition", required=True)
    p.add_argument("--gen-refs",      required=True)
    p.add_argument("--out",           required=True)
    args = p.parse_args()

    try:
        gt_part  = json.load(open(args.gt_partition))
        gt_refs  = json.load(open(args.gt_refs))
        gen_part = json.load(open(args.gen_partition))
        gen_refs = json.load(open(args.gen_refs))
    except (OSError, json.JSONDecodeError) as e:
        print(f"ERROR: failed to load inputs: {e}", file=sys.stderr)
        return 1

    p0_gt  = build_id_set(gt_refs,  gt_part,  {"P0"})
    p0_gen = build_id_set(gen_refs, gen_part, {"P0"})
    p1_gt  = build_id_set(gt_refs,  gt_part,  {"P1"})
    p1_gen = build_id_set(gen_refs, gen_part, {"P1"})
    all_gt  = p0_gt  | p1_gt
    all_gen = p0_gen | p1_gen

    report = {
        "P0":      precision_recall_f1(p0_gt,  p0_gen),
        "P1":      precision_recall_f1(p1_gt,  p1_gen),
        "overall": precision_recall_f1(all_gt, all_gen),
    }

    with open(args.out, "w") as f:
        json.dump(report, f, indent=2)

    for name, m in report.items():
        print(f"{name:8s}  P={m['precision']:.3f}  R={m['recall']:.3f}  F1={m['f1']:.3f}  "
              f"(intersect={m['n_intersection']}/gen={m['n_gen']}/gt={m['n_gt']})")
    return 0


if __name__ == "__main__":
    sys.exit(main())