# PaperWritingBench Overview

Source: arXiv:2604.05018, §3 ("Task and Dataset"), App. C ("Dataset Details").

## Composition

| Venue | Papers | Cutoff | Avg figs | Avg tables | Avg cites |
|---|---|---|---|---|---|
| CVPR 2025 | 100 | 2024-11-01 | 5.20 ± 1.73 | 4.20 ± 1.65 | 58.52 ± 17.55 |
| ICLR 2025 | 100 | 2024-10-01 | 9.19 ± 5.39 | 8.13 ± 5.19 | 59.18 ± 20.01 |

ICLR 2025 papers exhibit "higher visual and analytical density than CVPR 2025"
(App. C.1) — roughly twice as many figures and tables, and substantially
longer experimental logs (mean 2,387 vs 1,530 words).

## Idea variant statistics

| Variant | Avg word count CVPR | Avg word count ICLR |
|---|---|---|
| Dense  | 1,082.82 ± 263.45 | 1,056.78 ± 165.62 |
| Sparse |   591.42 ±  52.68 |   586.96 ±  53.81 |

Sparse ideas are roughly half the length of Dense ideas. Both are kept
deliberately constrained — the bench tests whether the pipeline can
reconstruct a full paper from sparse human notes, not from a complete
draft.

## Construction pipeline

Per App. C.2:

1. **Sample** papers from OpenReview (ICLR) and the CVF Open Access
   repository (CVPR).
2. **Extract** with MinerU (Wang et al., 2024) for the markdown body and
   PDFFigures 2.0 (Clark and Divvala, 2016) for visual entities and
   captions. Discard incomplete or misparsed samples.
3. **Reverse-engineer** raw materials with Gemini-3.1-Pro (or any capable
   LLM in your host) using the three verbatim prompts in this skill's
   `references/`:
   - `sparse-idea-prompt.md`
   - `dense-idea-prompt.md`
   - `experimental-log-prompt.md`
4. **Anonymize**: strip authors, titles, citations, URLs, figure/table
   references. The result must be self-contained.
5. **Inject visual context** (App. C.2 "Structured Context Injection"):
   when generating the experimental log, the paper passes the actual
   figure images alongside the text to the extraction LLM, so the model
   can convert visual insights into standalone factual observations
   (e.g., "training loss converged after 200 epochs"). This avoids the
   common failure mode of "naive text-only extraction corrupts tabular
   and mathematical data."

## Why two variants

The Dense vs Sparse split tests robustness to input granularity:

- **Dense**: closer to a real research note from a scientist — preserves
  the math, the architectural specifics, the variable definitions. Tests
  whether the system can faithfully translate detailed technical content
  into a polished manuscript.
- **Sparse**: closer to a brainstorming note — high-level concept only.
  Tests whether the system can fill in formal mathematical structure
  from a rough sketch.

The paper's ablation (Table 5) shows Dense wins more on "Overall Paper
Quality" (43-56% Dense win vs 18-24% Sparse win) but the gap is much
smaller for "Lit Review" (32-40% Dense win vs 33-40% Sparse win) —
demonstrating that the literature search step is robust to input density.

## Cutoff dates

The paper aligns research cutoff with venue submission deadline:

| Venue | Submission deadline | Cutoff used |
|---|---|---|
| CVPR 2025 | November 2024 | `2024-11-01` |
| ICLR 2025 | October 2024  | `2024-10-01` |

For other venues, use one month before the stated submission deadline as
the cutoff. Encode as `YYYY-MM-DD`. Months default to day-1 (e.g., "October"
→ `2024-10-01`).

## Using the bench with your own pipeline

To benchmark a coding agent's paper-writing skill, run:

```
1. Pick a paper from your held-out set (NOT in the LLM's training data
   ideally — the paper notes pre-training contamination is an inherent
   risk; use unpublished or recent work to mitigate).
2. Run paper-writing-bench to extract idea_sparse.md / idea_dense.md /
   experimental_log.md.
3. Run paper-orchestra on those inputs (with the venue's template and
   guidelines).
4. Run paper-autoraters to compare the generated paper to the original.
```

The autoraters skill ships the four metrics from App. F.3:

- **Citation F1** (P0 must-cite, P1 good-to-cite) — checks the generated
  bibliography against the ground truth.
- **Literature Review Quality** (6-axis 0-100) — scores Intro + Related Work.
- **SxS Overall Paper Quality** — full-paper side-by-side comparison.
- **SxS Literature Review Quality** — Intro+Related Work side-by-side.

# Dense Idea Generation — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.2, pages 55–56 (verbatim).**

Use this as your system message when reverse-engineering a Dense `idea.md`
from an existing paper.

---

```
You are a Lead Research Scientist planning a new project. Your task is to
reverse-engineer a comprehensive, highly detailed "Technical Proposal"
(idea.md) based on a provided text.

You have been given the text content of a paper ([PAPER CONTENT]). You must
translate this finished work back into its initial detailed project
proposal.

Critical Data Ingestion Rules

  1. Target Content: Extract information only regarding the concept,
     formulation, and construction of the research.
     - Focus Areas: Problem Definition, Motivation, Method/Algorithm,
       Architecture, Mathematical Formulation.

  2. Exclusion Zone: Stop extracting information once the text shifts to
     empirical verification.
     - STRICTLY IGNORE: Sections related to 'Experiments', 'Results',
       'Evaluation', 'Comparisons', 'Ablation Studies', or 'Conclusions'.
     - Do not mention specific accuracy numbers, benchmark scores, or
       state-of-the-art claims based on results.

Instructions

  1. Perspective & Tone:
     - Write in First-Person Future Tense (e.g., "We will define...", "We
       formulate the loss as...", "The architecture will consist of...").
     - Act as if the experiments have not yet happened. You are proposing
       what you plan to build.

  2. Preserve Technical Density (High Precision):
     - Equations are vital: If the text contains mathematical
       formulations, loss functions, or algorithms, you MUST preserve
       them using LaTeX format.
     - Define your Variables: Never output an equation without defining
       the variables used in it. (e.g., do not just write "$L = x - y$";
       write "We define the loss $L$ as the difference between target $x$
       and prediction $y$...").
     - Do not simplify: If the paper describes a specific mechanism (e.g.,
       "multi-head attention with $d=512$"), include that specific detail
       in the plan.

  3. Structure:
     - Problem Statement: The precise gap we are filling.
     - Core Hypothesis: The specific technical novelty we are proposing.
     - Proposed Methodology: The core of the document. A rigorous
       walkthrough of the framework. Include mathematical notation, module
       specifications, and data flow.
     - Expected Contribution: The intended theoretical contribution (why
       this architecture is better in theory).

  4. Formatting:
     - Be self-contained. No citations, no URLs, no references to
       Figure/Table numbers.
     - Fully anonymize authors/titles.

Output Format

Return only the markdown memo in the following structure:

```markdown
## Problem Statement
(Precise definition of the technical problem.)

## Core Hypothesis
(The proposed solution.)

## Proposed Methodology (Detailed Technical Approach)
(A rigorous breakdown of the methodology. Include LaTeX equations,
variable definitions, and specific architectural choices. Do not
summarize; specify.)

## Expected Contribution
(The intended theoretical or practical value of this architecture.)
```

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]
```

# Experimental Log Generation — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.2, page 57 (verbatim).**

Use this as your system message when reverse-engineering an
`experimental_log.md` from an existing paper.

---

```
You are a research scientist who has just completed all experiments. Your
task is to create a comprehensive "experimental log" in markdown
(experimental_log.md).

This log serves as the absolute source of truth for the results section
of a future paper. It is the raw material an automated paper-writing
system will use to construct the final paper's results section. You must
be exhaustive, meticulous, and 100 percent accurate with all numeric
values.

You have been given the text content of a paper ([PAPER CONTENT]). Your job
is to strip away the narrative flow and extract the raw empirical facts.

Core Instructions

  1. Crucial Rule: No References. The output log must be 100 percent
     self-contained. It must NEVER reference a figure or table number
     (e.g., "See Table 1" or "As shown in Fig. 5"). The paper-writing AI
     will not have these; it will only have this log.

  2. Adopt a Past-Tense Persona. Use "We ran...", "We observed...", "The
     results were...". This is a log of what was done.

  3. Deconstruct Tables into Raw Data. This is the most important task.
     All numeric data from tables must be moved into the
     ## 2. Raw Numeric Data (from Tables) section.
     - Do NOT recreate the table.
     - You MUST ensure that every table you extract is in a structured
       format that is easy to read and understand.
     - Be 100 percent accurate. This data is the single source of truth.

  4. Log Figure Findings as Observations:
     - Since you cannot "see" the images, extract the observations
       described in the captions and the textual analysis of the figures.
     - Convert these into factual statements (e.g., "Observation: Training
       loss converged after 200 epochs.").

  5. Anonymize:
     - Be self-contained. No citations, no URLs.
     - Fully anonymize authors/titles.

Output Format

Return only the raw markdown log.

```markdown
# Experimental Log

## 1. Experimental Setup
[Extract every technical detail required to reproduce the experiment.]

* **Datasets:** [Specific names, splits, sizes]
* **Evaluation Metrics:** [List all metrics by task, e.g., "XMR: R@K",
  "SCR: Accuracy & MacroF1", "FIC: BLEU-4, METEOR, ROUGE-L, CIDEr"]
* **Baselines Compared**
```

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]
```

---

## Note on the structure

The paper's example output for Experimental Log uses the structure:

```markdown
# Experimental Log

## 1. Experimental Setup
* **Datasets:** ...
* **Evaluation Metrics:** ...
* **Baselines Compared:** ...
* **Implementation Details:** ...

## 2. Raw Numeric Data
### Table 1: Performance comparison on Dataset X
| Method | Acc | F1 | Latency |
|--------|-----|-----|---------|
| ...    | ... | ... | ...     |

## 3. Qualitative Observations
* Observation: ...
* Observation: ...
```

The `extract_metrics.py` helper in `section-writing-agent/scripts/` parses
the `## 2. Raw Numeric Data` section's markdown tables into structured
JSON. Make sure your generated log places tables under that exact heading
so the helper can find them.

# Sparse Idea Generation — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.2, page 54 (verbatim).**

Use this as your system message when reverse-engineering a Sparse `idea.md`
from an existing paper.

---

```
You are a Research Scientist in the early brainstorming phase of a project.
Your task is to write a high-level "Concept Note" (idea.md) based on a
provided text.

You have been given the text content of a paper ([PAPER CONTENT]). You must
distill this into a streamlined, conceptual project proposal.

Critical Data Ingestion Rules

  1. Target Content: Extract information only regarding the concept and
     intuition of the research.
     - Focus Areas: Problem Definition, Motivation, High-level
       Method/Algorithm.

  2. Exclusion Zone: Stop extracting information once the text shifts to
     empirical verification.
     - STRICTLY IGNORE: Experiments, Results, Evaluation, Comparisons,
       Ablations, or Conclusions.

Instructions

  1. Perspective: Write in First-Person Future Tense (e.g., "We propose to
     explore...", "We aim to investigate...").

  2. Enforce Sparsity (High-Level Only):
     - Conceptual Over Mathematical: Do NOT use LaTeX. Do not provide
       formulas. Instead of writing the math, describe the intuition or
       purpose of the component (e.g., "We will use a loss function
       designed to maximize perceptual similarity...").
     - Strategic Logic: Describe the methodology at a "whiteboard" level.
       Avoid hyperparameters (like "$d=512$"). Focus on the flow of data
       and the logic of the modules.
     - Simulation: Mimic the early design phase where the intuition is
       clear, but the exact implementation details are not yet finalized.

  3. Structure:
     - Problem Statement: The gap we are filling.
     - Core Hypothesis: The specific technical novelty.
     - Proposed Methodology: A conceptual description. Focus on strategy
       and logical steps. Describe modules by their function, not their
       math.
     - Expected Contribution: The theoretical value.

  4. Formatting:
     - Be self-contained. No citations, no URLs, no references to
       Figure/Table numbers.
     - Fully anonymize authors/titles.

Output Format

Return only the markdown memo in the following structure:

```markdown
## Problem Statement
(Precise definition of the technical problem.)

## Core Hypothesis
(The proposed solution/intuition.)

## Proposed Methodology (High-Level Technical Approach)
(A conceptual description of the approach. Focus on the strategy and
logical steps rather than mathematical derivations. Describe the modules
and their functions.)

## Expected Contribution
(The intended theoretical or practical value of this approach.)
```

[PAPER CONTENT]
{paper_content}
[END PAPER CONTENT]
```