# Citation Density Rule

Source: arXiv:2604.05018, App. D.3.

## The 90% rule

> ...the system strictly constrains the model to cite only the provided
> verified papers, explicitly mandating that at least 90% of the gathered
> literature pool must be actively integrated and cited when synthesizing
> the Introduction and Related Work sections.

Why: this is the paper's core defense against citation inflation. The
literature review pool is built once via the rigorous discovery →
verification → dedup pipeline. The writing step must then *use* almost all
of it. This prevents the agent from gathering 50 papers and citing only the
3 most famous ones, which would defeat the entire literature search.

## Implementation

After the Lit Review writing call produces `intro_relwork.tex`:

```bash
python scripts/citation_coverage.py \
    --tex workspace/drafts/intro_relwork.tex \
    --pool workspace/citation_pool.json \
    --threshold 0.90
```

The script:

1. Reads `citation_pool.json` and counts `papers[]` (= N).
2. Computes `min_required = floor(0.90 * N)`.
3. Greps `intro_relwork.tex` for all `\cite{KEY}`, `\citep{KEY}`, `\citet{KEY}`,
   `\autocite{KEY}`, `\citeauthor{KEY}`, etc.
4. Counts the **unique** keys actually cited.
5. Reports `cited / N` and exits non-zero if `cited < min_required`.

## What to do on failure

The script prints the missing keys grouped by `discovered_for` cluster:

```
FAIL: 17/22 papers cited (77.3%, need ≥90%)
Uncited papers (5):
  - vaswani2017attention      [discovered_for: intro]       (Attention Is All You Need)
  - he2016deep                [discovered_for: intro]       (Deep Residual Learning ...)
  - liu2024video              [discovered_for: related_work[2.1]]  (Long Video Generation ...)
  - chen2024sparse            [discovered_for: related_work[2.2]]  (Sparse Attention Surveys ...)
  - kim2024transformer        [discovered_for: related_work[2.2]]  (Transformer Scaling Laws ...)
```

The host agent should then re-call the Lit Review writing step with an
appended instruction:

```
The previous draft cited only 17 out of 22 verified papers (77.3%, threshold
is 90%). You MUST integrate the following 5 papers into the appropriate
sections:
  - vaswani2017attention (intro): foundational attention reference
  - he2016deep (intro): foundational ResNet reference
  - liu2024video (related work 2.1): direct competing approach for long video
  - chen2024sparse (related work 2.2): sparse attention survey, group with [...]
  - kim2024transformer (related work 2.2): scaling-laws context

Do not remove any existing citations. Add new ones where contextually
appropriate. Re-emit the full template.tex with both sections updated.
```

After 2-3 re-prompts, if coverage still falls short, the pipeline should
emit a warning and proceed — the paper does not specify a hard halt on this,
only a strong constraint.

# Cross-Index Citation Verification

Supplementary verification layer that runs *after* the Semantic Scholar gate
(Rules 1–4 in `verification-rules.md`) and *before* `bibtex_format.py`. It
re-checks every S2-verified paper against two independent scholarly indices —
**Crossref** and **OpenAlex** — and flags any that cannot be corroborated.

## Why

A single index can return a plausible-looking record for a paper that does not
exist, or attach the wrong metadata to a real title. Triangulating across three
independent indices is the standard practical defense: a genuine paper appears
in all three with consistent metadata, while a fabricated or mis-attributed
record usually does not survive the cross-check. This directly targets the
documented failure mode where AI-assisted writing introduces hallucinated
citations into the bibliography.

S2 verification answers "does a paper with this title plausibly exist?"
Cross-index verification answers "do *other* indices agree it exists, with the
same year and DOI?"

## Scripts

| Script | Role |
|---|---|
| `scripts/crossref_client.py` | Crossref REST API title/DOI lookup, normalized output |
| `scripts/openalex_client.py` | OpenAlex API title/DOI lookup, normalized output |
| `scripts/cross_verify.py` | Orchestrates both, classifies confidence, writes report |

All three are stdlib-only (`urllib`), need **no API key**, and degrade
gracefully if an index is unreachable (the index is disabled for the run and
noted in the report; remaining indices still run).

## Polite pool (recommended, not required)

Crossref and OpenAlex give faster, more reliable service when you identify
yourself by email. Set one shared address:

```bash
export PAPER_ORCHESTRA_MAILTO="you@example.com"
```

or per-service `CROSSREF_MAILTO` / `OPENALEX_MAILTO`. The email is sent only as
a `mailto` query parameter / User-Agent per each service's etiquette docs. The
repo never commits an address.

## Confidence tiers

`cross_verify.py --inplace` writes a `cross_verification` object onto each pool
paper, and a summary report to `workspace/cross_verification_report.json`:

| Tier | Meaning | Host action |
|---|---|---|
| `high` | Corroborated by ≥1 external index, no metadata conflicts | keep |
| `medium` | Corroborated, but publication year disagrees beyond `--year-tolerance` | keep; spot-check the year used in prose |
| `low` | Not found in Crossref **or** OpenAlex | **review** — see false-positive note below |
| `conflict` | A DOI in the pool disagrees with the external index's DOI | **review** — likely wrong record |

Two thresholds keep the tiers honest:

- **Corroboration** uses the lenient `> 70` (`--threshold`, same as the S2
  gate): "an entry like this exists in the index."
- **Conflict downgrades** (`medium`/`conflict`) require a strict `>= 90`
  (`--strong-threshold`) or an exact DOI hit before an external record's year
  or DOI is trusted. This stops a noisy near-title hit — a *different* paper
  with a similar name — from polluting the metadata checks and falsely
  downgrading a correctly-matched paper.

A DOI present in the pool is looked up exactly first; only on a miss does the
script fall back to title search.

## This is a WARN gate, not a hard gate

`cross_verify.py` mirrors `validate_consistency.py`: it exits non-zero (1) when
anything is flagged or an index was unavailable, but it **does not block the
pipeline**. Hallucination removal is a judgment call, so the gate surfaces
candidates for the host agent to review — it never deletes citations itself.

Exit codes: `0` all corroborated · `1` flags present or an index unavailable
(WARN) · `2` usage error / unreadable pool.

## Known false positive: arXiv-only preprints

Crossref does not index most arXiv preprints (they have no Crossref DOI), and
OpenAlex title search may not rank an arXiv-only work in its top hits. A
legitimate, S2-verified preprint (e.g. *Proximal Policy Optimization
Algorithms*, arXiv:1707.06347) can therefore land in the `low` tier.

**`low` means "could not corroborate," not "fabricated."** S2 already confirmed
the record exists. Treat `low`/`conflict` as a prompt to look closer:
- If the paper is a well-known arXiv preprint you recognize → keep it.
- If you cannot find it anywhere by hand → drop it from the pool and re-run
  `dedupe_by_id.py` onward.

Do **not** auto-delete `low`-tier papers.

## Where it fits in the pipeline

```
dedupe_by_id → validate_pool --fix → cross_verify --inplace → bibtex_format → sync_keys
```

See SKILL.md Step 3.5.

# Discovery Pipeline (Phase 1 + Phase 2)

Source: arXiv:2604.05018, App. D.3 ("Citation Verification") and App. B
(LLM-call distribution).

## Phase 1 — Parallel Candidate Discovery

The paper uses 10 concurrent workers to fan out search-grounded LLM calls
("Gemini-3-Flash with Google Search grounding"). For our host-agent
implementation, the equivalent is: spawn up to 10 concurrent search queries
using the host's native web search tool.

### Inputs

From `outline.json`:

```
introduction_strategy:
  search_directions: [q1, q2, q3]              # 3-5 queries
related_work_strategy:
  subsections:
    - methodology_cluster: "..."
      sota_investigation_mission: "..."        # 1 derived query
      limitation_search_queries: [q4, q5]      # 1-3 queries
    - ...
```

Total query budget: typically 10-20 queries per paper.

### Per-query procedure

For each search query, instruct your host's search tool:

```
search("<query>", num_results=10)
```

Or, if you've enabled the optional Exa backend (see `exa-search-cookbook.md`):

```bash
python scripts/exa_search.py --query "<query>" --num-results 10
```

Both paths produce the same normalized candidate format. Collect the top
10 results per query. Each result should yield:

- `title` — the paper's title from the search snippet
- `snippet` — the abstract preview from the search snippet
- `source_url` — the result URL (often the arXiv abstract page)

Tag each result with `discovered_for: ["intro"]` or
`discovered_for: ["related_work[2.1]"]` so you can later trace which cluster
each citation supports.

Combine all results across all queries into a single `raw_candidates.json`:

```json
{
  "candidates": [
    {
      "title": "Attention Is All You Need",
      "snippet": "The dominant sequence transduction models...",
      "source_url": "https://arxiv.org/abs/1706.03762",
      "discovered_for": ["intro"]
    },
    ...
  ]
}
```

## Phase 2 — Sequential Verification via Semantic Scholar

The paper enforces strict sequential verification at ≤1 QPS via the public
Semantic Scholar API. We follow the same constraint.

### Per-candidate procedure

1. **Search S2 by title**. Use the host's URL fetch tool:
   ```
   GET https://api.semanticscholar.org/graph/v1/paper/search
       ?query=<URL-encoded(title)>
       &limit=5
       &fields=title,abstract,year,authors,venue,externalIds
   ```
   No API key required for the public endpoint. Be polite: 1 QPS.

2. **Take the top hit**. Compare `title` to the candidate `title` via the
   helper:
   ```bash
   python scripts/levenshtein_match.py --candidate "..." --found "..."
   ```
   The helper prints an integer 0-100 (the Levenshtein ratio).
   - **< 70 → discard the candidate.** Move on.
   - **≥ 70 → continue to checks 3-5.**

3. **Check abstract presence**. If `abstract` is null or empty → discard.
   The paper requires every cited entity to have a retrievable abstract for
   downstream context enrichment in the Section Writing Agent.

4. **Check temporal cutoff**:
   ```bash
   python scripts/check_cutoff.py \
       --paper-year <year> \
       --paper-month <month or omit> \
       --cutoff <YYYY-MM-DD>
   ```
   Exit 0 if strictly predates; exit 1 if not. Discard on exit 1.

5. **Year-alignment bonus**. If the candidate's `discovered_for` query
   mentioned a specific year and the S2 hit's year matches exactly, record
   `match_score = ratio + 5`. (This is a soft bonus used for tie-breaking
   when two candidates dedup to similar entries.)

6. **Append to verified pool** if all checks pass. Record:
   ```json
   {
     "paperId": "abc123...",
     "title": "...",
     "abstract": "...",
     "year": 2017,
     "venue": "NeurIPS",
     "authors": [{"name": "A. Vaswani"}, ...],
     "externalIds": {"DOI": "...", "ArXiv": "1706.03762"},
     "match_score": 100,
     "discovered_for": ["intro"]
   }
   ```

### Rate-limit etiquette

The S2 public endpoint enforces ~1 QPS without an API key. If you receive
HTTP 429, sleep 5 seconds and retry. Do not parallelize Phase 2 — verification
must be strictly sequential.

If your host has the patience for it, the paper measures ~20-30 LLM/API calls
total per Lit Review Agent invocation. With ~30 candidates that's roughly
30 seconds of verification wall-time. With 100 candidates it's ~100 seconds.

## Why two phases

The split exists because:

- **Discovery is high-throughput, low-stakes**. You want to cast a wide net
  fast. Search APIs accept high concurrency.
- **Verification is low-throughput, high-stakes**. The S2 API protects
  itself with QPS limits, and the verification step is what keeps the paper
  honest. Faking a citation is trivially easy without it.

The paper's design "successfully combines the high-concurrency tolerance of
the LLM API with the strict throughput limits of the Semantic Scholar API to
prevent quota-induced latency" (App. B).

# Exa Search Cookbook (optional Phase 1 backend)

[Exa](https://exa.ai) is a search engine optimized for finding academic
papers and other high-quality content. The `literature-review-agent` can
use Exa as an **OPTIONAL** backend for Phase 1 candidate discovery — useful
when your host coding agent has no native web search tool, or when you
want a research-paper-focused search backend with better signal-to-noise
than general web search.

> **Exa is opt-in.** The literature-review-agent's default Phase 1 path is
> "use your host agent's native web search tool" (`WebSearch` in Claude
> Code, `@web` in Cursor, the search tool in Antigravity, etc.). That
> requires zero configuration and no API key. Use Exa only if you want
> to.

## Why use it

Exa fills three gaps:

1. **Hosts with no built-in search.** Aider, OpenCode, and generic CLI
   agents often lack a native web search tool. Exa gives them one.
2. **Research-paper-focused results.** Exa's `category: "research paper"`
   filter returns higher signal-to-noise than general web search for
   academic queries. The example response (e.g., for the query
   "PaperOrchestra") returns arXiv pages, conference proceedings, and
   academic tools rather than general SEO content.
3. **Batch / non-interactive runs.** When you want a deterministic,
   scriptable backend rather than going through the host agent's tool
   interface.

Exa returns 10–20 results per call (the helper clamps to that range), and
each result includes a `title`, `url`, optional `publishedDate`, and a
list of `highlights` (snippets) which the helper joins into a `snippet`
field consumable by the rest of the Phase 1 pipeline.

## Get a key

1. Sign up at <https://dashboard.exa.ai/>.
2. Copy your API key (format: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`).
3. Set it in your environment:

   ```bash
   export EXA_API_KEY="paste-key-here"
   ```

   Or put it in a `.env` file (which is gitignored — the repo `.gitignore`
   blocks `*.env` and `.env*` patterns) and source it:

   ```bash
   set -a; source .env; set +a
   ```

**This repo never commits a key.** The helper reads `EXA_API_KEY` from the
environment at runtime. The key is your responsibility to provision and
secure.

## Run the helper

```bash
python skills/literature-review-agent/scripts/exa_search.py \
    --query "Sparse attention long context transformers" \
    --num-results 15 \
    --discovered-for "related_work[2.1]"
```

Output (default — normalized to the literature-review-agent candidate
format):

```json
{
  "candidates": [
    {
      "title": "Longformer: The Long-Document Transformer",
      "snippet": "We present the Longformer, a self-attention mechanism that scales linearly with sequence length...",
      "source_url": "https://arxiv.org/abs/2004.05150",
      "discovered_for": ["related_work[2.1]"],
      "_exa_id": "https://arxiv.org/abs/2004.05150",
      "_exa_published_date": "2020-04-10T00:00:00.000Z"
    },
    ...
  ]
}
```

This JSON can be merged directly into `workspace/raw_candidates.json`
before the Phase 2 sequential verification step.

### Useful flags

| Flag | Default | Purpose |
|---|---|---|
| `--query` | (required) | Search query string |
| `--num-results` | `10` | 1–20; the helper clamps to this range |
| `--category` | `"research paper"` | Pass `""` to disable category filtering for broader results |
| `--highlight-chars` | `4000` | Max characters per highlight (Exa parameter) |
| `--discovered-for` | `"intro"` | Tag attached to each candidate; use `"related_work[2.1]"` for cluster queries |
| `--raw` | off | Print the full Exa response JSON instead of normalized candidates |

## Direct curl recipe

If you'd rather not use the Python helper (for one-off testing, or to
invoke from a host agent's `Bash` / `WebFetch` tool directly):

```bash
curl -X POST https://api.exa.ai/search \
  --header "content-type: application/json" \
  --header "x-api-key: $EXA_API_KEY" \
  --data '{
    "query": "PaperOrchestra automated paper writing",
    "category": "research paper",
    "numResults": 10,
    "type": "auto",
    "contents": {
      "highlights": {
        "maxCharacters": 4000
      }
    }
  }'
```

The `$EXA_API_KEY` reference assumes the key is in your shell env. **Do
not** paste the literal key into the curl command in shell history or
chat — use the env var.

## Response shape

```json
{
  "requestId": "52fcb70256224863b33f356fdae37c7f",
  "resolvedSearchType": "neural",
  "results": [
    {
      "id": "https://arxiv.org/abs/2604.05018",
      "title": "PaperOrchestra: A Multi-Agent Framework for ...",
      "url": "https://arxiv.org/abs/2604.05018",
      "publishedDate": "2026-04-06T00:00:00.000Z",
      "highlights": ["...", "..."],
      "highlightScores": [0.4, 0.3],
      "image": "https://...",
      "favicon": "https://..."
    }
  ],
  "searchTime": 975.2,
  "costDollars": {
    "total":  0.007,
    "search": {"neural": 0.007}
  }
}
```

## Mapping Exa → literature-review-agent candidate format

Phase 2 verification (Semantic Scholar fuzzy match → cutoff check → dedup)
expects candidates in this shape:

```json
{
  "title":          "...",
  "snippet":        "...",
  "source_url":     "...",
  "discovered_for": ["intro"]
}
```

`exa_search.py --normalize` (the default mode) does this mapping:

| Exa field | Candidate field |
|---|---|
| `result.title` | `title` |
| `result.url` (fallback `result.id`) | `source_url` |
| `result.highlights` joined and capped at 1500 chars | `snippet` |
| `--discovered-for` flag | `discovered_for` |
| `result.id` | `_exa_id` (preserved for debugging) |
| `result.publishedDate` | `_exa_published_date` (preserved for tie-breaking) |

Phase 2 verification still goes through Semantic Scholar regardless of
whether the candidate came from Exa or from the host's native search.
Exa is ONLY a discovery backend; the verification chain
(`levenshtein_match.py` → `check_cutoff.py` → `dedupe_by_id.py` →
`bibtex_format.py` → `citation_coverage.py`) is unchanged.

## Query patterns

Match the literature-review-agent's outline-driven query design. Run one
Exa call per query, then merge all candidate lists:

| Query type | Source in `outline.json` | Example query | `--discovered-for` |
|---|---|---|---|
| Macro context | `introduction_strategy.search_directions[i]` | `"Survey of long-context attention mechanisms 2020-2024"` | `"intro"` |
| Foundational | same | `"Foundational papers transformer self-attention scaling laws"` | `"intro"` |
| SOTA scan | `related_work_strategy.subsections[i].sota_investigation_mission` | `"Recent SOTA sparse attention transformers 2024"` | `"related_work[2.1]"` |
| Limitation hunt | `related_work_strategy.subsections[i].limitation_search_queries[j]` | `"Block-sparse attention failure modes long sequences"` | `"related_work[2.1]"` |

For the related-work cluster queries, the `--discovered-for` tag matters
— the downstream `citation_coverage.py` gate uses it to attribute each
citation to the right cluster when reporting which papers were not yet
integrated.

## Cost and rate limits

Exa pricing is per-query (~$0.007 per neural search at the time of
writing). For a typical paper with ~15-20 search queries (3-5 intro
queries + 10-15 related-work queries), one full Lit Review Agent run
costs ~$0.10-$0.15. Check <https://exa.ai/pricing> for current rates.

Exa's rate limits are generous; the paper's 10-worker parallel discovery
pattern is well within them. The pipeline's wall-time floor is still set
by Semantic Scholar's 1 QPS verification limit, not by Exa.

## Security

- **NEVER commit `EXA_API_KEY` to git.** The repo's `.gitignore` blocks
  `.env`, `*.env`, and `secrets.json` patterns. Keep your key in your
  shell environment or your secrets manager (1Password CLI, op, doppler,
  etc.).
- The helper reads the key from the environment only. It does NOT accept
  the key as a command-line argument (which would expose it in shell
  history).
- Exa logs requests for billing and quality. Assume your queries are not
  private to Exa themselves. Don't include sensitive draft text in
  queries.

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `ERROR: EXA_API_KEY environment variable not set` | env var missing | `export EXA_API_KEY="..."` |
| `ERROR: Exa HTTP 401` | invalid or expired key | check the dashboard for the current key |
| `ERROR: Exa HTTP 429` | rate-limited | back off, lower concurrency |
| `WARN: Exa returned 0 results` | query too narrow or odd category | broaden the query or try `--category ""` |
| `Exa network error` | no internet, DNS issue | check your connection; the helper uses urllib stdlib only, no proxy support |

## When to prefer Exa vs the host's native search

| Use case | Recommended backend |
|---|---|
| Claude Code, Cursor, Antigravity (have native web search) | host's native search (free, integrated) |
| Aider, OpenCode, generic CLI agents | Exa (gives them search) |
| Batch reproducible runs | Exa (deterministic backend) |
| Research-paper-heavy queries | Exa (better academic signal) |
| One-off interactive runs | host's native search (less friction) |

You can also mix: use the host's web search for the broad intro queries
and Exa for the narrow limitation-search queries where the
research-paper-category filter helps the most.

# Literature Review Agent — verbatim prompt

**Source: arXiv:2604.05018, Appendix F.1, page 46 (verbatim).**

This is the exact prompt used by the Literature Review Agent in the paper.
Use it as your system message when drafting Introduction and Related Work.
Substitute the placeholders before sending. The Anti-Leakage Prompt
(`../paper-orchestra/references/anti-leakage-prompt.md`) MUST be prepended.

---

```
Role: Senior AI Researcher.

Task: Write the introduction and related work section of a paper.

You will be given a template.tex, this is the initial skeleton we outlined for
you. Your job is to fill in two sections: Introduction and Related Work.
Leave all the other sections untouched.

Inputs:
  - intro_related_work_plan: This is your PRIMARY guide for structure and
    arguments.
  - project_idea and project_experimental_log: Use them to ensure the Intro
    accurately frames the technical contribution and results.
  - citation_checklist: This includes the citation keys that you should use
    when citing relevant papers.
  - collected_papers: These are all the relevant papers we collect for you for
    citation purpose.

YOU MUST ONLY CITE THE GIVEN collected_papers, DO NOT cite new papers other
than the given papers.

Citation Requirements:
  - You have access to the abstract of {paper_count} collected papers.
  - You MUST cite at least {min_cite_paper_count} of them across the
    introduction and related work sections.
  - Introduction: Cite key statistics, foundational models (CLIP, etc.), and
    broad problem statements.
  - Related Work: Do deep comparative citations. Group distinct works (e.g.,
    "Several methods [A, B, C]...").
  - Ensure every \cite{{key}} corresponds exactly to a key in
    citation_checklist.
  - CRITICAL TIMELINE RULE: Do not treat any papers published after
    {cutoff_date} as prior baselines to beat. Treat them strictly as
    concurrent work.
  - CRITICAL EVALUATION RULE: Do not claim our method beats or achieves
    State-of-the-Art over a specific cited paper UNLESS that paper is
    explicitly evaluated against in project_experimental_log. Frame other
    recent papers strictly as concurrent, orthogonal, or conceptual work.
  - You need to return the full code for the new template.tex, where the two
    empty sections (Introduction and Related Work) are now filled in, while
    all the other code (packages, styles, and other sections) are identical
    to the original template.tex.

Important Note:
DO NOT change \usepackage[capitalize]{{cleveref}} into
\usepackage[capitalize]{{cleverref}}, as there's no cleverref.sty.

Output Format:
You must return the code for the updated template.tex. Make sure to wrap the
code with ```latex content ```.
```

---

## Placeholder substitution table

| Placeholder | Source |
|---|---|
| `{paper_count}` | `len(citation_pool.papers)` from `workspace/citation_pool.json` |
| `{min_cite_paper_count}` | `floor(0.9 * paper_count)` — the ≥90% rule |
| `{cutoff_date}` | Derived from `conference_guidelines.md` — see App. D.1 of the paper |

The other placeholders (`intro_related_work_plan`, `project_idea`,
`project_experimental_log`, `citation_checklist`, `collected_papers`) are
substituted by passing their full file/JSON contents into the user message.

# Semantic Scholar API Cookbook

How to verify a candidate paper via the Semantic Scholar Graph API.

Base: `https://api.semanticscholar.org/graph/v1`

Reference: <https://api.semanticscholar.org/api-docs/graph>

## API key (optional)

The pipeline uses the **public, unauthenticated endpoint** by default — no key
required.  If you have a Semantic Scholar API key you can pass it via the
`x-api-key` header to get higher rate limits (useful for large batches).

Get a free key at <https://api.semanticscholar.org/> then export it once:

```bash
export SEMANTIC_SCHOLAR_API_KEY="your-key-here"
```

The bundled `scripts/s2_search.py` helper picks this up automatically.  If the
variable is not set the script falls back to the unauthenticated endpoint — the
pipeline works fine either way; just keep to ≤1 QPS on live requests.

```bash
# check whether the key is configured
python skills/literature-review-agent/scripts/s2_search.py --check-key

# search by title (key used automatically if set)
python skills/literature-review-agent/scripts/s2_search.py \
    --query "Attention is All You Need" --limit 5

# print the raw S2 JSON
python skills/literature-review-agent/scripts/s2_search.py \
    --query "BERT pre-training" --raw
```

The repo never commits a key.  Key management is your responsibility (shell
environment, 1Password, doppler, etc.).

## Endpoint 1 — Search by title

```
GET /paper/search
    ?query=<URL-encoded title>
    &limit=5
    &fields=title,abstract,year,authors,venue,externalIds
```

Example:

```
GET https://api.semanticscholar.org/graph/v1/paper/search?query=Attention%20Is%20All%20You%20Need&limit=5&fields=title,abstract,year,authors,venue,externalIds
```

Response (truncated):

```json
{
  "total": 12345,
  "data": [
    {
      "paperId": "204e3073870fae3d05bcbc2f6a8e263d9b72e776",
      "title": "Attention is All you Need",
      "abstract": "The dominant sequence transduction models are based on...",
      "year": 2017,
      "venue": "NeurIPS",
      "authors": [{"name": "Ashish Vaswani"}, ...],
      "externalIds": {
        "DBLP": "conf/nips/VaswaniSPUJGKP17",
        "ArXiv": "1706.03762",
        "DOI": "10.5555/3295222.3295349"
      }
    },
    ...
  ]
}
```

## Endpoint 2 — Get a specific paper by ID

```
GET /paper/<paperId>?fields=title,abstract,year,authors,venue,externalIds,citationCount
```

## Useful identifiers

You can pass these as `<paperId>`:

- S2 internal: `204e3073870fae3d05bcbc2f6a8e263d9b72e776`
- DOI: `DOI:10.18653/v1/N18-3011`
- ArXiv: `ARXIV:1706.03762`
- Corpus ID: `CorpusId:13756489`
- URL: `URL:https://arxiv.org/abs/1706.03762`

## Rate limits

- Unauthenticated: ~1 QPS sustained. Bursts will get 429.
- Per the paper, "the strict throughput limits of the Semantic Scholar API
  (1 query per second)" — App. B.

If you get HTTP 429, sleep 5 seconds before retrying. Don't loop tightly.

## Fields cheat sheet

| Field | Type | Required by our pipeline? |
|---|---|---|
| `paperId` | string | yes (dedup key) |
| `title` | string | yes (Levenshtein match) |
| `abstract` | string | yes (rule 2: must exist) |
| `year` | int | yes (cutoff check) |
| `authors[].name` | string | yes (BibTeX author field) |
| `venue` | string | recommended (BibTeX journal/booktitle) |
| `externalIds.DOI` | string | recommended (dedup fallback, BibTeX doi) |
| `externalIds.ArXiv` | string | recommended (dedup fallback) |
| `publicationDate` | string `YYYY-MM-DD` | optional (more precise cutoff check) |
| `citationCount` | int | optional (could inform tie-breaking) |

Always pass `fields=...` explicitly — the default response is minimal and
will not include the abstract.

## Error handling

| Status | Meaning | What to do |
|---|---|---|
| 200 | OK | proceed |
| 400 | bad query syntax | URL-encode the title properly; retry once |
| 404 | not found | discard the candidate |
| 429 | rate limited | sleep 5s, retry |
| 500-503 | S2 down | sleep 30s, retry up to 3 times, then give up |

## Polite use

The S2 API is a public service. Do not hammer it. If you have many candidates:

- Throttle to 1 QPS.
- Cache hits (the dedup script already serves as a deduplication cache).
- Do not parallelize. Verification is sequential by design.

# Tavily Search Cookbook (optional Phase 1 backend)

[Tavily](https://tavily.com) is a search API designed for LLMs, enabling
AI applications to access real-time web data with high relevance scoring.
The `literature-review-agent` can use Tavily as an **OPTIONAL** backend
for Phase 1 candidate discovery — useful when your host coding agent has
no native web search tool, or when you want an LLM-optimized search
backend.

> **Tavily is opt-in.** The literature-review-agent's default Phase 1
> path is "use your host agent's native web search tool" (`WebSearch` in
> Claude Code, `@web` in Cursor, the search tool in Antigravity, etc.).
> That requires zero configuration and no API key. Use Tavily only if
> you want to.

## Why use it

Tavily fills three gaps:

1. **Hosts with no built-in search.** Aider, OpenCode, and generic CLI
   agents often lack a native web search tool. Tavily gives them one.
2. **LLM-optimized relevance.** Tavily's `search_depth: "advanced"`
   mode returns higher relevance results for complex queries. The
   `--academic` flag restricts results to academic domains (arxiv.org,
   scholar.google.com, semanticscholar.org, aclanthology.org,
   openreview.net) for research-focused discovery.
3. **Batch / non-interactive runs.** When you want a deterministic,
   scriptable backend rather than going through the host agent's tool
   interface.

Tavily returns up to 20 results per call (the helper clamps to that
range), and each result includes a `title`, `url`, `content` (snippet),
and a relevance `score` which the helper preserves as `_tavily_score`
for debugging.

## Get a key

1. Sign up at <https://app.tavily.com>.
2. Copy your API key (format: `tvly-xxxxxxxxxxxxxxxxxxxxxxxx`).
3. Set it in your environment:

   ```bash
   export TAVILY_API_KEY="tvly-your-key-here"
   ```

   Or put it in a `.env` file (which is gitignored — the repo `.gitignore`
   blocks `*.env` and `.env*` patterns) and source it:

   ```bash
   set -a; source .env; set +a
   ```

**This repo never commits a key.** The helper reads `TAVILY_API_KEY` from
the environment at runtime. The key is your responsibility to provision
and secure.

## Run the helper

```bash
python skills/literature-review-agent/scripts/tavily_search.py \
    --query "Sparse attention long context transformers" \
    --num-results 15 \
    --academic \
    --discovered-for "related_work[2.1]"
```

Output (default — normalized to the literature-review-agent candidate
format):

```json
{
  "candidates": [
    {
      "title": "Longformer: The Long-Document Transformer",
      "snippet": "We present the Longformer, a self-attention mechanism that scales linearly with sequence length...",
      "source_url": "https://arxiv.org/abs/2004.05150",
      "discovered_for": ["related_work[2.1]"],
      "_tavily_score": 0.92
    },
    ...
  ]
}
```

This JSON can be merged directly into `workspace/raw_candidates.json`
before the Phase 2 sequential verification step.

### Useful flags

| Flag | Default | Purpose |
|---|---|---|
| `--query` | (required) | Search query string |
| `--num-results` | `10` | 1–20; the helper clamps to this range |
| `--topic` | `"general"` | `"general"` or `"news"`; use `"news"` for recent results |
| `--academic` | off | Restrict to academic domains (arxiv.org, scholar.google.com, etc.) |
| `--discovered-for` | `"intro"` | Tag attached to each candidate; use `"related_work[2.1]"` for cluster queries |
| `--raw` | off | Print the full Tavily response JSON instead of normalized candidates |

## Direct curl recipe

If you'd rather not use the Python helper (for one-off testing, or to
invoke from a host agent's `Bash` / `WebFetch` tool directly):

```bash
curl -X POST https://api.tavily.com/search \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $TAVILY_API_KEY" \
  --data '{
    "query": "PaperOrchestra automated paper writing",
    "max_results": 10,
    "search_depth": "advanced",
    "topic": "general",
    "include_domains": ["arxiv.org", "scholar.google.com"]
  }'
```

The `$TAVILY_API_KEY` reference assumes the key is in your shell env.
**Do not** paste the literal key into the curl command in shell history
or chat — use the env var.

## Response shape

```json
{
  "query": "PaperOrchestra automated paper writing",
  "results": [
    {
      "title": "PaperOrchestra: A Multi-Agent Framework for ...",
      "url": "https://arxiv.org/abs/2604.05018",
      "content": "We present PaperOrchestra, a multi-agent framework...",
      "score": 0.95
    }
  ]
}
```

## Mapping Tavily → literature-review-agent candidate format

Phase 2 verification (Semantic Scholar fuzzy match → cutoff check → dedup)
expects candidates in this shape:

```json
{
  "title":          "...",
  "snippet":        "...",
  "source_url":     "...",
  "discovered_for": ["intro"]
}
```

`tavily_search.py` (the default mode) does this mapping:

| Tavily field | Candidate field |
|---|---|
| `result.title` | `title` |
| `result.url` | `source_url` |
| `result.content` capped at 1500 chars | `snippet` |
| `--discovered-for` flag | `discovered_for` |
| `result.score` | `_tavily_score` (preserved for debugging) |

Phase 2 verification still goes through Semantic Scholar regardless of
whether the candidate came from Tavily, Exa, or from the host's native
search. Tavily is ONLY a discovery backend; the verification chain
(`levenshtein_match.py` → `check_cutoff.py` → `dedupe_by_id.py` →
`bibtex_format.py` → `citation_coverage.py`) is unchanged.

## Query patterns

Match the literature-review-agent's outline-driven query design. Run one
Tavily call per query, then merge all candidate lists:

| Query type | Source in `outline.json` | Example query | `--discovered-for` |
|---|---|---|---|
| Macro context | `introduction_strategy.search_directions[i]` | `"Survey of long-context attention mechanisms 2020-2024"` | `"intro"` |
| Foundational | same | `"Foundational papers transformer self-attention scaling laws"` | `"intro"` |
| SOTA scan | `related_work_strategy.subsections[i].sota_investigation_mission` | `"Recent SOTA sparse attention transformers 2024"` | `"related_work[2.1]"` |
| Limitation hunt | `related_work_strategy.subsections[i].limitation_search_queries[j]` | `"Block-sparse attention failure modes long sequences"` | `"related_work[2.1]"` |

For the related-work cluster queries, the `--discovered-for` tag matters
— the downstream `citation_coverage.py` gate uses it to attribute each
citation to the right cluster when reporting which papers were not yet
integrated.

**Tip:** Use `--academic` for the SOTA scan and limitation hunt queries
to keep results focused on research papers. For broad intro queries,
omitting `--academic` may yield useful surveys and blog posts that
reference foundational work.

## Cost and rate limits

Tavily offers 1,000 free API credits per month (no credit card
required). The `search_depth: "advanced"` mode used by the helper costs
2 credits per query. For a typical paper with ~15-20 search queries
(3-5 intro queries + 10-15 related-work queries), one full Lit Review
Agent run costs ~30-40 credits — well within the free tier.

For higher volumes, see <https://tavily.com/pricing> for paid plans.
Tavily's rate limits are generous; the paper's 10-worker parallel
discovery pattern is well within them. The pipeline's wall-time floor is
still set by Semantic Scholar's 1 QPS verification limit, not by Tavily.

## SDK alternative

If `tavily-python` is installed (`pip install tavily-python`), you can
use the SDK directly instead of the bundled helper:

```python
from tavily import TavilyClient

client = TavilyClient()  # reads TAVILY_API_KEY from env
response = client.search(
    query="Sparse attention long context transformers",
    max_results=15,
    search_depth="advanced",
    include_domains=["arxiv.org", "scholar.google.com"],
)
```

The bundled `tavily_search.py` helper uses stdlib `urllib` only (like
`exa_search.py`) to avoid mandatory dependencies. The SDK is optional.

## Security

- **NEVER commit `TAVILY_API_KEY` to git.** The repo's `.gitignore`
  blocks `.env`, `*.env`, and `secrets.json` patterns. Keep your key in
  your shell environment or your secrets manager (1Password CLI, op,
  doppler, etc.).
- The helper reads the key from the environment only. It does NOT accept
  the key as a command-line argument (which would expose it in shell
  history).
- Tavily logs requests for billing and quality. Assume your queries are
  not private to Tavily themselves. Don't include sensitive draft text
  in queries.

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `ERROR: TAVILY_API_KEY environment variable not set` | env var missing | `export TAVILY_API_KEY="tvly-..."` |
| `ERROR: Tavily HTTP 401` | invalid or expired key | check your key at https://app.tavily.com |
| `ERROR: Tavily HTTP 429` | rate-limited | back off, lower concurrency |
| `WARN: Tavily returned 0 results` | query too narrow | broaden the query or remove `--academic` |
| `Tavily network error` | no internet, DNS issue | check your connection; the helper uses urllib stdlib only, no proxy support |

## When to prefer Tavily vs Exa vs the host's native search

| Use case | Recommended backend |
|---|---|
| Claude Code, Cursor, Antigravity (have native web search) | host's native search (free, integrated) |
| Aider, OpenCode, generic CLI agents | Tavily or Exa (gives them search) |
| Batch reproducible runs | Tavily or Exa (deterministic backend) |
| Research-paper-heavy queries | Exa (`category: "research paper"`) or Tavily (`--academic`) |
| Free tier / budget-conscious | Tavily (1,000 free credits/month) |
| LLM-optimized relevance scoring | Tavily (`search_depth: "advanced"`) |
| One-off interactive runs | host's native search (less friction) |

You can also mix: use the host's web search for the broad intro queries,
Exa for the narrow limitation-search queries where the research-paper
category filter helps the most, and Tavily for SOTA scan queries where
LLM-optimized relevance scoring shines.

# Verification Rules

Source: arXiv:2604.05018, App. D.3 ("Citation Verification"), verbatim
specifications below.

## Rule 1 — Fuzzy title match (Levenshtein > 70)

> Each candidate must resolve to a valid Semantic Scholar entity via a fuzzy
> title match (Levenshtein distance ratio > 70 [Levenshtein, 1965]),
> augmented by a point bonus for exact year alignment.

Implementation: `scripts/levenshtein_match.py` uses
`Levenshtein.ratio(a, b) * 100` from the `python-Levenshtein` package and
returns the integer ratio. Threshold: **strictly greater than 70**.

Examples:

| Candidate title | S2 title | Ratio | Verdict |
|---|---|---|---|
| "Attention Is All You Need" | "Attention Is All You Need" | 100 | accept |
| "Attention Is All You Need" | "Attention is All You Need." | 96 | accept |
| "Sparse Attention for Transformers" | "Sparse Attention in Transformers" | 88 | accept |
| "Self-Attention" | "Attention Is All You Need" | 47 | reject |
| "Linformer" | "Linformer: Self-Attention with Linear Complexity" | 28 | reject |

The Linformer case is the canonical false-negative: a short query against
a long title. Workaround: when the candidate title looks abbreviated
(< 4 words) and the S2 hit's title contains the candidate as a substring,
override the ratio check. The paper does not specify this workaround
explicitly; we add it as a soft safety net to avoid losing legitimate
short-title hits. See `levenshtein_match.py --substring-bypass`.

## Rule 2 — Abstract must exist

> To enter the final context pool, the entity must possess a retrievable
> abstract...

Discard any verified hit where `abstract` is null, empty, or `"N/A"`. The
Section Writing Agent uses the abstract to ground its citations contextually
(per the Section Writing Agent prompt: "Read the abstract provided in
citation_map.json for the papers you are citing. Use this context to write
accurate, specific sentences about those works.").

## Rule 3 — Strict temporal cutoff

> ...and strictly predate the research cutoff (when specified down to the
> month, the system defaults to the first day of that month).

Implementation: `scripts/check_cutoff.py`. Comparison rules:

- Cutoff is given as `YYYY-MM-DD`. The paper aligns it to venue submission
  deadline (Nov 2024 for CVPR 2025, Oct 2024 for ICLR 2025 — App. D.1).
- Paper year is required. Paper month is optional.
- If paper has only year: assume month=12, day=31 (worst case for the paper —
  must still be < cutoff).
- If paper has year + month: assume day=1 of that month.
- "Strictly predate" means `paper_date < cutoff_date`. Equality fails.

Examples (cutoff = 2024-10-01):

| Paper year | Paper month | Verdict |
|---|---|---|
| 2017 | — | accept |
| 2024 | 9 | accept (2024-09-01 < 2024-10-01) |
| 2024 | 10 | reject (2024-10-01 not strictly < 2024-10-01) |
| 2024 | — (only year) | reject (2024-12-31 ≥ 2024-10-01) |

The strict comparison is intentional: it prevents leakage of papers from
the same submission cycle as the target venue.

## Rule 4 — Dedup by Semantic Scholar paperId

> Finally, gathered citations are deduplicated using unique paper ID keys.

Implementation: `scripts/dedupe_by_id.py`. Key precedence:

1. `paperId` (S2's internal unique ID, always present on a verified hit)
2. `externalIds.DOI` (lowercased)
3. `externalIds.ArXiv` (without version suffix)
4. Normalized title (lowercased, alphanumeric only) — fallback only

When two candidates collide, keep the one with the higher `match_score`.

## Rule 5 — ≥90% citation integration

> The system constrains the model to cite only the provided verified papers,
> explicitly mandating that at least 90% of the gathered literature pool must
> be actively integrated and cited when synthesizing the Introduction and
> Related Work sections.

Implementation: `scripts/citation_coverage.py`. After the Lit Review writing
call produces `intro_relwork.tex`, this script:

1. Extracts every `\cite{KEY}` and `\citep{KEY}` (and variants) from the
   `.tex` file.
2. Counts unique cited keys against `len(citation_pool.papers)`.
3. Requires `cited / total ≥ 0.90`. Exits non-zero if not.

If the gate fails, the host agent must re-prompt the writing step,
explicitly listing the un-cited keys and asking the agent to integrate them.

## Rule 6 — Cross-index corroboration (supplementary)

Rules 1–4 verify candidates against a single index (Semantic Scholar). Rule 6
adds an independent second opinion: every paper that survives S2 verification is
re-checked against **Crossref** and **OpenAlex** before the bibliography is
built. This is not in the source paper — it is an addition that targets
hallucinated citations, which a single index can fail to catch.

Implementation: `scripts/cross_verify.py` (+ `crossref_client.py`,
`openalex_client.py`). It is a **WARN gate**: it annotates each pool paper with
a `cross_verification` confidence tier (`high` / `medium` / `low` / `conflict`)
and writes `cross_verification_report.json`, but never deletes citations — the
host agent reviews the `low` and `conflict` tiers and decides.

Matching reuses the same Levenshtein title threshold as Rule 1 (`> 70`); a DOI
present in the pool is looked up exactly first, falling back to title search.

⚠️ `low` means "could not corroborate," not "fabricated": arXiv-only preprints
without a DOI are a common benign cause. See `cross-index-verification.md` for
the full tier definitions, polite-pool env vars, and the false-positive note.

#!/usr/bin/env python3
"""
bibtex_format.py — Generate refs.bib from a verified citation pool.

Reads citation_pool.json (output of dedupe_by_id.py) and emits a BibTeX file
with deterministic citation keys derived from the first author + year +
first significant title word.

Never invents fields. Only writes fields that are actually present in the
S2 metadata. Writes one of:
    @article{ ... }      — when venue looks like a journal
    @inproceedings{ ... }— when venue looks like a conference
    @misc{ ... }         — fallback (e.g., arXiv-only papers)

Usage:
    python bibtex_format.py --pool citation_pool.json --out refs.bib
"""
import argparse
import json
import re
import sys

CONFERENCE_HINTS = {
    "neurips", "nips", "icml", "iclr", "cvpr", "iccv", "eccv", "aaai",
    "ijcai", "acl", "emnlp", "naacl", "kdd", "www", "sigir", "uai", "siggraph",
    "interspeech", "icassp", "miccai", "wacv", "bmvc", "coling", "conll",
}
STOPWORDS = {
    "a", "an", "and", "the", "of", "for", "to", "with", "on", "in", "by",
    "from", "as", "is", "are", "be", "via", "into", "their", "our", "we",
    "this", "that", "using", "use", "about", "at", "or", "if",
}


def normalize(s: str) -> str:
    return re.sub(r"[^a-z]", "", s.lower())


def first_significant_word(title: str) -> str:
    for w in re.findall(r"[A-Za-z][A-Za-z\-]*", title):
        wn = w.lower()
        if wn not in STOPWORDS and len(wn) > 2:
            return normalize(wn)
    return "paper"


def first_author_lastname(authors: list[dict]) -> str:
    if not authors:
        return "anon"
    name = authors[0].get("name", "").strip()
    if not name:
        return "anon"
    parts = name.replace(",", "").split()
    return normalize(parts[-1]) or "anon"


def make_key(paper: dict) -> str:
    last = first_author_lastname(paper.get("authors") or [])
    year = paper.get("year") or "0000"
    word = first_significant_word(paper.get("title", ""))
    return f"{last}{year}{word}"


def is_conference(venue: str) -> bool:
    if not venue:
        return False
    v = venue.lower()
    return any(h in v for h in CONFERENCE_HINTS)


def escape_bibtex(s: str) -> str:
    if not s:
        return ""
    return s.replace("{", "\\{").replace("}", "\\}").replace("&", "\\&")


def author_field(authors: list[dict]) -> str:
    names = [a.get("name", "").strip() for a in authors if a.get("name")]
    return " and ".join(escape_bibtex(n) for n in names)


def format_entry(paper: dict, key: str) -> str:
    venue = paper.get("venue") or ""
    if is_conference(venue):
        kind = "inproceedings"
        venue_key = "booktitle"
    elif venue:
        kind = "article"
        venue_key = "journal"
    else:
        kind = "misc"
        venue_key = None

    lines = [f"@{kind}{{{key},"]
    if title := paper.get("title"):
        lines.append(f'  title    = {{{escape_bibtex(title)}}},')
    if authors := paper.get("authors"):
        lines.append(f'  author   = {{{author_field(authors)}}},')
    if year := paper.get("year"):
        lines.append(f'  year     = {{{year}}},')
    if venue and venue_key:
        lines.append(f'  {venue_key:8s} = {{{escape_bibtex(venue)}}},')
    ext = paper.get("externalIds") or {}
    if doi := ext.get("DOI"):
        lines.append(f'  doi      = {{{doi}}},')
    if arxiv := ext.get("ArXiv"):
        lines.append(f'  eprint   = {{{arxiv}}},')
        lines.append(f'  archivePrefix = {{arXiv}},')
    # Strip trailing comma on last field
    if lines[-1].endswith(","):
        lines[-1] = lines[-1].rstrip(",")
    lines.append("}")
    return "\n".join(lines)


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--pool", required=True, help="citation_pool.json")
    p.add_argument("--out", required=True, help="output refs.bib")
    args = p.parse_args()

    with open(args.pool) as f:
        pool = json.load(f)
    papers = pool.get("papers", [])
    if not papers:
        print("ERROR: pool contains no papers", file=sys.stderr)
        return 1

    keys_used: dict[str, int] = {}
    entries: list[str] = []
    paper_keys: list[str] = []

    for paper in papers:
        base_key = make_key(paper)
        # Disambiguate collisions with letter suffix
        if base_key in keys_used:
            keys_used[base_key] += 1
            suffix = chr(ord("a") + keys_used[base_key] - 1)
            key = base_key + suffix
        else:
            keys_used[base_key] = 1
            key = base_key
        paper["bibtex_key"] = key
        paper_keys.append(key)
        entries.append(format_entry(paper, key))

    with open(args.out, "w") as f:
        f.write("% Generated by paper-orchestra literature-review-agent/bibtex_format.py\n")
        f.write(f"% {len(entries)} entries from citation_pool.json\n\n")
        f.write("\n\n".join(entries))
        f.write("\n")

    # Write the keys back into the pool so the writing step has the
    # citation_checklist mapping. (Idempotent — overwrites with same data.)
    with open(args.pool, "w") as f:
        json.dump(pool, f, indent=2, ensure_ascii=False)

    print(f"OK: {len(entries)} BibTeX entries → {args.out}")
    print(f"    keys: {', '.join(paper_keys[:5])}{'...' if len(paper_keys) > 5 else ''}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
check_cutoff.py — Strict temporal cutoff check for citation verification.

Implements the paper's Rule 3 (App. D.3): a paper passes only if its
publication date strictly predates the research cutoff. When only the year
is known, assume the worst case (Dec 31). When year + month are known,
assume day-1 of that month (per the paper's "first day of that month"
default).

Exit codes:
    0  paper strictly predates cutoff (PASS)
    1  paper does not strictly predate cutoff (FAIL)
    2  argument error

Usage:
    python check_cutoff.py --paper-year 2024 --paper-month 9 --cutoff 2024-10-01
    python check_cutoff.py --paper-year 2024 --cutoff 2024-10-01
    python check_cutoff.py --paper-date 2024-09-15 --cutoff 2024-10-01
"""
import argparse
import datetime as dt
import sys


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--paper-year", type=int, help="Paper publication year")
    p.add_argument("--paper-month", type=int, help="Paper publication month (1-12), optional")
    p.add_argument("--paper-date", help="Full paper date YYYY-MM-DD, overrides year/month")
    p.add_argument("--cutoff", required=True, help="Research cutoff date YYYY-MM-DD")
    args = p.parse_args()

    try:
        cutoff = dt.date.fromisoformat(args.cutoff)
    except ValueError:
        print(f"ERROR: --cutoff must be YYYY-MM-DD, got {args.cutoff}", file=sys.stderr)
        return 2

    if args.paper_date:
        try:
            paper_date = dt.date.fromisoformat(args.paper_date)
        except ValueError:
            print(f"ERROR: --paper-date must be YYYY-MM-DD, got {args.paper_date}",
                  file=sys.stderr)
            return 2
    elif args.paper_year:
        if args.paper_month:
            paper_date = dt.date(args.paper_year, args.paper_month, 1)
        else:
            paper_date = dt.date(args.paper_year, 12, 31)
    else:
        print("ERROR: must provide --paper-date OR --paper-year", file=sys.stderr)
        return 2

    if paper_date < cutoff:
        print(f"PASS  paper={paper_date}  <  cutoff={cutoff}")
        return 0
    print(f"FAIL  paper={paper_date}  not strictly before cutoff={cutoff}")
    return 1


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
citation_coverage.py — Enforce the paper's ≥90% citation integration rule
(App. D.3).

Greps a generated .tex file for all citation commands, counts the unique
keys actually cited, and compares against the verified citation pool.
Exits non-zero if coverage < 90%.

Usage:
    python citation_coverage.py --tex intro_relwork.tex --pool citation_pool.json
    python citation_coverage.py --tex intro_relwork.tex --pool citation_pool.json --threshold 0.85
"""
import argparse
import json
import re
import sys

CITE_RE = re.compile(
    r"\\(?:cite|citep|citet|citeauthor|citeyear|autocite|parencite|textcite)"
    r"(?:\[[^\]]*\])?"
    r"\{([^}]+)\}"
)


def extract_cited_keys(tex: str) -> set[str]:
    keys = set()
    for m in CITE_RE.finditer(tex):
        for k in m.group(1).split(","):
            k = k.strip()
            if k:
                keys.add(k)
    return keys


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--tex", required=True, help="LaTeX file to inspect")
    p.add_argument("--pool", required=True, help="citation_pool.json")
    p.add_argument("--threshold", type=float, default=0.90,
                   help="Minimum integration ratio (default 0.90 per paper)")
    args = p.parse_args()

    with open(args.tex) as f:
        tex = f.read()
    with open(args.pool) as f:
        pool = json.load(f)

    pool_papers = pool.get("papers", [])
    pool_keys = {p.get("bibtex_key") for p in pool_papers if p.get("bibtex_key")}
    if not pool_keys:
        print("ERROR: pool has no bibtex_keys. Run bibtex_format.py first.",
              file=sys.stderr)
        return 1

    cited = extract_cited_keys(tex)
    cited_in_pool = cited & pool_keys
    n_pool = len(pool_keys)
    n_cited = len(cited_in_pool)
    ratio = n_cited / n_pool if n_pool else 0.0
    threshold_n = int(args.threshold * n_pool)

    print(f"Coverage: {n_cited}/{n_pool} = {ratio*100:.1f}% "
          f"(threshold {args.threshold*100:.0f}% = {threshold_n})")

    # report keys cited but NOT in pool — those are forbidden by the prompt
    foreign = cited - pool_keys
    if foreign:
        print(f"\nWARNING: {len(foreign)} cited keys NOT in citation pool "
              f"(violates 'cite ONLY collected_papers' rule):")
        for k in sorted(foreign):
            print(f"  - {k}")

    if n_cited < threshold_n:
        uncited = pool_keys - cited
        print(f"\nFAIL: missing {len(uncited)} pool papers from .tex:")
        # show with title for actionable re-prompting
        title_by_key = {p.get("bibtex_key"): p.get("title", "")
                        for p in pool_papers if p.get("bibtex_key")}
        discovered_by_key = {p.get("bibtex_key"): p.get("discovered_for", [])
                             for p in pool_papers if p.get("bibtex_key")}
        for k in sorted(uncited):
            tag = ",".join(discovered_by_key.get(k, [])) or "?"
            t = title_by_key.get(k, "")
            print(f"  - {k:40s}  [{tag}]  {t[:60]}")
        return 1

    print("OK: citation coverage meets threshold")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
crossref_client.py — Crossref REST API title/DOI lookup for cross-index
citation corroboration.

Used by cross_verify.py as a second opinion alongside Semantic Scholar:
if a paper that S2 "verified" cannot be found in Crossref *or* OpenAlex, it is
flagged as a potential phantom record (hallucination risk). See
references/cross-index-verification.md for the rationale.

No API key is required. Crossref offers a "polite pool" with better
reliability when you identify yourself via an email address. Set one of:
    export CROSSREF_MAILTO="you@example.com"
    export PAPER_ORCHESTRA_MAILTO="you@example.com"   # shared fallback
The email is sent only as a `mailto` query parameter / User-Agent, per
Crossref's etiquette guidelines. The repo never commits an address.

Usage:
    # title search
    python crossref_client.py --query "Attention is All You Need"

    # direct DOI lookup (exact)
    python crossref_client.py --doi 10.5555/3295222.3295349

    # raw Crossref JSON
    python crossref_client.py --query "BERT pre-training" --raw

Output (normalized): {"total": N, "data": [{title, year, doi, venue, authors}, ...]}

Exit codes:
    0  at least one result returned
    1  HTTP error, network error, or zero results
    2  usage error (bad arguments)
"""
import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request

CROSSREF_BASE = "https://api.crossref.org/works"
DEFAULT_LIMIT = 5
MAX_LIMIT = 20
_RETRY_SLEEP = 5  # seconds to wait after a 429 before retrying
SELECT_FIELDS = "DOI,title,author,published,published-print,published-online,issued,container-title,type"


def _mailto() -> str:
    return (
        os.environ.get("CROSSREF_MAILTO", "").strip()
        or os.environ.get("PAPER_ORCHESTRA_MAILTO", "").strip()
    )


def _build_request(url: str) -> urllib.request.Request:
    mailto = _mailto()
    ua = "paper-orchestra/1.0 (https://github.com/Ar9av/paper-orchestra)"
    if mailto:
        ua = f"paper-orchestra/1.0 (mailto:{mailto})"
    return urllib.request.Request(
        url, headers={"Accept": "application/json", "User-Agent": ua}, method="GET"
    )


def _get(url: str, retries: int = 3) -> dict:
    for attempt in range(1, retries + 1):
        try:
            with urllib.request.urlopen(_build_request(url), timeout=30) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except urllib.error.HTTPError as exc:
            if exc.code == 404:
                return {"message": {"items": []}}
            if exc.code == 429 and attempt < retries:
                print(f"WARN: Crossref rate-limited (429). Sleeping {_RETRY_SLEEP}s "
                      f"before retry {attempt + 1}/{retries}.", file=sys.stderr)
                time.sleep(_RETRY_SLEEP)
                continue
            if exc.code in (500, 502, 503) and attempt < retries:
                print(f"WARN: Crossref server error ({exc.code}). Retrying.", file=sys.stderr)
                time.sleep(10)
                continue
            print(f"ERROR: Crossref HTTP {exc.code}", file=sys.stderr)
            sys.exit(1)
        except urllib.error.URLError as exc:
            print(f"ERROR: Network error reaching Crossref: {exc.reason}", file=sys.stderr)
            sys.exit(1)
    sys.exit(1)


def _year_from(work: dict) -> int | None:
    for key in ("published", "published-print", "published-online", "issued"):
        dp = (work.get(key) or {}).get("date-parts")
        if dp and dp[0] and dp[0][0]:
            return int(dp[0][0])
    return None


def _normalize_work(work: dict) -> dict:
    titles = work.get("title") or []
    venues = work.get("container-title") or []
    authors = []
    for a in work.get("author", []) or []:
        name = " ".join(p for p in [a.get("given"), a.get("family")] if p).strip()
        if name:
            authors.append(name)
    doi = (work.get("DOI") or "").lower().strip()
    return {
        "title": titles[0] if titles else "",
        "year": _year_from(work),
        "doi": doi,
        "venue": venues[0] if venues else "",
        "authors": authors,
        "type": work.get("type", ""),
    }


def search(query: str, limit: int) -> dict:
    params = {"query.bibliographic": query, "rows": limit, "select": SELECT_FIELDS}
    mailto = _mailto()
    if mailto:
        params["mailto"] = mailto
    url = f"{CROSSREF_BASE}?{urllib.parse.urlencode(params)}"
    resp = _get(url)
    items = (resp.get("message") or {}).get("items") or []
    return {"raw": resp, "data": [_normalize_work(w) for w in items]}


def lookup_doi(doi: str) -> dict:
    url = f"{CROSSREF_BASE}/{urllib.parse.quote(doi)}"
    mailto = _mailto()
    if mailto:
        url += f"?mailto={urllib.parse.quote(mailto)}"
    resp = _get(url)
    msg = resp.get("message")
    data = [_normalize_work(msg)] if msg else []
    return {"raw": resp, "data": data}


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("--query", help="Paper title (bibliographic search)")
    p.add_argument("--doi", help="Look up an exact DOI instead of a title search")
    p.add_argument("--limit", type=int, default=DEFAULT_LIMIT,
                   help=f"Max hits (default {DEFAULT_LIMIT}, max {MAX_LIMIT})")
    p.add_argument("--raw", action="store_true", help="Print full Crossref JSON")
    args = p.parse_args()

    if not args.query and not args.doi:
        print("ERROR: provide --query or --doi", file=sys.stderr)
        return 2

    if args.doi:
        result = lookup_doi(args.doi.lower())
    else:
        result = search(args.query, max(1, min(MAX_LIMIT, args.limit)))

    if args.raw:
        json.dump(result["raw"], sys.stdout, indent=2, ensure_ascii=False)
        sys.stdout.write("\n")
        return 0 if result["data"] else 1

    data = result["data"]
    json.dump({"total": len(data), "data": data}, sys.stdout, indent=2, ensure_ascii=False)
    sys.stdout.write("\n")
    return 0 if data else 1


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
cross_verify.py — Cross-index corroboration for a verified citation pool.

Semantic Scholar verification (Levenshtein title match + cutoff + dedup) is the
pipeline's first gate. This script adds a second: every paper that S2 accepted
is re-checked against two *independent* scholarly indices — Crossref and
OpenAlex. A genuine paper turns up in all three with matching metadata; a
hallucinated or mis-attributed record typically does not. This is the practical
defense against the well-documented problem of fabricated citations leaking
into AI-assisted writing.

The script is a WARN gate, not a hard gate (mirrors validate_consistency.py):
it annotates the pool and writes a report, but exits non-zero only to draw the
host agent's attention to flagged entries — it does not block the pipeline.
The host agent reviews flagged citations and decides whether to drop them.

Confidence tiers written onto each paper's `cross_verification` field:
    high      corroborated by >=1 external index, no metadata conflicts
    medium    corroborated, but publication year disagrees beyond tolerance
    low       NOT found in Crossref or OpenAlex — phantom/hallucination risk
    conflict  a DOI present in the pool disagrees with the external index's DOI

Network etiquette: queries are throttled (--sleep, default 1.0s between papers)
and identify via a polite-pool email if PAPER_ORCHESTRA_MAILTO (or the
per-service CROSSREF_MAILTO / OPENALEX_MAILTO) is set. If an index is
unreachable the script degrades gracefully — it disables that index, notes it
in the report, and continues with whatever indices remain.

Usage:
    python cross_verify.py --pool workspace/citation_pool.json
    python cross_verify.py --pool workspace/citation_pool.json --inplace
    python cross_verify.py --pool workspace/citation_pool.json \\
        --out workspace/cross_verification_report.json \\
        --indices crossref,openalex --threshold 70 --year-tolerance 1 --sleep 1.0

Exit codes:
    0  every paper corroborated (all high/medium), no flags
    1  one or more papers flagged low/conflict, OR an index was unavailable (WARN)
    2  usage error / pool unreadable
"""
import argparse
import json
import os
import re
import sys
import time

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import crossref_client  # noqa: E402
import openalex_client  # noqa: E402

try:
    import Levenshtein  # noqa: E402

    def _ratio(a: str, b: str) -> int:
        return int(round(Levenshtein.ratio(a, b) * 100))
except ImportError:  # graceful fallback — stdlib only
    from difflib import SequenceMatcher

    def _ratio(a: str, b: str) -> int:
        return int(round(SequenceMatcher(None, a, b).ratio() * 100))


def _normalize(s: str) -> str:
    s = (s or "").lower().strip()
    s = re.sub(r"[^a-z0-9\s]", " ", s)
    return re.sub(r"\s+", " ", s).strip()


def title_ratio(a: str, b: str) -> int:
    return _ratio(_normalize(a), _normalize(b))


def _bare_doi(doi: str | None) -> str:
    if not doi:
        return ""
    d = doi.strip().lower()
    for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
        if d.startswith(prefix):
            d = d[len(prefix):]
    return d


def pool_doi(paper: dict) -> str:
    """Extract a DOI from a pool record, tolerating several shapes."""
    ext = paper.get("externalIds") or {}
    for k in ("DOI", "doi", "Doi"):
        if ext.get(k):
            return _bare_doi(ext[k])
    if paper.get("doi"):
        return _bare_doi(paper["doi"])
    return ""


def best_match(title: str, hits: list[dict]) -> dict | None:
    """Pick the external hit whose title best matches `title`."""
    best, best_r = None, -1
    for h in hits:
        r = title_ratio(title, h.get("title", ""))
        if r > best_r:
            best, best_r = h, r
    if best is None:
        return None
    out = dict(best)
    out["title_ratio"] = best_r
    return out


def _safe(fn, *args):
    """Call a client function, converting its SystemExit (network/HTTP error)
    into a None so cross_verify can degrade instead of dying."""
    try:
        return fn(*args)
    except SystemExit:
        return None


def check_index(module, paper: dict, threshold: int, strong_threshold: int, limit: int) -> dict:
    """Look one paper up in a single external index. Returns a per-index dict,
    or {"available": False} if the index could not be reached.

    Two match strengths are recorded:
      found  — title_ratio > threshold (lenient): "something like this exists"
      strong — title_ratio >= strong_threshold, or an exact DOI hit: "this is
               confidently the same paper". Only `strong` matches are trusted
               for year/DOI conflict downgrades, so a noisy near-title hit
               (e.g. a different paper with a similar name) cannot pollute the
               metadata-conflict checks.
    """
    title = paper.get("title", "")
    doi = pool_doi(paper)

    result = None
    via = "title"
    if doi:
        r = _safe(module.lookup_doi, doi)
        if r is None:
            return {"available": False}
        if r["data"]:
            result = r["data"][0]
            result["title_ratio"] = title_ratio(title, result.get("title", ""))
            via = "doi"
    if result is None:
        r = _safe(module.search, title, limit)
        if r is None:
            return {"available": False}
        result = best_match(title, r["data"])

    if result is None:
        return {"available": True, "found": False, "strong": False}

    tr = result.get("title_ratio", 0)
    found = tr > threshold or via == "doi"
    strong = tr >= strong_threshold or via == "doi"
    ext_doi = _bare_doi(result.get("doi"))
    doi_state = "n/a"
    if doi and ext_doi:
        doi_state = "agree" if doi == ext_doi else "conflict"
    return {
        "available": True,
        "found": bool(found),
        "strong": bool(strong),
        "via": via,
        "matched_title": result.get("title", ""),
        "title_ratio": tr,
        "matched_year": result.get("year"),
        "matched_doi": ext_doi,
        "doi_state": doi_state,
    }


def classify(paper: dict, per_index: dict, year_tolerance: int) -> dict:
    found_in = [name for name, r in per_index.items() if r.get("found")]
    # Only confident same-paper matches can trigger a metadata-conflict downgrade.
    doi_conflict = any(
        r.get("doi_state") == "conflict" and r.get("strong") for r in per_index.values()
    )

    year = paper.get("year")
    year_conflict = False
    if year is not None:
        for r in per_index.values():
            my = r.get("matched_year")
            if r.get("strong") and my is not None and abs(int(my) - int(year)) > year_tolerance:
                year_conflict = True

    notes = []
    if doi_conflict:
        confidence = "conflict"
        notes.append("DOI in pool disagrees with external index — verify this record")
    elif not found_in:
        confidence = "low"
        notes.append("not found in Crossref or OpenAlex — possible phantom citation")
    elif year_conflict:
        confidence = "medium"
        notes.append("corroborated, but publication year disagrees beyond tolerance")
    else:
        confidence = "high"

    return {
        "confidence": confidence,
        "indices_found": found_in,
        "doi_conflict": doi_conflict,
        "year_conflict": year_conflict,
        "per_index": per_index,
        "notes": notes,
    }


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("--pool", required=True, help="citation_pool.json path")
    p.add_argument("--out", default=None,
                   help="Report path (default: <pool dir>/cross_verification_report.json)")
    p.add_argument("--inplace", action="store_true",
                   help="Annotate each paper in the pool with its cross_verification result")
    p.add_argument("--indices", default="crossref,openalex",
                   help="Comma-separated external indices to check (default both)")
    p.add_argument("--threshold", type=int, default=70,
                   help="Levenshtein title-match threshold for corroboration "
                        "(default 70, matches S2 gate)")
    p.add_argument("--strong-threshold", type=int, default=90,
                   help="Stricter title-match threshold before an external record's "
                        "year/DOI is trusted for a conflict downgrade (default 90)")
    p.add_argument("--year-tolerance", type=int, default=1,
                   help="Allowed |year| difference before flagging (default 1)")
    p.add_argument("--limit", type=int, default=5, help="Max hits per title search")
    p.add_argument("--sleep", type=float, default=1.0,
                   help="Seconds to pause between papers (politeness, default 1.0)")
    args = p.parse_args()

    try:
        with open(args.pool) as f:
            pool = json.load(f)
    except (OSError, json.JSONDecodeError) as exc:
        print(f"ERROR: cannot read pool: {exc}", file=sys.stderr)
        return 2

    papers = pool.get("papers", [])
    if not papers:
        print("ERROR: pool['papers'] is empty or missing", file=sys.stderr)
        return 2

    modules = {"crossref": crossref_client, "openalex": openalex_client}
    requested = [x.strip() for x in args.indices.split(",") if x.strip()]
    unknown = [x for x in requested if x not in modules]
    if unknown:
        print(f"ERROR: unknown index/indices: {', '.join(unknown)}", file=sys.stderr)
        return 2

    disabled: set[str] = set()
    summary = {"high": 0, "medium": 0, "low": 0, "conflict": 0}
    flagged = []

    for i, paper in enumerate(papers):
        per_index = {}
        for name in requested:
            if name in disabled:
                continue
            res = check_index(modules[name], paper, args.threshold,
                              args.strong_threshold, args.limit)
            if res.get("available") is False:
                disabled.add(name)
                print(f"WARN: index '{name}' unreachable — disabling for the rest of this run.",
                      file=sys.stderr)
                continue
            per_index[name] = res

        verdict = classify(paper, per_index, args.year_tolerance)
        summary[verdict["confidence"]] += 1
        if verdict["confidence"] in ("low", "conflict"):
            flagged.append({
                "bibtex_key": paper.get("bibtex_key") or paper.get("key"),
                "title": paper.get("title"),
                "confidence": verdict["confidence"],
                "notes": verdict["notes"],
            })
        if args.inplace:
            paper["cross_verification"] = verdict

        # Throttle between papers (skip after the last one).
        if i < len(papers) - 1 and args.sleep > 0 and requested != list(disabled):
            time.sleep(args.sleep)

    report = {
        "total_papers": len(papers),
        "indices_checked": [x for x in requested if x not in disabled],
        "indices_unavailable": sorted(disabled),
        "summary": summary,
        "flagged": flagged,
    }

    out_path = args.out or os.path.join(os.path.dirname(os.path.abspath(args.pool)),
                                        "cross_verification_report.json")
    with open(out_path, "w") as f:
        json.dump(report, f, indent=2, ensure_ascii=False)

    if args.inplace:
        with open(args.pool, "w") as f:
            json.dump(pool, f, indent=2, ensure_ascii=False)

    # Human-readable summary to stdout.
    print(f"Cross-index verification ({', '.join(report['indices_checked']) or 'none'}):")
    print(f"  {len(papers)} papers  |  high={summary['high']} "
          f"medium={summary['medium']} low={summary['low']} conflict={summary['conflict']}")
    print(f"  report → {out_path}")
    if flagged:
        print(f"\nWARN: {len(flagged)} citation(s) need review:")
        for f_ in flagged:
            print(f"  [{f_['confidence'].upper()}] {f_['bibtex_key']}: {f_['title']}")
            for note in f_["notes"]:
                print(f"      → {note}")
        print("\nReview these before building refs.bib. Drop any you cannot corroborate.")

    if disabled:
        print(f"\nNote: {', '.join(sorted(disabled))} unavailable this run — "
              "corroboration is partial.", file=sys.stderr)

    return 1 if (flagged or disabled) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
dedupe_by_id.py — Deduplicate a verified citation pool using Semantic Scholar
unique paperId, with DOI / ArXiv / normalized-title fallbacks.

Implements the paper's Rule 4 (App. D.3): "gathered citations are
deduplicated using unique paper ID keys".

Also computes `min_cite_paper_count = floor(0.9 * len(papers))` for the
≥90% citation integration rule.

Usage:
    python dedupe_by_id.py --in raw_pool.json --out citation_pool.json [--cutoff 2024-10-01]
"""
import argparse
import json
import math
import re
import sys


def norm_title(t: str) -> str:
    return re.sub(r"[^a-z0-9]", "", t.lower())


def make_key(paper: dict) -> str:
    if paper.get("paperId"):
        return f"s2:{paper['paperId']}"
    ext = paper.get("externalIds") or {}
    if ext.get("DOI"):
        return f"doi:{ext['DOI'].lower()}"
    if ext.get("ArXiv"):
        # strip version suffix if any
        a = ext["ArXiv"].split("v")[0] if "v" in ext["ArXiv"][-3:] else ext["ArXiv"]
        return f"arxiv:{a.lower()}"
    title = paper.get("title", "")
    return f"title:{norm_title(title)}"


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--in", dest="inp", required=True, help="Raw verified pool JSON")
    p.add_argument("--out", required=True, help="Deduped citation_pool.json")
    p.add_argument("--cutoff", help="Cutoff date YYYY-MM-DD (recorded in output)")
    args = p.parse_args()

    with open(args.inp) as f:
        raw = json.load(f)

    candidates = raw.get("papers") or raw.get("candidates") or []
    if not candidates:
        print("ERROR: input has neither 'papers' nor 'candidates' key", file=sys.stderr)
        return 1

    by_key: dict[str, dict] = {}
    collisions: list[tuple[str, str]] = []
    for c in candidates:
        key = make_key(c)
        if key in by_key:
            existing = by_key[key]
            score_new = c.get("match_score", 0)
            score_old = existing.get("match_score", 0)
            if score_new > score_old:
                # merge discovered_for
                merged = existing.get("discovered_for", []) + c.get("discovered_for", [])
                c["discovered_for"] = list(dict.fromkeys(merged))  # preserve order, dedupe
                by_key[key] = c
            else:
                merged = existing.get("discovered_for", []) + c.get("discovered_for", [])
                existing["discovered_for"] = list(dict.fromkeys(merged))
            collisions.append((key, c.get("title", "")))
        else:
            by_key[key] = c

    deduped = list(by_key.values())
    n = len(deduped)
    min_cite = math.floor(0.9 * n)

    out = {
        "papers": deduped,
        "min_cite_paper_count": min_cite,
        "n_total": n,
        "n_collisions_merged": len(collisions),
    }
    if args.cutoff:
        out["cutoff_date"] = args.cutoff

    with open(args.out, "w") as f:
        json.dump(out, f, indent=2, ensure_ascii=False)

    print(f"OK: {len(candidates)} candidates → {n} unique papers")
    print(f"    {len(collisions)} duplicates merged")
    print(f"    min_cite_paper_count (≥90%): {min_cite}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
exa_search.py — Optional Exa (https://exa.ai) backend for the literature
review agent's Phase 1 (parallel candidate discovery) step.

Exa is a search engine optimized for finding academic papers and other
high-quality content. It is OPTIONAL — the literature-review-agent works
fine with any host coding agent's native web search tool. Use Exa only if:

  - Your host has no built-in web search (e.g., Aider, OpenCode, generic
    CLI agents).
  - You want a research-paper-focused search backend with better
    signal-to-noise than general web search.
  - You're running the pipeline in batch / non-interactive mode and want
    a deterministic, scriptable backend.

This helper reads EXA_API_KEY from the environment. The key is YOUR
responsibility to provide; this repo never commits one. Get a key at
https://dashboard.exa.ai/.

Usage:
    export EXA_API_KEY="your-key-here"
    python exa_search.py --query "Sparse attention long context" --num-results 15
    python exa_search.py --query "..." --raw                       # full JSON
    python exa_search.py --query "..." --discovered-for "related_work[2.1]"

Default output: JSON candidates in the literature-review-agent format, ready
to be merged into raw_candidates.json before Phase 2 verification.

Exit codes:
    0  query succeeded
    1  EXA_API_KEY missing, HTTP error, network error, or empty results
"""
import argparse
import json
import os
import sys
import urllib.error
import urllib.request

EXA_ENDPOINT = "https://api.exa.ai/search"
DEFAULT_NUM = 10
MAX_NUM = 20      # the user explicitly asked for a 10-20 range
SNIPPET_CAP = 1500


def search(query: str, num_results: int, category: str | None,
           highlight_max_chars: int) -> dict:
    api_key = os.environ.get("EXA_API_KEY")
    if not api_key:
        print(
            "ERROR: EXA_API_KEY environment variable not set.\n"
            "Get a key at https://dashboard.exa.ai/ and run:\n"
            '  export EXA_API_KEY="your-key-here"\n'
            "Then retry. The literature-review-agent also works without\n"
            "Exa — see references/discovery-pipeline.md for the default\n"
            "host-native web search path.",
            file=sys.stderr,
        )
        sys.exit(1)

    body: dict = {
        "query":      query,
        "numResults": num_results,
        "type":       "auto",
        "contents":   {"highlights": {"maxCharacters": highlight_max_chars}},
    }
    if category:
        body["category"] = category

    req = urllib.request.Request(
        EXA_ENDPOINT,
        data=json.dumps(body).encode("utf-8"),
        headers={
            "content-type": "application/json",
            "x-api-key":    api_key,
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            return json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as e:
        body_text = e.read().decode("utf-8", errors="replace")[:500]
        print(f"ERROR: Exa HTTP {e.code}: {body_text}", file=sys.stderr)
        sys.exit(1)
    except urllib.error.URLError as e:
        print(f"ERROR: Exa network error: {e.reason}", file=sys.stderr)
        sys.exit(1)


def normalize(exa_response: dict, discovered_for: list[str]) -> list[dict]:
    """Convert Exa results into the literature-review-agent candidate format."""
    candidates: list[dict] = []
    for r in exa_response.get("results", []):
        title = (r.get("title") or "").strip()
        url = r.get("url") or r.get("id") or ""
        highlights = r.get("highlights") or []
        snippet = " ".join(h.strip() for h in highlights)[:SNIPPET_CAP]
        candidates.append({
            "title":          title,
            "snippet":        snippet,
            "source_url":     url,
            "discovered_for": list(discovered_for),
            "_exa_id":             r.get("id"),
            "_exa_published_date": r.get("publishedDate"),
        })
    return candidates


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    p.add_argument("--query", required=True, help="Search query")
    p.add_argument("--num-results", type=int, default=DEFAULT_NUM,
                   help=f"Number of results to fetch "
                        f"(default {DEFAULT_NUM}, clamped to [1, {MAX_NUM}])")
    p.add_argument("--category", default="research paper",
                   help='Exa category filter (default "research paper"; '
                        'pass an empty string to disable)')
    p.add_argument("--highlight-chars", type=int, default=4000,
                   help="Max characters per highlight (default 4000)")
    p.add_argument("--discovered-for", default="intro",
                   help='Tag to attach to each candidate '
                        '(default "intro"). Use "related_work[2.1]" or '
                        'similar for cluster-specific queries so the '
                        'downstream citation_coverage gate can attribute '
                        'the citation to the right section.')
    p.add_argument("--raw", action="store_true",
                   help="Print the full Exa response JSON unmodified "
                        "instead of normalized candidates")
    args = p.parse_args()

    n = max(1, min(MAX_NUM, args.num_results))
    category = args.category or None

    response = search(args.query, n, category, args.highlight_chars)
    if not response.get("results"):
        print(f"WARN: Exa returned 0 results for query: {args.query!r}",
              file=sys.stderr)
        return 1

    if args.raw:
        json.dump(response, sys.stdout, indent=2, ensure_ascii=False)
    else:
        candidates = normalize(response, [args.discovered_for])
        json.dump({"candidates": candidates}, sys.stdout, indent=2,
                  ensure_ascii=False)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
levenshtein_match.py — Fuzzy title match for citation verification.

Implements the paper's Rule 1 (App. D.3): a candidate paper passes only if
its title's Levenshtein ratio against the Semantic Scholar hit's title is
strictly greater than 70.

Includes a substring-bypass safety net for short candidate titles (the
Linformer false-negative case): if the candidate is < 4 words and is
contained as a substring in the S2 hit's title, return 100.

Exit code is always 0; the integer ratio is printed to stdout. The caller
parses it and decides whether to discard.

Usage:
    python levenshtein_match.py --candidate "..." --found "..."
    python levenshtein_match.py --candidate "..." --found "..." --substring-bypass
"""
import argparse
import re
import sys

try:
    import Levenshtein
except ImportError:
    print("ERROR: python-Levenshtein required. Install with: pip install python-Levenshtein",
          file=sys.stderr)
    sys.exit(2)


def normalize(s: str) -> str:
    s = s.lower().strip()
    s = re.sub(r"[^a-z0-9\s]", " ", s)
    s = re.sub(r"\s+", " ", s)
    return s


def ratio(a: str, b: str, substring_bypass: bool = False) -> int:
    na, nb = normalize(a), normalize(b)
    r = int(round(Levenshtein.ratio(na, nb) * 100))
    if substring_bypass and len(na.split()) < 4:
        if na in nb:
            return max(r, 95)
    return r


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--candidate", required=True,
                   help="The original candidate title (from web search)")
    p.add_argument("--found", required=True,
                   help="The title returned by Semantic Scholar")
    p.add_argument("--substring-bypass", action="store_true",
                   help="Bump short-candidate substring matches to 95")
    p.add_argument("--threshold", type=int, default=70,
                   help="Print PASS/FAIL alongside the ratio (default 70)")
    args = p.parse_args()

    r = ratio(args.candidate, args.found, args.substring_bypass)
    verdict = "PASS" if r > args.threshold else "FAIL"
    print(f"{r} {verdict}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
openalex_client.py — OpenAlex API title/DOI lookup for cross-index citation
corroboration.

The third index (alongside Semantic Scholar and Crossref) used by
cross_verify.py. Triangulating across three independent scholarly indices is
the practical defense against hallucinated citations: a fabricated paper may
slip past one index but is unlikely to appear in all three with matching
metadata. See references/cross-index-verification.md.

No API key is required. OpenAlex offers a faster "polite pool" when you
identify yourself via email. Set one of:
    export OPENALEX_MAILTO="you@example.com"
    export PAPER_ORCHESTRA_MAILTO="you@example.com"   # shared fallback
The email is sent only as the `mailto` query parameter, per OpenAlex docs.

Usage:
    python openalex_client.py --query "Attention is All You Need"
    python openalex_client.py --doi 10.5555/3295222.3295349
    python openalex_client.py --query "BERT pre-training" --raw

Output (normalized): {"total": N, "data": [{title, year, doi, venue, authors}, ...]}

Exit codes:
    0  at least one result returned
    1  HTTP error, network error, or zero results
    2  usage error (bad arguments)
"""
import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request

OPENALEX_BASE = "https://api.openalex.org/works"
DEFAULT_LIMIT = 5
MAX_LIMIT = 25
_RETRY_SLEEP = 5


def _mailto() -> str:
    return (
        os.environ.get("OPENALEX_MAILTO", "").strip()
        or os.environ.get("PAPER_ORCHESTRA_MAILTO", "").strip()
    )


def _build_request(url: str) -> urllib.request.Request:
    return urllib.request.Request(
        url,
        headers={
            "Accept": "application/json",
            "User-Agent": "paper-orchestra/1.0 (https://github.com/Ar9av/paper-orchestra)",
        },
        method="GET",
    )


def _get(url: str, retries: int = 3) -> dict:
    for attempt in range(1, retries + 1):
        try:
            with urllib.request.urlopen(_build_request(url), timeout=30) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except urllib.error.HTTPError as exc:
            if exc.code == 404:
                return {"results": []}
            if exc.code == 429 and attempt < retries:
                print(f"WARN: OpenAlex rate-limited (429). Sleeping {_RETRY_SLEEP}s "
                      f"before retry {attempt + 1}/{retries}.", file=sys.stderr)
                time.sleep(_RETRY_SLEEP)
                continue
            if exc.code in (500, 502, 503) and attempt < retries:
                print(f"WARN: OpenAlex server error ({exc.code}). Retrying.", file=sys.stderr)
                time.sleep(10)
                continue
            print(f"ERROR: OpenAlex HTTP {exc.code}", file=sys.stderr)
            sys.exit(1)
        except urllib.error.URLError as exc:
            print(f"ERROR: Network error reaching OpenAlex: {exc.reason}", file=sys.stderr)
            sys.exit(1)
    sys.exit(1)


def _bare_doi(doi_url: str | None) -> str:
    if not doi_url:
        return ""
    d = doi_url.strip().lower()
    for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
        if d.startswith(prefix):
            d = d[len(prefix):]
    return d


def _normalize_work(work: dict) -> dict:
    venue = ""
    loc = work.get("primary_location") or {}
    src = loc.get("source") or {}
    if src.get("display_name"):
        venue = src["display_name"]
    authors = []
    for a in work.get("authorships", []) or []:
        name = (a.get("author") or {}).get("display_name")
        if name:
            authors.append(name)
    return {
        "title": work.get("title") or work.get("display_name") or "",
        "year": work.get("publication_year"),
        "doi": _bare_doi(work.get("doi")),
        "venue": venue,
        "authors": authors,
        "type": work.get("type", ""),
    }


def _with_mailto(params: dict) -> dict:
    mailto = _mailto()
    if mailto:
        params["mailto"] = mailto
    return params


def search(query: str, limit: int) -> dict:
    params = _with_mailto({"search": query, "per-page": limit})
    url = f"{OPENALEX_BASE}?{urllib.parse.urlencode(params)}"
    resp = _get(url)
    results = resp.get("results") or []
    return {"raw": resp, "data": [_normalize_work(w) for w in results]}


def lookup_doi(doi: str) -> dict:
    # OpenAlex resolves a single work via the /works/doi:<doi> path.
    path = f"{OPENALEX_BASE}/doi:{urllib.parse.quote(doi)}"
    mailto = _mailto()
    if mailto:
        path += f"?mailto={urllib.parse.quote(mailto)}"
    resp = _get(path)
    # A single-work response is the work object itself, not a results list.
    if "results" in resp:
        works = resp["results"]
    elif resp.get("id"):
        works = [resp]
    else:
        works = []
    return {"raw": resp, "data": [_normalize_work(w) for w in works]}


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("--query", help="Paper title (full-text search)")
    p.add_argument("--doi", help="Look up an exact DOI instead of a title search")
    p.add_argument("--limit", type=int, default=DEFAULT_LIMIT,
                   help=f"Max hits (default {DEFAULT_LIMIT}, max {MAX_LIMIT})")
    p.add_argument("--raw", action="store_true", help="Print full OpenAlex JSON")
    args = p.parse_args()

    if not args.query and not args.doi:
        print("ERROR: provide --query or --doi", file=sys.stderr)
        return 2

    if args.doi:
        result = lookup_doi(args.doi.lower())
    else:
        result = search(args.query, max(1, min(MAX_LIMIT, args.limit)))

    if args.raw:
        json.dump(result["raw"], sys.stdout, indent=2, ensure_ascii=False)
        sys.stdout.write("\n")
        return 0 if result["data"] else 1

    data = result["data"]
    json.dump({"total": len(data), "data": data}, sys.stdout, indent=2, ensure_ascii=False)
    sys.stdout.write("\n")
    return 0 if data else 1


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
pre_dedup_candidates.py — Deduplicate Phase 1 raw candidates by normalized
title before Phase 2 Semantic Scholar verification.

Multiple search queries in Phase 1 often return the same papers. Verifying
duplicates wastes S2 quota (1 QPS hard cap) and adds 30-40% unnecessary
wall-time. This script removes obvious duplicates — same paper found via
multiple queries — before the sequential verification loop begins.

Dedup strategy (in order of preference):
1. Exact arXiv ID match extracted from source URL or snippet.
2. Levenshtein ratio >= 92 on normalized titles (high threshold to avoid
   false collisions between similarly-named papers).

When two candidates are considered the same, we keep the one that appeared
earlier in the list and merge their `discovered_for` attribution tags so
the surviving entry is credited to all originating queries.

Usage:
    python pre_dedup_candidates.py \\
        --in workspace/raw_candidates.json \\
        --out workspace/deduped_candidates.json

Input JSON shape:
    {"candidates": [{"title": "...", "url": "...", "snippet": "...",
                     "discovered_for": ["intro.1"]}, ...]}
    OR a bare list.
"""
import argparse
import json
import re
import sys

ARXIV_RE = re.compile(r"arxiv\.org/(?:abs|pdf)/(\d{4}\.\d{4,5})", re.IGNORECASE)


def norm_title(t: str) -> str:
    t = re.sub(r"[^a-z0-9 ]", " ", t.lower())
    return " ".join(t.split())


def levenshtein_ratio(a: str, b: str) -> float:
    if not a and not b:
        return 100.0
    if not a or not b:
        return 0.0
    la, lb = len(a), len(b)
    if la < lb:
        a, b = b, a
        la, lb = lb, la
    prev = list(range(lb + 1))
    for i, ca in enumerate(a):
        curr = [i + 1]
        for j, cb in enumerate(b):
            cost = 0 if ca == cb else 1
            curr.append(min(prev[j + 1] + 1, curr[j] + 1, prev[j] + cost))
        prev = curr
    dist = prev[lb]
    return (1.0 - dist / max(la, lb)) * 100.0


def extract_arxiv_id(candidate: dict) -> str | None:
    for text in (candidate.get("url", ""), candidate.get("snippet", "")):
        m = ARXIV_RE.search(text)
        if m:
            return m.group(1)
    return None


def make_exact_key(candidate: dict) -> str:
    """Canonical key: arXiv ID if extractable, else normalized title."""
    aid = extract_arxiv_id(candidate)
    if aid:
        return f"arxiv:{aid}"
    return f"title:{norm_title(candidate.get('title', ''))}"


def merge_discovered_for(a: dict, b: dict) -> list:
    df_a = a.get("discovered_for") or []
    df_b = b.get("discovered_for") or []
    return list(dict.fromkeys(df_a + df_b))


def dedup(candidates: list[dict], title_ratio_threshold: float = 92.0) -> list[dict]:
    # Pass 1: exact key dedup (arXiv ID or identical normalized title)
    by_key: dict[str, dict] = {}
    for c in candidates:
        key = make_exact_key(c)
        if key in by_key:
            by_key[key]["discovered_for"] = merge_discovered_for(by_key[key], c)
        else:
            by_key[key] = dict(c)

    deduped = list(by_key.values())

    # Pass 2: fuzzy title dedup — O(n²) but n is ~50-100 candidates max
    normed = [norm_title(c.get("title", "")) for c in deduped]
    drop: set[int] = set()
    for i in range(len(deduped)):
        if i in drop:
            continue
        for j in range(i + 1, len(deduped)):
            if j in drop:
                continue
            if levenshtein_ratio(normed[i], normed[j]) >= title_ratio_threshold:
                deduped[i]["discovered_for"] = merge_discovered_for(deduped[i], deduped[j])
                drop.add(j)

    return [c for idx, c in enumerate(deduped) if idx not in drop]


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--in", dest="inp", required=True, help="Raw Phase 1 candidates JSON")
    p.add_argument("--out", required=True, help="Deduped candidates JSON")
    p.add_argument("--title-ratio", type=float, default=92.0,
                   help="Levenshtein ratio threshold for fuzzy title match (default: 92)")
    args = p.parse_args()

    with open(args.inp) as f:
        raw = json.load(f)

    if isinstance(raw, list):
        candidates = raw
    else:
        candidates = raw.get("candidates") or raw.get("papers") or []

    if not isinstance(candidates, list):
        print("ERROR: input must be a JSON array or object with 'candidates' key",
              file=sys.stderr)
        return 1

    before = len(candidates)
    result = dedup(candidates, title_ratio_threshold=args.title_ratio)
    after = len(result)
    removed = before - after

    out_obj = {
        "candidates": result,
        "n_before_dedup": before,
        "n_after_dedup": after,
        "n_removed": removed,
    }
    with open(args.out, "w") as f:
        json.dump(out_obj, f, indent=2, ensure_ascii=False)

    print(f"OK: {before} candidates → {after} unique ({removed} duplicates removed)")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
s2_cache.py — Persistent Semantic Scholar verification cache.

Problem: Phase 2 verification is throttled to 1 QPS. If a pipeline run
fails partway through (gate error, network timeout, interrupted session),
re-running wastes the full S2 wait time again on already-verified papers.

Solution: a flat JSON cache at workspace/cache/s2_cache.json. On a cache
HIT the script emits the stored response and exits 0 so the caller can skip
the live S2 request. On a cache MISS it exits 1. After a live request the
caller stores the result with --store.

The cache key is derived from the normalized query title (lowercase,
alphanumeric only) so minor whitespace differences still hit.

Usage:

  CHECK mode — exits 0 + prints JSON if cached, else exits 1:
    python s2_cache.py --cache workspace/cache/s2_cache.json \\
        --check "Attention Is All You Need"

  STORE mode — write a response into the cache:
    python s2_cache.py --cache workspace/cache/s2_cache.json \\
        --store "Attention Is All You Need" \\
        --response '{"paperId": "...", "title": "..."}'

  STATS mode — print cache size and hit rate summary:
    python s2_cache.py --cache workspace/cache/s2_cache.json --stats
"""
import argparse
import json
import os
import re
import sys


def norm_key(title: str) -> str:
    """Lowercase, alphanumeric-only cache key."""
    return re.sub(r"[^a-z0-9]", "", title.lower())


def load_cache(path: str) -> dict:
    if os.path.isfile(path):
        with open(path) as f:
            try:
                return json.load(f)
            except json.JSONDecodeError:
                return {}
    return {}


def save_cache(path: str, cache: dict) -> None:
    os.makedirs(os.path.dirname(os.path.abspath(path)), exist_ok=True)
    with open(path, "w") as f:
        json.dump(cache, f, indent=2, ensure_ascii=False)


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--cache", required=True, help="Path to cache JSON file")

    mode = p.add_mutually_exclusive_group(required=True)
    mode.add_argument("--check", metavar="TITLE",
                      help="Check for title; exit 0 + print JSON if found, else exit 1")
    mode.add_argument("--store", metavar="TITLE",
                      help="Store a response for TITLE (requires --response)")
    mode.add_argument("--stats", action="store_true",
                      help="Print cache statistics")

    p.add_argument("--response", metavar="JSON",
                   help="S2 response JSON to store (used with --store)")
    args = p.parse_args()

    cache = load_cache(args.cache)

    if args.stats:
        print(f"Cache file : {args.cache}")
        print(f"Entries    : {len(cache)}")
        if cache:
            print("Sample keys:", list(cache.keys())[:5])
        return 0

    if args.check:
        key = norm_key(args.check)
        if key in cache:
            print(json.dumps(cache[key]))
            return 0  # HIT
        return 1  # MISS

    # --store mode
    if not args.response:
        print("ERROR: --store requires --response", file=sys.stderr)
        return 2
    try:
        response = json.loads(args.response)
    except json.JSONDecodeError as e:
        print(f"ERROR: invalid JSON in --response: {e}", file=sys.stderr)
        return 2

    key = norm_key(args.store)
    cache[key] = response
    save_cache(args.cache, cache)
    print(f"OK: cached '{args.store}' → key '{key}' ({len(cache)} total entries)")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
s2_search.py — Semantic Scholar title-search helper for Phase 2 verification.

Queries the Semantic Scholar Graph API for a paper by title and returns the
top candidate hits as JSON.  Used by the literature-review-agent to verify
each candidate from Phase 1 before adding it to citation_pool.json.

API key (optional):
    If SEMANTIC_SCHOLAR_API_KEY is set in the environment the key is forwarded
    via the ``x-api-key`` header, which raises the rate limit from ~100 req/5 min
    (unauthenticated) to 1 req/s sustained with higher burst headroom.
    If the variable is absent the script falls back to the public unauthenticated
    endpoint — the pipeline works fine without a key; just keep to ≤1 QPS.

    Get a free key at: https://api.semanticscholar.org/
    Then export it once before running the pipeline:
        export SEMANTIC_SCHOLAR_API_KEY="your-key-here"

Usage:
    # check for key and search
    python s2_search.py --query "Attention is All You Need"

    # request more hits and extra fields
    python s2_search.py --query "BERT pre-training" --limit 10 \\
        --fields title,abstract,year,authors,venue,externalIds,citationCount

    # pretty-print raw S2 JSON
    python s2_search.py --query "GPT-4 technical report" --raw

Exit codes:
    0  at least one result returned
    1  HTTP error, network error, or zero results
    2  usage error (bad arguments)
"""
import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request

S2_BASE = "https://api.semanticscholar.org/graph/v1"
DEFAULT_FIELDS = "title,abstract,year,authors,venue,externalIds"
DEFAULT_LIMIT = 5
MAX_LIMIT = 100
_RETRY_SLEEP = 5   # seconds to wait after a 429 before retrying


def _build_headers() -> dict:
    headers = {"Accept": "application/json"}
    api_key = os.environ.get("SEMANTIC_SCHOLAR_API_KEY", "").strip()
    if api_key:
        headers["x-api-key"] = api_key
    return headers


def search(query: str, limit: int, fields: str, retries: int = 3) -> dict:
    """
    Call /paper/search and return the parsed JSON response.

    Raises SystemExit on unrecoverable errors so the caller (or CLI) gets a
    clean non-zero exit code.
    """
    params = urllib.parse.urlencode({
        "query":  query,
        "limit":  limit,
        "fields": fields,
    })
    url = f"{S2_BASE}/paper/search?{params}"
    headers = _build_headers()

    for attempt in range(1, retries + 1):
        req = urllib.request.Request(url, headers=headers, method="GET")
        try:
            with urllib.request.urlopen(req, timeout=30) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except urllib.error.HTTPError as exc:
            if exc.code == 429:
                if attempt < retries:
                    print(
                        f"WARN: S2 rate-limited (429). Sleeping {_RETRY_SLEEP}s "
                        f"before retry {attempt + 1}/{retries}.",
                        file=sys.stderr,
                    )
                    time.sleep(_RETRY_SLEEP)
                    continue
                print(
                    "ERROR: S2 rate-limited (429) and retries exhausted.\n"
                    "Tip: set SEMANTIC_SCHOLAR_API_KEY to get a higher rate limit.\n"
                    "     See https://api.semanticscholar.org/ for a free key.",
                    file=sys.stderr,
                )
                sys.exit(1)
            if exc.code == 404:
                # not found — return an empty result set (caller handles this)
                return {"total": 0, "data": []}
            if exc.code in (500, 502, 503):
                if attempt < retries:
                    print(
                        f"WARN: S2 server error ({exc.code}). Sleeping 30s before "
                        f"retry {attempt + 1}/{retries}.",
                        file=sys.stderr,
                    )
                    time.sleep(30)
                    continue
                print(
                    f"ERROR: S2 server error ({exc.code}) after {retries} attempts.",
                    file=sys.stderr,
                )
                sys.exit(1)
            body = exc.read().decode("utf-8", errors="replace")[:400]
            print(f"ERROR: S2 HTTP {exc.code}: {body}", file=sys.stderr)
            sys.exit(1)
        except urllib.error.URLError as exc:
            print(f"ERROR: Network error reaching Semantic Scholar: {exc.reason}",
                  file=sys.stderr)
            sys.exit(1)

    # should never reach here
    sys.exit(1)


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    p.add_argument(
        "--query", required=True,
        help="Paper title (or search query) to look up on Semantic Scholar",
    )
    p.add_argument(
        "--limit", type=int, default=DEFAULT_LIMIT,
        help=f"Max hits to return (default {DEFAULT_LIMIT}, max {MAX_LIMIT})",
    )
    p.add_argument(
        "--fields", default=DEFAULT_FIELDS,
        help=f"Comma-separated S2 fields to request (default: {DEFAULT_FIELDS})",
    )
    p.add_argument(
        "--raw", action="store_true",
        help="Print the full S2 JSON response unmodified instead of normalized output",
    )
    p.add_argument(
        "--check-key", action="store_true",
        help="Print whether SEMANTIC_SCHOLAR_API_KEY is set and exit (no network call)",
    )
    args = p.parse_args()

    if args.check_key:
        key = os.environ.get("SEMANTIC_SCHOLAR_API_KEY", "").strip()
        if key:
            masked = key[:4] + "..." + key[-4:] if len(key) > 8 else "****"
            print(f"SEMANTIC_SCHOLAR_API_KEY is set ({masked}). "
                  "Authenticated mode: higher rate limits.")
        else:
            print(
                "SEMANTIC_SCHOLAR_API_KEY is NOT set. "
                "Unauthenticated mode: ~100 req/5 min, keep to ≤1 QPS.\n"
                "To enable higher rate limits:\n"
                "  1. Get a free key at https://api.semanticscholar.org/\n"
                '  2. export SEMANTIC_SCHOLAR_API_KEY="your-key-here"'
            )
        return 0

    limit = max(1, min(MAX_LIMIT, args.limit))
    response = search(args.query, limit, args.fields)

    if args.raw:
        json.dump(response, sys.stdout, indent=2, ensure_ascii=False)
        sys.stdout.write("\n")
        return 0

    data = response.get("data") or []
    if not data:
        print(
            f"WARN: Semantic Scholar returned 0 results for query: {args.query!r}",
            file=sys.stderr,
        )
        json.dump({"total": 0, "data": []}, sys.stdout, indent=2)
        sys.stdout.write("\n")
        return 1

    # Emit normalized output (subset of fields used by pipeline)
    out = {
        "total": response.get("total", len(data)),
        "authenticated": bool(os.environ.get("SEMANTIC_SCHOLAR_API_KEY", "").strip()),
        "data": data,
    }
    json.dump(out, sys.stdout, indent=2, ensure_ascii=False)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
r"""
sync_keys.py — Synchronize citation keys in a .tex file with the canonical
bibtex_key values stored in citation_pool.json.

Problem: The Literature Review Agent writes cite keys in its own format
(e.g. 'lewis2020rag'), while bibtex_format.py generates canonical keys from
author + year + first-significant-title-word (e.g. 'lewis2020retrievalaugmented').
After running bibtex_format.py these two sources are out of sync, causing the
citation_coverage gate to fail (it looks for \cite{canonical_key} in the .tex).

This script reads the 'key' -> 'bibtex_key' mapping from citation_pool.json
and performs a targeted substitution inside \cite{}, \citep{}, \citet{}
commands in the target .tex file. It handles multi-key citations like
\cite{a,b,c} correctly.

Run this immediately after bibtex_format.py, before Step 4 (Section Writing).

Usage:
    python sync_keys.py \
        --pool workspace/citation_pool.json \
        --tex  workspace/drafts/intro_relwork.tex \
        --inplace

    # Without --inplace: prints updated content to stdout (safe preview mode).
"""
import argparse
import json
import re
import sys

# Matches \cite, \citep, \citet, \citealt, \citealp, \citeauthor, \citeyear,
# starred variants like \cite*, and the optional [prenote][postnote] args.
CITE_RE = re.compile(
    r"(\\cite[a-zA-Z*]*)"          # command
    r"(?:\[[^\]]*\])*"             # optional bracket args (prenote/postnote)
    r"\{([^}]+)\}"                 # required brace arg with keys
)


def build_key_map(pool: dict) -> dict[str, str]:
    """Return {agent_key: bibtex_key} for every paper where they differ."""
    key_map: dict[str, str] = {}
    for paper in pool.get("papers", []):
        old = paper.get("key")
        new = paper.get("bibtex_key")
        if old and new and old != new:
            key_map[old] = new
    return key_map


def replace_keys(content: str, key_map: dict[str, str]) -> tuple[str, int]:
    if not key_map:
        return content, 0

    n_replaced = 0

    def replacer(m: re.Match) -> str:
        nonlocal n_replaced
        cmd = m.group(1)
        keys_str = m.group(2)
        keys = [k.strip() for k in keys_str.split(",")]
        new_keys: list[str] = []
        for k in keys:
            if k in key_map:
                new_keys.append(key_map[k])
                n_replaced += 1
            else:
                new_keys.append(k)
        # Reconstruct original bracket args (they were consumed by the regex
        # but we don't need to preserve them specially — re-emit as matched)
        full_match = m.group(0)
        # Rebuild: command + everything between command and { + new keys
        bracket_part = full_match[len(cmd):full_match.index("{")]
        return f"{cmd}{bracket_part}{{{', '.join(new_keys)}}}"

    updated = CITE_RE.sub(replacer, content)
    return updated, n_replaced


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--pool", required=True, help="citation_pool.json")
    p.add_argument("--tex", required=True, help="Target .tex file to update")
    p.add_argument("--inplace", action="store_true",
                   help="Overwrite --tex in place (default: print to stdout)")
    args = p.parse_args()

    with open(args.pool) as f:
        pool = json.load(f)
    key_map = build_key_map(pool)

    if not key_map:
        print("OK: no key differences in citation_pool.json — nothing to sync")
        return 0

    print(f"Key map ({len(key_map)} substitutions):")
    for old, new in key_map.items():
        print(f"  {old} → {new}")

    with open(args.tex) as f:
        content = f.read()

    updated, n = replace_keys(content, key_map)

    if args.inplace:
        with open(args.tex, "w") as f:
            f.write(updated)
        print(f"OK: {n} citation key(s) updated in {args.tex}")
    else:
        sys.stdout.write(updated)
        print(f"\n# sync_keys: {n} substitution(s) would be made", file=sys.stderr)

    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
tavily_search.py — Optional Tavily (https://tavily.com) backend for the
literature review agent's Phase 1 (parallel candidate discovery) step.

Tavily is a search API designed for LLMs, enabling AI applications to
access real-time web data. It is OPTIONAL — the literature-review-agent
works fine with any host coding agent's native web search tool. Use
Tavily only if:

  - Your host has no built-in web search (e.g., Aider, OpenCode, generic
    CLI agents).
  - You want an LLM-optimized search backend with high relevance scoring.
  - You're running the pipeline in batch / non-interactive mode and want
    a deterministic, scriptable backend.

This helper reads TAVILY_API_KEY from the environment. The key is YOUR
responsibility to provide; this repo never commits one. Get a key at
https://app.tavily.com (1,000 free credits/month).

Usage:
    export TAVILY_API_KEY="tvly-your-key-here"
    python tavily_search.py --query "Sparse attention long context" --num-results 15
    python tavily_search.py --query "..." --raw                       # full JSON
    python tavily_search.py --query "..." --discovered-for "related_work[2.1]"

Default output: JSON candidates in the literature-review-agent format, ready
to be merged into raw_candidates.json before Phase 2 verification.

Exit codes:
    0  query succeeded
    1  TAVILY_API_KEY missing, HTTP error, network error, or empty results
"""
import argparse
import json
import os
import sys
import urllib.error
import urllib.request

TAVILY_ENDPOINT = "https://api.tavily.com/search"
DEFAULT_NUM = 10
MAX_NUM = 20
SNIPPET_CAP = 1500

# Academic domains to boost research-paper results
ACADEMIC_DOMAINS = [
    "arxiv.org",
    "scholar.google.com",
    "semanticscholar.org",
    "aclanthology.org",
    "openreview.net",
]


def search(query: str, num_results: int, topic: str,
           include_domains: list[str] | None) -> dict:
    api_key = os.environ.get("TAVILY_API_KEY")
    if not api_key:
        print(
            "ERROR: TAVILY_API_KEY environment variable not set.\n"
            "Get a key at https://app.tavily.com and run:\n"
            '  export TAVILY_API_KEY="tvly-your-key-here"\n'
            "Then retry. The literature-review-agent also works without\n"
            "Tavily — see references/discovery-pipeline.md for the default\n"
            "host-native web search path.",
            file=sys.stderr,
        )
        sys.exit(1)

    body: dict = {
        "query":        query,
        "max_results":  num_results,
        "search_depth": "advanced",
        "topic":        topic,
    }
    if include_domains:
        body["include_domains"] = include_domains

    req = urllib.request.Request(
        TAVILY_ENDPOINT,
        data=json.dumps(body).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}",
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            return json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as e:
        body_text = e.read().decode("utf-8", errors="replace")[:500]
        print(f"ERROR: Tavily HTTP {e.code}: {body_text}", file=sys.stderr)
        sys.exit(1)
    except urllib.error.URLError as e:
        print(f"ERROR: Tavily network error: {e.reason}", file=sys.stderr)
        sys.exit(1)


def normalize(tavily_response: dict, discovered_for: list[str]) -> list[dict]:
    """Convert Tavily results into the literature-review-agent candidate format."""
    candidates: list[dict] = []
    for r in tavily_response.get("results", []):
        title = (r.get("title") or "").strip()
        url = r.get("url") or ""
        content = (r.get("content") or "").strip()
        snippet = content[:SNIPPET_CAP]
        candidates.append({
            "title":          title,
            "snippet":        snippet,
            "source_url":     url,
            "discovered_for": list(discovered_for),
            "_tavily_score":  r.get("score"),
        })
    return candidates


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    p.add_argument("--query", required=True, help="Search query")
    p.add_argument("--num-results", type=int, default=DEFAULT_NUM,
                   help=f"Number of results to fetch "
                        f"(default {DEFAULT_NUM}, clamped to [1, {MAX_NUM}])")
    p.add_argument("--topic", default="general",
                   choices=["general", "news"],
                   help='Tavily topic filter (default "general"; '
                        'use "news" for recent results)')
    p.add_argument("--academic", action="store_true",
                   help="Restrict results to academic domains "
                        "(arxiv.org, scholar.google.com, etc.)")
    p.add_argument("--discovered-for", default="intro",
                   help='Tag to attach to each candidate '
                        '(default "intro"). Use "related_work[2.1]" or '
                        'similar for cluster-specific queries so the '
                        'downstream citation_coverage gate can attribute '
                        'the citation to the right section.')
    p.add_argument("--raw", action="store_true",
                   help="Print the full Tavily response JSON unmodified "
                        "instead of normalized candidates")
    args = p.parse_args()

    n = max(1, min(MAX_NUM, args.num_results))
    include_domains = ACADEMIC_DOMAINS if args.academic else None

    response = search(args.query, n, args.topic, include_domains)
    if not response.get("results"):
        print(f"WARN: Tavily returned 0 results for query: {args.query!r}",
              file=sys.stderr)
        return 1

    if args.raw:
        json.dump(response, sys.stdout, indent=2, ensure_ascii=False)
    else:
        candidates = normalize(response, [args.discovered_for])
        json.dump({"candidates": candidates}, sys.stdout, indent=2,
                  ensure_ascii=False)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
validate_pool.py — Validate and auto-fix citation_pool.json before it is
passed to bibtex_format.py or the Section Writing Agent.

Catches the two most common schema errors produced by the Literature Review
Agent and fixes them in place with --fix.

Error 1 — Authors as plain strings (WRONG format for bibtex_format.py):
    WRONG:   "authors": ["Alice Smith", "Bob Jones"]
    CORRECT: "authors": [{"name": "Alice Smith"}, {"name": "Bob Jones"}]

Error 2 — Missing required fields (title, year). These cause bibtex_format.py
    to emit incomplete entries. Reported as errors, not auto-fixed.

Also checks that the pool has the top-level keys that downstream scripts
expect: "papers", "min_cite_paper_count".

Exit codes:
    0  Pool is valid (or was fully fixed with --fix)
    1  Unrecoverable errors remain (missing required fields, no papers)

Usage:
    python validate_pool.py --pool workspace/citation_pool.json
    python validate_pool.py --pool workspace/citation_pool.json --fix
"""
import argparse
import json
import sys

REQUIRED_PAPER_FIELDS = ["title", "year"]
RECOMMENDED_PAPER_FIELDS = ["paperId", "abstract", "venue", "authors"]
REQUIRED_TOP_FIELDS = ["papers", "min_cite_paper_count"]


def validate_and_fix(pool: dict, fix: bool) -> tuple[list[str], list[str], int]:
    """
    Returns (errors, warnings, n_fixed).
    If fix=True, mutates pool in place where possible.
    """
    errors: list[str] = []
    warnings: list[str] = []
    n_fixed = 0

    # Top-level structure
    for field in REQUIRED_TOP_FIELDS:
        if field not in pool:
            warnings.append(f"top-level field '{field}' missing — was dedupe_by_id.py run?")

    papers = pool.get("papers", [])
    if not papers:
        errors.append("pool['papers'] is empty or missing")
        return errors, warnings, n_fixed

    for i, paper in enumerate(papers):
        label = paper.get("title") or f"paper #{i}"

        # --- Authors format check ---
        authors = paper.get("authors")
        if authors is not None:
            if not isinstance(authors, list):
                errors.append(f"[{label}] 'authors' must be a list, got {type(authors).__name__}")
            elif authors:
                if isinstance(authors[0], str):
                    if fix:
                        paper["authors"] = [{"name": a} for a in authors]
                        n_fixed += 1
                    else:
                        errors.append(
                            f"[{label}] authors are plain strings "
                            f"(e.g. \"{authors[0]}\") — run with --fix to auto-convert"
                        )
                elif not isinstance(authors[0], dict):
                    errors.append(
                        f"[{label}] authors[0] is {type(authors[0]).__name__}, "
                        f"expected dict with 'name' key"
                    )

        # --- Required fields ---
        for field in REQUIRED_PAPER_FIELDS:
            if not paper.get(field):
                errors.append(f"[{label}] missing required field '{field}'")

        # --- Recommended fields ---
        for field in RECOMMENDED_PAPER_FIELDS:
            if not paper.get(field):
                warnings.append(f"[{label}] missing recommended field '{field}'")

    return errors, warnings, n_fixed


def main() -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("--pool", required=True, help="citation_pool.json path")
    p.add_argument("--fix", action="store_true",
                   help="Auto-fix recoverable errors (authors format) and write back")
    p.add_argument("--quiet", action="store_true",
                   help="Suppress warnings, only show errors")
    args = p.parse_args()

    with open(args.pool) as f:
        pool = json.load(f)

    errors, warnings, n_fixed = validate_and_fix(pool, fix=args.fix)

    if not args.quiet:
        for w in warnings:
            print(f"WARN: {w}")

    had_errors = bool(errors)
    for e in errors:
        print(f"ERROR: {e}", file=sys.stderr)

    if had_errors and not args.fix:
        print(
            "\nTip: re-run with --fix to auto-correct recoverable issues (authors format).",
            file=sys.stderr,
        )
        return 1

    if n_fixed > 0:
        with open(args.pool, "w") as f:
            json.dump(pool, f, indent=2, ensure_ascii=False)
        print(f"OK: {n_fixed} paper(s) auto-fixed and written back to {args.pool}")

    n = len(pool.get("papers", []))
    if not had_errors and n_fixed == 0:
        print(f"OK: {n} papers validated — no errors")
    elif n_fixed > 0 and not errors:
        print(f"OK: {n} papers validated after auto-fix")

    return 0 if (not errors or (args.fix and n_fixed > 0 and not [e for e in errors if "missing required" in e])) else 1


if __name__ == "__main__":
    sys.exit(main())