---
name: cite-check
description: Verify that citations actually exist and that the claims they support are faithful to the cited source. Runs deterministic existence checks (Crossref / OpenAlex / Semantic Scholar / arXiv) plus a claim-faithfulness pass via the alterlab-citation-verifier skill.
argument-hint: [path to .bib / manuscript, or pasted references]
disable-model-invocation: true
allowed-tools: Read Write Edit Bash WebFetch WebSearch
---

**Cite-check** the citations in: $ARGUMENTS

Use the `alterlab-citation-verifier` skill. Decide each verdict from the script
output and retrieved records, not from whether a reference looks plausible or
familiar: a model's memory shares the training data that produces fabricated
references, so it cannot confirm them.

Steps:
1. **Collect** — Parse the references from $ARGUMENTS (a `.bib` file, a manuscript
   path, or a pasted list). If nothing was given, ask for the references or file.
2. **Existence check** — Run `scripts/verify_citations.py` over the references. It
   resolves each one against Crossref, OpenAlex, Semantic Scholar, and arXiv (no API
   key required; optional `OPENALEX_API_KEY` / `S2_API_KEY` raise rate limits),
   checks DOI registration at doi.org, matches title and authors with the difflib
   `SequenceMatcher` ratio (≥ 0.70), and maps each entry to the TF / PAC / IH / PH
   taxonomy.
3. **Retraction screen** — Surface every `RETRACTED` flag (Crossref `updated-by` /
   `update-to` notices, including Retraction Watch data, or OpenAlex
   `is_retracted`) and any expression of concern.
4. **Claim faithfulness** — Where a sentence is tied to a citation, run
   `scripts/claim_faithfulness.py` on the (claim, DOI) pair; report `contradict` as
   SH and escalate `unsupported` claims (the abstract does not establish them) to the
   `--tier llm` judge or full-text reading.
5. **Report** — A per-citation table: verdict code (`verified` / `TF` / `PAC` / `IH`
   / `PH` / `SH` / `unverified`), flags, the source that confirmed it, and the matched
   DOI/ID. Summarize how many of N citations could not be verified and why
   (`source_status`).

If the network or a source is unavailable, say so explicitly: the script returns
`unverified` rather than a pass. For those entries, run the WebSearch fallback from
the SKILL.md "Graceful Degradation and the Fallback" section (three distinct
queries); anything still not found is reported as TF (NOT_FOUND), never as
"difficult to verify".

{
  "skill": "alterlab-citation-verifier",
  "evals": [
    {
      "id": "verify-real-citation-passes",
      "prompt": "Before I add this to my reference list, can you confirm this paper actually exists and the metadata is right? Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.",
      "expected_output": "Invokes alterlab-citation-verifier in existence-check mode: runs scripts/verify_citations.py to resolve the citation against Crossref / OpenAlex / Semantic Scholar / arXiv, confirms title/author match above the 0.70 difflib SequenceMatcher ratio threshold, checks for retractions (Crossref updated-by / update-to, including Retraction Watch data, and OpenAlex is_retracted), and returns a VERIFIED verdict with the matched canonical record (DOI/arXiv ID and the source database that confirmed it). Does not fabricate a verdict from model memory.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "behavior", "value": "Reaches a VERIFIED verdict for a genuinely real paper and reports which authoritative source (Crossref/OpenAlex/Semantic Scholar/arXiv) confirmed it, rather than asserting existence from model memory." }
      ]
    },
    {
      "id": "catch-total-fabrication",
      "prompt": "My co-author handed me these references for our manuscript and I'm worried some were generated by ChatGPT. Can you check whether each one actually exists? Start with: Henderson, M. K., & Albrecht, J. P. (2021). Transformer architectures for low-resource morphological parsing in agglutinative languages. Journal of Computational Linguistics Research, 14(3), 220-241.",
      "expected_output": "Invokes alterlab-citation-verifier to run an existence check on the supplied reference: queries Crossref / OpenAlex / Semantic Scholar / arXiv via scripts/verify_citations.py, finds no resolvable DOI and no matching title+author record across all sources, and returns a NOT_FOUND / Total Fabrication (TF) verdict. Refuses any 'difficult to verify' gray-zone classification and flags the entry for removal or replacement.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "TF" },
        { "type": "behavior", "value": "Classifies an unresolvable, non-existent reference as Total Fabrication (NOT_FOUND), never as 'difficult to verify', and does NOT invent plausible-looking metadata to make it pass." }
      ]
    },
    {
      "id": "catch-identifier-hijacking",
      "prompt": "Can you double-check this citation? The DOI looks fine but something feels off. Park, S., & Oktay, F. (2020). Curriculum design for hybrid studio teaching. Computers & Education, 158, 103998. https://doi.org/10.1016/j.compedu.2020.103998",
      "expected_output": "Invokes alterlab-citation-verifier in identifier-resolution mode: scripts/verify_citations.py resolves the DOI against Crossref, finds that it points to a real but completely different article (different title and authors than cited), and returns an Identifier Hijacking (IH) verdict. Reports the true record the DOI resolves to so the user can correct or remove the borrowed identifier.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "IH" },
        { "type": "behavior", "value": "Resolves the DOI and detects that it points to a real but unrelated paper, returning Identifier Hijacking and surfacing the true record the identifier resolves to." }
      ]
    },
    {
      "id": "frankenstein-real-paper-wrong-claim",
      "prompt": "I want to write: 'Randomized controlled trials have shown that smartphone note-taking apps improve undergraduate exam scores by 31% compared to handwriting (Mueller & Oppenheimer, 2014).' Is that citation supported by the source? Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological Science, 25(6), 1159-1168.",
      "expected_output": "Invokes alterlab-citation-verifier in claim-faithfulness mode: scripts/verify_citations.py confirms the Mueller & Oppenheimer (2014) paper genuinely EXISTS (real DOI, real authors), so this is NOT a fabrication; then scripts/claim_faithfulness.py compares the user's claim against the source and finds a Frankenstein mismatch — the real paper reports the OPPOSITE (longhand handwriting outperforms laptop note-taking, with no 31% smartphone-app RCT result). Returns a Semantic Hallucination (SH) FAIL: real source, unsupported/contradicted claim, fabricated statistic. Distinguishes 'the citation is real' from 'the citation supports the claim'.",
      "files": [
        {
          "name": "claim_under_review.md",
          "content": "## Draft sentence under review\n\n> Randomized controlled trials have shown that smartphone note-taking apps improve undergraduate exam scores by 31% compared to handwriting (Mueller & Oppenheimer, 2014).\n\n## Attached source (real)\n\nMueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. *Psychological Science, 25*(6), 1159-1168. https://doi.org/10.1177/0956797614524581\n\nSource abstract (paraphrased): Across three studies, students who took notes by hand (longhand) performed better on conceptual questions than students who took notes on laptops. The studies concern laptop-vs-longhand note-taking; they do not test smartphone apps and report no 31% exam-score figure.\n"
        }
      ],
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "SH" },
        { "type": "behavior", "value": "Confirms the paper is REAL (passes existence check) yet returns a Semantic Hallucination FAIL because the source does not support — in fact contradicts — the attached claim, and explicitly separates 'citation exists' from 'citation supports the claim'. Flags the 31% figure as fabricated." }
      ]
    },
    {
      "id": "batch-bibliography-audit",
      "prompt": "Here's the full reference list from my literature review draft (24 entries, APA 7.0). Run a citation-existence audit over all of them and tell me which ones I can't trust before I submit.",
      "expected_output": "Invokes alterlab-citation-verifier in batch / bibliography-audit mode: parses all 24 references, runs scripts/verify_citations.py against Crossref / OpenAlex / Semantic Scholar / arXiv for each (title+author difflib SequenceMatcher matching, DOI/arXiv resolution, retraction flag from Crossref updated-by (including Retraction Watch data) / OpenAlex is_retracted), and returns a per-entry verdict table mapping each problem to the TF/PAC/IH/PH/SH taxonomy with a VERIFIED/NOT_FOUND/MISMATCH status. Reports entries the script could not check as unverified (naming the failing sources), runs the WebSearch fallback for them rather than silently passing them, and surfaces the retracted Wakefield (1998) entry.",
      "files": ["fixtures/bibliography_24_entries.md"],
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "behavior", "value": "Produces a per-entry verdict over the whole reference list and documents fallback to WebSearch when an API/network is unavailable, instead of silently marking unverifiable entries as passing." }
      ]
    },
    {
      "id": "near-miss-alterlab-link-health",
      "prompt": "Several of the URLs in my reference list are returning 404s and some redirect to parked domains. Can you check every hyperlink in my bibliography and tell me which ones are dead so I can swap in archived copies?",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-link-health. The user wants HTTP reachability / dead-link and redirect detection (link rot) across URLs, not verification that the cited works exist or that claims are faithful to their sources. alterlab-citation-verifier validates citation existence and claim faithfulness against bibliographic databases; checking whether a hyperlink resolves to a live page is alterlab-link-health's job.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-link-health" }
      ]
    },
    {
      "id": "near-miss-alterlab-paper-reviewer",
      "prompt": "Here is a colleague's finished manuscript. Give me a structured peer review with section-by-section comments, an assessment of the methodology and contribution, and an accept/revise/reject verdict.",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-paper-reviewer. The user wants a holistic scholarly quality judgment of an entire manuscript (novelty, rigor, methodology, contribution) with an editorial decision, not a factual existence/faithfulness check of its citations. alterlab-citation-verifier only performs deterministic citation-existence and claim-faithfulness verification and makes no quality judgments.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-paper-reviewer" }
      ]
    },
    {
      "id": "near-miss-alterlab-citation-mgmt",
      "prompt": "Find me five recent peer-reviewed papers on retrieval-augmented generation in higher education and give me clean BibTeX entries for each so I can drop them into my .bib file.",
      "expected_output": "Does NOT invoke this skill; defers to alterlab-citation-mgmt. The user wants to discover papers and generate BibTeX metadata for new sources, not to audit an existing bibliography for fabricated, hijacked, or retracted references. alterlab-citation-verifier checks references the user already has.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "alterlab-citation-mgmt" }
      ]
    }
  ]
}

---
scenario: A real, existing paper is cited to support a claim it does not make (Frankenstein / Semantic Hallucination)
mode: existence check (verify_citations.py) + claim faithfulness (claim_faithfulness.py)
demonstrates: Why "the citation is real" and "the citation supports the claim" are two separate verdicts, and how the SH verdict is grounded in the retrieved source
taxonomy_hit: SH (Semantic Hallucination)
---

# Caught-Hallucination Walk-Through: A Real Paper, the Wrong Claim

This is the failure mode that fools careful humans and shared-training-data LLMs alike: the
cited paper is **completely real** — correct authors, correct year, resolvable DOI, thousands
of citations — and yet it **does not support the sentence it is attached to.** The reference
passes every existence check, so a reviewer skimming the bibliography sees nothing wrong. The
distortion lives in the gap between the claim and the source.

`alterlab-citation-verifier` is built around that gap. It runs **two** deterministic checks
and keeps their verdicts separate:

1. `scripts/verify_citations.py` — *does this work exist?* (Crossref / OpenAlex / Semantic
   Scholar / arXiv resolution, title+author similarity >= 0.70 via difflib `SequenceMatcher`,
   DOI/arXiv-ID resolution, retraction flag from Crossref `updated-by` / `update-to` —
   including Retraction Watch data — and OpenAlex `is_retracted`).
2. `scripts/claim_faithfulness.py` — *does this work support the claim?* (compares the user's
   sentence against the cited work's retrieved abstract and returns `support` /
   `contradict` / `unsupported`; `contradict` maps to SH).

A reference can PASS check 1 and FAIL check 2. That is exactly what happens here.

---

## The Request

> **Researcher:** I want to write this sentence in my methods discussion:
>
> > "Randomized controlled trials have shown that smartphone note-taking apps improve
> > undergraduate exam scores by 31% compared to handwriting (Mueller & Oppenheimer, 2014)."
>
> Is that citation supported by the source?
>
> Source: Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard:
> Advantages of longhand over laptop note taking. *Psychological Science, 25*(6), 1159-1168.

A reviewer would recognize Mueller & Oppenheimer (2014) instantly — it is a famous,
frequently cited study. That recognition is the trap: "I know this paper is real" silently
becomes "so the citation must be fine."

---

## Step 1 — Existence Check (`verify_citations.py`)

```
$ echo "Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological Science, 25(6), 1159-1168. https://doi.org/10.1177/0956797614524581" \
  | uv run python skills/core/alterlab-citation-verifier/scripts/verify_citations.py -

# Summary of the JSON entry:
  verdict:        verified   (severity NONE)
  detail:         Matched in crossref, openalex, semanticscholar.
  source_status:  crossref=record  openalex=record  semanticscholar=record  arxiv=no_record
  title_ratio:    1.0   (Crossref stores the subtitle separately; the script rejoins it)
  author_overlap: 1.0   (Mueller, Oppenheimer)
  retracted:      false (no retraction notice in Crossref updated-by / OpenAlex is_retracted)
```

So far, **everything is green.** The paper is real, the DOI resolves, the metadata matches,
it is not retracted. If the verifier stopped here — the way a bibliography-only check does —
the citation would pass and the fabricated claim would sail into the manuscript.

This is the key design point: **VERIFIED existence is necessary but not sufficient.** The
verifier does not return a final PASS on existence alone when the user has supplied a *claim*.

---

## Step 2 — Claim Faithfulness (`claim_faithfulness.py`)

The verifier now compares the **claim** against what the source actually says. The default
heuristic tier runs first:

```
$ uv run python skills/core/alterlab-citation-verifier/scripts/claim_faithfulness.py \
    --claim "Randomized controlled trials have shown that smartphone note-taking apps improve undergraduate exam scores by 31% compared to handwriting" \
    --doi 10.1177/0956797614524581

verdict=unsupported  taxonomy=UNVERIFIABLE  conf=0.20  tier=heuristic  source=crossref
  why: Lexical overlap 0% is below the support threshold and no clear contradiction signal.
       Heuristic abstains: the abstract does not establish the claim (this is non-coverage,
       not refutation). Use the llm tier or full text to resolve.
```

That abstention is the heuristic doing its job: word overlap cannot tell a swapped intervention
or an inverted effect from mere silence, so it refuses to guess `support`. An `unsupported`
claim still cannot be cited as-is, so the check escalates to the LLM-judge tier, which reads
the same retrieved abstract (the output below is illustrative; the rationale wording varies
by run):

```
$ uv run python skills/core/alterlab-citation-verifier/scripts/claim_faithfulness.py \
    --claim "Randomized controlled trials have shown that smartphone note-taking apps improve undergraduate exam scores by 31% compared to handwriting" \
    --doi 10.1177/0956797614524581 --tier llm

verdict=contradict  taxonomy=MAJOR_DISTORTION/SH  tier=llm  source=crossref
  why: The abstract reports three laptop-vs-longhand studies in which laptop note-takers did
       worse on conceptual questions; it tests no smartphone apps and reports no 31% figure.
```

Reading the claim against the abstract:

| Claim asserts | Abstract reports |
|---------------|------------------|
| smartphone note-taking **apps** | **laptop** note-taking |
| randomized controlled trials of apps | "three studies" of laptop vs. longhand note-taking |
| digital note-taking **improves** exam scores | **longhand outperforms** laptops on conceptual questions |
| a **31%** improvement | no such figure |

> Note on tool honesty: both tiers see only the abstract (`abstract_only: true`), so no one
> mistakes an abstract-level check for full-text verification. The heuristic never cries
> "contradiction" on silence; here the abstract actively asserts the opposite direction
> (handwriting wins), which is what makes `contradict` — and therefore SH — the right call.

The claim is a **Frankenstein**: a real, recognizable citation stitched onto a statistic the
source never reports, in a direction the source actively contradicts. The source is about
**longhand vs. laptop** note-taking and finds **handwriting wins**; the claim invents a
**smartphone-app RCT** that boosts scores **by 31%**. Three separate distortions, one fake
number, zero support.

---

## The Verdict

```markdown
# Citation Verification Report

## Citation
Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard:
Advantages of longhand over laptop note taking. Psychological Science, 25(6), 1159-1168.

## Existence:  ✅ VERIFIED   (real paper, DOI 10.1177/0956797614524581, not retracted)
## Faithfulness: ❌ FAIL — SH (Semantic Hallucination)

## Why it fails despite being a real paper
The reference EXISTS, but it does NOT support the claim attached to it. The source studies
longhand vs. laptop note-taking and finds handwriting superior; the claim asserts a 31%
exam-score gain from smartphone APPS via RCTs. The intervention is wrong, the effect
direction is inverted, and the 31% figure is fabricated (not present in the source).

## Taxonomy
SH — Semantic Hallucination: citation resolves but does not support the claim it is attached to.

## Recommended action
Do NOT cite Mueller & Oppenheimer (2014) for this sentence. Either:
  (a) restate the claim to match the source ("longhand note-taking improved performance on
      conceptual test questions relative to laptop note-taking (Mueller & Oppenheimer, 2014)"),
      removing the smartphone-app framing and the fabricated 31%; or
  (b) if you genuinely need a smartphone-app exam-score result, find and verify a source that
      actually reports it — this paper is not it.

## Audit trail
- verify_citations.py: verified in Crossref/OpenAlex/Semantic Scholar; title ratio 1.00; no retraction flag.
- claim_faithfulness.py: heuristic tier abstained (unsupported, 0% overlap); llm tier returned contradict —
  intervention mismatch, effect-direction contradiction, "31%" absent from the abstract (abstract_only: true).
```

---

## Why This Matters

### 1. "Real citation" and "supported claim" are different questions
Bibliography-only checks — and most human reviewers — answer only the first. The dangerous
hallucinations of 2024-2026 increasingly pass the first check: the model attaches a famous,
real paper to a claim that paper never made. Separating the two verdicts is the whole point.

### 2. The SH verdict is grounded in the retrieved abstract, not in memory
The verifier never relies on the model "knowing" what Mueller & Oppenheimer found. It
resolves the DOI, fetches the cited work's abstract (Crossref/OpenAlex), and compares that
retrieved text against the claim. Same-source hallucination (the verifier and the writer
sharing training data) cannot launder a false claim through, because the comparison is
against the retrieved abstract, not recall. (The abstract is the ceiling of what the tool
sees; the verdict is stamped `abstract_only: true` so it is never mistaken for full-text
adjudication, and an abstract that is merely silent yields `unsupported`, not a false pass.)

### 3. The fabricated statistic is the tell
The "31%" appears nowhere in the source abstract. Invented precision — a specific percentage,
effect size, or sample count with no counterpart in the cited work — is one of the strongest
signals of a semantic hallucination. The verifier surfaces it rather than rounding past it.

### 4. Graceful degradation keeps the gate honest offline
When the network is unavailable, `verify_citations.py` emits an `unverified` verdict per entry
(repo verdict `UNVERIFIED`) with manual-check instructions, rather than silently returning a
pass; the agent then falls back to a WebSearch pass. An entry it genuinely cannot check is
reported as `unverified`, never quietly marked `verified` — the same zero-gray-zone discipline
the integrity agent enforces.

### 5. This is the case prompt-only taxonomies miss
A prose taxonomy can *describe* SH, but it cannot *retrieve and compare* the source. Wiring
the integrity and bibliography agents to call these two scripts turns the taxonomy from a
description into an executable gate — which is the entire reason this skill exists.

#!/usr/bin/env python3
"""claim_faithfulness.py — score claim support against a cited work's abstract.

Given one or more (claim, DOI) pairs, this tool fetches the cited work's abstract
from Crossref (primary) or OpenAlex (fallback, with inverted-index reconstruction)
and scores whether the abstract **supports**, **contradicts**, or leaves the claim
**unsupported**. It is the Phase-E ("Semantic Hallucination", SH) half of the
AlterLab citation gate; existence/metadata checks (TF/PAC/IH/PH) live in
``verify_citations.py``.

Two scoring tiers are available:

* ``heuristic`` (default, deterministic, offline-friendly once the abstract is
  fetched): a transparent keyword-overlap + lightweight negation/antonym signal.
  It is **honest about its limits** — it cannot do real entailment, so its
  default verdict is ``unsupported`` (abstain) rather than a false ``support``.
* ``llm`` (optional): an LLM-judge tier that asks the model configured via the
  ``ALTERLAB_MODEL`` convention (see ``skills/core/shared/model_env.md``) to
  classify support/contradict/unsupported. NEVER hardcodes a model id. Falls back
  to the heuristic verdict (clearly flagged) if the ``claude`` CLI is unavailable.

The abstract is the ceiling of what this tool can see. An abstract that does not
mention the claim is **not** evidence of contradiction — only of non-coverage.
The tool therefore returns ``unsupported`` (not ``contradict``) in that case, and
the verdict object records ``abstract_only: true`` so downstream consumers never
mistake an abstract-level pass for full-text verification.

Verdict vocabulary (this tool) and its mapping to the corpus claim taxonomy
(``integrity_verification_agent.md`` Phase E / SH; see ``docs/integrity.md``):

    support      -> VERIFIED            (claim consistent with abstract)
    contradict   -> MAJOR_DISTORTION/SH (abstract asserts the opposite)
    unsupported  -> UNVERIFIABLE (abstain) — abstract does not establish the claim

Usage
-----
    # single pair
    uv run python claim_faithfulness.py \
        --claim "Transformers outperform RNNs on translation" \
        --doi 10.5555/3295222.3295349

    # batch from JSON ([{"claim": "...", "doi": "..."}, ...]) -> JSON report
    uv run python claim_faithfulness.py --input pairs.json --json

    # LLM-judge tier (uses $ALTERLAB_MODEL, default reviewed 2026-09-23)
    uv run python claim_faithfulness.py --input pairs.json --tier llm --json

    # offline self-test (no network) on a toy pair
    uv run python claim_faithfulness.py --self-test

Dependencies: Python standard library only (urllib/json/re). No ``requests``,
no third-party packages — runs under ``uv run python`` with no extra installs.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import subprocess
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import asdict, dataclass, field
from typing import Any, Dict, List, Optional, Tuple

# --------------------------------------------------------------------------- #
# Model convention — default reviewed 2026-09-23; override via ALTERLAB_MODEL.  #
# See skills/core/shared/model_env.md before changing the default.             #
# This is the ONLY place a model id literal may appear (rule 1 of model_env).  #
# --------------------------------------------------------------------------- #
DEFAULT_MODEL = "claude-opus-5-5"


def alterlab_model() -> str:
    """Return $ALTERLAB_MODEL if set/non-empty, else the dated default."""
    return os.environ.get("ALTERLAB_MODEL") or DEFAULT_MODEL


# Crossref routes requests that carry a mailto to its "polite" pool (higher limits).
# OpenAlex has ignored mailto since Feb 2026, but the single-DOI lookups used here
# are free without an API key, so no key is needed for abstract retrieval.
CONTACT_EMAIL = os.environ.get("ALTERLAB_CONTACT_EMAIL", "alterlab.ieu@gmail.com")
USER_AGENT = f"AlterLab-CitationVerifier/2.0 (mailto:{CONTACT_EMAIL})"

VERDICTS = ("support", "contradict", "unsupported")

# Verdict -> corpus taxonomy mapping (see module docstring / docs/integrity.md).
TAXONOMY_MAP = {
    "support": "VERIFIED",
    "contradict": "MAJOR_DISTORTION/SH",
    "unsupported": "UNVERIFIABLE",
}

# Lightweight English stopword set for keyword extraction. Intentionally small
# and transparent — this is a heuristic, not an NLP pipeline.
STOPWORDS = frozenset(
    """
    a an and are as at be been being but by for from had has have he her his
    in into is it its of on or our that the their them they this to was were
    what when where which who will with we you your i s t can may might must
    not no nor than then these those such over under more most less least
    using used use based study paper results show shows showed found find
    """.split()
)

# Cheap negation / polarity signals for the heuristic contradiction check.
NEGATION_TOKENS = frozenset(
    "not no never none cannot without fails failed fail lacks lack absent".split()
)
# Antonym-ish stems that frequently flip a claim's truth value in abstracts.
# Matched at the STEM level (see ``_stem``) so inflections like
# "increase/increases/increasing" and "decrease/decreasing" all collapse and
# compare correctly. Keep these as lowercase stems, not full inflected forms.
ANTONYM_PAIRS = [
    ("increas", "decreas"), ("higher", "lower"), ("more", "fewer"),
    ("positive", "negative"), ("significant", "insignificant"),
    ("effective", "ineffective"), ("improv", "worsen"),
    ("outperform", "underperform"), ("better", "worse"), ("gain", "loss"),
    ("rise", "fall"), ("present", "absent"), ("support", "refut"),
    ("confirm", "refut"), ("reduc", "rais"),
]


def _stem(token: str) -> str:
    """Crude suffix stripper so antonym matching tolerates inflection.

    Deliberately simple and transparent — this is a heuristic, not a lemmatizer.
    """
    for suf in ("ing", "ed", "es", "s", "er", "ly"):
        if token.endswith(suf) and len(token) - len(suf) >= 4:
            return token[: -len(suf)]
    return token


# --------------------------------------------------------------------------- #
# Data model                                                                   #
# --------------------------------------------------------------------------- #
@dataclass
class FaithfulnessResult:
    claim: str
    doi: str
    verdict: str  # one of VERDICTS
    taxonomy: str  # corpus mapping (see TAXONOMY_MAP)
    confidence: float  # 0.0-1.0; heuristic confidence is deliberately capped
    tier: str  # "heuristic" | "llm" | "llm->heuristic-fallback"
    rationale: str
    abstract_only: bool = True  # abstracts only — never claim full-text coverage
    abstract_found: bool = True
    source: str = ""  # "crossref" | "openalex" | ""
    signals: Dict[str, Any] = field(default_factory=dict)


# --------------------------------------------------------------------------- #
# Abstract fetching (Crossref primary, OpenAlex fallback)                       #
# --------------------------------------------------------------------------- #
def _http_get_json(url: str, timeout: int = 20) -> Optional[dict]:
    req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            if resp.status != 200:
                return None
            return json.loads(resp.read().decode("utf-8", errors="replace"))
    except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError,
            json.JSONDecodeError, ValueError):
        return None


_TAG_RE = re.compile(r"<[^>]+>")


def _strip_jats(text: str) -> str:
    """Crossref abstracts are JATS-XML; strip tags to plain text."""
    return re.sub(r"\s+", " ", _TAG_RE.sub(" ", text)).strip()


def _reconstruct_openalex_abstract(inv: Dict[str, List[int]]) -> str:
    """Rebuild plain text from OpenAlex abstract_inverted_index."""
    if not inv:
        return ""
    positions: List[Tuple[int, str]] = []
    for word, idxs in inv.items():
        for i in idxs:
            positions.append((i, word))
    positions.sort(key=lambda p: p[0])
    return " ".join(word for _, word in positions)


def fetch_abstract(doi: str, timeout: int = 20) -> Tuple[Optional[str], str]:
    """Return (abstract_text_or_None, source). Crossref first, then OpenAlex."""
    doi_clean = doi.strip()
    doi_clean = re.sub(r"^(https?://(dx\.)?doi\.org/|doi:)", "", doi_clean, flags=re.I)
    enc = urllib.parse.quote(doi_clean, safe="")

    # 1) Crossref
    cr = _http_get_json(
        f"https://api.crossref.org/works/{enc}?mailto={urllib.parse.quote(CONTACT_EMAIL)}",
        timeout=timeout,
    )
    if cr:
        abstract = (cr.get("message") or {}).get("abstract")
        if abstract:
            text = _strip_jats(abstract)
            if text:
                return text, "crossref"

    # 2) OpenAlex (reconstruct from inverted index)
    oa = _http_get_json(
        f"https://api.openalex.org/works/doi:{enc}?mailto={urllib.parse.quote(CONTACT_EMAIL)}",
        timeout=timeout,
    )
    if oa:
        inv = oa.get("abstract_inverted_index")
        if inv:
            text = _reconstruct_openalex_abstract(inv)
            if text:
                return text, "openalex"

    return None, ""


# --------------------------------------------------------------------------- #
# Tier 1 — heuristic (keyword overlap + entailment-lite). Honest about limits.  #
# --------------------------------------------------------------------------- #
def _tokenize(text: str) -> List[str]:
    return re.findall(r"[a-z][a-z0-9\-]+", text.lower())


def _content_terms(text: str) -> List[str]:
    return [t for t in _tokenize(text) if t not in STOPWORDS and len(t) > 2]


def _negation_count(text: str) -> int:
    toks = set(_tokenize(text))
    return sum(1 for n in NEGATION_TOKENS if n in toks)


def heuristic_score(claim: str, abstract: str) -> FaithfulnessResult:
    """Transparent keyword-overlap + lightweight polarity heuristic.

    Design principle: this tier CANNOT do real entailment, so it is biased toward
    abstaining (``unsupported``) rather than emitting a false ``support``. It only
    asserts ``support`` on strong lexical overlap, and ``contradict`` only when an
    explicit antonym/negation-polarity flip co-occurs with topical overlap.

    Known blind spots (documented honestly, not worked around): bag-of-words
    overlap is invariant to word order, so SUBJECT-OBJECT ROLE REVERSAL ("A
    outperforms B" vs "B outperforms A") looks identical and will read as
    ``support``; it also misses numeric mismatches, scope/qualifier changes, and
    sarcasm. Use the ``llm`` tier or full-text verification when role/number
    fidelity matters. The heuristic's job is to cheaply triage, not to adjudicate.
    """
    claim_terms = set(_content_terms(claim))
    abs_terms = set(_content_terms(abstract))

    if not claim_terms:
        return FaithfulnessResult(
            claim=claim, doi="", verdict="unsupported",
            taxonomy=TAXONOMY_MAP["unsupported"], confidence=0.0,
            tier="heuristic",
            rationale="Claim contains no content-bearing terms to match.",
            signals={"overlap": 0.0},
        )

    overlap_terms = claim_terms & abs_terms
    overlap = len(overlap_terms) / len(claim_terms)

    # Polarity / antonym-flip signal: does the abstract assert the opposite of a
    # directional word in the claim, given topical overlap? Compared on stemmed
    # token sets so inflections (increase/increases/increasing) line up.
    claim_stems = {_stem(t) for t in _tokenize(claim)}
    abs_stems = {_stem(t) for t in _tokenize(abstract)}
    flips: List[str] = []
    for a, b in ANTONYM_PAIRS:
        ca, cb = a in claim_stems, b in claim_stems
        aa, ab = a in abs_stems, b in abs_stems
        # Claim says one pole, abstract says the other and NOT the claim's pole.
        if (ca and ab and not aa) or (cb and aa and not ab):
            flips.append(f"{a}/{b}")

    neg_delta = abs(_negation_count(claim) - _negation_count(abstract))

    signals = {
        "overlap": round(overlap, 3),
        "overlap_terms": sorted(overlap_terms),
        "n_claim_terms": len(claim_terms),
        "antonym_flips": flips,
        "negation_delta": neg_delta,
    }

    # Decision logic — deliberately conservative.
    if overlap >= 0.30 and flips:
        verdict = "contradict"
        # confidence capped: abstract-level lexical flip is suggestive, not proof
        conf = min(0.55, 0.30 + 0.10 * len(flips))
        rationale = (
            f"Topical overlap ({overlap:.0%}) with polarity flip on "
            f"{', '.join(flips)} — abstract appears to assert the opposite. "
            "Heuristic only; confirm against full text."
        )
    elif overlap >= 0.60 and not flips and neg_delta == 0:
        verdict = "support"
        conf = min(0.60, 0.30 + overlap * 0.40)  # cap heuristic 'support' at 0.60
        rationale = (
            f"Strong lexical overlap ({overlap:.0%}) and no polarity conflict. "
            "Heuristic lexical match only — NOT semantic entailment; verify key "
            "numbers/qualifiers against full text."
        )
    else:
        verdict = "unsupported"
        conf = 0.20 if overlap < 0.30 else 0.30
        if overlap >= 0.60 and neg_delta != 0:
            why = (
                f"Lexical overlap is high ({overlap:.0%}) but a negation "
                f"mismatch (delta={neg_delta}) makes naive 'support' unsafe"
            )
        elif overlap >= 0.60:
            why = (
                f"Lexical overlap is high ({overlap:.0%}) but a polarity/"
                "negation signal prevents a confident 'support'"
            )
        else:
            why = f"Lexical overlap {overlap:.0%} is below the support threshold"
        rationale = (
            f"{why} and no clear contradiction signal. Heuristic abstains: the "
            "abstract does not establish the claim (this is non-coverage, not "
            "refutation). Use the llm tier or full text to resolve."
        )

    return FaithfulnessResult(
        claim=claim, doi="", verdict=verdict, taxonomy=TAXONOMY_MAP[verdict],
        confidence=round(conf, 3), tier="heuristic", rationale=rationale,
        signals=signals,
    )


# --------------------------------------------------------------------------- #
# Tier 2 — optional LLM judge (uses ALTERLAB_MODEL; never hardcodes an id)      #
# --------------------------------------------------------------------------- #
_LLM_PROMPT = """You are a strict citation-faithfulness judge.

Decide whether the cited work's ABSTRACT supports, contradicts, or leaves the \
CLAIM unsupported. You can see ONLY the abstract, not the full text. If the \
abstract does not contain enough information to establish the claim, answer \
"unsupported" (this is non-coverage, NOT contradiction). Answer "contradict" \
only if the abstract asserts something logically incompatible with the claim.

Respond with a single minified JSON object and nothing else:
{{"verdict": "support|contradict|unsupported", "confidence": 0.0-1.0, \
"rationale": "one sentence"}}

CLAIM:
{claim}

ABSTRACT:
{abstract}
"""


def _claude_cli_available() -> bool:
    try:
        subprocess.run(
            ["claude", "--version"],
            capture_output=True, timeout=15, check=False,
        )
        return True
    except (FileNotFoundError, subprocess.SubprocessError):
        return False


def _extract_json_object(text: str) -> Optional[dict]:
    m = re.search(r"\{.*\}", text, re.DOTALL)
    if not m:
        return None
    try:
        return json.loads(m.group(0))
    except json.JSONDecodeError:
        return None


def llm_score(claim: str, abstract: str, timeout: int = 120) -> FaithfulnessResult:
    """LLM-judge tier. Falls back to the heuristic (flagged) if CLI is missing."""
    fallback = heuristic_score(claim, abstract)

    if not _claude_cli_available():
        fallback.tier = "llm->heuristic-fallback"
        fallback.rationale = (
            "LLM tier requested but `claude` CLI unavailable; "
            "degraded to heuristic. " + fallback.rationale
        )
        return fallback

    prompt = _LLM_PROMPT.format(claim=claim, abstract=abstract[:6000])
    try:
        proc = subprocess.run(
            ["claude", "--model", alterlab_model(), "-p", prompt],
            capture_output=True, text=True, timeout=timeout, check=False,
        )
    except subprocess.SubprocessError as exc:
        fallback.tier = "llm->heuristic-fallback"
        fallback.rationale = (
            f"LLM tier failed ({type(exc).__name__}); degraded to heuristic. "
            + fallback.rationale
        )
        return fallback

    parsed = _extract_json_object(proc.stdout or "")
    if not parsed or parsed.get("verdict") not in VERDICTS:
        fallback.tier = "llm->heuristic-fallback"
        fallback.rationale = (
            "LLM tier returned unparseable output; degraded to heuristic. "
            + fallback.rationale
        )
        return fallback

    verdict = parsed["verdict"]
    try:
        conf = float(parsed.get("confidence", 0.5))
    except (TypeError, ValueError):
        conf = 0.5
    conf = max(0.0, min(1.0, conf))

    return FaithfulnessResult(
        claim=claim, doi="", verdict=verdict, taxonomy=TAXONOMY_MAP[verdict],
        confidence=round(conf, 3), tier="llm",
        rationale=str(parsed.get("rationale", "")).strip()[:400]
        or "(no rationale returned)",
        signals={"model": alterlab_model()},
    )


# --------------------------------------------------------------------------- #
# Orchestration                                                                #
# --------------------------------------------------------------------------- #
def score_pair(
    claim: str, doi: str, tier: str = "heuristic",
    abstract_override: Optional[str] = None, timeout: int = 20,
) -> FaithfulnessResult:
    """Score one (claim, DOI) pair. ``abstract_override`` skips network (tests)."""
    if abstract_override is not None:
        abstract, source = abstract_override, "override"
    else:
        abstract, source = fetch_abstract(doi, timeout=timeout)

    if not abstract:
        return FaithfulnessResult(
            claim=claim, doi=doi, verdict="unsupported",
            taxonomy=TAXONOMY_MAP["unsupported"], confidence=0.0,
            tier=tier, abstract_found=False, source=source,
            rationale=(
                "No abstract available from Crossref or OpenAlex for this DOI. "
                "Cannot assess faithfulness — treat as UNVERIFIABLE_ACCESS and "
                "fall back to full-text / WebSearch verification."
            ),
            signals={},
        )

    if tier == "llm":
        result = llm_score(claim, abstract, timeout=max(timeout, 120))
    else:
        result = heuristic_score(claim, abstract)

    result.doi = doi
    result.source = source
    return result


def score_pairs(
    pairs: List[Dict[str, str]], tier: str = "heuristic", timeout: int = 20,
) -> List[FaithfulnessResult]:
    out: List[FaithfulnessResult] = []
    for p in pairs:
        out.append(score_pair(p.get("claim", ""), p.get("doi", ""), tier=tier, timeout=timeout))
    return out


# --------------------------------------------------------------------------- #
# Self-test (offline, deterministic; no network)                               #
# --------------------------------------------------------------------------- #
_TOY_ABSTRACT = (
    "We introduce the Transformer, a model architecture based solely on "
    "attention mechanisms, dispensing with recurrence entirely. Experiments on "
    "machine translation tasks show these models outperform recurrent networks "
    "in quality while being more parallelizable and requiring less time to "
    "train, decreasing the training time needed relative to prior approaches."
)


def run_self_test() -> int:
    cases = [
        # claim, abstract_override, expected_verdict
        (
            "Transformers outperform recurrent networks on machine translation",
            _TOY_ABSTRACT, "support",
        ),
        # Polarity flip the bag-of-words heuristic CAN catch: the claim asserts
        # an *increase* in training time; the abstract reports a *decrease*.
        (
            "The Transformer architecture increases the training time required "
            "for machine translation models relative to recurrent networks",
            _TOY_ABSTRACT, "contradict",
        ),
        (
            "Transformers reduce carbon emissions in data centers by 40 percent",
            _TOY_ABSTRACT, "unsupported",
        ),
    ]
    ok = True
    for claim, abstract, expected in cases:
        r = score_pair(claim, "10.0000/self-test", tier="heuristic",
                       abstract_override=abstract)
        status = "PASS" if r.verdict == expected else "FAIL"
        if r.verdict != expected:
            ok = False
        print(f"[{status}] expected={expected:<12} got={r.verdict:<12} "
              f"conf={r.confidence:.2f} :: {claim}")
        print(f"        rationale: {r.rationale}")
    print("\nSELF-TEST:", "OK" if ok else "FAILED")
    return 0 if ok else 1


# --------------------------------------------------------------------------- #
# CLI                                                                          #
# --------------------------------------------------------------------------- #
def _load_pairs(args: argparse.Namespace) -> List[Dict[str, str]]:
    if args.input:
        with open(args.input, "r", encoding="utf-8") as fh:
            data = json.load(fh)
        if not isinstance(data, list):
            raise SystemExit("--input JSON must be a list of {claim, doi} objects")
        return data
    if args.claim and args.doi:
        return [{"claim": args.claim, "doi": args.doi}]
    raise SystemExit("Provide --claim and --doi, or --input pairs.json, or --self-test")


def main(argv: Optional[List[str]] = None) -> int:
    ap = argparse.ArgumentParser(
        description="Score (claim, DOI) faithfulness against the cited abstract.",
    )
    ap.add_argument("--claim", help="A single claim string.")
    ap.add_argument("--doi", help="DOI for the single claim.")
    ap.add_argument("--input", help="JSON file: [{\"claim\":..., \"doi\":...}, ...]")
    ap.add_argument("--tier", choices=["heuristic", "llm"], default="heuristic",
                    help="Scoring tier (default: heuristic).")
    ap.add_argument("--timeout", type=int, default=20, help="HTTP timeout seconds.")
    ap.add_argument("--json", action="store_true", help="Emit JSON report.")
    ap.add_argument("--self-test", action="store_true",
                    help="Run offline deterministic self-test and exit.")
    args = ap.parse_args(argv)

    if args.self_test:
        return run_self_test()

    pairs = _load_pairs(args)
    results = score_pairs(pairs, tier=args.tier, timeout=args.timeout)

    if args.json:
        print(json.dumps([asdict(r) for r in results], indent=2, ensure_ascii=False))
    else:
        for r in results:
            print(f"verdict={r.verdict}  taxonomy={r.taxonomy}  "
                  f"conf={r.confidence:.2f}  tier={r.tier}  source={r.source}")
            print(f"  claim: {r.claim}")
            print(f"  doi:   {r.doi}")
            print(f"  why:   {r.rationale}\n")

    # Exit non-zero if any pair contradicts — useful as a CI gate.
    return 1 if any(r.verdict == "contradict" for r in results) else 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""verify_citations.py — Existence-verify a bibliography against public scholarly APIs.

Given a bibliography (BibTeX, a list of DOIs/arXiv IDs, or free-form references),
this checks whether each entry ACTUALLY EXISTS by querying four public scholarly
APIs (Crossref, OpenAlex, Semantic Scholar, arXiv), plus the doi.org Handle API to
confirm whether a cited DOI is registered at all. It resolves DOI / arXiv
identifiers, fuzzy-matches title and authors (difflib SequenceMatcher ratio —
Ratcliff/Obershelp, not edit-distance — default threshold 0.70), flags retractions
(Crossref ``updated-by`` / ``update-to`` records, which since 2025 include the
Retraction Watch database, and OpenAlex ``is_retracted``), and emits a JSON verdict
per entry mapped to the AlterLab citation hallucination taxonomy (TF / PAC / IH / PH / SH).

Design constraints:
- NO API key required. Optional keys only raise rate limits: ``OPENALEX_API_KEY``
  (OpenAlex retired its ``mailto`` polite pool in Feb 2026; keyless calls share a
  small per-IP daily budget in which DOI lookups are free but searches are metered)
  and ``S2_API_KEY`` (Semantic Scholar). Keys are sent as headers, never logged.
- NO third-party deps required: uses ``requests`` if present, else the stdlib
  (``urllib``). Mirrors the integrity_verification_agent taxonomy exactly (see
  ../SKILL.md and skills/core/alterlab-research-pipeline/agents/integrity_verification_agent.md).
- GRACEFUL DEGRADATION: with no network, or when every source errors (rate limits,
  5xx), the entry gets an ``unverified`` verdict plus manual-verification
  instructions. It never silently passes an entry, and it never reports a Total
  Fabrication that rests only on sources that failed to answer.

Taxonomy (codes mirror the canonical Five-Type Taxonomy):
  TF  Total Fabrication           — entry exists in no source
  PAC Partial Attribute Corruption — entry found but ≥1 metadata field is wrong
  IH  Identifier Hijacking         — DOI/arXiv ID resolves to an unrelated paper
  PH  Placeholder Hallucination    — entry is an unresolved template/placeholder
  SH  Semantic Hallucination       — entry resolves but does not support its claim
                                      (claim-vs-source check is out of scope here;
                                       emitted only as an advisory flag)

Usage:
  uv run python verify_citations.py INPUT [--format auto|bibtex|doi|freeform]
                                          [--mailto you@example.com]
                                          [--threshold 0.70]
                                          [--out report.json] [--offline]
  uv run python verify_citations.py - < refs.bib        # read stdin

Optional environment: OPENALEX_API_KEY (free key, openalex.org/settings/api; sent
as an Authorization: Bearer header), S2_API_KEY (Semantic Scholar, x-api-key header).
A rate-limited source (HTTP 429) is reported in ``source_status`` and
``summary.notes``; it never counts as "not found".

Exit codes: 0 = ran (see JSON ``summary.verdict``); 2 = bad input/usage.
"""

from __future__ import annotations

import argparse
import json
import os
import re
import socket
import sys
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
from difflib import SequenceMatcher
from typing import Any, Callable, Optional

# --------------------------------------------------------------------------- #
# HTTP layer — prefer requests, fall back to urllib (stdlib). Keys optional.  #
# --------------------------------------------------------------------------- #

TOOL_VERSION = "1.1.0"
DEFAULT_MAILTO = "alterlab.ieu@gmail.com"
USER_AGENT = (
    "alterlab-citation-verifier/" + TOOL_VERSION + " (https://github.com/AlterLab-IEU/"
    "AlterLab-Academic-Skills; mailto:{mailto})"
)
HTTP_TIMEOUT = 15
RETRIES = 2
BACKOFF = 1.5
# A 429 asking us to wait longer than this (e.g. OpenAlex's exhausted daily budget
# returns Retry-After of many hours) is reported as a source error, not retried.
MAX_RETRY_AFTER = 30

# Optional credentials — never required; they only raise rate limits.
OPENALEX_API_KEY = os.environ.get("OPENALEX_API_KEY", "").strip()
S2_API_KEY = (os.environ.get("S2_API_KEY") or os.environ.get("SEMANTIC_SCHOLAR_API_KEY") or "").strip()

# Minimum spacing between calls to one host. arXiv's API terms ask for >= 3 s
# between calls; Semantic Scholar's key tier is 1 request/second.
MIN_INTERVAL = {"export.arxiv.org": 3.0, "arxiv.org": 3.0, "api.semanticscholar.org": 1.0}
_LAST_CALL: dict[str, float] = {}

try:  # pragma: no cover - environment dependent
    import requests as _requests  # type: ignore
    _HAS_REQUESTS = True
except Exception:  # pragma: no cover
    _requests = None
    _HAS_REQUESTS = False

import urllib.error
import urllib.parse
import urllib.request


class NetworkUnavailable(Exception):
    """Raised when a request cannot reach the network (DNS / connection error)."""


class SourceError(Exception):
    """One source could not answer (rate limit, 5xx, auth, timeout).

    Distinct from "no match": a source that errored says nothing about whether
    the cited work exists, so it must never count as evidence of fabrication.
    """


class _TransientHTTP(Exception):
    def __init__(self, status: int, retry_after: float = 0.0) -> None:
        super().__init__(f"transient HTTP {status}")
        self.status = status
        self.retry_after = retry_after


def _retry_after(value: Optional[str]) -> float:
    try:
        return float(value) if value else 0.0
    except ValueError:
        return 0.0


def _throttle(url: str) -> None:
    host = urllib.parse.urlsplit(url).hostname or ""
    gap = MIN_INTERVAL.get(host, 0.0)
    if gap:
        wait = _LAST_CALL.get(host, 0.0) + gap - time.monotonic()
        if wait > 0:
            time.sleep(wait)
        _LAST_CALL[host] = time.monotonic()


# What a rate limit means per host, so a 429 is reported with its remedy instead
# of silently reading as "no match".
_RATE_LIMIT_HINTS = {
    "api.openalex.org": ("OpenAlex rate limit: the keyless daily budget shared by this IP is "
                         "spent (or the request rate is too high); set a free OPENALEX_API_KEY "
                         "or retry after the reset at midnight UTC"),
    "api.semanticscholar.org": ("Semantic Scholar rate limit: the keyless pool is shared by all "
                                "unauthenticated users; set S2_API_KEY or retry later"),
    "api.crossref.org": "Crossref rate limit: slow down (polite pool allows 10 req/s, 3 concurrent)",
}


def _http_get(url: str, mailto: str, accept: str = "application/json",
              extra_headers: Optional[dict] = None) -> Any:
    """GET ``url`` and parse JSON (or return raw text for XML endpoints).

    Returns None when the API answered "not found / no match" (404, other 4xx).
    Raises SourceError when the API could not answer (429/5xx after retries,
    401/403, timeout) and NetworkUnavailable on a DNS/connection failure, so the
    caller can tell "offline" and "this source failed" apart from "not found".
    """
    headers = {"User-Agent": USER_AGENT.format(mailto=mailto), "Accept": accept}
    headers.update(extra_headers or {})
    last_exc: Optional[Exception] = None
    for attempt in range(RETRIES + 1):
        _throttle(url)
        try:
            if _HAS_REQUESTS:
                resp = _requests.get(url, headers=headers, timeout=HTTP_TIMEOUT)
                status = resp.status_code
                if status == 429 or 500 <= status < 600:
                    raise _TransientHTTP(status, _retry_after(resp.headers.get("Retry-After")))
                if status in (401, 403):
                    raise SourceError(f"HTTP {status} (access denied)")
                if status >= 400:
                    return None  # 404 and other 4xx → the API answered "no match"
                return resp.json() if "json" in accept else resp.text
            req = urllib.request.Request(url, headers=headers)
            with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT) as r:
                body = r.read().decode("utf-8", "replace")
                return json.loads(body) if "json" in accept else body
        except _TransientHTTP as exc:
            last_exc = exc
            if exc.retry_after > MAX_RETRY_AFTER:
                break
        except urllib.error.HTTPError as exc:  # urllib path
            if exc.code == 429 or 500 <= exc.code < 600:
                last_exc = exc
                if _retry_after(exc.headers.get("Retry-After") if exc.headers else None) > MAX_RETRY_AFTER:
                    break
            elif exc.code in (401, 403):
                raise SourceError(f"HTTP {exc.code} (access denied)") from exc
            else:
                return None  # 404 and other 4xx → "no match"
        except urllib.error.URLError as exc:
            if isinstance(exc.reason, (socket.timeout, TimeoutError)):
                last_exc = exc  # slow source, not an offline machine
            else:
                raise NetworkUnavailable(str(exc.reason)) from exc  # DNS / no route
        except (socket.timeout, TimeoutError) as exc:
            last_exc = exc
        except ValueError as exc:  # malformed JSON body
            raise SourceError(f"unparseable response: {exc}") from exc
        except SourceError:
            raise
        except Exception as exc:  # requests ConnectionError, Timeout, SSLError, etc.
            name = type(exc).__name__
            if "Timeout" in name:
                last_exc = exc
            elif any(k in name for k in ("Connection", "DNS", "SSL")):
                raise NetworkUnavailable(str(exc)) from exc
            else:
                last_exc = exc
        if attempt < RETRIES:
            time.sleep(BACKOFF * (attempt + 1))
    code = getattr(last_exc, "status", None) or getattr(last_exc, "code", None)
    if code == 429:
        host = urllib.parse.urlsplit(url).hostname or ""
        raise SourceError("HTTP 429 — " + _RATE_LIMIT_HINTS.get(host, "rate limited; retry later"))
    raise SourceError(f"no answer after retries ({last_exc})")


# --------------------------------------------------------------------------- #
# Fuzzy matching                                                              #
# --------------------------------------------------------------------------- #

_WS = re.compile(r"\s+")
_NONWORD = re.compile(r"[^\w\s]")


def _normalize(text: str) -> str:
    text = (text or "").lower().strip()
    text = _NONWORD.sub(" ", text)
    return _WS.sub(" ", text).strip()


def title_ratio(a: str, b: str) -> float:
    """Similarity ratio (0..1) via difflib SequenceMatcher on normalized titles.

    Some records drop the subtitle (OpenAlex / Semantic Scholar often store only
    "Main title"), so a title with a subtitle also counts as matching when its main
    title is essentially identical (ratio >= 0.9) to the other title *as a whole*.
    Comparing main titles loosely would match unrelated works that merely share a
    generic main title ("Education and technology: ..."), so that is not done.
    """
    na, nb = _normalize(a), _normalize(b)
    best = SequenceMatcher(None, na, nb).ratio()
    for x, other in ((a, nb), (b, na)):
        m = re.match(r"(.+?)(?::|\s[-—]\s|\?)", x or "")
        if m:
            head = _normalize(m.group(1))
            if len(head) >= 20 and SequenceMatcher(None, head, other).ratio() >= 0.9:
                best = max(best, SequenceMatcher(None, head, other).ratio())
    return best


def _surname(name: str) -> str:
    """Best-effort surname extraction from 'Last, First' or 'First Last'."""
    name = name.strip()
    if "," in name:
        return _normalize(name.split(",", 1)[0])
    parts = _normalize(name).split()
    return parts[-1] if parts else ""


def author_overlap(cited: list[str], found_surnames: list[str]) -> Optional[float]:
    """Fraction of cited author surnames that appear among the found surnames.

    None when either side has no author list: the record can neither confirm nor
    disprove the cited authors, which is different from a 0% overlap.
    """
    cited_s = {_surname(a) for a in cited if a.strip()}
    found_s = {_normalize(s) for s in found_surnames if s.strip()}
    if not cited_s or not found_s:
        return None
    hits = sum(1 for c in cited_s if any(c and (c in f or f in c) for f in found_s))
    return hits / len(cited_s)


def _year_match(entry: "Entry", rec: dict) -> bool:
    return bool(entry.year) and entry.year in (rec.get("years") or [rec.get("year")])


def _rank(entry: "Entry", rec: dict) -> tuple:
    """Sort key for "which returned record is the cited work": title match first,
    then author agreement, then year — so a same-titled paper by other authors, or
    another edition, loses to the record that matches the citation."""
    r = title_ratio(entry.title, rec.get("title") or "") if entry.title else 1.0
    ov = author_overlap(entry.authors, rec.get("authors") or [])
    return (r >= 0.7, 0.5 if ov is None else ov, _year_match(entry, rec), r)


# --------------------------------------------------------------------------- #
# Parsing: BibTeX / DOI list / free-form                                      #
# --------------------------------------------------------------------------- #

DOI_RE = re.compile(r"10\.\d{4,9}/[-._;()/:A-Z0-9]+", re.IGNORECASE)
# New-style IDs (2007+) anywhere; old-style (archive/YYMMNNN) only with an explicit
# arXiv prefix or abs/ URL, so ordinary URL paths are not mistaken for arXiv IDs.
ARXIV_RE = re.compile(
    r"(?<![\w.])(\d{4}\.\d{4,5})(?:v\d+)?(?![\w.])"
    r"|(?:arxiv:\s*|arxiv\.org/(?:abs|pdf)/)([a-z\-]+(?:\.[A-Z]{2})?/\d{7})",
    re.IGNORECASE,
)
OLD_ARXIV_BARE_RE = re.compile(r"^\s*([a-z\-]+(?:\.[A-Z]{2})?/\d{7})(?:v\d+)?\s*$", re.IGNORECASE)
# "M. K." / "J. P" / "A.-B." / "L" — author initials, not a surname ("OECD" is kept).
_INITIALS_RE = re.compile(r"^(?:[A-Z]\.?-?\s*){1,4}$")


# Start of a new reference on its own line: "[12] ", "12. ", "Surname, I." /
# "Surname, Name", or a capitalised author/organisation followed soon by "(YYYY".
_NEW_REF_LINE_RE = re.compile(
    r"^\s*(?:\[?\d{1,3}[\].)]\s|[A-Z][\w'’\-]+(?:\s[A-Z][\w'’\-]+)?,\s+[A-Z]"
    r"|[A-Z][^\n]{0,120}?\((?:19|20)\d{2}[a-z]?[),])"
)
PLACEHOLDER_RE = re.compile(
    r"(\[(?:citation|ref|cite|todo|xx+|author|year)[^\]]*\]"
    r"|\\cite\{[^}]*\}"
    r"|\bTODO\b|\bTKTK\b|\bXX+\b"
    r"|et al\.,?\s*\(?(?:YYYY|n\.d\.|\?\?\?\?)\)?"
    r"|\(\s*(?:year|date)\s*\)"
    r"|\bforthcoming\b|\bin press\b)",
    re.IGNORECASE,
)


def _is_initials(token: str) -> bool:
    letters = re.sub(r"[\s.\-]", "", token)
    if not _INITIALS_RE.match(token):
        return False
    return "." in token or len(letters) <= 2


@dataclass
class Entry:
    raw: str
    key: str = ""
    title: str = ""
    authors: list[str] = field(default_factory=list)
    year: str = ""
    doi: str = ""
    arxiv_id: str = ""
    venue: str = ""


def _looks_like_bibtex(text: str) -> bool:
    return bool(re.search(r"@\w+\s*\{", text))


def parse_bibtex(text: str) -> list[Entry]:
    entries: list[Entry] = []
    for m in re.finditer(r"@(\w+)\s*\{([^,]*),(.*?)\n\}", text, re.DOTALL):
        body = m.group(3)
        key = m.group(2).strip()

        def fld(name: str) -> str:
            # (?<![\w-]) keeps "title" from matching inside "booktitle"/"shorttitle";
            # braces/quotes are optional so bare values such as `year = 2019,` parse.
            fm = re.search(
                rf"(?<![\w-]){name}\s*=\s*[{{\"]?(.+?)[}}\"]?\s*,?\s*$",
                body,
                re.IGNORECASE | re.MULTILINE | re.DOTALL,
            )
            return _WS.sub(" ", fm.group(1)).strip() if fm else ""

        authors_raw = fld("author")
        authors = [a.strip() for a in re.split(r"\s+and\s+", authors_raw) if a.strip()]
        year_raw = fld("year") or fld("date")
        ym = re.search(r"(?:19|20)\d{2}", year_raw)
        eprint = fld("eprint")
        bare_old = OLD_ARXIV_BARE_RE.match(eprint) if eprint else None
        e = Entry(
            raw=m.group(0),
            key=key,
            title=fld("title").replace("{", "").replace("}", ""),
            authors=authors,
            year=ym.group(0) if ym else year_raw,
            doi=_first_doi(fld("doi") or body),
            arxiv_id=bare_old.group(1) if bare_old else _first_arxiv(eprint or body),
            venue=fld("journal") or fld("booktitle") or fld("publisher"),
        )
        entries.append(e)
    return entries


def _first_doi(text: str) -> str:
    m = DOI_RE.search(text or "")
    return m.group(0).rstrip(".,;") if m else ""


def _first_arxiv(text: str) -> str:
    # arXiv's own DOIs (10.48550/arXiv.<id>) name the ID; every other DOI is
    # stripped first so "10.1016/j.compedu.2020.103998" cannot yield "2020.10399".
    text = re.sub(r"10\.48550/arxiv\.", "arXiv:", text or "", flags=re.IGNORECASE)
    text = DOI_RE.sub(" ", text)
    m = ARXIV_RE.search(text)
    if not m:
        return ""
    return (m.group(1) or m.group(2) or "").strip()


def _is_bare_identifier_line(line: str) -> bool:
    """True for lines that are just an identifier ("doi:10.x/y", a doi.org URL, an arXiv ID)."""
    rest = DOI_RE.sub(" ", line)
    rest = ARXIV_RE.sub(" ", rest)
    rest = re.sub(r"(?i)https?://(dx\.)?doi\.org/|https?://arxiv\.org/(abs|pdf)/|\bdoi:|\barxiv:", " ", rest)
    return len(re.sub(r"[\W_]+", "", rest)) <= 3


def _clean_authors(names: list[str]) -> list[str]:
    """Drop initials-only fragments ("M. K.") and 'et al.' produced by comma splitting."""
    out = []
    for n in names:
        n = n.strip(" ,&")
        if not n or n.lower().startswith("et al") or _is_initials(n):
            continue
        n = n.strip(" .")
        out.append(n)
    return out


def parse_doi_list(text: str) -> list[Entry]:
    entries: list[Entry] = []
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        doi = _first_doi(line)
        arxiv = _first_arxiv(line) if not doi else ""
        if doi or arxiv:
            entries.append(Entry(raw=line, doi=doi, arxiv_id=arxiv))
    return entries


def _split_references(text: str) -> list[str]:
    """One block per reference: blank lines, numbered markers, or — for a list pasted
    one reference per line — a new line that starts like a reference ("Surname, I.")."""
    text = text.strip()
    if re.search(r"\n\s*\n", text):
        return re.split(r"\n\s*\n|\n(?=\s*\[?\d{1,3}[\].)]\s)", text)
    blocks: list[str] = []
    for line in text.splitlines():
        if not line.strip():
            continue
        starts_new = (_NEW_REF_LINE_RE.match(line) or _is_bare_identifier_line(line)
                      or (blocks and _is_bare_identifier_line(blocks[-1])))
        if blocks and not starts_new:
            blocks[-1] += " " + line.strip()  # hard-wrapped continuation line
        else:
            blocks.append(line.strip())
    return blocks


def parse_freeform(text: str) -> list[Entry]:
    """One reference per block (see _split_references); heuristic field extraction."""
    blocks = _split_references(text)
    entries: list[Entry] = []
    for block in blocks:
        block = _WS.sub(" ", block).strip()
        if not block:
            continue
        if _is_bare_identifier_line(block):  # a lone DOI / arXiv ID inside a pasted list
            entries.append(Entry(raw=block, doi=_first_doi(block), arxiv_id=_first_arxiv(block)))
            continue
        ym = re.search(r"\(?((?:19|20)\d{2})\)?", block)
        # Title heuristic: first quoted span, else the sentence after the year
        # (split on ". " but not after an initial, so "Web 2.0" / "J. Smith" survive).
        tm = re.search(r"[\"“](.+?)[\"”]", block)
        if tm:
            title = tm.group(1)
        elif ym:
            after = block[ym.end():].lstrip(" .)")
            title = re.split(r"(?<!\s[A-Z])\.\s", after, maxsplit=1)[0] if after else block
        else:
            title = block
        title = re.sub(r"[*_]", "", title)  # Markdown emphasis around book titles
        title = re.sub(r"\s*\(\s*(?:\d+(?:st|nd|rd|th)|rev(?:ised)?\.?)\s*(?:ed\.?|edition)?\s*\)?\s*$",
                       "", title, flags=re.IGNORECASE)
        authors: list[str] = []
        if ym and ym.start() > 0:
            head = block[: ym.start()].strip(" .,(")
            head = re.sub(r"^\s*\[?\d{1,3}[\].)]\s+", "", head)  # drop "12." / "[12]" markers
            authors = _clean_authors(re.split(r",| & | and ", head))[:6]
        entries.append(
            Entry(
                raw=block,
                title=title.strip(" .,"),
                authors=authors,
                year=ym.group(1) if ym else "",
                doi=_first_doi(block),
                arxiv_id=_first_arxiv(block),
            )
        )
    return entries


def parse_bibliography(text: str, fmt: str) -> list[Entry]:
    if fmt == "auto":
        if _looks_like_bibtex(text):
            fmt = "bibtex"
        else:
            # "doi" only when the lines are bare identifiers; a full reference that
            # merely contains a DOI stays free-form, so its title and authors are kept
            # for the Identifier Hijacking check.
            non_empty = [ln for ln in text.splitlines() if ln.strip() and not ln.strip().startswith("#")]
            id_lines = sum(1 for ln in non_empty
                           if (_first_doi(ln) or _first_arxiv(ln)) and _is_bare_identifier_line(ln))
            fmt = "doi" if non_empty and id_lines >= max(1, len(non_empty) * 0.6) else "freeform"
    if fmt == "bibtex":
        return parse_bibtex(text)
    if fmt == "doi":
        return parse_doi_list(text)
    if fmt == "freeform":
        return parse_freeform(text)
    raise ValueError(f"unknown format: {fmt}")


# --------------------------------------------------------------------------- #
# API adapters → normalized record {title, authors[surnames], year, doi, retracted} #
# --------------------------------------------------------------------------- #


def crossref_by_doi(doi: str, mailto: str) -> Optional[dict]:
    q_mailto = urllib.parse.quote(mailto)
    url = f"https://api.crossref.org/works/{urllib.parse.quote(doi)}?mailto={q_mailto}"
    data = _http_get(url, mailto)
    if not data or "message" not in data:
        return None
    rec = _crossref_norm(data["message"])
    rec["method"] = "id"  # resolved BY the cited identifier
    return rec


def crossref_by_title(entry: "Entry", mailto: str) -> Optional[dict]:
    # query.bibliographic is built for citation strings: title + first author + year
    # ranks the cited work above same-titled works and other editions.
    biblio = " ".join(x for x in (entry.title, _surname(entry.authors[0]) if entry.authors else "",
                                  entry.year) if x)
    url = (
        f"https://api.crossref.org/works?query.bibliographic={urllib.parse.quote(biblio)}"
        f"&rows=5&mailto={urllib.parse.quote(mailto)}"
    )
    data = _http_get(url, mailto)
    items = (data or {}).get("message", {}).get("items", []) if data else []
    if not items:
        return None
    # Relevance order is not match order: keep the record that best matches the citation.
    rec = max((_crossref_norm(it) for it in items[:5]), key=lambda r: _rank(entry, r))
    rec["method"] = "title"  # found by a fuzzy title/biblio search
    return rec


# Crossref update types that mean "do not rely on this work" (retraction notices,
# withdrawals, removals). Expressions of concern are surfaced as a separate flag.
_RETRACTION_TYPES = ("retraction", "partial_retraction", "withdrawal", "removal")


def _crossref_title(msg: dict) -> str:
    title = (msg.get("title") or [""])[0] or ""
    sub = (msg.get("subtitle") or [""])[0] or ""
    return f"{title}: {sub}" if title and sub else title


def _crossref_norm(msg: dict) -> dict:
    # Organisational authors ("OECD") carry `name` instead of `family`.
    surnames = [a.get("family") or a.get("name") or "" for a in msg.get("author", []) or []]
    surnames = [s for s in surnames if s]
    years = []
    for key in ("issued", "published-print", "published-online", "published"):
        parts = (msg.get(key, {}) or {}).get("date-parts", [[None]])
        if parts and parts[0] and parts[0][0]:
            years.append(str(parts[0][0]))
    year = years[0] if years else ""  # `issued` = earliest of print/online
    # `updated-by` on the cited work lists notices that update it, from the
    # publisher or (source "retraction-watch") the Retraction Watch database,
    # which Crossref serves in its REST API since 2025. `update-to` appears when the
    # matched record is itself a notice (or a publisher deposited the link there).
    updates = list(msg.get("updated-by", []) or []) + list(msg.get("update-to", []) or [])
    types = {(u.get("type") or "").lower() for u in updates}
    retracted = any(t in _RETRACTION_TYPES or "retract" in t for t in types)
    return {
        "source": "crossref",
        "title": _crossref_title(msg),
        "authors": surnames,
        "year": year,
        "years": sorted(set(years)),  # online-first vs print years can differ by one
        "doi": (msg.get("DOI") or "").lower(),
        "retracted": retracted,
        "concern": "expression_of_concern" in types,
    }


def _openalex_headers() -> dict:
    # OpenAlex ignores `mailto` since Feb 2026; a free key raises the daily budget.
    return {"Authorization": f"Bearer {OPENALEX_API_KEY}"} if OPENALEX_API_KEY else {}


def openalex_lookup(entry: Entry, mailto: str) -> Optional[dict]:
    if entry.doi:
        # Singleton DOI lookups are free even without a key.
        url = f"https://api.openalex.org/works/doi:{urllib.parse.quote(entry.doi)}"
        data = _http_get(url, mailto, extra_headers=_openalex_headers())
        if data and data.get("id"):
            rec = _openalex_norm(data)
            rec["method"] = "id"
            return rec
    if entry.title:
        # Searches draw on the metered budget (shared per IP when keyless).
        q = urllib.parse.quote(entry.title)
        url = f"https://api.openalex.org/works?search={q}&per-page=5"
        data = _http_get(url, mailto, extra_headers=_openalex_headers())
        results = (data or {}).get("results", []) if data else []
        if results:
            rec = max((_openalex_norm(w) for w in results[:5]), key=lambda r: _rank(entry, r))
            rec["method"] = "title"
            return rec
    return None


def _openalex_norm(w: dict) -> dict:
    surnames = []
    for au in w.get("authorships", []):
        disp = (au.get("author", {}) or {}).get("display_name", "")
        if disp:
            surnames.append(disp.split()[-1])
    return {
        "source": "openalex",
        "title": w.get("title") or w.get("display_name") or "",
        "authors": surnames,
        "year": str(w.get("publication_year") or ""),
        "doi": (w.get("doi") or "").replace("https://doi.org/", "").lower(),
        "retracted": bool(w.get("is_retracted")),
    }


def semanticscholar_lookup(entry: Entry, mailto: str) -> Optional[dict]:
    fields = "title,year,authors,externalIds"
    hdrs = {"x-api-key": S2_API_KEY} if S2_API_KEY else {}
    if entry.doi:
        url = f"https://api.semanticscholar.org/graph/v1/paper/DOI:{urllib.parse.quote(entry.doi)}?fields={fields}"
        data = _http_get(url, mailto, extra_headers=hdrs)
        if data and data.get("title"):
            rec = _ss_norm(data)
            rec["method"] = "id"
            return rec
    if entry.arxiv_id:
        url = f"https://api.semanticscholar.org/graph/v1/paper/ARXIV:{entry.arxiv_id}?fields={fields}"
        data = _http_get(url, mailto, extra_headers=hdrs)
        if data and data.get("title"):
            rec = _ss_norm(data)
            rec["method"] = "id"
            return rec
    if entry.title:
        q = urllib.parse.quote(entry.title)
        url = (
            f"https://api.semanticscholar.org/graph/v1/paper/search?query={q}"
            f"&limit=5&fields={fields}"
        )
        data = _http_get(url, mailto, extra_headers=hdrs)
        items = (data or {}).get("data", []) if data else []
        if items:
            rec = max((_ss_norm(p) for p in items[:5]), key=lambda r: _rank(entry, r))
            rec["method"] = "title"
            return rec
    return None


def _ss_norm(p: dict) -> dict:
    surnames = []
    for au in p.get("authors", []) or []:
        nm = au.get("name", "")
        if nm:
            surnames.append(nm.split()[-1])
    ext = p.get("externalIds", {}) or {}
    return {
        "source": "semanticscholar",
        "title": p.get("title") or "",
        "authors": surnames,
        "year": str(p.get("year") or ""),
        "doi": (ext.get("DOI") or "").lower(),
        "retracted": False,  # S2 does not expose a retraction flag
    }


def arxiv_lookup(entry: Entry, mailto: str) -> Optional[dict]:
    if entry.arxiv_id:
        url = f"https://export.arxiv.org/api/query?id_list={urllib.parse.quote(entry.arxiv_id)}&max_results=1"
    elif entry.title:
        q = urllib.parse.quote(f'ti:"{entry.title}"')
        url = f"https://export.arxiv.org/api/query?search_query={q}&max_results=5"
    else:
        return None
    text = _http_get(url, mailto, accept="application/atom+xml")
    if not text or "<entry>" not in text:
        return None
    method = "id" if entry.arxiv_id else "title"
    recs = []
    for block in text.split("<entry>")[1:]:
        block = block.split("</entry>", 1)[0]
        tm = re.search(r"<title>(.*?)</title>", block, re.DOTALL)
        title = _WS.sub(" ", tm.group(1)).strip() if tm else ""
        surnames = [
            _WS.sub(" ", n).strip().split()[-1]
            for n in re.findall(r"<name>(.*?)</name>", block, re.DOTALL)
            if n.strip()
        ]
        ym = re.search(r"<published>(\d{4})", block)
        aid = re.search(r"<id>.*?abs/([^<]+)</id>", block)
        recs.append({
            "source": "arxiv",
            "title": title,
            "authors": surnames,
            "year": ym.group(1) if ym else "",
            "doi": "",
            "arxiv_id": aid.group(1) if aid else (entry.arxiv_id or ""),
            "retracted": False,
            "method": method,
        })
    if not recs:
        return None
    if method == "title" and entry.title:
        return max(recs, key=lambda r: _rank(entry, r))
    return recs[0]


def arxiv_id_exists(arxiv_id: str, mailto: str) -> Optional[bool]:
    """The arXiv abstract page answers 200 for a real ID and 404 for a nonexistent
    one — authoritative, unlike the query API, which can return an empty feed
    under load. True / False, or None if arXiv could not answer."""
    try:
        page = _http_get(f"https://arxiv.org/abs/{urllib.parse.quote(arxiv_id, safe='/.')}",
                         mailto, accept="text/html")
    except (SourceError, NetworkUnavailable):
        return None
    return page is not None


def doi_registered(doi: str, mailto: str) -> Optional[bool]:
    """Ask the doi.org Handle API whether a DOI exists with ANY registration agency
    (Crossref, DataCite, mEDRA, ...). True / False, or None if it could not answer."""
    url = f"https://doi.org/api/handles/{urllib.parse.quote(doi)}"
    try:
        data = _http_get(url, mailto)
    except (SourceError, NetworkUnavailable):
        return None
    if data is None:  # HTTP 404 (responseCode 100) is how the Handle API says "not found"
        return False
    code = data.get("responseCode")
    # 1 = found; 200 = handle exists but has no values of the queried type; 100 = not found.
    return True if code in (1, 200) else False if code == 100 else None


# --------------------------------------------------------------------------- #
# Verdict engine                                                              #
# --------------------------------------------------------------------------- #

VERDICTS = {
    "verified": "Entry exists; title+authors match an authoritative record.",
    "TF": "Total Fabrication — entry found in NO source.",
    "PAC": "Partial Attribute Corruption — entry found but metadata fields disagree.",
    "IH": "Identifier Hijacking — DOI/arXiv ID resolves to an unrelated paper.",
    "PH": "Placeholder Hallucination — unresolved citation template/placeholder.",
    "SH": "Semantic Hallucination — resolves but claim-support unverified (advisory).",
    "unverified": "Could not verify (offline or all APIs failed); manual check required.",
}
SEVERITY = {
    "verified": "NONE",
    "TF": "SERIOUS",
    "PAC": "MEDIUM",
    "IH": "SERIOUS",
    "PH": "SERIOUS",
    "SH": "SERIOUS",
    "unverified": "MEDIUM",
}


def _is_placeholder(entry: Entry) -> bool:
    if PLACEHOLDER_RE.search(entry.raw):
        return True
    if not entry.doi and not entry.arxiv_id and len(_normalize(entry.title)) < 6 and not entry.authors:
        return True
    return False


def verify_entry(entry: Entry, mailto: str, threshold: float, offline: bool) -> dict:
    ref_id = entry.key or entry.doi or entry.arxiv_id or (entry.title[:40] or entry.raw[:40])

    # PH check first — placeholders never reach the network.
    if _is_placeholder(entry):
        return _verdict(entry, ref_id, "PH",
                        detail="Citation is an unresolved placeholder/template.",
                        matches=[])

    if offline:
        return _unverified(entry, ref_id, reason="offline mode requested (--offline)")

    matches: list[dict] = []
    status: dict[str, str] = {}  # source -> record | no_record | error: ... | offline
    network_failed = False
    has_identifier = bool(entry.doi or entry.arxiv_id)

    def _crossref() -> Optional[dict]:
        if entry.doi:
            rec = crossref_by_doi(entry.doi, mailto)
            # A DOI that does not resolve may still belong to a real work cited
            # with a wrong identifier; the title search lets that surface as PAC.
            return rec or (crossref_by_title(entry, mailto) if entry.title else None)
        return crossref_by_title(entry, mailto) if entry.title else None

    sources: list[tuple[str, Callable[[], Optional[dict]]]] = [
        ("crossref", _crossref),
        ("openalex", lambda: openalex_lookup(entry, mailto)),
        ("semanticscholar", lambda: semanticscholar_lookup(entry, mailto)),
        ("arxiv", lambda: arxiv_lookup(entry, mailto)),
    ]
    for name, fn in sources:
        try:
            rec = fn()
        except NetworkUnavailable:
            network_failed = True
            status[name] = "offline"
            break
        except SourceError as exc:
            status[name] = f"error: {exc}"
            continue
        status[name] = "record" if rec else "no_record"
        if rec:
            matches.append(rec)

    def answered(name: str) -> bool:
        return status.get(name) in ("record", "no_record")

    # An arXiv ID that no source resolved: settle existence on arxiv.org/abs, and
    # retry the query API once if the ID is real (its feed is sometimes empty).
    arxiv_exists: Optional[bool] = None
    if entry.arxiv_id and not network_failed and not any(m.get("method") == "id" for m in matches):
        arxiv_exists = arxiv_id_exists(entry.arxiv_id, mailto)
        status["arxiv.org/abs"] = {True: "exists", False: "not_found", None: "error"}[arxiv_exists]
        if arxiv_exists:
            try:
                rec = arxiv_lookup(entry, mailto)
            except (SourceError, NetworkUnavailable):
                rec = None
            if rec:
                matches.append(rec)
                status["arxiv"] = "record"

    if network_failed and not matches:
        return _unverified(entry, ref_id, reason="network unavailable (DNS/connection failure)",
                           status=status)
    if not any(answered(s) for s in status):
        return _unverified(entry, ref_id,
                           reason="every source failed to answer (rate limits / server errors)",
                           status=status)

    id_matches = [m for m in matches if m.get("method") == "id"]
    title_matches = [m for m in matches if m.get("method") == "title"]

    def r_of(m: dict) -> float:
        return title_ratio(entry.title, m["title"]) if entry.title and m.get("title") else 0.0

    # Identifier Hijacking: the cited DOI/arXiv id actually RESOLVED (method=id),
    # but the resolved record's title does not match the cited title.
    # This must be checked against id-resolved records only — a coincidental
    # title-search hit on a fabricated DOI is NOT hijacking, it is fabrication.
    if has_identifier and entry.title and id_matches:
        best_id = max(id_matches, key=r_of)
        if best_id["title"]:
            r = r_of(best_id)
            if r < threshold:
                return _verdict(entry, ref_id, "IH",
                                detail=(f"Cited identifier resolved to '{best_id['title']}' "
                                        f"(title ratio {r:.2f} < {threshold:.2f}) — unrelated paper."),
                                matches=matches, status=status)

    pac_reasons: list[str] = []

    # The cited identifier resolved nowhere. Decide TF only on authoritative
    # evidence: doi.org says the DOI is unregistered, or arXiv says the ID does
    # not exist. A registered DOI that no index returned, or an identifier whose
    # authoritative source errored, is `unverified` (manual check), never TF.
    if has_identifier and not id_matches:
        best_loose = max(title_matches, key=r_of, default=None) if entry.title else None
        loose_ok = bool(best_loose and r_of(best_loose) >= threshold)
        if loose_ok:
            pac_reasons.append(
                "cited identifier does not resolve to this work"
                + (f" (records list DOI {best_loose['doi']})" if best_loose.get("doi") else ""))
        else:
            if entry.doi:
                registered = doi_registered(entry.doi, mailto)
                status["doi.org"] = {True: "registered", False: "not_registered", None: "error"}[registered]
                if registered is False:
                    return _verdict(entry, ref_id, "TF",
                                    detail=(f"DOI {entry.doi} is not registered with any DOI agency "
                                            "(doi.org Handle API) and no close title match was found "
                                            "— entry appears fabricated."),
                                    matches=matches, status=status)
                why = ("DOI is registered at doi.org but no metadata source returned it "
                       "(non-Crossref DOI or index gap)" if registered
                       else "the cited DOI did not resolve and doi.org could not be reached")
                return _unverified(entry, ref_id, reason=why, status=status, matches=matches)
            if arxiv_exists is False:
                return _verdict(entry, ref_id, "TF",
                                detail=(f"arXiv ID {entry.arxiv_id} does not exist (arxiv.org/abs "
                                        "returns 404) and no close title match was found — entry "
                                        "appears fabricated."),
                                matches=matches, status=status)
            why = ("the arXiv ID exists but its record could not be retrieved" if arxiv_exists
                   else "arXiv could not be reached to resolve the cited ID")
            return _unverified(entry, ref_id, reason=why, status=status, matches=matches)

    # Which returned record is the cited work? Rank by title, then authors, then
    # year. A fuzzy search always returns its closest hit, so a hit only counts as
    # "found" when the title matches — and a same-titled record by entirely
    # different authors is a different work unless the titles are near-identical.
    pool = id_matches or matches  # a resolved identifier anchors the comparison
    best = max(pool, key=lambda m: _rank(entry, m)) if pool else None
    t_ratio = r_of(best) if (best and entry.title) else (1.0 if best else 0.0)
    a_overlap = author_overlap(entry.authors, best["authors"]) if best else None
    found = best is not None and t_ratio >= threshold and not (a_overlap == 0.0 and t_ratio < 0.95)
    if not has_identifier and not found:
        # Garbled/mashup title whose closest record shares the cited authors → PAC.
        near_mashup = (best is not None and a_overlap is not None and a_overlap >= 0.5
                       and t_ratio >= 0.5)
        if not near_mashup:
            # TF needs Crossref plus a broad index (OpenAlex or Semantic Scholar) to
            # have answered: Crossref alone misses books and grey literature.
            if answered("crossref") and (answered("openalex") or answered("semanticscholar")):
                closest = (f" Closest record: '{best['title']}' (title ratio {t_ratio:.2f})."
                           if best else "")
                return _verdict(entry, ref_id, "TF",
                                detail=("No source returned a record matching the cited title and "
                                        "authors — entry appears fabricated." + closest),
                                matches=matches, status=status)
            limited = [f"{src} {'rate-limited (HTTP 429)' if '429' in st else 'failed'}"
                       for src, st in status.items()
                       if src in ("openalex", "semanticscholar") and st.startswith("error")]
            return _unverified(entry, ref_id,
                               reason=("no matching record in the sources that answered, and "
                                       "OpenAlex / Semantic Scholar could not be queried to rule "
                                       "out books or grey literature"
                                       + (" (" + ", ".join(limited) + "; see summary.notes)"
                                          if limited else "")),
                               status=status, matches=matches)
        pac_reasons.append(f"title ratio {t_ratio:.2f} < {threshold:.2f} but the closest record "
                           f"shares the cited authors — possible mashup of real references")

    def same_work(m: dict) -> bool:
        if m is best or m.get("method") == "id":
            return True  # id matches already passed the IH title check
        return r_of(m) >= threshold and author_overlap(entry.authors, m["authors"]) != 0.0

    same = [m for m in matches if same_work(m)]
    # Retraction / concern flags come only from records of the cited work itself,
    # never from an unrelated closest hit.
    retracted = any(m.get("retracted") for m in same)
    concern = any(m.get("concern") for m in same)

    # Metadata corruption checks (PAC): year, authors.
    years = {y for m in same for y in (m.get("years") or [m.get("year")]) if y}
    if entry.year and years and entry.year not in years:
        pac_reasons.append(f"year cited={entry.year} vs source={'/'.join(sorted(years))}")
    if a_overlap is not None and a_overlap < 0.5:
        pac_reasons.append(f"author overlap {a_overlap:.0%} below 50%")

    verdict_code = "verified"
    detail = f"Matched in {', '.join(sorted({m['source'] for m in same}))}."
    if pac_reasons:
        verdict_code = "PAC"
        detail = "Found but metadata disagrees: " + "; ".join(pac_reasons) + "."

    result = _verdict(entry, ref_id, verdict_code, detail=detail, matches=matches, status=status)
    result["title_ratio"] = round(t_ratio, 3)
    result["author_overlap"] = round(a_overlap, 3) if a_overlap is not None else None
    result["retracted"] = retracted
    if retracted:
        result["flags"] = result.get("flags", []) + ["RETRACTED"]
        # Retraction does not change existence verdict but is a SERIOUS flag.
        result["severity"] = "SERIOUS"
        result["detail"] += (" RETRACTED: Crossref (publisher or Retraction Watch notice) and/or "
                             "OpenAlex mark this work as retracted.")
    if concern:
        result["flags"] = result.get("flags", []) + ["EXPRESSION_OF_CONCERN"]
        result["detail"] += " An expression of concern is on record for this work."
    return result


def _verdict(entry: Entry, ref_id: str, code: str, detail: str, matches: list[dict],
             status: Optional[dict] = None) -> dict:
    return {
        "ref_id": ref_id,
        "verdict": code,
        "verdict_meaning": VERDICTS[code],
        "severity": SEVERITY[code],
        "detail": detail,
        "cited": {
            "title": entry.title,
            "authors": entry.authors,
            "year": entry.year,
            "doi": entry.doi,
            "arxiv_id": entry.arxiv_id,
        },
        "sources_checked": sorted(status or {}),
        "source_status": dict(status or {}),
        "matches": [{k: m.get(k) for k in ("source", "method", "title", "year", "doi", "retracted")}
                    for m in matches],
        "flags": [],
    }


def _unverified(entry: Entry, ref_id: str, reason: str, status: Optional[dict] = None,
                matches: Optional[list[dict]] = None) -> dict:
    r = _verdict(entry, ref_id, "unverified",
                 detail=f"{reason}. NOT confirmed — do not treat as passing.",
                 matches=matches or [], status=status)
    r["manual_instructions"] = (
        "Verify manually: (1) search the exact title + first author + year on Google "
        "Scholar; (2) if a DOI is given, resolve https://doi.org/<DOI> and confirm the "
        "landing page title matches; (3) for arXiv IDs, open https://arxiv.org/abs/<id>; "
        "(4) confirm the venue, year, and author list field-by-field. Re-run this script "
        "with network access to obtain an automated verdict."
    )
    return r


# --------------------------------------------------------------------------- #
# Report assembly                                                             #
# --------------------------------------------------------------------------- #


def build_report(results: list[dict], mailto: str, threshold: float, offline: bool) -> dict:
    counts = {k: 0 for k in VERDICTS}
    sev = {"SERIOUS": 0, "MEDIUM": 0, "MINOR": 0}
    for r in results:
        counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
        s = r["severity"]
        if s in sev:
            sev[s] += 1
    n = len(results) or 1
    fabricated = counts["TF"] + counts["IH"] + counts["PH"]
    fabrication_risk = round(fabricated / n, 3)
    integrity = round((counts["verified"]) / n, 3)

    # SERIOUS findings (TF / IH / PH / retraction) dominate; otherwise any entry that
    # could not be checked makes the run UNVERIFIED — never a (conditional) pass.
    if sev["SERIOUS"] > 0:
        verdict = "FAIL"
    elif counts["unverified"]:
        verdict = "UNVERIFIED"
    elif counts["PAC"] > 0 or sev["MEDIUM"] > 0:
        verdict = "PASS_WITH_CONDITIONS"
    else:
        verdict = "PASS"

    # Surface source failures at the top of the report, so a rate-limited index
    # is visible even when other sources carried the verdicts.
    source_errors: dict[str, int] = {}
    notes: list[str] = []
    for r in results:
        for src, st in (r.get("source_status") or {}).items():
            if isinstance(st, str) and st.startswith("error"):
                source_errors[src] = source_errors.get(src, 0) + 1
                hint = st.split("HTTP 429 — ", 1)[1] if "HTTP 429 — " in st else ""
                if hint and hint not in notes:
                    notes.append(hint)

    return {
        "tool": "alterlab-citation-verifier/verify_citations.py",
        "version": TOOL_VERSION,
        "timestamp": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "config": {"mailto": mailto, "threshold": threshold, "offline": offline,
                   "http_backend": "requests" if _HAS_REQUESTS else "urllib",
                   # whether optional keys were present — never the keys themselves
                   "openalex_api_key": bool(OPENALEX_API_KEY),
                   "s2_api_key": bool(S2_API_KEY)},
        "taxonomy": VERDICTS,
        "summary": {
            "total": len(results),
            "verdict": verdict,
            "verdict_counts": counts,
            "severity_counts": sev,
            "citation_integrity_score": integrity,
            "fabrication_risk_score": fabrication_risk,
            "retracted": sum(1 for r in results if r.get("retracted")),
            "source_errors": source_errors,
            "notes": notes,
        },
        "entries": results,
    }


# --------------------------------------------------------------------------- #
# CLI                                                                         #
# --------------------------------------------------------------------------- #


def _read_input(path: str) -> str:
    if path == "-":
        return sys.stdin.read()
    import os
    if os.path.exists(path):
        with open(path, "r", encoding="utf-8") as fh:
            return fh.read()
    return path  # treat the argument itself as inline text


def main(argv: Optional[list[str]] = None) -> int:
    ap = argparse.ArgumentParser(description="Verify that bibliography entries exist via public scholarly APIs.")
    ap.add_argument("input", help="Path to a .bib/.txt file, '-' for stdin, or inline text.")
    ap.add_argument("--format", choices=["auto", "bibtex", "doi", "freeform"], default="auto")
    ap.add_argument("--mailto", default=DEFAULT_MAILTO, help="Polite-pool contact email.")
    ap.add_argument("--threshold", type=float, default=0.70, help="Fuzzy title-match ratio (0..1).")
    ap.add_argument("--out", default=None, help="Write JSON report to this path (default stdout).")
    ap.add_argument("--offline", action="store_true", help="Skip the network; emit 'unverified' verdicts.")
    args = ap.parse_args(argv)

    text = _read_input(args.input)
    try:
        entries = parse_bibliography(text, args.format)
    except Exception as exc:  # noqa: BLE001
        print(f"error: could not parse bibliography: {exc}", file=sys.stderr)
        return 2
    if not entries:
        print("error: no bibliography entries parsed from input.", file=sys.stderr)
        return 2

    results = [verify_entry(e, args.mailto, args.threshold, args.offline) for e in entries]
    report = build_report(results, args.mailto, args.threshold, args.offline)
    out = json.dumps(report, indent=2, ensure_ascii=False)
    if args.out:
        with open(args.out, "w", encoding="utf-8") as fh:
            fh.write(out + "\n")
        print(f"wrote {report['summary']['total']} verdicts → {args.out} "
              f"(verdict={report['summary']['verdict']})", file=sys.stderr)
    else:
        print(out)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())