alterlab-citation-verifier
Verifies that every bibliography entry actually exists by cross-checking Crossref, OpenAlex, Semantic Scholar, and arXiv (no API key required) plus doi.org DOI registration, fuzzy-matching title and authors (difflib ratio >= 0.70), flagging retractions recorded by Crossref (inclu
Install
npx skills add https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/tree/main/skills/core/alterlab-citation-verifier
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install alterlab-ieu-alterlab-academic-skills@llmmart
git clone https://github.com/AlterLab-IEU/AlterLab-Academic-Skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole alterlab-ieu/alterlab-academic-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Citation Verifier — Existence-Verify a Bibliography Against Public Scholarly APIs
The headline existence-verification skill: given a bibliography in any common
form, it proves entry-by-entry whether each reference actually exists by
querying public scholarly APIs, then maps each result to the canonical AlterLab
citation-hallucination taxonomy. It is the deterministic, network-grounded
companion to the LLM-driven integrity_verification_agent — where that agent
uses WebSearch + judgment, this skill uses authoritative API records and a
reproducible Python script, so every verdict rests on retrieved records and the
same input and source responses yield the same verdicts.
Verify references with the script, not from memory. A model checking citations against what it "remembers" shares the training data that produced the hallucination in the first place, so a fabricated reference that feels right passes undetected; only an external record settles existence.
Quick Start
Verify the citations in references.bib
Check whether these DOIs resolve to the papers I cited
Audit my bibliography for fabricated / hallucinated references
Does this reference list contain any fake citations or retractions?
→ Run scripts/verify_citations.py over the bibliography, read the JSON, then
present a verdict table grouped by severity. State the offline/degraded status
explicitly if the network or a source was unavailable.
When to Use This Skill
- "Verify / check / audit my citations or references exist"
- "Did the AI hallucinate any of these references?" / "Is this DOI fake?"
- "Do these DOIs resolve to the papers I cited?"
- "Check this bibliography for retractions"
- "Does this source actually support the sentence I cite it for?" (SH, abstract level)
- A reproducible, scriptable existence check with an explicit offline mode
Does NOT Trigger
| Scenario | Use Instead |
|---|---|
| Writing or revising the paper itself (its citation-check mode formats citations) | alterlab-paper-writer |
| Finding papers, extracting metadata, or generating BibTeX for new sources | alterlab-citation-mgmt |
| Dead or redirected hyperlinks (link rot) in docs or a reference list | alterlab-link-health |
| Full pre-/post-review integrity gate (citation context, data, originality) | alterlab-research-pipeline |
| Holistic peer review of a manuscript with an editorial decision | alterlab-paper-reviewer |
Hand-off rule. When the request matches a row above, open your answer by naming that skill by its exact id (for example alterlab-paper-writer) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work.
Grading source quality or predatory venues belongs to alterlab-deep-research
(source_verification_agent). For a whole-manuscript audit in Claude Code where every
flag should be re-checked by separate agents, offer the packaged
/alterlab-workflows:citation-audit workflow (alterlab-research-workflows, roughly
10–40 agents), which runs this skill's checks at scale.
What This Does
For each bibliography entry the script:
- Parses the input (auto-detects BibTeX / a bare DOI-or-arXiv list / free-form
references, including one-reference-per-line pastes), extracting title, authors,
year, venue, DOI, and arXiv ID. A full reference that merely contains a DOI stays
free-form, so its title and authors are kept for the Identifier Hijacking check.
Every detected entry ends up parsed, rejected (malformed — with line, key, and
reason) or unsupported; see
input_accounting. Mixed BibTeX + free-form/DOI input is handled piecewise, and a broken entry cannot swallow the ones after it. Text that is not a reference — a Markdown heading, an intro or prose paragraph, a line with no year, no DOI/arXiv/ISBN/URL and no author name — is skipped (input_accounting.ignored_text, reason "skipped: not a reference") and gets no verdict. A year in running text ("compiled in 2019 for the review") does not make prose a reference; a terse real one (Smith J. 2019. Title. Journal 3:1-2.) is checked. - Resolves identifiers — looks up the cited DOI/arXiv ID directly when present, and asks the doi.org Handle API whether a DOI is registered with any agency.
- Searches by title + first author + year as a fallback across the sources
(Crossref
query.bibliographic, Semantic Scholar, Europe PMCTITLE:"…" AND AUTH:…, arXiv, and Open Library for entries with no identifier, which are mostly books). - Fuzzy-matches the cited title (difflib
SequenceMatcherratio, default threshold 0.70; a main title that is near-identical to a record stored without its subtitle also counts) and computes author-surname overlap. Among the returned records it keeps the one that matches title, then authors, then year — so a same-titled paper by other authors or another edition does not win. - Flags retractions on the matched work: Crossref
updated-by/update-tonotices of type retraction, withdrawal, or removal (from the publisher or, withsource: retraction-watch, the Retraction Watch database that Crossref serves in its REST API since 2025). Expressions of concern get a separate flag. - Emits a verdict per entry mapped to the taxonomy below, with a
source_statusmap showing which sources answered, separateexistence,identifier,metadataandretractionblocks, plus a repo-levelsummary.verdict(FAIL / PARTIAL / REVIEW_REQUIRED / UNVERIFIED / PASS_WITH_CONDITIONS / PASS).
Sources (no key required; checked 2026-11-02)
| Source | Endpoint | Used for | Access notes |
|---|---|---|---|
| Crossref | api.crossref.org/works |
DOI resolution, bibliographic search, retraction notices | Public pool: 5 req/s, one request in flight (the script spaces calls and backs off on 429); an opt-in --mailto routes to the polite pool (10 req/s) |
| Semantic Scholar | api.semanticscholar.org/graph/v1 |
DOI/arXiv resolution, title search (best effort) | Keyless pool is shared. DOI/arXiv lookups mostly answer (backoff 2 + 4 + 8 s, then a 30 s pause); its search pool is usually spent from busy networks, so search gets one retry and a 5-minute pause. Either way it is recorded as not checked, never "not found", and no verdict depends on it. S2_API_KEY gives a dedicated 1 req/s |
| Europe PMC | ebi.ac.uk/europepmc/webservices/rest/search |
DOI lookup, title phrase + first-author search | Keyless; life sciences first, but also many other journals and preprint servers |
| arXiv | export.arxiv.org/api/query |
arXiv ID resolution, preprint title search | ≥ 3 s between calls (the script throttles) |
| Open Library | openlibrary.org/search.json |
Books (entries with no DOI/arXiv ID): main title + first author; every edition's year is kept | Keyless; 1 s between calls |
| doi.org Handle API | doi.org/api/handles/<doi> |
Is the DOI registered with any agency (Crossref, DataCite, mEDRA…)? | Authoritative for DOI existence |
Keys are optional, read from the environment, sent as headers, and never written
to the report (the report only records whether one was present). A source that
errors (rate limit, 5xx) is recorded in source_status and is never counted as
evidence that a work does not exist. One dropped connection is retried before a source
is marked offline. OpenAlex is no longer queried: since 2026 its keyless searches draw on
a small per-IP daily budget, so dependable use needed an account.
This skill answers "does the cited work exist, and does its identifier point
to it?" From API metadata alone it cannot establish Semantic Hallucination
(does the source support the claim?); scripts/claim_faithfulness.py gives an
abstract-level SH triage (see below), and full-text claim verification is the
research pipeline's claim_verification_protocol (Phase E).
Verdict Taxonomy (mirrors the canonical Five-Type Taxonomy)
Identical codes and definitions to
alterlab-research-pipeline/agents/integrity_verification_agent.md
(GPTZero × NeurIPS 2025; Ansari, 2026). Severity feeds the same
SERIOUS / MEDIUM / MINOR scale used in the Integrity Report schema.
| Code | Name | Severity | Script trigger |
|---|---|---|---|
verified |
— (exists, matches) | NONE | Found in >=1 authoritative source and every supplied field that could be checked (title ratio >= threshold, authors, year) agrees |
PAC |
Partial Attribute Corruption | MEDIUM | Entry found but >=1 supplied field disagrees: a cited author the record does not list (metadata.authors.unsupported, flag UNSUPPORTED_AUTHOR), a wrong year, a cited identifier that does not resolve to it (IDENTIFIER_DOES_NOT_RESOLVE), or a garbled title whose closest record shares the authors (POSSIBLE_MASHUP). "et al." / BibTeX "and others" is honoured; an abbreviated list without it is noted, not an error |
ID_INVALID |
Invalid identifier | SERIOUS | Cited DOI is not registered at doi.org, or cited arXiv ID does not exist on arXiv, and no close title match exists. The identifier is wrong; whether the work exists is undetermined |
NOT_FOUND |
Not found in the checked sources | MEDIUM | No identifier, and no record matches title + authors in Crossref plus at least one other index that actually answered (Semantic Scholar, Europe PMC, arXiv or Open Library); detail names the indexes that answered and those not checked. Absence from indexes is not proof of fabrication (books, grey literature, theses, regional venues) — check manually. If only Crossref answered, or a free-form entry lacks a title plus an author or year, the entry is unverified |
IH |
Identifier Hijacking | SERIOUS | Cited DOI/arXiv ID resolved (method=id) but the resolved record's title is unrelated (ratio < threshold) |
PH |
Placeholder Hallucination | SERIOUS | Unresolved template/placeholder ([CITATION NEEDED], \cite{}, et al., YYYY, TODO, forthcoming, in press) — caught pre-network |
SH |
Semantic Hallucination | SERIOUS | Entry resolves but does not support its claim — advisory only here; asserted via claim_faithfulness.py / Phase E |
unverified |
— (could not check) | MEDIUM | Offline, the sources needed for a decision errored, or a registered DOI that no index returned. Never treated as passing. |
TF |
Total Fabrication | SERIOUS | Never emitted by the script. Reserved for an analyst's conclusion after a documented manual search (the fallback below) finds no trace of a NOT_FOUND / ID_INVALID work |
A RETRACTED flag is attached (and severity bumped to SERIOUS) whenever Crossref
(publisher notice or Retraction Watch) marks the matched work as retracted, independent of the existence
verdict; an unrelated closest hit never lends its flag to the cited entry. Every found entry
carries retraction.status — flagged, not_flagged_in_checked_sources (Crossref answered
with no notice; other retraction databases were not checked) or unknown (Crossref did not
answer or holds no record) — and the v1 retracted key is true / false / null
(null = unknown). Never report unknown / null as "not retracted";
summary.retraction counts the three states.
Since report schema 2 (verify_citations.py 2.0.0) the script no longer reports TF for a
reference absent from the indexes (v1 did): that is NOT_FOUND, and an unregistered DOI or
nonexistent arXiv ID is ID_INVALID. TF is a serious accusation that only a documented
manual search can support. When the deciding sources were rate-limited or down, the entry is
unverified with the reason and source_status, and it goes to the fallback below.
Repo-level verdict
Most severe first: FAIL > PARTIAL > REVIEW_REQUIRED > UNVERIFIED > PASS_WITH_CONDITIONS > PASS. Only PASS and PASS_WITH_CONDITIONS are passes.
- PASS — every entry
verified, no SERIOUS/MEDIUM flags. Retraction status is reported separately (summary.retraction) and may beunknownfor some entries. - PASS_WITH_CONDITIONS — only
PAC/ MEDIUM metadata issues (fixable), and every entry was found. - FAIL — any SERIOUS finding (
IH/ID_INVALID/PH/ retraction). - REVIEW_REQUIRED — nothing SERIOUS, but at least one entry is
NOT_FOUND. Not a pass: check those entries manually (absence is not proof of fabrication). - UNVERIFIED — nothing SERIOUS was found but at least one entry could not be checked (e.g. an offline run or rate-limited sources). This is not a pass — re-run with network access or an API key, or resolve those entries manually.
- PARTIAL — at least one detected entry was rejected, unsupported, or left
unprocessed (
status: partial, exit code 3). Not a pass: fix the entries listed ininput_accountingand re-run. FAIL still takes precedence; PARTIAL outranks REVIEW_REQUIRED.
Pipeline (how to run it)
1. Locate or capture the bibliography
Accept any of: a .bib file, a .txt list of DOIs/arXiv IDs, a pasted
reference list, or inline text. The script auto-detects the format; override
with --format bibtex|doi|freeform if detection is wrong.
2. Run the verifier
uv run python scripts/verify_citations.py \
path/to/references.bib \
--mailto <contact-email> \
--threshold 0.70 \
--out citation_report.json
path/to/references.bibmay also be-(stdin) or inline text.--mailtois optional: the user's own contact email for Crossref's polite pool (or setALTERLAB_CONTACT_EMAIL). Without it no address is sent and Crossref's public pool answers. Never fill in an address the user did not give.--thresholdtunes the fuzzy title-match ratio (0..1; default 0.70).--offlineskips the network and emitsunverifiedverdicts deliberately.- Omit
--outto print the JSON report to stdout. - Optional:
export S2_API_KEY=…before large bibliographies — the keyless Semantic Scholar pool is shared and busy. Never needed for a verdict. - Exit codes: 0 complete, 2 bad input / nothing parseable, 3 partial (the report is still written).
The script uses requests if installed, else the Python stdlib (urllib), so it
runs with no extra dependencies in a bare uv environment. Expect roughly 3–10 s
per entry: it spaces arXiv calls 3 s apart and backs off on rate limits.
3. Read the JSON and report
Check status first: if it is partial, list input_accounting.rejected_entries and
unsupported_entries with their line numbers and reasons before anything else. Then parse
summary.verdict and the per-entry verdict codes. Present:
- The headline verdict and counts (
verdict_counts,severity_counts). - A table of every non-
verifiedentry with its code, severity, anddetail. - For each
ID_INVALID/IH/PH/NOT_FOUND: quote the cited entry and explain the evidence (e.g. "DOI 10.x resolved to an unrelated paper titled '…'", "DOI not registered at doi.org — identifier wrong, work undetermined", "not found in crossref, europepmc, arxiv — needs a manual check"). - Any
RETRACTED/EXPRESSION_OF_CONCERNflags, prominently, and how many found entries haveretraction.status: unknown(not checked — not "not retracted"). - For
unverifiedentries: say which sources failed (source_status), relay themanual_instructionsand anysummary.notes(for example, a Semantic Scholar HTTP 429 means its shared keyless pool was busy: that source was not checked, which is not "not found"), and run the fallback below. For aNOT_FOUNDentry, quote the indexes itsdetailsays answered with no match, and run the same fallback.
4. Route fixes
PH→ the reference must be completed, or removed or replaced.ID_INVALID→ the identifier is wrong; find the correct DOI/arXiv ID by title, or check whether the work exists at all. Do not call it fabricated on this evidence alone.NOT_FOUND→ manual check (fallback below). Only a documented manual search that finds nothing supportsTF(remove or replace the reference).IH→ the DOI/arXiv ID is wrong; find and substitute the correct identifier.PAC→ correct the specific metadata field(s) named indetail; a possible mashup needs the real source identified before it can stay.RETRACTED→ flag to the author; cite the retraction notice or drop the source.
Graceful Degradation and the Fallback
Failures are never swallowed into a pass:
- A DNS/connection failure raises
NetworkUnavailable; the entry becomesunverifiedwith a populatedmanual_instructionsfield. - A rate-limited or failing source is recorded in
source_status; if the sources needed for a decision did not answer, the entry isunverified, notNOT_FOUND. --offlineforces every networked entry tounverifiedup front (placeholders are still caught locally asPH).- The repo-level verdict becomes
UNVERIFIEDwhenever an entry is unverified and nothing SERIOUS was found.
For each unverified or NOT_FOUND entry, re-run with connectivity or a key if possible;
otherwise look it up with the plugin's keyless crossref MCP tools when they
are available (setup: mcp_setup.md in the alterlab-core plugin's references/ folder), then fall back to WebSearch with
three distinct queries (exact title in quotes; title + first author; first author +
venue + year) and, for a DOI, open https://doi.org/<DOI>. If a record is found, report the verdict it supports
and say it came from the fallback. If nothing is found, report it as NOT_FOUND with the
documented queries; concluding TF is the analyst's call on that documented search, never the
script's. There is no "difficult to verify" outcome: the Lin et al.
mashup this skill was built around slipped through three integrity rounds
precisely because it was parked in that gray zone and never searched.
Claim Faithfulness (SH) — scripts/claim_faithfulness.py
Existence is half of the gate; a real paper cited for something it never said
(the "Frankenstein" pattern) passes every existence check. For (claim, DOI) pairs,
claim_faithfulness.py fetches the cited abstract (Crossref, falling back to Semantic
Scholar, then Europe PMC; signals.abstract_sources says what each answered, and a
rate-limited source is reported as not checked) and returns support / contradict /
unsupported:
uv run python scripts/claim_faithfulness.py \
--claim "<the sentence as written>" --doi <doi>
uv run python .../claim_faithfulness.py --input pairs.json --json # batch
uv run python .../claim_faithfulness.py --input pairs.json --tier llm --json
- The default
heuristictier is lexical and deliberately abstains (unsupported) rather than guessingsupport; it cannot see role reversals or a changed population reliably, so itssupportcarries taxonomyLEXICAL_MATCH_REQUIRES_REVIEW— report it as "needs review", never as verified. Guards make it abstain on a figure the abstract lacks, a reversed or unmatched comparison, a population mismatch (people vs animals or cells; age group), and a definite or causal claim against tentative or associational evidence. An llm-tiersupportisMODEL_JUDGED_SUPPORT_REQUIRES_REVIEW— one model's reading, also not verification. The script never returns VERIFIED for a claim. Thellmtier asks the model set byALTERLAB_MODEL(seeshared/model_env.md) and falls back to the heuristic, flagged, if theclaudeCLI is unavailable. - It only sees the abstract (
abstract_only: true).unsupportedmeans the abstract does not establish the claim — non-coverage, not refutation. Reportcontradictas SH; sendunsupportedclaims that matter to full-text checking. - Always report existence and faithfulness separately: "the citation is real" and "the citation supports this sentence" are different findings.
Output Shape (excerpt)
{
"tool": "alterlab-citation-verifier/verify_citations.py",
"version": "2.0.0",
"schema_version": 2,
"status": "complete",
"warning": null,
"summary": {
"total": 2,
"input": {"detected": 2, "parsed": 2, "rejected": 0, "unsupported": 0},
"verdict": "FAIL",
"verdict_counts": {"verified": 1, "PAC": 0, "IH": 0, "ID_INVALID": 1, "NOT_FOUND": 0, "PH": 0, "TF": 0, "SH": 0, "unverified": 0},
"severity_counts": {"SERIOUS": 1, "MEDIUM": 0, "MINOR": 0},
"citation_integrity_score": 0.5,
"fabrication_risk_score": 0.5,
"retracted": 0,
"retraction": {"flagged": 0, "not_flagged_in_checked_sources": 1, "unknown": 0},
"existence": {"found": 1, "identifier_invalid": 1},
"source_errors": {"semanticscholar": 1},
"notes": ["Semantic Scholar rate limit: not checked (which is not 'not found'); the keyless pool is shared ..."]
},
"entries": [
{"ref_id": "walters2023", "verdict": "verified", "severity": "NONE",
"existence": {"status": "found", "sources_answered": ["arxiv", "crossref", "europepmc"], "sources_not_checked": ["semanticscholar"]},
"identifier": {"doi": {"status": "resolves_to_cited_work"}, "arxiv": {"status": "not_cited"}},
"metadata": {"title": {"status": "agrees"}, "authors": {"status": "agrees"}, "year": {"status": "agrees"}},
"retraction": {"status": "not_flagged_in_checked_sources", "sources_answered": ["crossref"]},
"retracted": false, "title_ratio": 1.0, "author_overlap": 1.0, "matches": [{"source": "crossref"}],
"source_status": {"crossref": "record", "semanticscholar": "error: HTTP 429 — Semantic Scholar rate limit: not checked ...", "europepmc": "no_record", "arxiv": "no_record"}},
{"ref_id": "ghostpaper2021", "verdict": "ID_INVALID", "severity": "SERIOUS",
"existence": {"status": "identifier_invalid"},
"identifier": {"doi": {"status": "unregistered"}},
"detail": "DOI 10.xxxx/... is not registered with any DOI agency (doi.org Handle API), so the identifier is wrong ... Whether the work itself exists is undetermined ..."}
]
}
citation_integrity_score and fabrication_risk_score (both 0..1) align with
the Integrity Report schema fields of the same name, so the report can feed
alterlab-research-pipeline's integrity gate directly. fabrication_risk_score is
(IH + PH + ID_INVALID + NOT_FOUND [+ legacy TF]) / total — a risk indicator, not proof.
Self-Check Before Reporting
- Is
statuscomplete? A PARTIAL run (rejected or unsupported entries) is never reported as a pass. - Did the run reach the network, and which sources answered? If entries are
unverified, name the failing sources fromsource_status; do not imply a pass. - Are there any
RETRACTEDflags? Surface them even on otherwise-verifiedentries. - Did any entry score
IH? Confirm the detail shows an id-resolved mismatch, not a loose title-search coincidence (the script enforces this distinction). - Is the headline verdict consistent with the per-entry codes (any SERIOUS → FAIL;
any
NOT_FOUNDwithout SERIOUS → REVIEW_REQUIRED; anyunverifiedwithout SERIOUS orNOT_FOUND→ UNVERIFIED)? - Did every
unverifiedandNOT_FOUNDentry get the WebSearch fallback and a final verdict? - Did you call anything "fabricated" or
TF? Only after a documented manual search; never onNOT_FOUNDorID_INVALIDalone. Did you presentretraction.status: unknownas "not retracted"? Don't.
References
alterlab-research-pipeline/agents/integrity_verification_agent.md— canonical Five-Type Taxonomy, compound-deception patterns, and the Lin et al. (2020) mashup case study this skill is built to catch.alterlab-research-pipeline/references/claim_verification_protocol.md— Phase E full-text claim-vs-source verification (beyond the abstract-level SH triage here).shared/schemas/integrity_report.schema.json— the integrity-report shape whosecitation_integrity_score/fabrication_risk_scorethis skill mirrors.examples/caught_hallucination_walkthrough.md— end-to-end run on a real mashup.- Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13, 14045. https://doi.org/10.1038/s41598-023-41032-5
- Ansari, S. (2026). Compound deception in elite peer review: A failure mode taxonomy of 100 fabricated citations at NeurIPS 2025 (arXiv:2602.05930). https://arxiv.org/abs/2602.05930
- Rittman, M. (2025, January 29). Retraction Watch retractions now in the Crossref API [Blog post]. Crossref. https://www.crossref.org/blog/retraction-watch-retractions-now-in-the-crossref-api/
Part of the AlterLab Academic Skills suite.
Files (alterlab-academic-skills)
-
commands
-
cite-check.md 3.7 KB
Loading…
-
-
evals
-
fixtures
-
bibliography_24_entries.md 6.8 KB
Loading… -
bibtex_edge_cases.bib 1.7 KB · in bundle
-
bibtex_mixed_malformed.bib 1.4 KB · in bundle
-
claim_faithfulness_adversarial.json 3.1 KB
Loading…
-
-
evals.json 10.5 KB
Loading…
-
-
examples
-
caught_hallucination_walkthrough.md 10.5 KB
Loading…
-
-
scripts
-
claim_faithfulness.py 39.6 KB
Loading… -
verify_citations.py 130.6 KB
Loading…
-
-
shared
-
schemas
-
integrity_report.schema.json 9.6 KB
Loading…
-
-
model_env.md 8.2 KB
Loading…
-
-
SKILL.md 26.3 KB
--- name: alterlab-citation-verifier description: "Verifies every bibliography entry actually exists by checking Crossref, Semantic Scholar, Europe PMC, and arXiv, with Open Library for books (no API key required), plus doi.org DOI registration, fuzzy-matching title and authors, flagging retractions recorded by Crossref (incl. Retraction Watch), and emitting per-entry JSON verdicts in the AlterLab citation-hallucination taxonomy (PAC/IH/PH/SH/ID_INVALID; index absence is NOT_FOUND, never fabrication); a companion script scores whether a cited abstract supports a claim (SH). Accepts BibTeX, DOI/arXiv lists, or pasted references, and reports unverified instead of passing anything it could not check. Use when the request mentions verify citations, check references, fabricated or hallucinated references, fake DOI, retraction check, bibliography audit, reference existence check, or whether a source supports a claim. For drafting a manuscript prefer alterlab-paper-writer; for dead hyperlinks prefer alterlab-link-health. Part of the AlterLab Academic Skills suite." license: MIT allowed-tools: Read Write Edit Bash(python:*) Bash WebSearch WebFetch compatibility: "No API key required — stdlib Python (`requests` optional) via `uv run python` against Crossref, Semantic Scholar, Europe PMC, arXiv, Open Library and the doi.org Handle API; optional S2_API_KEY only raises Semantic Scholar's rate limit; offline runs return 'unverified' verdicts" metadata: skill-author: AlterLab version: "1.3.0" last_updated: "2026-11-05" credentials: "optional: S2_API_KEY or SEMANTIC_SCHOLAR_API_KEY (raises Semantic Scholar's shared keyless rate limit: a dedicated 1 request/s; never needed for a verdict)" depends_on: "alterlab-research-pipeline (shares the integrity taxonomy), alterlab-deep-research" --- # Citation Verifier — Existence-Verify a Bibliography Against Public Scholarly APIs The headline existence-verification skill: given a bibliography in any common form, it proves entry-by-entry whether each reference **actually exists** by querying public scholarly APIs, then maps each result to the canonical AlterLab citation-hallucination taxonomy. It is the deterministic, network-grounded companion to the LLM-driven `integrity_verification_agent` — where that agent uses WebSearch + judgment, this skill uses authoritative API records and a reproducible Python script, so every verdict rests on retrieved records and the same input and source responses yield the same verdicts. Verify references with the script, not from memory. A model checking citations against what it "remembers" shares the training data that produced the hallucination in the first place, so a fabricated reference that feels right passes undetected; only an external record settles existence. ## Quick Start ``` Verify the citations in references.bib Check whether these DOIs resolve to the papers I cited Audit my bibliography for fabricated / hallucinated references Does this reference list contain any fake citations or retractions? ``` → Run `scripts/verify_citations.py` over the bibliography, read the JSON, then present a verdict table grouped by severity. State the offline/degraded status explicitly if the network or a source was unavailable. ## When to Use This Skill - "Verify / check / audit my citations or references exist" - "Did the AI hallucinate any of these references?" / "Is this DOI fake?" - "Do these DOIs resolve to the papers I cited?" - "Check this bibliography for retractions" - "Does this source actually support the sentence I cite it for?" (SH, abstract level) - A reproducible, scriptable existence check with an explicit offline mode ### Does NOT Trigger | Scenario | Use Instead | |----------|-------------| | Writing or revising the paper itself (its citation-check mode formats citations) | `alterlab-paper-writer` | | Finding papers, extracting metadata, or generating BibTeX for new sources | `alterlab-citation-mgmt` | | Dead or redirected hyperlinks (link rot) in docs or a reference list | `alterlab-link-health` | | Full pre-/post-review integrity gate (citation context, data, originality) | `alterlab-research-pipeline` | | Holistic peer review of a manuscript with an editorial decision | `alterlab-paper-reviewer` | **Hand-off rule.** When the request matches a row above, open your answer by naming that skill by its exact id (for example `alterlab-paper-writer`) and saying in one line why it fits better, then stop applying this skill's procedure. Name it even when you also need to ask for missing input. If only part of the request is out of scope, name the skill for that part and do only the in-scope part. A brief general pointer is fine; the named skill does the work. Grading source quality or predatory venues belongs to `alterlab-deep-research` (`source_verification_agent`). For a whole-manuscript audit in Claude Code where every flag should be re-checked by separate agents, offer the packaged `/alterlab-workflows:citation-audit` workflow (`alterlab-research-workflows`, roughly 10–40 agents), which runs this skill's checks at scale. --- ## What This Does For each bibliography entry the script: 1. **Parses** the input (auto-detects BibTeX / a bare DOI-or-arXiv list / free-form references, including one-reference-per-line pastes), extracting title, authors, year, venue, DOI, and arXiv ID. A full reference that merely contains a DOI stays free-form, so its title and authors are kept for the Identifier Hijacking check. Every detected entry ends up parsed, **rejected** (malformed — with line, key, and reason) or **unsupported**; see `input_accounting`. Mixed BibTeX + free-form/DOI input is handled piecewise, and a broken entry cannot swallow the ones after it. Text that is not a reference — a Markdown heading, an intro or prose paragraph, a line with no year, no DOI/arXiv/ISBN/URL and no author name — is **skipped** (`input_accounting.ignored_text`, reason "skipped: not a reference") and gets no verdict. A year in running text ("compiled in 2019 for the review") does not make prose a reference; a terse real one (`Smith J. 2019. Title. Journal 3:1-2.`) is checked. 2. **Resolves identifiers** — looks up the cited DOI/arXiv ID directly when present, and asks the doi.org Handle API whether a DOI is registered with any agency. 3. **Searches by title + first author + year** as a fallback across the sources (Crossref `query.bibliographic`, Semantic Scholar, Europe PMC `TITLE:"…" AND AUTH:…`, arXiv, and Open Library for entries with no identifier, which are mostly books). 4. **Fuzzy-matches** the cited title (difflib `SequenceMatcher` ratio, default threshold **0.70**; a main title that is near-identical to a record stored without its subtitle also counts) and computes author-surname overlap. Among the returned records it keeps the one that matches title, then authors, then year — so a same-titled paper by other authors or another edition does not win. 5. **Flags retractions** on the matched work: Crossref `updated-by` / `update-to` notices of type retraction, withdrawal, or removal (from the publisher or, with `source: retraction-watch`, the Retraction Watch database that Crossref serves in its REST API since 2025). Expressions of concern get a separate flag. 6. **Emits a verdict** per entry mapped to the taxonomy below, with a `source_status` map showing which sources answered, separate `existence`, `identifier`, `metadata` and `retraction` blocks, plus a repo-level `summary.verdict` (FAIL / PARTIAL / REVIEW_REQUIRED / UNVERIFIED / PASS_WITH_CONDITIONS / PASS). ### Sources (no key required; checked 2026-11-02) | Source | Endpoint | Used for | Access notes | |--------|----------|----------|--------------| | Crossref | `api.crossref.org/works` | DOI resolution, bibliographic search, retraction notices | Public pool: 5 req/s, one request in flight (the script spaces calls and backs off on 429); an opt-in `--mailto` routes to the polite pool (10 req/s) | | Semantic Scholar | `api.semanticscholar.org/graph/v1` | DOI/arXiv resolution, title search (best effort) | Keyless pool is shared. DOI/arXiv lookups mostly answer (backoff 2 + 4 + 8 s, then a 30 s pause); its **search** pool is usually spent from busy networks, so search gets one retry and a 5-minute pause. Either way it is recorded as *not checked*, never "not found", and no verdict depends on it. `S2_API_KEY` gives a dedicated 1 req/s | | Europe PMC | `ebi.ac.uk/europepmc/webservices/rest/search` | DOI lookup, title phrase + first-author search | Keyless; life sciences first, but also many other journals and preprint servers | | arXiv | `export.arxiv.org/api/query` | arXiv ID resolution, preprint title search | ≥ 3 s between calls (the script throttles) | | Open Library | `openlibrary.org/search.json` | Books (entries with no DOI/arXiv ID): main title + first author; every edition's year is kept | Keyless; 1 s between calls | | doi.org Handle API | `doi.org/api/handles/<doi>` | Is the DOI registered with *any* agency (Crossref, DataCite, mEDRA…)? | Authoritative for DOI existence | Keys are optional, read from the environment, sent as headers, and never written to the report (the report only records whether one was present). A source that errors (rate limit, 5xx) is recorded in `source_status` and is never counted as evidence that a work does not exist. One dropped connection is retried before a source is marked `offline`. OpenAlex is no longer queried: since 2026 its keyless searches draw on a small per-IP daily budget, so dependable use needed an account. This skill answers **"does the cited work exist, and does its identifier point to it?"** From API metadata alone it cannot establish Semantic Hallucination (does the source support the claim?); `scripts/claim_faithfulness.py` gives an abstract-level SH triage (see below), and full-text claim verification is the research pipeline's `claim_verification_protocol` (Phase E). --- ## Verdict Taxonomy (mirrors the canonical Five-Type Taxonomy) Identical codes and definitions to `alterlab-research-pipeline/agents/integrity_verification_agent.md` (GPTZero × NeurIPS 2025; Ansari, 2026). Severity feeds the same SERIOUS / MEDIUM / MINOR scale used in the Integrity Report schema. | Code | Name | Severity | Script trigger | |------|------|----------|----------------| | `verified` | — (exists, matches) | NONE | Found in >=1 authoritative source and every supplied field that could be checked (title ratio >= threshold, authors, year) agrees | | `PAC` | Partial Attribute Corruption | MEDIUM | Entry found but >=1 supplied field disagrees: a cited author the record does not list (`metadata.authors.unsupported`, flag `UNSUPPORTED_AUTHOR`), a wrong year, a cited identifier that does not resolve to it (`IDENTIFIER_DOES_NOT_RESOLVE`), or a garbled title whose closest record shares the authors (`POSSIBLE_MASHUP`). "et al." / BibTeX "and others" is honoured; an abbreviated list without it is noted, not an error | | `ID_INVALID` | Invalid identifier | **SERIOUS** | Cited DOI is not registered at doi.org, or cited arXiv ID does not exist on arXiv, and no close title match exists. The identifier is wrong; **whether the work exists is undetermined** | | `NOT_FOUND` | Not found in the checked sources | MEDIUM | No identifier, and no record matches title + authors in Crossref plus at least one other index that actually answered (Semantic Scholar, Europe PMC, arXiv or Open Library); `detail` names the indexes that answered and those not checked. **Absence from indexes is not proof of fabrication** (books, grey literature, theses, regional venues) — check manually. If only Crossref answered, or a free-form entry lacks a title plus an author or year, the entry is `unverified` | | `IH` | Identifier Hijacking | **SERIOUS** | Cited DOI/arXiv ID **resolved** (method=id) but the resolved record's title is unrelated (ratio < threshold) | | `PH` | Placeholder Hallucination | **SERIOUS** | Unresolved template/placeholder (`[CITATION NEEDED]`, `\cite{}`, `et al., YYYY`, `TODO`, `forthcoming`, `in press`) — caught pre-network | | `SH` | Semantic Hallucination | **SERIOUS** | Entry resolves but does not support its claim — **advisory only** here; asserted via `claim_faithfulness.py` / Phase E | | `unverified` | — (could not check) | MEDIUM | Offline, the sources needed for a decision errored, or a registered DOI that no index returned. **Never treated as passing.** | | `TF` | Total Fabrication | **SERIOUS** | **Never emitted by the script.** Reserved for an analyst's conclusion after a documented manual search (the fallback below) finds no trace of a `NOT_FOUND` / `ID_INVALID` work | A `RETRACTED` flag is attached (and severity bumped to SERIOUS) whenever Crossref (publisher notice or Retraction Watch) marks the *matched* work as retracted, independent of the existence verdict; an unrelated closest hit never lends its flag to the cited entry. Every found entry carries `retraction.status` — `flagged`, `not_flagged_in_checked_sources` (Crossref answered with no notice; other retraction databases were not checked) or `unknown` (Crossref did not answer or holds no record) — and the v1 `retracted` key is `true` / `false` / `null` (`null` = unknown). Never report `unknown` / `null` as "not retracted"; `summary.retraction` counts the three states. Since report schema 2 (`verify_citations.py` 2.0.0) the script no longer reports `TF` for a reference absent from the indexes (v1 did): that is `NOT_FOUND`, and an unregistered DOI or nonexistent arXiv ID is `ID_INVALID`. `TF` is a serious accusation that only a documented manual search can support. When the deciding sources were rate-limited or down, the entry is `unverified` with the reason and `source_status`, and it goes to the fallback below. ### Repo-level verdict Most severe first: FAIL > PARTIAL > REVIEW_REQUIRED > UNVERIFIED > PASS_WITH_CONDITIONS > PASS. Only PASS and PASS_WITH_CONDITIONS are passes. - **PASS** — every entry `verified`, no SERIOUS/MEDIUM flags. Retraction status is reported separately (`summary.retraction`) and may be `unknown` for some entries. - **PASS_WITH_CONDITIONS** — only `PAC` / MEDIUM metadata issues (fixable), and every entry was found. - **FAIL** — any SERIOUS finding (`IH` / `ID_INVALID` / `PH` / retraction). - **REVIEW_REQUIRED** — nothing SERIOUS, but at least one entry is `NOT_FOUND`. Not a pass: check those entries manually (absence is not proof of fabrication). - **UNVERIFIED** — nothing SERIOUS was found but at least one entry could not be checked (e.g. an offline run or rate-limited sources). This is **not** a pass — re-run with network access or an API key, or resolve those entries manually. - **PARTIAL** — at least one detected entry was rejected, unsupported, or left unprocessed (`status: partial`, exit code 3). Not a pass: fix the entries listed in `input_accounting` and re-run. FAIL still takes precedence; PARTIAL outranks REVIEW_REQUIRED. --- ## Pipeline (how to run it) ### 1. Locate or capture the bibliography Accept any of: a `.bib` file, a `.txt` list of DOIs/arXiv IDs, a pasted reference list, or inline text. The script auto-detects the format; override with `--format bibtex|doi|freeform` if detection is wrong. ### 2. Run the verifier ```bash uv run python scripts/verify_citations.py \ path/to/references.bib \ --mailto <contact-email> \ --threshold 0.70 \ --out citation_report.json ``` - `path/to/references.bib` may also be `-` (stdin) or inline text. - `--mailto` is optional: the user's own contact email for Crossref's polite pool (or set `ALTERLAB_CONTACT_EMAIL`). Without it no address is sent and Crossref's public pool answers. Never fill in an address the user did not give. - `--threshold` tunes the fuzzy title-match ratio (0..1; default 0.70). - `--offline` skips the network and emits `unverified` verdicts deliberately. - Omit `--out` to print the JSON report to stdout. - Optional: `export S2_API_KEY=…` before large bibliographies — the keyless Semantic Scholar pool is shared and busy. Never needed for a verdict. - Exit codes: 0 complete, 2 bad input / nothing parseable, 3 partial (the report is still written). The script uses `requests` if installed, else the Python stdlib (`urllib`), so it runs with no extra dependencies in a bare `uv` environment. Expect roughly 3–10 s per entry: it spaces arXiv calls 3 s apart and backs off on rate limits. ### 3. Read the JSON and report Check `status` first: if it is `partial`, list `input_accounting.rejected_entries` and `unsupported_entries` with their line numbers and reasons before anything else. Then parse `summary.verdict` and the per-entry `verdict` codes. Present: 1. The **headline verdict** and counts (`verdict_counts`, `severity_counts`). 2. A **table of every non-`verified` entry** with its code, severity, and `detail`. 3. For each `ID_INVALID` / `IH` / `PH` / `NOT_FOUND`: quote the cited entry and explain the evidence (e.g. "DOI 10.x resolved to an unrelated paper titled '…'", "DOI not registered at doi.org — identifier wrong, work undetermined", "not found in crossref, europepmc, arxiv — needs a manual check"). 4. Any `RETRACTED` / `EXPRESSION_OF_CONCERN` flags, prominently, and how many found entries have `retraction.status: unknown` (not checked — not "not retracted"). 5. For `unverified` entries: say which sources failed (`source_status`), relay the `manual_instructions` and any `summary.notes` (for example, a Semantic Scholar HTTP 429 means its shared keyless pool was busy: that source was not checked, which is not "not found"), and run the fallback below. For a `NOT_FOUND` entry, quote the indexes its `detail` says answered with no match, and run the same fallback. ### 4. Route fixes - `PH` → the reference must be completed, or removed or replaced. - `ID_INVALID` → the identifier is wrong; find the correct DOI/arXiv ID by title, or check whether the work exists at all. Do not call it fabricated on this evidence alone. - `NOT_FOUND` → manual check (fallback below). Only a documented manual search that finds nothing supports `TF` (remove or replace the reference). - `IH` → the DOI/arXiv ID is wrong; find and substitute the correct identifier. - `PAC` → correct the specific metadata field(s) named in `detail`; a possible mashup needs the real source identified before it can stay. - `RETRACTED` → flag to the author; cite the retraction notice or drop the source. --- ## Graceful Degradation and the Fallback Failures are never swallowed into a pass: - A DNS/connection failure raises `NetworkUnavailable`; the entry becomes `unverified` with a populated `manual_instructions` field. - A rate-limited or failing source is recorded in `source_status`; if the sources needed for a decision did not answer, the entry is `unverified`, not `NOT_FOUND`. - `--offline` forces every networked entry to `unverified` up front (placeholders are still caught locally as `PH`). - The repo-level verdict becomes `UNVERIFIED` whenever an entry is unverified and nothing SERIOUS was found. For each `unverified` or `NOT_FOUND` entry, re-run with connectivity or a key if possible; otherwise look it up with the plugin's keyless `crossref` MCP tools when they are available (setup: `mcp_setup.md` in the alterlab-core plugin's `references/` folder), then fall back to WebSearch with three distinct queries (exact title in quotes; title + first author; first author + venue + year) and, for a DOI, open `https://doi.org/<DOI>`. If a record is found, report the verdict it supports and say it came from the fallback. If nothing is found, report it as `NOT_FOUND` with the documented queries; concluding `TF` is the analyst's call on that documented search, never the script's. There is no "difficult to verify" outcome: the Lin et al. mashup this skill was built around slipped through three integrity rounds precisely because it was parked in that gray zone and never searched. --- ## Claim Faithfulness (SH) — `scripts/claim_faithfulness.py` Existence is half of the gate; a real paper cited for something it never said (the "Frankenstein" pattern) passes every existence check. For (claim, DOI) pairs, `claim_faithfulness.py` fetches the cited abstract (Crossref, falling back to Semantic Scholar, then Europe PMC; `signals.abstract_sources` says what each answered, and a rate-limited source is reported as not checked) and returns `support` / `contradict` / `unsupported`: ```bash uv run python scripts/claim_faithfulness.py \ --claim "<the sentence as written>" --doi <doi> uv run python .../claim_faithfulness.py --input pairs.json --json # batch uv run python .../claim_faithfulness.py --input pairs.json --tier llm --json ``` - The default `heuristic` tier is lexical and deliberately abstains (`unsupported`) rather than guessing `support`; it cannot see role reversals or a changed population reliably, so its `support` carries taxonomy `LEXICAL_MATCH_REQUIRES_REVIEW` — report it as "needs review", never as verified. Guards make it abstain on a figure the abstract lacks, a reversed or unmatched comparison, a population mismatch (people vs animals or cells; age group), and a definite or causal claim against tentative or associational evidence. An llm-tier `support` is `MODEL_JUDGED_SUPPORT_REQUIRES_REVIEW` — one model's reading, also not verification. The script never returns VERIFIED for a claim. The `llm` tier asks the model set by `ALTERLAB_MODEL` (see `shared/model_env.md`) and falls back to the heuristic, flagged, if the `claude` CLI is unavailable. - It only sees the abstract (`abstract_only: true`). `unsupported` means the abstract does not establish the claim — non-coverage, not refutation. Report `contradict` as SH; send `unsupported` claims that matter to full-text checking. - Always report existence and faithfulness separately: "the citation is real" and "the citation supports this sentence" are different findings. --- ## Output Shape (excerpt) ```json { "tool": "alterlab-citation-verifier/verify_citations.py", "version": "2.0.0", "schema_version": 2, "status": "complete", "warning": null, "summary": { "total": 2, "input": {"detected": 2, "parsed": 2, "rejected": 0, "unsupported": 0}, "verdict": "FAIL", "verdict_counts": {"verified": 1, "PAC": 0, "IH": 0, "ID_INVALID": 1, "NOT_FOUND": 0, "PH": 0, "TF": 0, "SH": 0, "unverified": 0}, "severity_counts": {"SERIOUS": 1, "MEDIUM": 0, "MINOR": 0}, "citation_integrity_score": 0.5, "fabrication_risk_score": 0.5, "retracted": 0, "retraction": {"flagged": 0, "not_flagged_in_checked_sources": 1, "unknown": 0}, "existence": {"found": 1, "identifier_invalid": 1}, "source_errors": {"semanticscholar": 1}, "notes": ["Semantic Scholar rate limit: not checked (which is not 'not found'); the keyless pool is shared ..."] }, "entries": [ {"ref_id": "walters2023", "verdict": "verified", "severity": "NONE", "existence": {"status": "found", "sources_answered": ["arxiv", "crossref", "europepmc"], "sources_not_checked": ["semanticscholar"]}, "identifier": {"doi": {"status": "resolves_to_cited_work"}, "arxiv": {"status": "not_cited"}}, "metadata": {"title": {"status": "agrees"}, "authors": {"status": "agrees"}, "year": {"status": "agrees"}}, "retraction": {"status": "not_flagged_in_checked_sources", "sources_answered": ["crossref"]}, "retracted": false, "title_ratio": 1.0, "author_overlap": 1.0, "matches": [{"source": "crossref"}], "source_status": {"crossref": "record", "semanticscholar": "error: HTTP 429 — Semantic Scholar rate limit: not checked ...", "europepmc": "no_record", "arxiv": "no_record"}}, {"ref_id": "ghostpaper2021", "verdict": "ID_INVALID", "severity": "SERIOUS", "existence": {"status": "identifier_invalid"}, "identifier": {"doi": {"status": "unregistered"}}, "detail": "DOI 10.xxxx/... is not registered with any DOI agency (doi.org Handle API), so the identifier is wrong ... Whether the work itself exists is undetermined ..."} ] } ``` `citation_integrity_score` and `fabrication_risk_score` (both 0..1) align with the Integrity Report schema fields of the same name, so the report can feed `alterlab-research-pipeline`'s integrity gate directly. `fabrication_risk_score` is (IH + PH + ID_INVALID + NOT_FOUND [+ legacy TF]) / total — a risk indicator, not proof. --- ## Self-Check Before Reporting - Is `status` complete? A PARTIAL run (rejected or unsupported entries) is never reported as a pass. - Did the run reach the network, and which sources answered? If entries are `unverified`, name the failing sources from `source_status`; do not imply a pass. - Are there any `RETRACTED` flags? Surface them even on otherwise-`verified` entries. - Did any entry score `IH`? Confirm the detail shows an **id-resolved** mismatch, not a loose title-search coincidence (the script enforces this distinction). - Is the headline verdict consistent with the per-entry codes (any SERIOUS → FAIL; any `NOT_FOUND` without SERIOUS → REVIEW_REQUIRED; any `unverified` without SERIOUS or `NOT_FOUND` → UNVERIFIED)? - Did every `unverified` and `NOT_FOUND` entry get the WebSearch fallback and a final verdict? - Did you call anything "fabricated" or `TF`? Only after a documented manual search; never on `NOT_FOUND` or `ID_INVALID` alone. Did you present `retraction.status: unknown` as "not retracted"? Don't. --- ## References - `alterlab-research-pipeline/agents/integrity_verification_agent.md` — canonical Five-Type Taxonomy, compound-deception patterns, and the Lin et al. (2020) mashup case study this skill is built to catch. - `alterlab-research-pipeline/references/claim_verification_protocol.md` — Phase E full-text claim-vs-source verification (beyond the abstract-level SH triage here). - `shared/schemas/integrity_report.schema.json` — the integrity-report shape whose `citation_integrity_score` / `fabrication_risk_score` this skill mirrors. - `examples/caught_hallucination_walkthrough.md` — end-to-end run on a real mashup. - Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. *Scientific Reports, 13*, 14045. https://doi.org/10.1038/s41598-023-41032-5 - Ansari, S. (2026). *Compound deception in elite peer review: A failure mode taxonomy of 100 fabricated citations at NeurIPS 2025* (arXiv:2602.05930). https://arxiv.org/abs/2602.05930 - Rittman, M. (2025, January 29). *Retraction Watch retractions now in the Crossref API* [Blog post]. Crossref. https://www.crossref.org/blog/retraction-watch-retractions-now-in-the-crossref-api/ Part of the AlterLab Academic Skills suite.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.