# Reviewing a sentence against its cited source

Use the existing claim-fidelity report to connect a manuscript sentence to the
source actually inspected. This workflow does not fetch papers, invent source
excerpts, or turn word/number matches into semantic support. It adds no detector
or submission-blocking verdict; `--strict` retains the existing probe behavior.

## Create the review surface

Run the command in the skill's “Sentence-level source evidence table” section.
`--retrieval-report` consumes the existing `fetch_oa.py` report, including each
DOI's PDF filename, hash and advisory identity assessment. PDFs are looked up next
to that report unless `--pdf-dir` selects another directory. The converted texts
can be in a different `--fulltext-dir`.

`--reference-audit` reads existing `records[]` from the metadata audit. Its recorded
OK status has no input hash binding in that artifact and is shown as context only.
It cannot establish PDF identity, a current bibliography audit, or source support.
Missing/ambiguous links remain visible. No network or bibliography writes occur.

`qc/claim_fidelity.json` schema 2 retains the existing `findings`, probe counts,
and source-resolution fields, and adds:

| Field | Meaning |
|---|---|
| `evidence_rows[]` | One row per recognized prose sentence/citation pair, including citations outside the three probes. Duplicate occurrences have distinct IDs. |
| `manuscript` | Exact text, character offsets and line coordinates in the `read_text` representation. DOCX coordinates refer to extracted paragraph/table text, not rendered pages. |
| `source` | Selected converted text and hash, actual PDF hash, report hash, advisory identity and hash status. Filename/DOI resolution is not identity verification. |
| `reference_audit` | Linked metadata status with its binding limitation. |
| `automatic_finding_indices` | Same-sentence or quoted-span links to existing `findings`; these do not assign the manual assessment. |
| `assessment` | Editable, attributed review record described below. Initially `not_assessed`. |
| `binding` / `binding_sha256` | Current manuscript, bibliography, mapping, report and source hashes. |
| `verdict` / `review_state` | Effective recorded outcome and whether the assessment is current, incomplete or stale. Never a machine-certified fact. |
| `orphaned_evidence_reviews` | Preserved prior rows that no longer match a current sentence/citation pair. Excluded from current assessment counts. |

The old `sources_resolved` and `claims_checked` fields still describe probe
execution; use `evidence_counts` for the broader sentence/citation inventory.
Neither count covers uncited statements, Markdown tables, blockquotes, code or
the bibliography. Citation recognition is syntactic and can miss unsupported
formats. A complete inventory is not a complete scientific review.

## Enter an assessment after reading the source

Edit only the row's `assessment` in the JSON. Supply:

- `verdict`: `supported`, `contradicted`, `unresolved` or `not_assessed`.
- `assessor`, `assessed_at`, `method`, `rationale`: who made the judgment, when,
  how it was made (including AI assistance), and why. These are attributed
  declarations, not authenticated human signatures or approval.
- `source_scope`: `full_text`, `abstract_only`, `partial` or `unknown`;
  `identity_checked`: whether the assessor actually checked the paper identity.
- `source_pages`: a nonempty list of one-based PDF page indices, not printed
  journal page numbers; `source_excerpt`: the relevant text actually inspected.
- `claim` and `source`: compare `metric`, `unit`, `denominator`, `population`,
  and `direction` separately. Use an explicit `not applicable` explanation for a
  qualitative field; an empty field is not a completed comparison.

For example, a synthetic software experiment can report the same latency
percentage for a subgroup that the manuscript attributes to the whole sample.
The equal number does not support the manuscript's population claim. A direction
reversal also needs a source-level judgment despite matching terms and numbers.

Do not fill pages or excerpts from memory. The tool does not check that an
excerpt appears on the declared PDF page, authenticate an assessor, or adjudicate
the meaning of a comparison. Complete fields allow a **recorded judgment**, not
an independent source verification. Check the PDF itself, including tables,
subgroups and qualification of the claim, before using a recorded conclusion.

The extraction tool does not currently bind the converted text to the PDF with a
conversion manifest. The report explicitly marks that derivation as unrecorded;
matching filenames are insufficient. Source-text and PDF hashes identify the
files reviewed, not proof that one was faithfully extracted from the other.

## Carry reviews forward without losing their history

Rerun the original command with `--reviewed-report qc/claim_fidelity.json` and the
same `--out`. The old JSON is read first; findings and the Markdown view are
regenerated. `--evidence-table` requires `--out`, and output aliases of selected
source inputs are rejected. Input files are not edited.

The assessment's `binding_sha256` stays pinned to its original inputs. Changes to
the manuscript, bibliography, source text, PDF, input reports or reference map
make the old conclusion `unresolved` with `stale_inputs`. Repeating the command
does not make it current again. Inspect the changed source/context, update the
assessment, and only then copy the row's current `binding_sha256` into
`assessment.binding_sha256`. Changed sentence IDs retain the old row under
`orphaned_evidence_reviews` for explicit reconciliation; they are not silently
discarded or transferred to a different sentence.

A supported/contradicted outcome additionally requires a source text, a PDF
matching the retrieval hash, no reported identity conflict, full-text inspection,
pages, an excerpt and a populated comparison. Missing PDFs, legacy reports
without hashes, DOI-unresolved sources, abstract-only evidence, and incomplete
reviews remain unresolved. A consistent retrieval identity is advisory, not
enough by itself. Resolve conflicting source identity before recording support;
this workflow does not override the retrieval assessment automatically.

The Markdown file is a derived view. Its source excerpts and project context may
be private; do not publish generated reports without reviewing their contents.

# Manual Reference Verification Checkpoint Guide

**Scope**: medsci-skills v1.1.1 Phase 1A.5
**Audience**: Project owner before journal submission or before circulating a near-final draft to co-authors.

`/verify-refs` runs automatically inside `/write-paper` Step 7.3 and via the
pre-save hook `~/.claude/hooks/verify-refs-guard.sh` when a
`submission/*/manuscript/*.docx` or `revision/R*/*circulation*.docx` is saved.
This guide documents the **manual strict-mode run** the owner should perform
immediately before submitting or before the manuscript leaves the project
workspace.

## When to run manually

Run `verify-refs --strict` manually at every one of these checkpoints:

1. **Before first circulation to co-authors.** Catches hallucinations that
   slipped past Step 7.3 in earlier drafts.
2. **Before cover-letter + manuscript package freeze** (`/sync-submission`).
3. **Before each revision resubmission** (R1, R2, ...). Revisions are a common
   site of new citation drift because reviewer responses introduce new
   references.
4. **Before preprint posting** (medRxiv / arXiv). Preprints are public and hard
   to correct.
5. **After any external edit** (co-author returns a `.docx` with tracked
   changes, or an external editor touches references).

## Command

From the project root:

```bash
python3 "${CLAUDE_SKILL_DIR}/scripts/verify_refs.py" \
  manuscript/index.qmd \
  --project-root . \
  --strict
```

Or via the wrapper:

```bash
"${CLAUDE_SKILL_DIR}/scripts/verify_cli.sh" manuscript/index.qmd --strict
```

Docx input is also accepted:

```bash
"${CLAUDE_SKILL_DIR}/scripts/verify_cli.sh" \
  submission/radiology_ai/manuscript_main.docx --strict
```

## What `--strict` changes

- Exit code is non-zero if `submission_safe: false` (any FABRICATED / MISMATCH).
- Exit code is non-zero if any UNVERIFIED row remains.
- Offline verification is not tolerated — the script requires live PubMed /
  CrossRef reachability.

## Reading the output

The only output file is `qc/reference_audit.json`. Inspect with:

```bash
python3 -c "import json; a=json.load(open('qc/reference_audit.json')); \
print('safe=', a['submission_safe'], 'counts=', a['counts'])"
```

Then jump to the records:

```bash
python3 -c "import json; a=json.load(open('qc/reference_audit.json')); \
[print(r['ref_id'], r['status'], r.get('note','')) \
  for r in a['records'] if r['status']!='OK']"
```

## Failure actions

| Status | Action |
|---|---|
| `FABRICATED` | STOP. Locate the citation in the manuscript. Either remove it or replace with a verified entry via `/search-lit` + `/lit-sync`. Never patch `refs.bib` by hand. |
| `MISMATCH` | STOP. Usually a copy-paste error (wrong DOI for the title). Confirm the author's intent and correct via Zotero. |
| `UNVERIFIED` | Review. If the reference genuinely lacks DOI/PMID (rare: old conference abstracts, grey literature), mark `verified: manual` in `refs.bib` via Zotero and re-run. Never keep UNVERIFIED rows in a submission package. |

## Relationship with `/lit-sync`

`/verify-refs` is audit-only. All bibliographic corrections flow through
`/lit-sync` (owner-only) → Zotero → Better BibTeX auto-export →
`manuscript/_src/refs.bib`. Never edit `refs.bib` to satisfy `/verify-refs`.

## Automation checkpoint (informational)

The pre-save hook `verify-refs-guard.sh` already runs on every `.docx` save
under `submission/` or `revision/R*/`. A `FABRICATED` status blocks the save.
It additionally runs **warn-only** (never blocks) on pre-submission and
mentor-circulation drafts that used to skip the audit — `outgoing/`,
`8_Review_Comments/*/outgoing/`, and any `circulation/` path (issue #14) — so a
missing citation audit is surfaced there without interrupting rapid iteration.
The manual strict run is a belt-and-suspenders check at the checkpoints listed
above; it is NOT a substitute for the inline hook.

## Change log

- **2026-04-24 v1.1.1 Phase 1A.5** Initial manual checkpoint guide. Aligned to
  audit-only verify-refs contract (no `references/*` writes).

#!/usr/bin/env python3
"""Claim fidelity — does the cited work actually say what the manuscript says it says?

WHAT THIS ADDS TO `verify_refs.py`

`verify_refs.py` answers "does this reference exist, and are its authors who you say they
are". It cannot answer the question a reader actually asks, which is whether the *sentence
citing it* is true of it. A citation can be perfectly real and still be attached to a claim
the source never makes. That failure survives every existing gate: the DOI resolves, the
authors match, the reference list renders, and the sentence is wrong.

A source can discuss one concept while the manuscript attributes a different concept to it.
Bibliographic identity checks do not inspect that attribution; source reading is still needed.

THREE PROBES, ORDERED BY HOW CHECKABLE THE CLAIM IS

  1. QUOTED TEXT (`CITED_QUOTE_ABSENT`, major)
     A quotation attributed to a source is the one claim with an exact answer: the words are
     in that document or they are not. This is where a quote-operator inversion lives — a
     manuscript that quotes a source as saying "only if" where the source said "so long as"
     has reversed its meaning while looking verbatim.

  2. CONCEPT ATTRIBUTION (`ATTRIBUTION_UNSUPPORTED`, prompt)
     "X proposes <thing> [12]". Paraphrase is legitimate and ordinary, so this cannot be a
     blocker and cannot fire on mere wording differences. It fires only in the extreme case:
     NOT ONE content word of the attributed span appears anywhere in the source, in any
     morphological form. A real paraphrase almost always keeps at least one content stem
     ("propose a framework for oversight" keeps "oversight"); an attribution to the wrong
     concept keeps none. That is the whole precision argument, and it is why the verdict is
     phrased as a prompt to go read the source rather than as a defect.

  3. CARDINAL CLAIMS (`ORDINAL_CLAIM_UNSUPPORTED`, prompt)
     "reports three strategies [12]" is cheap to get wrong and cheap to check. Fires only
     when the noun IS present in the source (so we know the concept was located) but the
     stated cardinal never appears near it. If the noun is absent entirely that is probe 2's
     business, not this one.

DELIBERATELY NOT CHECKED: float ordinals ("as in Table 2 of [12]"). A manuscript refers to
its OWN Table 2 constantly, and often in a sentence that also carries a citation, so the
probe would fire mostly on correct prose. A check that is right in principle and wrong in
practice is a check that gets switched off, and takes the honest ones with it.

KNOWN LIMITATION, STATED SO IT IS NOT MISTAKEN FOR COVERAGE: a *small* alteration inside a
long quotation — flipping "so long as" to "only if", two tokens in twenty — lands inside the
20% loss that `_quote_match` tolerates as extraction damage, so it is reported as
`CITED_QUOTE_UNRESOLVED` (a prompt) rather than as a fabrication. That is not a bug to fix by
tightening the threshold: nothing in a single extracted text distinguishes "the author changed
two words" from "the extractor dropped two words". The claim is still surfaced for a human;
only the severity is withheld. A heavier rewrite falls outside the tolerance and is major. Both
sides of that boundary are pinned in the challenge card.

WHY ABSENCE IS ONLY EVIDENCE FROM A WHOLE DOCUMENT

Every verdict here is an argument from absence, so it is only as good as the text it
searched. Two guards make that argument honest:

  * The source text must be substantial (`MIN_SOURCE_TOKENS`). A PDF that converted to its
    abstract alone would make every probe scream; such a source is reported as unusable
    (`sources_too_short`) and produces no findings. A quote graded absent against a short
    source is downgraded to unresolved, never to a defect.
  * Quote matching goes through `_quote_match.py`, so a correct quote read through a dirty
    extraction (a bled reference column, PDF line numbers, a hyphen split across a line) is
    reported as UNRESOLVED rather than as a fabrication. That module exists because
    contiguous-substring tests can mistake extraction damage for an absent quotation.

INPUT CONTRACT

Full texts are supplied as a directory of ALREADY-CONVERTED text — this detector never
fetches. `/fulltext-retrieval` produces exactly that (`fetch_oa.py` writes
`<doi-with-unsafe-chars-underscored>.pdf`, `pdf_to_md.py` converts it alongside), and a
detector that reached the network would be neither deterministic nor CI-runnable.

A citation is resolved to a file by, in order: an explicit `--refmap`; the DOI from the
`--bib` entry for `[@key]`; the DOI parsed out of a numbered reference list in the
manuscript for `[N]`; or a file whose stem is the citation key itself. Citations that
resolve to nothing are counted, never guessed at, and never produce a finding.

Usage:
    check_claim_fidelity.py --manuscript m.md --fulltext-dir fulltext/ [--bib refs.bib]
    check_claim_fidelity.py --manuscript m.md --fulltext-dir ft/ --out qc/claim_fidelity.json --strict

Exit 0 when no major verdict fires (prompts alone do not fail). With --strict, exit 1 if any
major fires. Stdlib only; .docx read via python-docx when available.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))
from _quote_match import match_quality, normalize, tokens  # noqa: E402  (vendored, same-dir)
from _claim_evidence import build_evidence, render_table, sha256  # noqa: E402

DETECTOR = "check_claim_fidelity"

# A source shorter than this is an extraction stub (an abstract, a landing page, a paywall
# interstitial). Absence proves nothing against it, so it produces no findings at all.
MIN_SOURCE_TOKENS = 400
# A quoted span shorter than this is not a quotation, it is a scare-quoted term.
MIN_QUOTE_TOKENS = 6
# How far after a closing quote a citation may sit and still be its attribution.
CITE_WINDOW_CHARS = 120
# Attributed span: tokens between the attribution verb and the citation that are examined.
MAX_SPAN_TOKENS = 14
# A stem must be this long before a prefix match is meaningful ("dat" matches too much).
MIN_STEM = 4
# How far from an occurrence of the noun the source may state the cardinal.
CARDINAL_WINDOW = 15

# Verbs that assign a claim to a cited work. The citation in the same sentence is the real
# filter; this list only has to be broad enough to find the point where attribution starts.
ATTRIB_VERB = re.compile(
    r"\b("
    r"propos\w*|offer\w*|introduc\w*|argu\w*|report\w*|show\w*|demonstrat\w*|describ\w*|"
    r"suggest\w*|conclud\w*|observ\w*|advocat\w*|recommend\w*|claim\w*|assert\w*|"
    r"present\w*|identif\w*|characteri[sz]\w*|defin\w*|coin\w*|"
    r"found|finds|note[ds]?|term(?:s|ed)|call(?:s|ed)|frame[sd]?"
    r")\b",
    re.IGNORECASE,
)

CITE_KEY = re.compile(r"\[([^\]]*@[^\]]+)\]")
CITE_NUM = re.compile(r"\[(\d{1,3}(?:\s*[-–,;]\s*\d{1,3})*)\]")
DOI_RE = re.compile(r"\b10\.\d{4,9}/[^\s\"<>,;\]]+", re.IGNORECASE)
QUOTE_RE = re.compile(r"[\"“]([^\"“”]{25,700})[\"”]")

NUM_WORDS = {
    "one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6,
    "seven": 7, "eight": 8, "nine": 9, "ten": 10, "eleven": 11, "twelve": 12,
}
CARDINAL_CLAIM = re.compile(
    r"\b(one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|\d{1,3})\s+"
    r"((?:[a-z]+[- ]){0,2}[a-z]{3,}s)\b",
    re.IGNORECASE,
)

# Function words plus the vocabulary every paper shares. A span made only of these carries no
# checkable content, so it is skipped rather than flagged.
STOPWORDS = frozenset("""
a an the this that these those there here it its it's they them their theirs we our us you your
and or but nor for so yet as if then than because while whereas although though however
of in on at by to from with within without into onto over under between among across during
is are was were be been being being has have had having do does did done
can could may might must shall should will would
not no none nor only just also both each either neither all any some many much more most few less least
such same other others another new recent previous prior earlier later further additional
which who whom whose what when where why how
study studies paper papers work works article articles author authors research researchers
result results finding findings data dataset datasets analysis analyses approach approaches
method methods methodology model models framework frameworks evidence report reports
review reviews literature field fields context setting settings case cases example examples
use used using uses need needs based given note noted show shown seen make made take taken
group groups patient patients subject subjects participant participants sample samples
value values level levels rate rates number numbers effect effects
""".split())


# --------------------------------------------------------------------------- reading input

def read_text(path: Path) -> str:
    if path.suffix.lower() == ".docx":
        try:
            import docx  # type: ignore
        except ImportError:
            print(f"ERROR: reading {path.name} needs python-docx (pip install python-docx)",
                  file=sys.stderr)
            raise SystemExit(2)
        d = docx.Document(str(path))
        parts = [p.text for p in d.paragraphs]
        for t in d.tables:
            for row in t.rows:
                parts += [c.text for c in row.cells]
        return "\n".join(parts)
    return path.read_text(encoding="utf-8", errors="replace")


def strip_noise(md: str) -> str:
    """Remove regions whose text is not manuscript prose making claims.

    Fenced code, reviewer/quoted blockquote lines and table rows are dropped; the reference
    list is truncated away because a numbered bibliography would feed probe 3 a cardinal per
    line ("12 Smith J, 4 Brown K") and probe 2 a citation per line.
    """
    md = re.sub(r"```.*?```", " ", md, flags=re.S)
    md = re.sub(r"^\s*>.*$", " ", md, flags=re.M)
    md = re.sub(r"^\s*\|.*$", " ", md, flags=re.M)
    cut = re.search(r"^#{1,3}\s*\**\s*(References|REFERENCES|Bibliography|Works Cited)\b",
                    md, flags=re.M)
    return md[:cut.start()] if cut else md


def reference_section(md: str) -> str:
    m = re.search(r"^#{1,3}\s*\**\s*(References|REFERENCES|Bibliography|Works Cited)\b",
                  md, flags=re.M)
    return md[m.start():] if m else ""


# ------------------------------------------------------------------- citation -> source file

def parse_bib(path: Path) -> dict[str, str]:
    """citation key -> DOI, for the entries that carry one."""
    out: dict[str, str] = {}
    text = path.read_text(encoding="utf-8", errors="replace")
    for chunk in re.split(r"\n(?=@)", text):
        key_m = re.match(r"@\w+\s*\{\s*([^,\s]+)\s*,", chunk)
        if not key_m:
            continue
        doi_m = re.search(r"\bdoi\s*=\s*[{\"]\s*(?:https?://doi\.org/)?([^}\"\s]+)",
                          chunk, re.IGNORECASE)
        if doi_m:
            out[key_m.group(1)] = doi_m.group(1).rstrip(".,;")
    return out


REF_ENTRY_START = re.compile(r"^\s*\[?(\d{1,3})[.)\]]\s+\S")


def parse_numbered_refs(ref_text: str) -> dict[str, str]:
    """reference number -> DOI, read off a numbered reference list in the manuscript.

    Entries are accumulated as BLOCKS rather than lines: a rendered reference wraps, and the DOI
    is usually on the continuation line, so a line-scoped read finds the number and the DOI in
    different rows and resolves nothing.
    """
    out: dict[str, str] = {}
    num: str | None = None
    buf: list[str] = []

    def flush() -> None:
        if num is None:
            return
        doi = DOI_RE.search(" ".join(buf))
        if doi:
            out[num] = doi.group(0).rstrip(".,;")

    for line in ref_text.splitlines():
        m = REF_ENTRY_START.match(line)
        if m:
            flush()
            num, buf = m.group(1), [line]
        elif num is not None:
            if not line.strip():
                flush()
                num, buf = None, []
            else:
                buf.append(line)
    flush()
    return out


def safe_doi_name(doi: str) -> str:
    """Filesystem-safe stem for a DOI — identical to /fulltext-retrieval fetch_oa.py, so the
    files that skill writes are found without any hand-built map."""
    return re.sub(r"[^\w\-.]", "_", doi)


def index_fulltext(d: Path) -> dict[str, Path]:
    """lowercased stem -> file, over the text formats pdf_to_md.py and friends emit."""
    idx: dict[str, Path] = {}
    for p in sorted(d.rglob("*")):
        if p.is_file() and p.suffix.lower() in (".md", ".txt", ".markdown", ".text"):
            idx.setdefault(p.stem.lower(), p)
    return idx


def expand_numeric(raw: str) -> list[str]:
    """"4-11, 15" -> ['4'..'11', '15']. Ranges must expand or a cited number inside one reads
    as uncited — the same trap that produced false gaps in the citation-order work."""
    out: list[str] = []
    for part in re.split(r"\s*[,;]\s*", raw):
        rng = re.match(r"^(\d{1,3})\s*[-–]\s*(\d{1,3})$", part.strip())
        if rng:
            lo, hi = int(rng.group(1)), int(rng.group(2))
            if lo <= hi and hi - lo <= 60:
                out += [str(n) for n in range(lo, hi + 1)]
                continue
        p = part.strip()
        if p.isdigit():
            out.append(p)
    return out


class Resolver:
    """Turns the citation tokens in a sentence into readable source texts."""

    def __init__(self, fulltext: dict[str, Path], key_doi: dict[str, str],
                 num_doi: dict[str, str], refmap: dict[str, str]):
        self.fulltext = fulltext
        self.key_doi = key_doi
        self.num_doi = num_doi
        self.refmap = refmap
        self.unresolved: set[str] = set()
        self._cache: dict[str, tuple[Path, str] | None] = {}
        self._source_hashes: dict[Path, str | None] = {}

    def _file_for(self, token: str) -> Path | None:
        mapped = self.refmap.get(token)
        candidates: list[str] = []
        if mapped:
            candidates += [safe_doi_name(mapped), mapped]
        doi = self.key_doi.get(token) or self.num_doi.get(token)
        if doi:
            candidates.append(safe_doi_name(doi))
        candidates.append(token)
        for c in candidates:
            hit = self.fulltext.get(c.lower())
            if hit:
                return hit
        return None

    def resolve(self, token: str) -> tuple[Path, str] | None:
        if token in self._cache:
            return self._cache[token]
        path = self._file_for(token)
        before = sha256(path)
        got = (path, read_text(path)) if path else None
        if path:
            if before != sha256(path):
                raise ValueError("source changed while reading")
            self._source_hashes[path] = before
        if got is None:
            self.unresolved.add(token)
        self._cache[token] = got
        return got

    @staticmethod
    def citations_in(text: str) -> list[str]:
        out: list[str] = []
        for m in CITE_KEY.finditer(text):
            out += re.findall(r"@([A-Za-z0-9_][A-Za-z0-9_:.#$%&+?<>~/\-]*)", m.group(1))
        for m in CITE_NUM.finditer(text):
            out += expand_numeric(m.group(1))
        return [t for t in out if t]


# ------------------------------------------------------------------------- text utilities

def stem(word: str) -> str:
    """Crude suffix strip — enough to let 'strategies' find 'strategy' by prefix."""
    w = word.lower()
    for suf, repl in (("ies", "i"), ("ing", ""), ("ies", "y"), ("es", ""), ("ed", ""), ("s", "")):
        if w.endswith(suf) and len(w) - len(suf) >= MIN_STEM:
            return w[: len(w) - len(suf)] + repl
    return w


class Source:
    """A resolved full text, prepared once for repeated membership questions."""

    def __init__(self, path: Path, raw: str):
        self.path = path
        self.raw = raw
        self.tokens = tokens(raw)
        self.token_set = frozenset(self.tokens)
        self.prefixes = frozenset(t[:MIN_STEM] for t in self.tokens if len(t) >= MIN_STEM)
        self.usable = len(self.tokens) >= MIN_SOURCE_TOKENS

    def has_term(self, word: str) -> bool:
        """Morphology-tolerant presence: exact token, or any source token sharing this word's
        stem prefix and extending it plausibly."""
        w = word.lower()
        if w in self.token_set:
            return True
        s = stem(w)
        if len(s) < MIN_STEM:
            return False
        if s in self.token_set:
            return True
        if s[:MIN_STEM] not in self.prefixes:
            return False
        return any(t.startswith(s) and len(t) - len(s) <= 3 for t in self.token_set)

    def positions(self, word: str) -> list[int]:
        s = stem(word.lower())
        if len(s) < MIN_STEM:
            return []
        return [i for i, t in enumerate(self.tokens) if t.startswith(s) and len(t) - len(s) <= 3]


ABBREV = re.compile(r"\b(?:et al|e\.g|i\.e|cf|vs|Fig|Figs|Tab|No|Dr|Prof|approx)\.$",
                    re.IGNORECASE)


def sentences(text: str) -> list[str]:
    out: list[str] = []
    buf = ""
    for piece in re.split(r"(?<=[.!?])\s+", text):
        buf = f"{buf} {piece}".strip() if buf else piece
        if ABBREV.search(buf.strip()):
            continue
        out.append(buf)
        buf = ""
    if buf:
        out.append(buf)
    return [s.strip() for s in out if s.strip()]


def content_words(span: str) -> list[str]:
    ws = re.findall(r"[A-Za-z][A-Za-z'-]{2,}", span)
    return [w for w in ws if w.lower() not in STOPWORDS and len(w) >= MIN_STEM]


# ------------------------------------------------------------------------------ the probes

def probe_quotes(md: str, res: Resolver, findings: list[dict]) -> tuple[int, list[tuple[int, int]]]:
    """Quoted text attributed to a source must be in that source. Returns (checked, spans)."""
    checked = 0
    spans: list[tuple[int, int]] = []
    for m in QUOTE_RE.finditer(md):
        quote = m.group(1).strip()
        if len(tokens(quote)) < MIN_QUOTE_TOKENS:
            continue
        tail = md[m.end(): m.end() + CITE_WINDOW_CHARS]
        cites = Resolver.citations_in(tail)
        if not cites:
            continue
        spans.append((m.start(), m.end()))
        for token in cites[:3]:
            got = res.resolve(token)
            if not got:
                continue
            src = Source(*got)
            checked += 1
            grade = match_quality(quote, src.raw)
            if grade["grade"] in ("EXACT", "INTERLEAVED"):
                continue
            absent = grade["grade"] == "ABSENT" and src.usable
            findings.append({
                "verdict": "CITED_QUOTE_ABSENT" if absent else "CITED_QUOTE_UNRESOLVED",
                "severity": "major" if absent else "minor",
                "citation": token,
                "source": src.path.name,
                "quote": quote[:200],
                "match_grade": grade["grade"],
                "coverage": grade["coverage"],
                "message": (
                    f"Quoted text attributed to [{token}] is not in {src.path.name} in any "
                    f"reading order (coverage {grade['coverage']}). Read the source before "
                    f"keeping the quotation marks."
                ) if absent else (
                    f"Quoted text attributed to [{token}] matched {src.path.name} only as "
                    f"{grade['grade']} (coverage {grade['coverage']})"
                    + ("" if src.usable else "; the extracted source is too short to judge")
                    + " — verify by eye rather than treating this as a defect."
                ),
            })
            break
    return checked, spans


def probe_attribution(md: str, res: Resolver, quote_spans: list[tuple[int, int]],
                      findings: list[dict]) -> int:
    """A claim assigned to a source whose every content word is absent from that source."""
    checked = 0
    offset = 0
    for sent in sentences(md):
        idx = md.find(sent, offset)
        if idx >= 0:
            offset = idx + len(sent)
            # A sentence that carries a quotation is probe 1's — attributing the whole quoted
            # span to probe 2 as well would report one claim twice, under two verdicts.
            if any(idx < b and a < idx + len(sent) for a, b in quote_spans):
                continue
        cite_m = CITE_KEY.search(sent) or CITE_NUM.search(sent)
        if not cite_m:
            continue
        verbs = [v for v in ATTRIB_VERB.finditer(sent) if v.end() <= cite_m.start()]
        if not verbs:
            continue
        span = sent[verbs[-1].end(): cite_m.start()]
        words = content_words(span)[:MAX_SPAN_TOKENS]
        if not words:
            continue
        for token in Resolver.citations_in(cite_m.group(0))[:2]:
            got = res.resolve(token)
            if not got:
                continue
            src = Source(*got)
            if not src.usable:
                continue
            checked += 1
            if any(src.has_term(w) for w in words):
                continue
            findings.append({
                "verdict": "ATTRIBUTION_UNSUPPORTED",
                "severity": "minor",
                "citation": token,
                "source": src.path.name,
                "attributed_verb": verbs[-1].group(0),
                "attributed_terms": words,
                "sentence": sent.strip()[:300],
                "message": (
                    f"Nothing attributed to [{token}] here — {', '.join(words)} — appears "
                    f"anywhere in {src.path.name}, in any form. Paraphrase normally keeps at "
                    f"least one of the source's own terms; none survived. Read the source and "
                    f"confirm this is the claim it makes."
                ),
            })
            break
    return checked


def probe_cardinal(md: str, res: Resolver, findings: list[dict]) -> int:
    """"reports three strategies [12]" where the source names that noun but never that count."""
    checked = 0
    for sent in sentences(md):
        cite_m = CITE_KEY.search(sent) or CITE_NUM.search(sent)
        if not cite_m:
            continue
        if not any(v.end() <= cite_m.start() for v in ATTRIB_VERB.finditer(sent)):
            continue
        claims = [c for c in CARDINAL_CLAIM.finditer(sent[: cite_m.start()])
                  if c.group(2).split()[-1].lower() not in STOPWORDS]
        if not claims:
            continue
        for token in Resolver.citations_in(cite_m.group(0))[:2]:
            got = res.resolve(token)
            if not got:
                continue
            src = Source(*got)
            if not src.usable:
                continue
            for c in claims:
                raw_num, noun_phrase = c.group(1).lower(), c.group(2)
                noun = noun_phrase.split()[-1].split("-")[-1]
                value = NUM_WORDS.get(raw_num, None)
                if value is None:
                    if not raw_num.isdigit():
                        continue
                    value = int(raw_num)
                forms = {str(value)} | {w for w, v in NUM_WORDS.items() if v == value}
                spots = src.positions(noun)
                if not spots:
                    continue  # concept not located: probe 2's territory, not this one
                checked += 1
                near = any(
                    src.tokens[j] in forms
                    for i in spots
                    for j in range(max(0, i - CARDINAL_WINDOW),
                                   min(len(src.tokens), i + CARDINAL_WINDOW + 1))
                )
                if near:
                    continue
                findings.append({
                    "verdict": "ORDINAL_CLAIM_UNSUPPORTED",
                    "severity": "minor",
                    "citation": token,
                    "source": src.path.name,
                    "claimed": f"{raw_num} {noun_phrase}",
                    "sentence": sent.strip()[:300],
                    "message": (
                        f"This attributes \"{raw_num} {noun_phrase}\" to [{token}], but "
                        f"{src.path.name} discusses \"{noun}\" without ever stating that count "
                        f"nearby. Count them in the source."
                    ),
                })
            break
    return checked


# ------------------------------------------------------------------------------------ main

def build_report(manuscript: Path, fulltext_dir: Path, bib: Path | None,
                 refmap: dict[str, str], *, retrieval_report: Path | None = None,
                 pdf_dir: Path | None = None, reference_audit: Path | None = None,
                 reviewed_report: Path | None = None) -> dict:
    def optional_json(path):
        if path is None:
            return None
        value = json.loads(path.read_text(encoding="utf-8"))
        if not isinstance(value, dict):
            raise ValueError(f"expected a JSON object: {path.name}")
        return value

    inputs = [p for p in (manuscript, bib, retrieval_report, reference_audit, reviewed_report) if p]
    initial_hashes = {p: sha256(p) for p in inputs}
    # Read review input before writing --out, which may intentionally be the same report.
    retrieval = optional_json(retrieval_report)
    audit = optional_json(reference_audit)
    reviewed = optional_json(reviewed_report)
    raw = read_text(manuscript)
    body = strip_noise(raw)
    res = Resolver(
        fulltext=index_fulltext(fulltext_dir),
        key_doi=parse_bib(bib) if bib else {},
        num_doi=parse_numbered_refs(reference_section(raw)),
        refmap=refmap,
    )

    findings: list[dict] = []
    n_quote, spans = probe_quotes(body, res, findings)
    n_attr = probe_attribution(body, res, spans, findings)
    n_card = probe_cardinal(body, res, findings)

    resolved = {t: v for t, v in res._cache.items() if v}
    short = sorted({v[0].name for t, v in resolved.items()
                    if len(tokens(v[1])) < MIN_SOURCE_TOKENS})
    report = {
        "schema_version": 2,
        "detector": DETECTOR,
        "manuscript": str(manuscript),
        "fulltext_dir": str(fulltext_dir),
        "sources_resolved": len({v[0] for v in resolved.values()}),
        "sources_unresolved": sorted(res.unresolved),
        "sources_too_short": short,
        "claims_checked": {"quotes": n_quote, "attributions": n_attr, "cardinals": n_card},
        "findings": findings,
    }
    report.update(build_evidence(
        raw, manuscript, fulltext_dir, res, ABBREV,
        binding={"manuscript_sha256": initial_hashes[manuscript], "bib_sha256": initial_hashes.get(bib),
                 "retrieval_report_sha256": initial_hashes.get(retrieval_report),
                 "reference_audit_sha256": initial_hashes.get(reference_audit),
                 "refmap": refmap},
        retrieval=retrieval, pdf_dir=pdf_dir or (retrieval_report.parent if retrieval_report else None),
        reference_audit=audit, reviewed_report=reviewed,
    ))
    for row in report["evidence_rows"]:
        sentence = " ".join(row["manuscript"]["text"].split())
        row["automatic_finding_indices"] = [
            i for i, finding in enumerate(findings)
            if finding["citation"] == row["citation"] and (
                (finding.get("sentence") and " ".join(finding["sentence"].split()) == sentence[:300])
                or (finding.get("quote") and " ".join(finding["quote"].split()) in sentence))]
    if any(sha256(p) != digest for p, digest in initial_hashes.items()):
        raise ValueError("input changed while building report")
    if any(sha256(p) != digest for p, digest in res._source_hashes.items()):
        raise ValueError("source changed while building report")
    return report


def main() -> int:
    ap = argparse.ArgumentParser(description="Verify what a manuscript says a cited work says.")
    ap.add_argument("--manuscript", required=True, type=Path)
    ap.add_argument("--fulltext-dir", required=True, type=Path,
                    help="directory of ALREADY-CONVERTED source texts (see /fulltext-retrieval)")
    ap.add_argument("--bib", type=Path, help="refs.bib, for [@key] -> DOI resolution")
    ap.add_argument("--refmap", type=Path,
                    help='JSON {"41": "10.1000/xyz"} for citations neither bib nor list resolves')
    ap.add_argument("--out", type=Path, help="write the JSON report here")
    ap.add_argument("--retrieval-report", type=Path,
                    help="existing fetch_oa report; advisory identity and PDF hashes, not claim support")
    ap.add_argument("--pdf-dir", type=Path,
                    help="PDF directory, when different from the retrieval report's directory")
    ap.add_argument("--reference-audit", type=Path, help="existing reference_audit.json (read-only)")
    ap.add_argument("--reviewed-report", type=Path,
                    help="prior claim_fidelity.json containing assessor-entered evidence_rows")
    ap.add_argument("--evidence-table", type=Path, help="optional Markdown view derived from this report")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any major verdict fires")
    args = ap.parse_args()

    if args.evidence_table and not args.out:
        print("ERROR: --evidence-table requires --out to preserve the source JSON", file=sys.stderr)
        return 2

    if not args.manuscript.is_file():
        print(f"ERROR: manuscript not found: {args.manuscript}", file=sys.stderr)
        return 2
    if not args.fulltext_dir.is_dir():
        print(f"ERROR: --fulltext-dir not a directory: {args.fulltext_dir}", file=sys.stderr)
        return 2
    protected = [args.manuscript, args.bib, args.refmap, args.retrieval_report, args.reference_audit]
    protected += list(index_fulltext(args.fulltext_dir).values())
    protected = {p.resolve() for p in protected if p}
    outputs = [p.resolve() for p in (args.out, args.evidence_table) if p]
    if any(p in protected for p in outputs) or len(outputs) != len(set(outputs)):
        print("ERROR: output paths must be distinct and must not overwrite source inputs", file=sys.stderr)
        return 2
    if args.reviewed_report and args.evidence_table and args.reviewed_report.resolve() == args.evidence_table.resolve():
        print("ERROR: the table must not overwrite the reviewed JSON", file=sys.stderr)
        return 2
    try:
        refmap: dict[str, str] = {}
        if args.refmap:
            refmap = {str(k): str(v)
                      for k, v in json.loads(args.refmap.read_text(encoding="utf-8")).items()}
        report = build_report(args.manuscript, args.fulltext_dir, args.bib, refmap,
                              retrieval_report=args.retrieval_report, pdf_dir=args.pdf_dir,
                              reference_audit=args.reference_audit, reviewed_report=args.reviewed_report)
        selected_pdf_dir = args.pdf_dir or (args.retrieval_report.parent if args.retrieval_report else None)
        if selected_pdf_dir:
            protected.update((selected_pdf_dir / row["source"]["pdf"]["file"]).resolve()
                             for row in report["evidence_rows"] if row["source"]["pdf"]["file"])
        if any(p in protected for p in outputs):
            raise ValueError("output would overwrite a source PDF")
        table_text = render_table(report) if args.evidence_table else None
    except (OSError, ValueError, TypeError, KeyError, AttributeError) as exc:
        print(f"ERROR: cannot build claim evidence report: {exc}", file=sys.stderr)
        return 2

    if args.out:
        args.out.parent.mkdir(parents=True, exist_ok=True)
        args.out.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
    if args.evidence_table:
        args.evidence_table.parent.mkdir(parents=True, exist_ok=True)
        args.evidence_table.write_text(table_text, encoding="utf-8")

    c = report["claims_checked"]
    print(f"{DETECTOR}: {report['sources_resolved']} source(s) resolved; "
          f"checked {c['quotes']} quote(s), {c['attributions']} attribution(s), "
          f"{c['cardinals']} cardinal claim(s)")
    if report["sources_unresolved"]:
        print(f"  not resolved to a full text (not checked): "
              f"{', '.join(report['sources_unresolved'][:12])}")
    if report["sources_too_short"]:
        print(f"  extracted text too short to judge absence: "
              f"{', '.join(report['sources_too_short'][:8])}")

    majors = [f for f in report["findings"] if f["severity"] == "major"]
    e = report["evidence_counts"]
    print(f"  evidence table: {e['sentence_citation_pairs']} sentence/citation pair(s); "
          f"recorded review outcomes: {json.dumps(e['verdicts'], sort_keys=True)}")
    if not report["findings"]:
        print("No automated findings. Claim support has not been established by this check.")
        return 0
    for f in report["findings"]:
        print(f"  [{f['severity']}] {f['verdict']}: {f['message']}")
    return 1 if (args.strict and majors) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# verify-refs CLI wrapper — thin shell in front of verify_refs.py for use by
# the pre-submission hook and ad-hoc manual runs.
#
# Usage:   verify_cli.sh <manuscript.docx|.md|.bib|.txt|.tsv> [--offline]
# Exit:    0 = submission-safe (no FABRICATED/MISMATCH)
#          1 = FABRICATED or MISMATCH found -> hook should block
#          2 = input missing / usage error
#          3 = no references detected (unusual; treated as non-blocking)
#
# Project root resolution: the first ancestor directory containing a
# `submission/` sibling, else the manuscript's parent directory. Output
# artifacts are written under `<project_root>/references/` and
# `<project_root>/qc/` per the skill's output contract.

set -euo pipefail

if [ $# -lt 1 ]; then
  echo "Usage: verify_cli.sh <manuscript> [--offline]" >&2
  exit 2
fi

MANUSCRIPT="$1"; shift || true
EXTRA_ARGS=()
if [ $# -gt 0 ]; then
  EXTRA_ARGS=("$@")
fi

if [ ! -f "$MANUSCRIPT" ]; then
  echo "Not found: $MANUSCRIPT" >&2
  exit 2
fi

ABS="$(cd "$(dirname "$MANUSCRIPT")" && pwd)/$(basename "$MANUSCRIPT")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Resolve project root: nearest ancestor whose path contains /submission/,
# otherwise the manuscript's parent directory.
PROJECT_ROOT=""
dir="$(dirname "$ABS")"
while [ "$dir" != "/" ] && [ -n "$dir" ]; do
  case "$dir" in
    */submission|*/submission/*)
      PROJECT_ROOT="${dir%/submission*}/submission/$(basename "$(dirname "$dir")")"
      [ -d "$PROJECT_ROOT" ] || PROJECT_ROOT="$dir"
      break
      ;;
  esac
  dir="$(dirname "$dir")"
done
if [ -z "$PROJECT_ROOT" ]; then
  PROJECT_ROOT="$(dirname "$ABS")"
fi

set +e
python3 "$SCRIPT_DIR/verify_refs.py" "$ABS" --project-root "$PROJECT_ROOT" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
EXIT_CODE=$?
set -e

AUDIT="$PROJECT_ROOT/qc/reference_audit.json"
echo "[verify-refs] exit=$EXIT_CODE; audit=$AUDIT" >&2
exit "$EXIT_CODE"

#!/usr/bin/env python3
"""Reference verification helper for medsci-skills.

The script is deliberately stdlib-only. It extracts reference-like entries from
Markdown, DOCX, BibTeX, plain text, or TSV, verifies DOI/PMID when possible, and
writes a single audit artifact: qc/reference_audit.json. Per v1.1.1 artifact
contract, this skill is sole writer of that file and MUST NOT touch references/.
"""

from __future__ import annotations

import argparse
import csv
import html
import json
import os
import re
import sys
import time
import urllib.parse
import urllib.request
import zipfile
from dataclasses import dataclass, asdict, field
from pathlib import Path
from xml.etree import ElementTree as ET


DOI_RE = re.compile(r"\b10\.\d{4,9}/[-._;()/:A-Z0-9]+\b", re.I)
PMID_RE = re.compile(r"\bPMID\s*:?\s*(\d{5,9})\b", re.I)
YEAR_RE = re.compile(r"\b(19|20)\d{2}\b")


@dataclass
class RefRecord:
    ref_id: str
    raw: str
    title_guess: str = ""
    doi: str = ""
    pmid: str = ""
    year_guess: str = ""
    first_author_guess: str = ""  # back-compat (= cited_authors[0] when available)
    # v1.3.0: full author cross-check (AI-assisted-drafting hallucination motivation)
    cited_authors: list = field(default_factory=list)        # family names parsed from bib/tsv/text
    actual_authors: list = field(default_factory=list)        # family names from authoritative source
    cited_author_count: int = 0
    actual_author_count: int = 0
    # v1.3.0: intentional truncate marker. Set via BibTeX field `_audit_truncated = N`
    # (any non-empty value); when present, count mismatch is downgraded to a note
    # and does not trigger MISMATCH status. Use when CSL renders first-1 or first-5
    # + et al. and the trailing authors are deliberately omitted from the bib.
    #
    # It is ALSO set by the standard BibTeX `and others` sentinel. `_audit_truncated`
    # is a field this toolkit invented; `and others` is the one every reference
    # manager already writes and BibTeX/CSL already render as "et al.". Treating it
    # as a count mismatch fired MISMATCH — the render-aborting verdict — on any
    # consortium or multisociety-guideline reference, which is the class clinical
    # manuscripts cite most.
    audit_truncated: bool = False
    # Collective / corporate author (EASL, KDIGO, AHA/ACC, WHO, a named working
    # group / consortium). BibTeX convention double-braces these
    # (`author = {{KDIGO Working Group}}`) and PubMed returns them as
    # <CollectiveName>; the personal-name family cross-check does not apply and
    # must not fire MISMATCH (which would abort render on every guideline-citing
    # cohort manuscript).
    corporate_author: bool = False
    status: str = "UNVERIFIED"
    evidence: str = ""
    note: str = ""


def normalize_space(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()


# Organization / collective-author signal in an author field. Used (a) when a
# brace survives in the parsed BibTeX author field (double-brace convention) and
# (b) as a keyword fallback for single-braced or plain-text collective names.
_ORG_AUTHOR_RE = re.compile(
    r"\b(?:Group|Committee|Society|Association|Collaborat\w+|Consortium|Network|"
    r"Panel|Initiative|Organization|Organisation|Investigators|Trialists|Task\s+Force|"
    r"Working\s+Group|Study\s+Group|Foundation|Institute|Council|Federation|College|"
    r"WHO|EASL|EASD|EASO|KDIGO|AHA|ACC|ESC|NICE|AASLD|KASL)\b", re.IGNORECASE)


def is_corporate_author_field(author_field: str) -> bool:
    """A collective/corporate author (a guideline body, working group, consortium)
    rather than a list of people.

    Braces alone do NOT make an author corporate. Better BibTeX brace-protects a
    hyphenated or particle SURNAME so BibTeX will not re-case or re-split it —
    `author = {{Eckel-Passow}, Jeanette E. and {von Deimling}, Andreas}` — and the
    old brace-only rule read those as organizations and *skipped* the author
    cross-check entirely. That is a false PASS, not a false alarm: the check this
    tool exists to perform was silently not performed, and nothing said so.

    A field is corporate when it carries an organizational keyword, or when it is a
    single braced blob with no personal-name structure at all (no comma separating a
    surname from a given name, no `and`-joined list).
    """
    if not author_field:
        return False
    if _ORG_AUTHOR_RE.search(author_field):
        return True

    braced = "{" in author_field or "}" in author_field
    if not braced:
        return False

    # Personal-name structure survives brace-stripping -> these are people.
    bare = author_field.replace("{", "").replace("}", "")
    if "," in bare or re.search(r"\band\b", bare):
        return False

    # A lone braced blob with no keyword and no name structure (e.g. `{{ADNI}}`):
    # still treat as collective — skipping is the conservative outcome there.
    return True


def clean_doi(doi: str) -> str:
    return doi.rstrip(".,;)].").lower()


def normalize_doi_for_dup(doi: str) -> str:
    """Strict DOI normalization for duplicate detection.

    Beyond clean_doi(): strips common URL prefixes and trailing slashes so that
    `https://doi.org/10.1234/abc/` and `10.1234/abc` collapse to the same key.
    """
    if not doi:
        return ""
    s = doi.strip().lower()
    for prefix in ("https://doi.org/", "http://doi.org/",
                   "https://dx.doi.org/", "http://dx.doi.org/", "doi:"):
        if s.startswith(prefix):
            s = s[len(prefix):]
            break
    s = s.strip().rstrip("/")
    return clean_doi(s)


def read_docx(path: Path) -> str:
    with zipfile.ZipFile(path) as zf:
        xml = zf.read("word/document.xml")
    root = ET.fromstring(xml)
    ns = {"w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main"}
    paragraphs = []
    for p in root.findall(".//w:p", ns):
        parts = [t.text or "" for t in p.findall(".//w:t", ns)]
        if parts:
            paragraphs.append("".join(parts))
    return "\n".join(paragraphs)


def read_input(path: Path) -> str:
    if path.suffix.lower() == ".docx":
        return read_docx(path)
    return path.read_text(encoding="utf-8", errors="replace")


def brace_field(entry: str, name: str) -> str:
    """Read a brace-delimited BibTeX field by counting braces, not by regex.

    A non-greedy `{(.+?)}` stops at the first `}`, which truncates any brace-protected inner group
    (`title = {A multisociety {Delphi} consensus}`). Anchoring the match on a trailing comma avoids
    that but silently returns nothing when the field is the entry's LAST one, where BibTeX makes the
    comma optional. Counting depth is correct in both cases. Returns "" when the field is absent or
    is not brace-delimited.
    """
    m = re.search(rf"{name}\s*=\s*\{{", entry, re.I)
    if not m:
        return ""
    start = m.end()
    depth, j = 1, start
    while j < len(entry) and depth > 0:
        if entry[j] == "{":
            depth += 1
        elif entry[j] == "}":
            depth -= 1
        j += 1
    return entry[start : j - 1]


def parse_bib(text: str) -> list[RefRecord]:
    records: list[RefRecord] = []
    entries = re.split(r"\n(?=@\w+\{)", "\n" + text)
    for entry in entries:
        entry = entry.strip()
        if not entry.startswith("@"):
            continue
        key_match = re.match(r"@\w+\{([^,]+),", entry)
        title_match = re.search(r"title\s*=\s*[\{\"](.+?)[\}\"]\s*,", entry, re.I | re.S)
        # The trailing comma is OPTIONAL: BibTeX does not require one after an entry's LAST field,
        # and `doi` is very often that last field. Requiring it left `record.doi` empty for those
        # entries, which skips the CrossRef check outright and drops the record onto the soft-flagged
        # OpenAlex title path where the author cross-check is disabled — so the anti-fabrication
        # check this skill exists for was silently off for exactly those references. Unlike `title`
        # on the line above, a DOI never carries brace-protected inner groups (`{Delphi}`), so
        # stopping at the first `}` is correct here and would truncate there.
        doi_match = re.search(r"doi\s*=\s*[\{\"](.+?)[\}\"]\s*,?", entry, re.I | re.S)
        pmid_match = re.search(r"pmid\s*=\s*[\{\"]?(\d{5,9})", entry, re.I)
        year_match = re.search(r"year\s*=\s*[\{\"]?((?:19|20)\d{2})", entry, re.I)
        raw = normalize_space(entry)
        author_field = brace_field(entry, "author")
        # `title` needs the same balanced-brace read, and for the same reason `doi` needed an
        # optional comma: the regex above requires a trailing comma, so a title that is the entry's
        # LAST field came back empty. It cannot simply take `,?` — the non-greedy match would then
        # stop at the inner `}` of a brace-protected group and return "A multisociety {Delphi".
        # Prefer the balanced read whenever the field is brace-delimited; keep the regex for the
        # quote-delimited form it already handled.
        title_braced = brace_field(entry, "title")
        if title_braced:
            title_match = None
            title_text = title_braced
        else:
            title_text = title_match.group(1) if title_match else ""
        cited = parse_bib_authors(author_field)
        # v1.3.0: intentional truncate marker (any non-empty `_audit_truncated`)
        trunc_match = re.search(r"_audit_truncated\s*=\s*[\{\"]?([^,}\"]+)", entry, re.I)
        records.append(
            RefRecord(
                ref_id=key_match.group(1) if key_match else f"ref_{len(records)+1}",
                raw=raw,
                title_guess=normalize_space(title_text),
                doi=clean_doi(doi_match.group(1)) if doi_match else "",
                pmid=pmid_match.group(1) if pmid_match else "",
                year_guess=year_match.group(1) if year_match else "",
                first_author_guess=cited[0] if cited else (parse_first_author(author_field) if author_field else ""),
                cited_authors=cited,
                cited_author_count=len(cited),
                audit_truncated=(
                    bool(trunc_match and trunc_match.group(1).strip().lower() not in ("", "false", "0", "no"))
                    or has_bibtex_et_al(author_field)
                ),
                corporate_author=is_corporate_author_field(author_field),
            )
        )
    return records


def parse_tsv(text: str) -> list[RefRecord]:
    rows = list(csv.DictReader(text.splitlines(), delimiter="\t"))
    records: list[RefRecord] = []
    for i, row in enumerate(rows, 1):
        joined = " ".join(str(v) for v in row.values() if v)
        doi = ""
        pmid = ""
        for key, value in row.items():
            lk = (key or "").lower()
            if lk == "doi" and value:
                doi = clean_doi(value)
            if lk == "pmid" and value:
                pmid = re.sub(r"\D", "", value)
        title = row.get("title") or row.get("Title") or ""
        author_field = row.get("author") or row.get("authors") or row.get("Author") or row.get("Authors") or ""
        records.append(
            RefRecord(
                ref_id=f"ref_{i}",
                raw=normalize_space(joined),
                title_guess=title,
                doi=doi,
                pmid=pmid,
                first_author_guess=parse_first_author(author_field) if author_field else "",
                corporate_author=is_corporate_author_field(author_field),
            )
        )
    return records


def reference_section(text: str) -> str:
    match = re.search(r"(?im)^\s*(references|bibliography|reference list)\s*$", text)
    if match:
        return text[match.end() :]
    return text


def parse_reference_lines(text: str) -> list[RefRecord]:
    section = reference_section(text)
    lines = [normalize_space(line) for line in section.splitlines()]
    candidates: list[str] = []
    current = ""
    for line in lines:
        if not line:
            continue
        starts_ref = bool(re.match(r"^(\[\d+\]|\d+[\.\)]|\-\s+)", line))
        if starts_ref and current:
            candidates.append(current)
            current = line
        else:
            current = f"{current} {line}".strip() if current else line
    if current:
        candidates.append(current)

    if len(candidates) < 2:
        candidates = [line for line in lines if DOI_RE.search(line) or PMID_RE.search(line) or len(line) > 60]

    records: list[RefRecord] = []
    for i, raw in enumerate(candidates, 1):
        raw = normalize_space(raw)
        doi_match = DOI_RE.search(raw)
        pmid_match = PMID_RE.search(raw)
        year_match = YEAR_RE.search(raw)
        records.append(
            RefRecord(
                ref_id=f"ref_{i}",
                raw=raw,
                title_guess=guess_title(raw),
                doi=clean_doi(doi_match.group(0)) if doi_match else "",
                pmid=pmid_match.group(1) if pmid_match else "",
                year_guess=year_match.group(0) if year_match else "",
                first_author_guess=parse_first_author(raw),
            )
        )
    return records


_NAME_PARTICLES = {"von", "van", "de", "del", "della", "dos", "da", "le", "la", "du", "den", "der", "ten"}


def has_bibtex_et_al(author_field: str) -> bool:
    """True when the author field ends in the BibTeX `and others` sentinel.

    `and others` is BibTeX's own way of saying "et al." — it is what Zotero,
    Mendeley, JabRef and a hand-written .bib all emit for a list the author chose
    not to type out, and what BibTeX and CSL both render as an et-al. It is a
    declaration that the list is deliberately short, so a cited-vs-source count
    difference is expected and must not be reported as a mismatch.

    Matched only in the trailing position, where BibTeX defines it. A real surname
    "Others" mid-list (or a first name "Others") is not this sentinel and is left
    to the ordinary family cross-check.
    """
    if not author_field:
        return False
    return bool(re.search(r"\s+and\s+others\s*$", re.sub(r"\s+", " ", author_field).strip(), re.I))


def parse_bib_authors(author_field: str) -> list:
    """Parse BibTeX author field into a list of family-name strings.

    Handles "Last, First and Last, First" and "First Last and First Last" forms.
    Strips simple LaTeX accents and braces.
    """
    if not author_field:
        return []
    raw = re.sub(r"\s+", " ", author_field).strip()
    parts = re.split(r"\s+and\s+", raw)
    families: list[str] = []
    for name in parts:
        n = name.strip()
        if not n:
            continue
        if "," in n:
            family = n.split(",", 1)[0].strip()
        else:
            toks = n.split()
            family = toks[-1] if toks else ""
        # Strip simple LaTeX accents: \~{n}, \"{o}, \`{a} → underlying char
        family = re.sub(r"\\[\"'`~^=.]?\{?([A-Za-zà-ÿ])\}?", r"\1", family)
        family = re.sub(r"[{}]", "", family).strip()
        if family and family != "others":
            families.append(family)
    return families


def parse_first_author(raw: str) -> str:
    """Extract first-author surname from a Vancouver/AMA/BibTeX-style citation.

    Conservative: returns "" when the format is ambiguous so author-mismatch
    checks degrade gracefully rather than firing false MISMATCH alerts.
    """
    text = re.sub(r"^\s*(\[\d+\]|\d+[\.\)])\s*", "", raw).strip()
    bib_m = re.search(r"author\s*=\s*[{\"]([^}\"]+)", text, re.I)
    if bib_m:
        text = bib_m.group(1)
    text = re.split(r"\s+and\s+", text, maxsplit=1)[0]
    parts = [p.strip() for p in text.split(",") if p.strip()]
    if not parts:
        return ""
    # "Lastname, Firstname H." style (BibTeX expanded)
    if len(parts) >= 2 and re.match(r"^[A-Z][a-zA-Z .\-']*$", parts[1]) and not re.search(r"\d", parts[1]):
        if re.match(r"^[A-Z][a-zA-Zà-ÿ'\- ]+$", parts[0]):
            return parts[0].strip()
    first = parts[0]
    # "Surname Initials" — strip trailing initials block (e.g., "DH", "J", "F.D.")
    m = re.match(
        r"^((?:(?:" + "|".join(_NAME_PARTICLES) + r")\s+)?[A-Zà-ÿ][\wà-ÿ'\-]*(?:\s+[A-Zà-ÿ][\wà-ÿ'\-]*)?)\s+(?:[A-Z]\.?\s*){1,4}$",
        first,
    )
    if m:
        return m.group(1).strip()
    tokens = first.split()
    if tokens and tokens[0].lower() in _NAME_PARTICLES and len(tokens) >= 2:
        return f"{tokens[0]} {tokens[1]}"
    return tokens[0] if tokens else ""


def _normalize_surname(name: str) -> str:
    """Strip diacritics + lowercase for surname comparison.

    Coverage (v1.3.0): Latin-with-accents (NFKD decomposes), Turkish
    (ş→s, ğ→g, ı→i), Polish/Czech (ł, đ — not NFKD-decomposable, handled below),
    German ß→ss, Nordic ø/æ/œ → o/ae/oe. Motivation: a Turkish surname
    `Çolakoğlu` vs PubMed `Colakoglu` false-positive MISMATCH.

    Unicode dashes are folded to ASCII `-` FIRST. The final filter keeps `[a-z\\s-]`
    and deletes everything else, so a publisher-supplied U+2010 in a hyphenated
    surname was *deleted* rather than matched: CrossRef `Foltyn‐Dumitru` normalized
    to `foltyndumitru` while the identical ASCII bib entry gave `foltyn-dumitru`,
    and the audit fired MISMATCH — its loudest verdict — on a clean reference.
    """
    import unicodedata
    n = unicodedata.normalize("NFKD", name)
    n = "".join(c for c in n if not unicodedata.combining(c))
    n = n.lower().strip()
    # Unicode dash/hyphen variants -> ASCII hyphen. Publisher metadata uses these
    # freely in hyphenated surnames (U+2010 HYPHEN, U+2011 NON-BREAKING HYPHEN,
    # U+2012 FIGURE DASH, U+2013 EN DASH, U+2014 EM DASH, U+2212 MINUS).
    n = re.sub(r"[‐‑‒–—―−﹘﹣－]", "-", n)
    # Multi-char + non-NFKD-decomposable mappings
    multi = {
        "ß": "ss", "þ": "th", "ł": "l", "đ": "d", "ı": "i",
        "ø": "o", "æ": "ae", "œ": "oe",
    }
    for k, v in multi.items():
        n = n.replace(k, v)
    n = re.sub(r"[^a-z\s\-]", "", n)
    n = re.sub(r"\s+", " ", n).strip()
    return n


def author_surnames_match(cited: str, actual: str) -> bool:
    """Tolerant comparison: handles particle variants and hyphenation."""
    if not cited or not actual:
        return True  # cannot judge → do not flag
    a = _normalize_surname(cited)
    b = _normalize_surname(actual)
    if not a or not b:
        return True
    if a == b:
        return True
    # Particle-stripped variants ("von elm" vs "elm")
    a_core = re.sub(r"^(?:" + "|".join(_NAME_PARTICLES) + r")\s+", "", a)
    b_core = re.sub(r"^(?:" + "|".join(_NAME_PARTICLES) + r")\s+", "", b)
    if a_core and b_core and (a_core == b_core or a_core in b_core or b_core in a_core):
        return True
    # Hyphen vs space ("Abd-alrazaq" vs "abd alrazaq")
    if a.replace("-", " ") == b.replace("-", " "):
        return True
    return False


def guess_title(raw: str) -> str:
    no_prefix = re.sub(r"^(\[\d+\]|\d+[\.\)]|\-\s+)\s*", "", raw)
    parts = [p.strip() for p in re.split(r"\.\s+", no_prefix) if p.strip()]
    for part in parts:
        words = part.split()
        if 4 <= len(words) <= 30 and not re.search(r"\b(doi|pmid|journal|vol)\b", part, re.I):
            return part.strip('"')
    return ""


def _contact_email() -> str:
    """User-supplied contact email (courtesy for NCBI/CrossRef), never a credential."""
    return (os.environ.get("MEDSCI_CONTACT_EMAIL")
            or os.environ.get("NCBI_EMAIL")
            or "medsci-skills@users.noreply.github.com")


def _user_agent() -> str:
    return f"medsci-skills/verify-refs (mailto:{_contact_email()})"


def _ncbi_extras() -> dict:
    """Optional NCBI E-utilities etiquette / rate-limit params, all from env.
    Setting NCBI_API_KEY raises the PubMed rate limit from 3 to 10 requests/s;
    absent it, the calls stay keyless. `tool`/`email` are NCBI-recommended courtesy."""
    extra = {"tool": "medsci-skills", "email": _contact_email()}
    key = os.environ.get("NCBI_API_KEY")
    if key:
        extra["api_key"] = key
    return extra


def http_json(url: str, timeout: int) -> dict | None:
    req = urllib.request.Request(url, headers={"User-Agent": _user_agent()})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return json.loads(resp.read().decode("utf-8", "replace"))
    except Exception:
        return None


def verify_crossref(doi: str, timeout: int) -> tuple[str, str, list]:
    """Returns (status, evidence, family_names).

    v1.3.0: returns full author family list instead of first-author only.
    CrossRef API is not authoritative for given names (documented case: CrossRef
    returned "Vasileios", PubMed efetch & the curated record = "Victoria").
    Use verify_pubmed_efetch as the truth source when PMID is available.
    """
    url = "https://api.crossref.org/works/" + urllib.parse.quote(doi)
    data = http_json(url, timeout)
    if not data or data.get("status") != "ok":
        return "UNVERIFIED", "CrossRef DOI lookup failed", []
    msg = data.get("message", {})
    title = " ".join(msg.get("title") or [])
    year_parts = (((msg.get("issued") or {}).get("date-parts") or [[None]])[0])
    year = str(year_parts[0]) if year_parts and year_parts[0] else ""
    authors_raw = msg.get("author") or []
    families: list[str] = []
    for a in authors_raw:
        fam = (a.get("family") or a.get("name") or "").strip()
        if fam:
            families.append(fam)
    evidence = "CrossRef DOI OK"
    if title:
        evidence += f"; title={title[:120]}"
    if year:
        evidence += f"; year={year}"
    if families:
        evidence += f"; authors={len(families)} (first={families[0]})"
    return "OK", evidence, families


def verify_pubmed_pmid(pmid: str, timeout: int) -> tuple[str, str, list]:
    """Returns (status, evidence, family_names).

    Uses esummary (fast). Returns family-name approximation by stripping trailing
    initial block from "Surname Initials" form. Authoritative names → call
    verify_pubmed_efetch().
    """
    url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?" + urllib.parse.urlencode(
        {"db": "pubmed", "id": pmid, "retmode": "json", **_ncbi_extras()}
    )
    data = http_json(url, timeout)
    if not data:
        return "UNVERIFIED", "PubMed PMID lookup failed", []
    result = data.get("result", {})
    item = result.get(pmid)
    if not item:
        return "FABRICATED", "PMID not found in PubMed", []
    if item.get("error"):
        return "FABRICATED", f"PubMed PMID error: {item['error']}", []
    title = html.unescape(item.get("title", ""))
    authors_raw = item.get("authors") or []
    families: list[str] = []
    for a in authors_raw:
        if a.get("authtype") not in (None, "Author"):
            continue
        full = (a.get("name") or "").strip()
        # esummary "name" is "Surname Initials" e.g. "Reichheld FF"
        m = re.match(r"^(.+?)\s+[A-Z]{1,4}$", full)
        fam = m.group(1).strip() if m else full
        if fam:
            families.append(fam)
    evidence = f"PubMed PMID OK; title={title[:120]}; authors={len(families)}"
    if families:
        evidence += f" (first={families[0]})"
    return "OK", evidence, families


def verify_pubmed_efetch(pmid: str, timeout: int) -> tuple[str, str, list, list]:
    """Authoritative PubMed full author record via efetch.fcgi (XML).

    Returns (status, evidence, family_names, given_names). Use given_names for
    given-name cross-check (CrossRef-vs-PubMed disagreement, e.g. a documented
    case: CrossRef "Vasileios" vs PubMed "Victoria" — PubMed is authoritative).
    """
    url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?" + urllib.parse.urlencode(
        {"db": "pubmed", "id": pmid, "retmode": "xml", **_ncbi_extras()}
    )
    req = urllib.request.Request(
        url,
        headers={"User-Agent": _user_agent()},
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            xml_text = resp.read().decode("utf-8", "replace")
    except Exception:
        return "UNVERIFIED", "PubMed efetch failed", [], []
    families: list[str] = []
    givens: list[str] = []
    # Per-Author block: <Author ValidYN="Y"><LastName>X</LastName><ForeName>Y</ForeName>...
    for am in re.finditer(
        r'<Author\s+ValidYN="Y"[^>]*>(.*?)</Author>', xml_text, re.S
    ):
        block = am.group(1)
        lm = re.search(r"<LastName>([^<]+)</LastName>", block)
        fm = re.search(r"<ForeName>([^<]+)</ForeName>", block)
        if lm:
            families.append(html.unescape(lm.group(1)).strip())
            givens.append(html.unescape(fm.group(1)).strip() if fm else "")
    if not families:
        return "UNVERIFIED", "PubMed efetch returned no author elements", [], []
    return (
        "OK",
        f"PubMed efetch OK; authors={len(families)} (first={families[0]})",
        families,
        givens,
    )


def verify_pubmed_title(title: str, timeout: int) -> tuple[str, str, list]:
    """Title-only search returns no confident author list."""
    if not title:
        return "UNVERIFIED", "No DOI, PMID, or usable title", []
    url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?" + urllib.parse.urlencode(
        {"db": "pubmed", "term": title, "retmode": "json", "retmax": "3", **_ncbi_extras()}
    )
    data = http_json(url, timeout)
    if not data:
        return "UNVERIFIED", "PubMed title search failed", []
    ids = data.get("esearchresult", {}).get("idlist", [])
    if not ids:
        return "UNVERIFIED", "No PubMed title match", []
    return "OK", f"PubMed title match; PMID candidates={','.join(ids)}", []


def _title_similarity(a: str, b: str) -> float:
    """Token Jaccard on normalized titles (stdlib-only).

    Guards OpenAlex title matches: a fabricated title must not earn a spurious OK
    just because a full-text search returned some unrelated work. Stop-short tokens
    (<=2 chars) are dropped so connective words do not inflate similarity.
    """
    def toks(s: str) -> set:
        s = re.sub(r"[^a-z0-9 ]", " ", s.lower())
        return {w for w in s.split() if len(w) > 2}
    ta, tb = toks(a), toks(b)
    if not ta or not tb:
        return 0.0
    return len(ta & tb) / len(ta | tb)


def _openalex_families(work: dict) -> list:
    """Best-effort family-name list from an OpenAlex work's authorships.

    OpenAlex exposes only `author.display_name` with NO structured family/given
    split, and the live data mixes "First Last" and "Last, First" forms within a
    single record (observed: 'Noah Shinn' alongside 'Cassano, Federico'). This list
    is therefore informational only — it is NOT used to drive the authoritative
    family-by-family MISMATCH cross-check (that stays reserved for PubMed efetch /
    CrossRef, which carry a structured family field). See verify_record.
    """
    families: list[str] = []
    for au in work.get("authorships") or []:
        name = ((au.get("author") or {}).get("display_name") or "").strip()
        if not name:
            continue
        if "," in name:
            # "Last, First" → family is the part before the comma.
            fam = name.split(",", 1)[0].strip()
        else:
            toks = name.split()
            fam = toks[-1] if toks else ""
            # Strip a trailing initials block ("Madaan A").
            if fam and re.match(r"^[A-Z]{1,4}$", fam) and len(toks) >= 2:
                fam = toks[-2]
        if fam:
            families.append(fam)
    return families


def verify_openalex(doi: str, title: str, timeout: int) -> tuple[str, str, list]:
    """Tertiary index for conference proceedings / non-DOI / non-biomedical works.

    PubMed covers only biomedical literature and CrossRef's proceedings coverage is
    spotty, so NeurIPS / ICLR / ACL-style citations (common in medical-AI papers)
    fall through both. OpenAlex (https://api.openalex.org) is free and key-less and
    ingests those venues, so it recovers them — the free analogue of the second
    index (e.g. Scopus) that journal portals use alongside CrossRef.

    Resolves by DOI when available (exact); otherwise by title.search with a
    similarity guard so a fabricated title cannot earn a spurious OK. Returns
    (status, evidence, family_names). Never returns FABRICATED: an OpenAlex miss is
    a coverage gap, not proof of fabrication.
    """
    work = None
    via = ""
    if doi:
        data = http_json(
            "https://api.openalex.org/works/https://doi.org/" + urllib.parse.quote(doi),
            timeout,
        )
        if data and data.get("id"):
            work = data
            via = "doi"
    if work is None and title:
        url = "https://api.openalex.org/works?" + urllib.parse.urlencode(
            {"filter": "title.search:" + title, "per-page": "5"}
        )
        data = http_json(url, timeout)
        results = (data or {}).get("results") or []
        best, best_sim = None, 0.0
        for w in results:
            sim = _title_similarity(title, w.get("title") or w.get("display_name") or "")
            if sim > best_sim:
                best, best_sim = w, sim
        if best is not None and best_sim >= 0.8:
            work = best
            via = f"title(sim={best_sim:.2f})"
    if work is None:
        return "UNVERIFIED", "OpenAlex: no confident match", []
    families = _openalex_families(work)
    wtitle = (work.get("title") or work.get("display_name") or "")[:120]
    year = work.get("publication_year")
    ev = f"OpenAlex OK via {via}; title={wtitle}"
    if year:
        ev += f"; year={year}"
    if families:
        ev += f"; authors={len(families)} (first={families[0]})"
    return "OK", ev, families


def author_cross_check(
    cited_authors: list,
    actual_authors: list,
    cited_author_count: int,
    actual_author_count: int,
    *,
    corporate: bool = False,
    soft: bool = False,
    audit_truncated: bool = False,
    first_author_guess: str = "",
) -> tuple[list, list]:
    """Pure family-by-family + author-count cross-check (no network, no RefRecord).

    Compares every cited author family against the authoritative list positionally,
    flags any cited author beyond the source list, and compares total counts (a count
    mismatch is downgraded to a note when ``audit_truncated`` and cited < actual — an
    intentional CSL et-al truncation). When no cited list was parsed (TSV / plain
    text) it degrades to a first-author surname check. Corporate/collective authors
    and OpenAlex-``soft`` lists are exempt (they must never fire MISMATCH).

    Returns ``(mismatches, notes)``: ``mismatches`` — human-readable mismatch strings
    (empty = clean); ``notes`` — non-mismatch evidence strings (the truncation note).
    This is the sole decision surface behind an AUTHOR MISMATCH status, extracted so
    the fabricated-co-author path has a network-free regression test
    (``tests/test_fabricated_author.sh``). Behaviour is identical to the inline logic
    it replaced in ``verify_record``.
    """
    mismatches: list = []
    notes: list = []
    if not corporate and cited_authors and actual_authors and not soft:
        compare_n = min(len(cited_authors), len(actual_authors))
        for i in range(compare_n):
            cited = cited_authors[i]
            if not author_surnames_match(cited, actual_authors[i]):
                mismatches.append(
                    f"#{i+1} family: cited='{cited}' vs source='{actual_authors[i]}'"
                )
        # cited has more authors than source — always flag (cannot be intentional)
        for i in range(compare_n, len(cited_authors)):
            mismatches.append(
                f"#{i+1} extra cited='{cited_authors[i]}' (source has only {len(actual_authors)} authors)"
            )
        # source has more authors than cited — count mismatch, suppressed when the
        # bib declares the truncation, either with the `_audit_truncated` marker or
        # with BibTeX's own `and others` sentinel (intentional CSL et-al truncation).
        if cited_author_count != actual_author_count:
            if audit_truncated and cited_author_count < actual_author_count:
                notes.append(
                    f"NOTE: intentional truncate ({cited_author_count} of {actual_author_count}; "
                    f"declared by `and others` or `_audit_truncated`)"
                )
            else:
                mismatches.append(
                    f"AUTHOR COUNT: cited={cited_author_count} vs source={actual_author_count}"
                )
    elif not corporate and first_author_guess and actual_authors:
        # No parsed cited author list (TSV / plain-text input) — degrade to the
        # first-author surname cross-check (Gate 4 behaviour).
        if not any(author_surnames_match(first_author_guess, a) for a in actual_authors):
            mismatches.append(
                f"#1 family: cited='{first_author_guess}' vs source='{actual_authors[0]}'"
            )
    return mismatches, notes


def verify_record(record: RefRecord, offline: bool, timeout: int,
                  use_openalex: bool = True) -> RefRecord:
    """v1.3.0: full-author cross-check.

    Authoritative source priority for the actual author list:
      1. PubMed efetch (XML full-record) — best (motivation: CrossRef returned a
         wrong given name "Vasileios" vs PubMed efetch authoritative "Victoria";
         also catches AI-generated bib entries with hallucinated #2..#N family
         names).
      2. CrossRef DOI (fallback when no PMID).
      3. OpenAlex (tertiary; conference proceedings / non-DOI / non-biomedical works
         that PubMed and CrossRef miss — the free analogue of a portal's Scopus pass).
      4. PubMed esummary (fast count check; family-name approximation only).
    All cited authors (BibTeX) are compared family-by-family against the
    authoritative list AND total counts are compared. Any cited author beyond
    the actual list, any per-index family mismatch, and any count mismatch are
    each reported. When no full cited list was parsed (TSV / plain text), the
    check degrades to the first-author surname comparison (Gate 4 behaviour).
    """
    if offline:
        if record.doi or record.pmid:
            record.status = "UNVERIFIED"
            record.evidence = "Identifier extracted; offline mode"
        else:
            record.status = "UNVERIFIED"
            record.evidence = "No identifier; offline mode"
        if record.corporate_author:
            record.note = "corporate/collective author — personal-name cross-check skipped"
            record.evidence += " | CORPORATE AUTHOR (collective/organization)"
        return record

    statuses: list[str] = []
    evidence_parts: list[str] = []
    actual_authors: list[str] = []
    actual_givens: list[str] = []
    sources_consulted: list[str] = []
    # True when the actual_authors list came from OpenAlex, whose display names carry
    # no structured family field and mix "First Last" / "Last, First" forms. Such a
    # list can support a tolerant first-author membership check but NOT the strict
    # positional + author-count cross-check (which would mis-fire on the format noise).
    actual_authors_soft = False

    # Step 1 — PubMed efetch (authoritative) when PMID present.
    if record.pmid:
        st, ev, fams, givens = verify_pubmed_efetch(record.pmid, timeout)
        time.sleep(0.2)
        statuses.append(st)
        evidence_parts.append(ev)
        if st == "OK" and fams:
            actual_authors = fams
            actual_givens = givens
            sources_consulted.append("pubmed_efetch")
        # also run esummary for FABRICATED detection (efetch returns valid XML even for
        # unknown PMIDs in some edge cases; esummary's "error" field is decisive).
        st_es, ev_es, fams_es = verify_pubmed_pmid(record.pmid, timeout)
        time.sleep(0.2)
        statuses.append(st_es)
        evidence_parts.append(ev_es)
        if not actual_authors and st_es == "OK" and fams_es:
            actual_authors = fams_es
            sources_consulted.append("pubmed_esummary")

    # Step 2 — CrossRef DOI (used only when efetch did not provide a list).
    if record.doi:
        st_cr, ev_cr, fams_cr = verify_crossref(record.doi, timeout)
        time.sleep(0.2)
        statuses.append(st_cr)
        evidence_parts.append(ev_cr)
        if not actual_authors and st_cr == "OK" and fams_cr:
            actual_authors = fams_cr
            sources_consulted.append("crossref")

    # Step 3 — OpenAlex tertiary index. Fires only when no authoritative author list
    # was obtained yet (no PMID/DOI, or those lookups returned no authors), so a
    # biomedical reference already resolved by PubMed/CrossRef incurs no extra call.
    # Recovers conference proceedings and non-biomedical works (NeurIPS/ICLR/ACL) and
    # retries DOIs that CrossRef missed.
    if use_openalex and not actual_authors:
        st_oa, ev_oa, fams_oa = verify_openalex(record.doi, record.title_guess, timeout)
        time.sleep(0.2)
        statuses.append(st_oa)
        evidence_parts.append(ev_oa)
        if st_oa == "OK":
            sources_consulted.append("openalex")
            if fams_oa:
                actual_authors = fams_oa
                actual_authors_soft = True

    # Step 4 — PubMed title-only final fallback when nothing confident resolved.
    if "OK" not in statuses and not actual_authors:
        st_t, ev_t, _ = verify_pubmed_title(record.title_guess, timeout)
        time.sleep(0.2)
        statuses.append(st_t)
        evidence_parts.append(ev_t)

    # Full-author cross-check
    record.actual_authors = actual_authors
    record.actual_author_count = len(actual_authors)
    if record.cited_authors and not record.cited_author_count:
        record.cited_author_count = len(record.cited_authors)

    # Collective/corporate author (guideline body, working group): PubMed returns
    # it as <CollectiveName> and the BibTeX double-braces it, so the personal-name
    # family cross-check does not apply. Detect it on the source side too (no parsed
    # personal authors but a title/DOI verified, or the source author looks like an
    # organization), so a guideline cite is VERIFIED, never a render-aborting MISMATCH.
    source_corporate = bool(actual_authors) and any(_ORG_AUTHOR_RE.search(a) for a in actual_authors)
    if record.corporate_author or source_corporate:
        if not record.note:
            record.note = "corporate/collective author — personal-name cross-check skipped"
        evidence_parts.append("CORPORATE AUTHOR (collective/organization; family cross-check skipped)")

    mismatches, xcheck_notes = author_cross_check(
        record.cited_authors,
        actual_authors,
        record.cited_author_count,
        record.actual_author_count,
        corporate=(record.corporate_author or source_corporate),
        soft=actual_authors_soft,
        audit_truncated=record.audit_truncated,
        first_author_guess=record.first_author_guess,
    )
    evidence_parts.extend(xcheck_notes)

    author_mismatch = bool(mismatches)
    if author_mismatch:
        evidence_parts.append("AUTHOR MISMATCH | " + " | ".join(mismatches))

    # Status precedence
    if "OK" in statuses and "FABRICATED" in statuses:
        record.status = "MISMATCH"
    elif "OK" in statuses:
        record.status = "MISMATCH" if author_mismatch else "OK"
    elif "FABRICATED" in statuses:
        record.status = "FABRICATED"
    else:
        record.status = "UNVERIFIED"

    # Note classification (most informative wins)
    if author_mismatch and not record.note:
        # Distinguish first-author hallucination (high reviewer salience)
        first_cited = record.cited_authors[0] if record.cited_authors else record.first_author_guess
        first_bad = (
            first_cited
            and actual_authors
            and not author_surnames_match(first_cited, actual_authors[0])
        )
        if first_bad:
            record.note = "first-author hallucination suspected (DOI/PMID correct, family differs)"
        else:
            record.note = "non-first-author hallucination or count mismatch (DOI/PMID correct)"
    record.evidence = " | ".join(p for p in evidence_parts if p)
    if sources_consulted:
        record.evidence += f" | source={'+'.join(sources_consulted)}"
    return record


def detect_duplicates(records: list[RefRecord]) -> list[dict]:
    """Detect verbatim PMID or DOI duplicates within the reference list.

    Verbatim duplicates (same PMID or normalized DOI) are a common LLM
    citation-compilation artifact and require cite renumbering before
    submission.
    """
    seen_pmids: dict[str, str] = {}
    seen_dois: dict[str, str] = {}
    findings: list[dict] = []
    for rec in records:
        rec_id = rec.ref_id or "<unknown>"
        pmid = (rec.pmid or "").strip()
        if pmid:
            if pmid in seen_pmids:
                findings.append({
                    "severity": "MAJOR",
                    "category": "duplicate_pmid",
                    "ref_ids": [seen_pmids[pmid], rec_id],
                    "pmid": pmid,
                    "note": "Verbatim duplicate reference. Cite renumbering required.",
                })
            else:
                seen_pmids[pmid] = rec_id
        doi = normalize_doi_for_dup(rec.doi or "")
        if doi:
            if doi in seen_dois:
                findings.append({
                    "severity": "MAJOR",
                    "category": "duplicate_doi",
                    "ref_ids": [seen_dois[doi], rec_id],
                    "doi": doi,
                    "note": "Verbatim duplicate reference. Cite renumbering required.",
                })
            else:
                seen_dois[doi] = rec_id
    return findings


# Pagination / publication-stage placeholders. A reference whose pages or status is
# still "e000–e000", "in press", "TBD", or "forthcoming" is not yet a fully citable
# record. verify-refs is manuscript-agnostic, so it only flags these as UNVERIFIED
# with note="pagination_placeholder"; the centrality call (is this a method- or
# headline-load-bearing cite, hence a P0 blocker?) is made by /self-review Phase 2.5c,
# which has the manuscript in hand. (Gate 6, added 2026-06.)
PAGINATION_PLACEHOLDER_RE = re.compile(
    r"e0{3}.{0,3}e0{3}|in[ .]?press|\bTBD\b|forthcoming", re.I)


def flag_pagination_placeholder(record: RefRecord) -> None:
    """If the raw entry carries a pagination/publication-stage placeholder, attach a
    note and downgrade a would-be VERIFIED record to UNVERIFIED (an in-press/e000
    citation is not yet locatable to the page). Worse statuses are left unchanged."""
    if not PAGINATION_PLACEHOLDER_RE.search(record.raw or ""):
        return
    tag = "pagination_placeholder"
    record.note = f"{record.note} | {tag}".strip(" |") if record.note else tag
    if record.status == "VERIFIED":
        record.status = "UNVERIFIED"
        ev = "identifier resolved but pagination/publication-stage placeholder unresolved"
        record.evidence = f"{record.evidence} | {ev}".strip(" |") if record.evidence else ev


def portable_source(source: Path, project_root: Path) -> str:
    """Render the audited file's path so the audit can travel with the manuscript.

    `source` was written with `str()`, i.e. whatever the caller passed — and the caller
    resolves it, so on a real run that is an absolute path. On a maintainer's machine an
    absolute path contains the home directory, and therefore a username, inside
    `qc/reference_audit.json`, which is a COMMITTED artifact. The public-surface PII gate
    caught exactly that three times between 2026-07-31 and 2026-08-01; the field is
    rewritten on every render, so hand-scrubbing it lost twice.

    Nothing consumes the absolute form. The audit file lives beside the manuscript and
    moves with it, so a path relative to the project root is not merely PII-free, it is
    the correct referent. Falls back to the CWD, then to the bare filename, so a source
    outside the project (a shared .bib) still yields something readable and still carries
    no home directory.
    """
    for base in (project_root, Path.cwd()):
        try:
            return source.resolve().relative_to(base.resolve()).as_posix()
        except ValueError:
            continue
    return source.name


def write_outputs(records: list[RefRecord], project_root: Path, source: Path,
                  duplicate_findings: list[dict]) -> None:
    """Audit-only writer (v1.3.0).

    Per docs/artifact_contract.md, /verify-refs is sole writer of qc/reference_audit.json
    only. It MUST NOT write to references/ (that directory is owned by /search-lit and
    /lit-sync). All per-record details live inside reference_audit.json.

    v1.2.0 (2026-05): adds duplicate_findings[] for PMID/DOI duplicate detection
    (Gate 5; resolves /peer-review Phase 2A P7). submission_safe and fully_verified
    both require duplicate_findings to be empty.

    v1.3.0 (2026-05): full-author cross-check. records[] now carry cited_authors[],
    actual_authors[], and author counts; schema_version bumps to 4. MISMATCH now
    fires on any #2..#N family hallucination or author-count mismatch, not just the
    first author. A correct lead author does not verify the remaining author list.
    """
    qc_dir = project_root / "qc"
    qc_dir.mkdir(parents=True, exist_ok=True)

    counts: dict[str, int] = {}
    for rec in records:
        counts[rec.status] = counts.get(rec.status, 0) + 1
    audit = {
        "schema_version": 4,
        "source": portable_source(source, project_root),
        "total_references": len(records),
        "counts": counts,
        "duplicate_findings": duplicate_findings,
        "submission_safe": (
            counts.get("FABRICATED", 0) == 0
            and counts.get("MISMATCH", 0) == 0
            and len(duplicate_findings) == 0
        ),
        "fully_verified": (
            counts.get("UNVERIFIED", 0) == 0
            and counts.get("FABRICATED", 0) == 0
            and counts.get("MISMATCH", 0) == 0
            and len(duplicate_findings) == 0
        ),
        "requires_manual_reference_check": counts.get("UNVERIFIED", 0) > 0,
        "records": [asdict(rec) for rec in records],
    }
    (qc_dir / "reference_audit.json").write_text(json.dumps({"detector": "verify_refs", **audit}, indent=2, ensure_ascii=False), encoding="utf-8")


def main() -> int:
    parser = argparse.ArgumentParser(description="Verify manuscript references.")
    parser.add_argument("input", help="Input .md, .docx, .bib, .txt, or .tsv file")
    parser.add_argument("--project-root", default=".", help="Project root for output artifacts")
    parser.add_argument("--offline", action="store_true", help="Do not call PubMed/CrossRef/OpenAlex APIs")
    parser.add_argument("--no-openalex", action="store_true",
                        help="Disable the OpenAlex tertiary index (restrict to PubMed + CrossRef)")
    parser.add_argument("--timeout", type=int, default=10, help="HTTP timeout seconds")
    parser.add_argument("--strict", action="store_true", help="Exit non-zero on any UNVERIFIED row, and forbid --offline")
    args = parser.parse_args()

    if args.strict and args.offline:
        print("--strict is incompatible with --offline", file=sys.stderr)
        return 2

    input_path = Path(args.input).resolve()
    project_root = Path(args.project_root).resolve()
    if not input_path.exists():
        print(f"Input not found: {input_path}", file=sys.stderr)
        return 2

    text = read_input(input_path)
    suffix = input_path.suffix.lower()
    if suffix == ".bib":
        records = parse_bib(text)
    elif suffix == ".tsv":
        records = parse_tsv(text)
    else:
        records = parse_reference_lines(text)

    if not records:
        print("No references detected.", file=sys.stderr)
        return 3

    verified = [
        verify_record(rec, args.offline, args.timeout, use_openalex=not args.no_openalex)
        for rec in records
    ]
    for rec in verified:
        flag_pagination_placeholder(rec)
    duplicate_findings = detect_duplicates(verified)
    write_outputs(verified, project_root, input_path, duplicate_findings)

    counts: dict[str, int] = {}
    for rec in verified:
        counts[rec.status] = counts.get(rec.status, 0) + 1
    print(json.dumps({
        "total": len(verified),
        "counts": counts,
        "duplicate_findings_count": len(duplicate_findings),
    }, indent=2))
    if counts.get("FABRICATED", 0) or counts.get("MISMATCH", 0) or duplicate_findings:
        return 1
    if args.strict and counts.get("UNVERIFIED", 0):
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Report helpers for claim fidelity; no clinical verdicts or network calls.

An assessment is an attributed review record, not a mechanically verified fact.
The helpers bind that record to inputs and render a view of the existing JSON.
"""
from __future__ import annotations

from collections import Counter
import copy
import hashlib
import json
from pathlib import Path
import re

DIMENSIONS = ("metric", "unit", "denominator", "population", "direction")
VERDICTS = ("supported", "contradicted", "unresolved", "not_assessed")


def sha256(path: Path | None) -> str | None:
    return hashlib.sha256(path.read_bytes()).hexdigest() if path and path.is_file() else None


def normalized_doi(value: str) -> str:
    return re.sub(r"^(?:https?://(?:dx\.)?doi\.org/|doi:\s*)", "", value.strip(),
                  flags=re.I).lower().rstrip(".,;")


def prose_spans(raw: str, abbrev: re.Pattern):
    """Prose sentence offsets in the read_text representation, not PDF coordinates.

    Mask excluded regions without shifting positions. As with the existing probes,
    prose inventory excludes code, blockquotes, markdown tables and references.
    """
    def mask(match):
        return re.sub(r"[^\n]", " ", match.group())

    body = re.sub(r"\A---[^\n]*\n.*?\n(?:---|\.\.\.)[^\n]*(?:\n|$)", mask, raw, flags=re.S)
    body = re.sub(r"(?m)^[ \t]*(```|~~~)[^\n]*\n.*?^[ \t]*\1[^\n]*(?:\n|$)",
                  mask, body, flags=re.S)
    cut = re.search(r"(?im)^#{1,3}\s*\**\s*(references|bibliography|works cited)\b", body)
    if cut:
        body = body[:cut.start()]
    body = re.sub(r"(?m)^[ \t]*(?:>|\||#{1,6}\s)[^\n]*", mask, body)
    bracket_spans = [(m.start(), m.end()) for m in re.finditer(r"\[[^\]]*\]", body)]
    start = 0
    for boundary in re.finditer(r"(?<=[.!?])\s+|\n[ \t]*\n|\Z", body):
        end = boundary.start()
        if any(a < end < b for a, b in bracket_spans):
            continue  # e.g. a page locator inside [see @key, p. 3]
        if abbrev.search(body[start:end].strip()) and "\n\n" not in boundary.group():
            continue
        part = body[start:end]
        left = len(part) - len(part.lstrip())
        right = len(part.rstrip())
        if right > left:
            yield start + left, start + right
        start = boundary.end()


def blank_assessment() -> dict:
    return {"verdict": "not_assessed", "assessor": "", "assessed_at": "", "method": "",
            "source_scope": "unknown", "identity_checked": False,
            "source_pages": [], "source_excerpt": "", "rationale": "",
            "claim": {key: "" for key in DIMENSIONS},
            "source": {key: "" for key in DIMENSIONS}}


def binding_hash(binding: dict) -> str:
    return hashlib.sha256(json.dumps(binding, sort_keys=True).encode()).hexdigest()


def reference_record(audit: dict | None, token: str, doi: str) -> dict:
    """Metadata audit is contextual, never proof of source identity or claim support."""
    records = (audit or {}).get("records", [])
    matches = [r for r in records if r.get("ref_id") == token]
    if not matches and doi:
        matches = [r for r in records if normalized_doi(r.get("doi", "")) == doi]
    if len(matches) != 1:
        return {"link": "ambiguous" if matches else "not_available", "recorded_status": None}
    row = matches[0]
    same_doi = not doi or normalized_doi(row.get("doi", "")) == doi
    return {"link": "matched_identifier" if same_doi else "identifier_conflict",
            "recorded_status": row.get("status"), "doi": row.get("doi", ""),
            "input_binding": "not_recorded_by_reference_audit"}


def pdf_record(retrieval: dict | None, pdf_dir: Path | None, doi: str) -> dict:
    rows = [r for r in (retrieval or {}).get("items", [])
            if doi and normalized_doi(r.get("doi", "")) == doi]
    result = {"file": None, "sha256": None, "report_sha256": None,
              "hash_status": "not_available", "identity": {"status": "unavailable"},
              "text_derivation": "not_recorded; equal filenames do not establish PDF-to-text provenance"}
    if len(rows) != 1:
        result["hash_status"] = "ambiguous_report" if rows else "not_available"
        return result
    row = rows[0]
    identity = row.get("source_identity")
    if not isinstance(identity, dict) or identity.get("status") not in {"consistent", "conflict", "unresolved", "unavailable"}:
        identity = {"status": "unavailable", "reason": "missing_or_invalid_identity_record"}
    result["identity"] = copy.deepcopy(identity)
    result["report_sha256"] = row.get("file_sha256") or None
    name = row.get("file", "")
    # Retrieval reports contain basenames, not arbitrary local read instructions.
    if not name or Path(name).name != name or "/" in name or "\\" in name or not pdf_dir:
        result["hash_status"] = "invalid_or_missing_filename"
        return result
    path = pdf_dir / name
    if not path.resolve().is_relative_to(pdf_dir.resolve()):
        result["hash_status"] = "path_outside_pdf_directory"
        return result
    result["file"] = name
    result["sha256"] = sha256(path)
    if result["sha256"] is None:
        result["hash_status"] = "file_missing"
    elif result["report_sha256"] is None:
        result["hash_status"] = "report_hash_missing"
    elif result["sha256"] != result["report_sha256"]:
        result["hash_status"] = "changed_since_retrieval"
    else:
        result["hash_status"] = "matched"
    return result


def nonempty(value) -> bool:
    return isinstance(value, str) and bool(value.strip())


def assessment_state(assessment: dict, row: dict, binding_matches: bool) -> tuple[str, str]:
    """Validate record completeness and freshness, not scientific interpretation."""
    verdict = assessment.get("verdict")
    if verdict == "not_assessed":
        return "not_assessed", "not_assessed"
    if not binding_matches:
        return "unresolved", "stale_inputs"
    if verdict not in VERDICTS:
        return "unresolved", "invalid_verdict"
    if not all(nonempty(assessment.get(key)) for key in ("assessor", "assessed_at", "method", "rationale")):
        return "unresolved", "incomplete_assessment"
    if verdict == "unresolved":
        return verdict, "recorded"
    if row["source"]["text_sha256"] is None:
        return "unresolved", "source_not_available"
    if row["source"]["pdf"]["hash_status"] != "matched":
        return "unresolved", "pdf_not_bound"
    if row["source"]["pdf"]["identity"].get("status") == "conflict":
        return "unresolved", "source_identity_conflict"
    if assessment.get("source_scope") != "full_text" or assessment.get("identity_checked") is not True:
        return "unresolved", "source_review_incomplete"
    pages = assessment.get("source_pages")
    if not isinstance(pages, list) or not pages or any(type(p) is not int or p < 1 for p in pages):
        return "unresolved", "missing_pdf_pages"
    if not nonempty(assessment.get("source_excerpt")):
        return "unresolved", "missing_source_excerpt"
    for side in ("claim", "source"):
        values = assessment.get(side)
        if not isinstance(values, dict) or not all(nonempty(values.get(key)) for key in DIMENSIONS):
            return "unresolved", "incomplete_comparison"
    return verdict, "recorded"


def build_evidence(raw: str, manuscript: Path, fulltext_dir: Path, resolver,
                   abbrev: re.Pattern, *, binding: dict, retrieval: dict | None = None,
                   pdf_dir: Path | None = None, reference_audit: dict | None = None,
                   reviewed_report: dict | None = None) -> dict:
    previous_rows = (reviewed_report or {}).get("evidence_rows", [])
    previous = {r["id"]: r for r in previous_rows}
    if len(previous) != len(previous_rows):
        raise ValueError("duplicate evidence row IDs in reviewed report")
    rows = []
    occurrences = Counter()
    pdf_cache = {}
    for start, end in prose_spans(raw, abbrev):
        sentence = raw[start:end]
        for token in dict.fromkeys(resolver.citations_in(sentence)):
            sentence_hash = hashlib.sha256(sentence.encode()).hexdigest()
            base = hashlib.sha256((sentence + "\0" + token).encode()).hexdigest()[:20]
            occurrences[base] += 1
            ident = f"claim-{base}-{occurrences[base]}"
            doi = normalized_doi(resolver.key_doi.get(token) or resolver.num_doi.get(token) or "")
            mapped = resolver.refmap.get(token, "")
            if not doi and normalized_doi(mapped).startswith("10."):
                doi = normalized_doi(mapped)
            source = resolver.resolve(token)
            if doi not in pdf_cache:
                pdf_cache[doi] = pdf_record(retrieval, pdf_dir, doi)
            pdf = copy.deepcopy(pdf_cache[doi])
            source_hash = resolver._source_hashes[source[0]] if source else None
            current_binding = {**binding, "sentence_sha256": sentence_hash, "citation": token,
                               "doi": doi, "text_sha256": source_hash, "pdf_sha256": pdf["sha256"]}
            row = {"id": ident, "citation": token, "doi": doi or None,
                   "manuscript": {"file": manuscript.name, "text": sentence,
                                  "line_start": raw.count("\n", 0, start) + 1,
                                  "line_end": raw.count("\n", 0, end) + 1,
                                  "char_start": start, "char_end": end,
                                  "coordinate_system": "read_text character offsets; lines are not rendered pages"},
                   "source": {"text_file": source[0].relative_to(fulltext_dir).as_posix() if source else None,
                              "text_sha256": source_hash, "pdf": pdf},
                   "reference_audit": reference_record(reference_audit, token, doi),
                   "binding": current_binding, "assessment": blank_assessment(),
                   "verdict": "not_assessed", "review_state": "not_assessed"}
            row["binding_sha256"] = binding_hash(current_binding)
            row["assessment"]["binding_sha256"] = row["binding_sha256"]
            prior = previous.get(ident)
            if prior:
                assessment = prior.get("assessment")
                if not isinstance(assessment, dict):
                    raise ValueError("assessment must be an object")
                row["assessment"] = copy.deepcopy(assessment)
                # Preserve the assessment's original binding through repeated reruns.
                # Comparing only the prior report's current inputs would make a stale
                # assessment look current on the second rerun.
                matches = assessment.get("binding_sha256") == binding_hash(current_binding)
                row["verdict"], row["review_state"] = assessment_state(assessment, row, matches)
                if not matches:
                    row["previous_binding"] = prior.get("binding")
            rows.append(row)
    present = {r["id"] for r in rows}
    # Do not silently drop a review when its sentence/citation disappears on rerun.
    orphaned = [copy.deepcopy(r) for r in previous_rows if r["id"] not in present]
    orphaned += copy.deepcopy((reviewed_report or {}).get("orphaned_evidence_reviews", []))
    return {"evidence_rows": rows, "orphaned_evidence_reviews": orphaned,
            "evidence_counts": {"sentence_citation_pairs": len(rows),
                                "verdicts": dict(Counter(r["verdict"] for r in rows)),
                                "orphaned_reviews": len(orphaned)},
            "evidence_scope": "Recognized prose citations only; tables, blockquotes, code, bibliography and uncited claims are not inventoried. Recorded assessments are attributed judgments, not automatically verified facts. PDF pages and excerpts require assessor inspection; PDF-to-text derivation is not inferred."}


def render_table(report: dict) -> str:
    def cell(value):
        # Literal user/source text must not become HTML, links or multiline table syntax.
        import html
        value = html.escape(str(value), quote=False)
        value = re.sub(r"([\\`*\[\]_])", r"\\\1", value)
        return value.replace("|", "&#124;").replace("\n", " ").replace("\r", " ")

    lines = ["# Claim-source evidence table", "",
             "Derived from claim_fidelity.json. Recorded judgments are not automatic source verification.",
             "Automated findings and manual assessments are separate. Unknown or stale evidence remains visible.",
             "", "| ID / citation | Manuscript location and claim | Source / PDF SHA256 / identity | Pages and excerpt | Claim vs source | Assessment |",
             "|---|---|---|---|---|---|"]
    for row in report["evidence_rows"]:
        assessment = row["assessment"]; source = row["source"]; pdf = source["pdf"]
        comparison = "; ".join(f"{k}: {assessment.get('claim', {}).get(k, '')} / {assessment.get('source', {}).get(k, '')}" for k in DIMENSIONS)
        columns = [f"{row['id']} / {row['citation']}",
                   f"{row['manuscript']['file']} L{row['manuscript']['line_start']}: {row['manuscript']['text']}",
                   f"Text: {source['text_file'] or 'missing'}; PDF: {pdf['file'] or 'missing'} / {pdf['sha256'] or 'unbound'} / {pdf['identity'].get('status', 'unavailable')} ({pdf['hash_status']})",
                   f"{assessment.get('source_pages', [])}: {assessment.get('source_excerpt', '')}",
                   comparison,
                   f"{row['verdict']} ({row['review_state']}); {assessment.get('assessor', '')}; {assessment.get('assessed_at', '')}; {assessment.get('method', '')}; {assessment.get('rationale', '')}"]
        lines.append("| " + " | ".join(cell(c) for c in columns) + " |")
    lines += ["", report["evidence_scope"], "",
              f"Orphaned review records retained in JSON: {len(report['orphaned_evidence_reviews'])}.",
              f"Automated findings retained in JSON: {len(report['findings'])}.", ""]
    return "\n".join(lines)

"""Quote matching that survives an extraction layer — the substrate under quote gates.

WHY THIS EXISTS (the failure it removes)

Verifying "the manuscript contains this quoted sentence" by searching a CONTIGUOUS string
is wrong whenever the haystack came out of an extractor, because extractors interleave
tokens the source never had. In one submission-day session that single assumption produced
thirteen false positives, all the same shape:

  * a two-column PDF bled reference-list text into the middle of a sentence
    ("learners form independent" | "civile." | "assessments before seeing AI output");
  * a line-numbered supplement PDF put the line number inside the sentence
    ("were" | "86" | "performed");
  * superscript markers and footnote references landed mid-clause;
  * hyphenation across a line break split one word into two ("assess-" + "ments").

Every one of those quotes was CORRECT and present. The contiguous check called them absent.
It came within one step of instructing an author to delete two accurate verbatim quotes.

THE RULE THIS ENCODES

A quote that cannot be matched contiguously is not thereby "not in the source". It is
UNRESOLVED until something stronger says otherwise. So this module grades a match instead
of answering yes/no:

  EXACT        the normalized quote is a contiguous substring — verified, no doubt.
  INTERLEAVED  every quote token appears IN ORDER, with only a bounded number of foreign
               tokens wedged between them — the text is there and the extraction is dirty.
  PARTIAL      most quote tokens appear in order but some are missing — consistent with
               extraction damage (hyphen splits, dropped glyphs); too weak to call absent.
  ABSENT       not even a partial ordered run — the text really is not there.

Only ABSENT justifies a "you claimed an edit you did not make" verdict. INTERLEAVED and
PARTIAL are reported as unresolved so a human looks, rather than as a defect.

WHY THE GAPS ARE BOUNDED (the precision that makes this safe)

An unbounded subsequence match is worthless: the tokens of almost any short sentence appear
"in order" somewhere in a long document if you allow arbitrary distance. The bound that works
is not a token budget but an INTERRUPTION COUNT, because the two cases differ in shape:

    a real extraction artifact interrupts a sentence once or twice, and each interruption can
    be long (a bled reference line is a dozen tokens);

    a spurious "match" interrupts at nearly every token, each time by a little.

So the limits are: at most MAX_GAP foreign tokens at any single join, at most
MAX_INTERRUPTIONS joins that are interrupted at all, and a total-insertion sanity cap. A
quote whose words are scattered one-by-one across a Discussion section needs an interruption
at every join and fails, while a quote split once by a column bleed passes.

Not a detector: a helper imported by the gates that need it (leading underscore keeps it out
of the detector catalog glob). Stdlib only.
"""

from __future__ import annotations

import re
import unicodedata

# At most this many foreign tokens may sit at ONE join. A bled reference line ("civile. Rev
# Med Suisse 2019;15:1122.") is around a dozen tokens; a running header a handful.
MAX_GAP = 25
# At most this many joins may be interrupted AT ALL. This is the limit that separates a dirty
# extraction (one or two interruptions) from a spurious scatter (an interruption per token).
MAX_INTERRUPTIONS = 4
# Sanity cap on total foreign tokens, so a short quote cannot absorb an entire paragraph.
MAX_TOTAL_INSERT_FRAC = 5.0
MIN_TOTAL_INSERT = 20
# A PARTIAL match must still account for this share of the quote's tokens; below it, ABSENT.
PARTIAL_COVERAGE = 0.80

_TOKEN_RE = re.compile(r"[0-9a-z]+(?:'[a-z]+)?", re.IGNORECASE)


def normalize(s: str) -> str:
    """Casefold, unify quotes/dashes, drop markdown emphasis, repair line-break hyphenation,
    and collapse whitespace. Hyphenation repair matters: an extractor that wraps "assess-
    ments" across a line otherwise destroys the token the quote is looking for."""
    s = unicodedata.normalize("NFKC", s)
    s = s.replace("’", "'").replace("‘", "'")
    s = s.replace("“", '"').replace("”", '"')
    # join a word split by a hyphen at a line break: "assess-\n  ments" -> "assessments"
    s = re.sub(r"(\w)[-‐‑]\s*\n\s*(\w)", r"\1\2", s)
    s = re.sub(r"[*_`]", "", s)
    s = re.sub(r"\s+", " ", s)
    return s.casefold().strip()


def tokens(s: str) -> list[str]:
    """Normalized word/number tokens. Punctuation is dropped, so an injected '.' or a stray
    bracket never breaks a match on its own."""
    return _TOKEN_RE.findall(normalize(s))


def _ordered_run(needle: list[str], hay: list[str], allow_missing: bool):
    """Best ordered match of `needle` inside `hay`.

    Walks every candidate start and consumes needle tokens in order, skipping at most
    MAX_GAP foreign tokens per join, at most MAX_INTERRUPTIONS interrupted joins, and a
    total-insertion sanity cap. With allow_missing, a needle token that cannot be found
    within the gap window is skipped (counted as missing) instead of failing the run.

    Returns (matched_count, inserted_count) for the best run, or (0, 0)."""
    if not needle or not hay:
        return (0, 0)
    budget = max(MIN_TOTAL_INSERT, int(len(needle) * MAX_TOTAL_INSERT_FRAC))
    max_missing = len(needle) - int(len(needle) * PARTIAL_COVERAGE)
    best = (0, 0)
    first = needle[0]
    starts = [i for i, t in enumerate(hay) if t == first]
    if allow_missing and not starts:
        # the opening token itself may be the damaged one — try any token of the quote
        wanted = set(needle)
        starts = [i for i, t in enumerate(hay) if t in wanted]
    for start in starts:
        hi = start
        matched = inserted = missing = interruptions = 0
        for tok in needle:
            found = -1
            for j in range(hi, min(hi + MAX_GAP + 1, len(hay))):
                if hay[j] == tok:
                    found = j
                    break
            if found < 0:
                if not allow_missing:
                    break
                missing += 1
                if missing > max_missing:
                    break
                continue
            gap = found - hi
            if gap:
                interruptions += 1
                if interruptions > MAX_INTERRUPTIONS:
                    break
            inserted += gap
            if inserted > budget:
                break
            matched += 1
            hi = found + 1
        if matched > best[0]:
            best = (matched, inserted)
        if matched == len(needle):
            break
    return best


def match_quality(quote: str, haystack: str) -> dict:
    """Grade how well `quote` is present in `haystack`.

    Returns {"grade": EXACT|INTERLEAVED|PARTIAL|ABSENT, "matched", "total", "inserted",
             "coverage"}. Only ABSENT means "this text is not in the document"."""
    nq, nh = normalize(quote), normalize(haystack)
    q_tok = tokens(quote)
    total = len(q_tok)
    if total == 0:
        return {"grade": "ABSENT", "matched": 0, "total": 0, "inserted": 0, "coverage": 0.0}
    if nq and nq in nh:
        return {"grade": "EXACT", "matched": total, "total": total, "inserted": 0, "coverage": 1.0}

    h_tok = tokens(haystack)
    matched, inserted = _ordered_run(q_tok, h_tok, allow_missing=False)
    if matched == total:
        return {"grade": "INTERLEAVED", "matched": matched, "total": total,
                "inserted": inserted, "coverage": 1.0}

    matched, inserted = _ordered_run(q_tok, h_tok, allow_missing=True)
    coverage = matched / total
    grade = "PARTIAL" if coverage >= PARTIAL_COVERAGE else "ABSENT"
    return {"grade": grade, "matched": matched, "total": total,
            "inserted": inserted, "coverage": round(coverage, 3)}

#!/usr/bin/env bash
# Regression test: the audit's `source` field must never carry an absolute path.
#
# `qc/reference_audit.json` is a committed artifact that travels with the manuscript, and its
# `source` was written as `str(source)` — the caller resolves the input, so on a real run that is
# an absolute path. On a maintainer's machine an absolute path contains the home directory, and
# therefore a username. The public-surface PII gate caught it three times between 2026-07-31 and
# 2026-08-01: the field is rewritten on every render, so hand-scrubbing lost twice.
#
# The fix is `portable_source()`. This test pins BOTH directions: the path is relative (so the
# audit is portable and PII-free) AND it still identifies the file (so the field keeps its
# meaning). The last two cases are negative controls — a name that merely *looks* like a home
# directory, and a source outside the project, which must degrade to a bare filename rather than
# leaking the way there.
#
# Network-free: nothing here calls PubMed, CrossRef or OpenAlex.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
V="$REPO_ROOT/skills/verify-refs/scripts/verify_refs.py"

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-52s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-52s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

out="$(python3 - "$V" <<'PY'
import importlib.util, json, os, sys, tempfile
from pathlib import Path

spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec)
sys.modules["vr"] = m
spec.loader.exec_module(m)

results = {}
with tempfile.TemporaryDirectory() as td:
    td = Path(td).resolve()
    # A directory tree shaped like the real thing: a home, a project inside it, a bib inside that.
    home = td / "Users" / "someone"
    project = home / "workspace" / "05_Paper"
    (project / "manuscript").mkdir(parents=True)
    bib = project / "manuscript" / "refs.bib"
    bib.write_text("@article{k,\n  title = {A title}\n}\n", encoding="utf-8")

    results["inside_project"] = m.portable_source(bib, project)

    # Source outside the project entirely — a shared library .bib two levels up.
    shared = home / "shared" / "library.bib"
    shared.parent.mkdir(parents=True)
    shared.write_text("", encoding="utf-8")
    results["outside_project"] = m.portable_source(shared, project)

    # The end-to-end path: write_outputs is what actually produces the committed file.
    m.write_outputs([], project, bib, [])
    audit = json.loads((project / "qc" / "reference_audit.json").read_text(encoding="utf-8"))
    results["written"] = audit["source"]
    results["written_is_absolute"] = str(Path(audit["source"]).is_absolute()).lower()
    # The whole point: the serialized artifact must not contain the tree above the project.
    blob = (project / "qc" / "reference_audit.json").read_text(encoding="utf-8")
    results["home_leaked"] = str("someone" in blob).lower()

    # Relative-to-CWD fallback: a source under the CWD but outside --project-root.
    cwd_before = Path.cwd()
    try:
        os.chdir(home)
        results["under_cwd"] = m.portable_source(shared, project)
    finally:
        os.chdir(cwd_before)

print(json.dumps(results))
PY
)"

get() { python3 -c "import json,sys; print(json.loads(sys.argv[1])[sys.argv[2]])" "$out" "$1"; }

echo "==== the source is relative, and still names the file ===="
ck "source inside the project root"        "manuscript/refs.bib"  "$(get inside_project)"
ck "as written into reference_audit.json"  "manuscript/refs.bib"  "$(get written)"

echo "==== NEGATIVE CONTROLS — nothing above the project may appear ===="
ck "written path is absolute"              "false"                "$(get written_is_absolute)"
ck "home directory name in the artifact"   "false"                "$(get home_leaked)"

echo "==== a source outside the project degrades to a name, not a route ===="
ck "outside project, no CWD match"         "library.bib"          "$(get outside_project)"
ck "outside project, but under the CWD"    "shared/library.bib"   "$(get under_cwd)"

echo
echo "  passed=$pass failed=$fail"
[ "$fail" -eq 0 ] || exit 1
echo "OK: reference_audit.json names its source without naming the machine."

#!/usr/bin/env bash
# Regression test for two precision defects in verify_refs.py's author cross-check.
#
# Both were found on the same clean bibliography, and they fail in opposite directions:
#
#   1. FALSE ALARM. A publisher-supplied Unicode hyphen (U+2010) in a hyphenated surname was
#      DELETED by the normalizer's final `[^a-z\s-]` filter rather than folded, so CrossRef's
#      `Foltyn‐Dumitru` normalized to `foltyndumitru` while the identical ASCII bib entry gave
#      `foltyn-dumitru` — and the audit fired MISMATCH, its loudest verdict, on a correct entry.
#
#   2. FALSE PASS, which is worse. Better BibTeX brace-protects a hyphenated or particle surname
#      (`{Eckel-Passow}, Jeanette E.`) so BibTeX will not re-split it. The corporate-author
#      heuristic treated any brace as an organization and SKIPPED the author cross-check — the
#      one thing this tool exists to do — without the user noticing.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
V="$REPO_ROOT/skills/verify-refs/scripts/verify_refs.py"

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-56s exit=%s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-56s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

# 1) Unicode dash variants fold to ASCII — no false MISMATCH on a hyphenated surname
python3 -B - "$V" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec); sys.modules["vr"] = m; spec.loader.exec_module(m)
n = m._normalize_surname
# every Unicode dash the publishers actually emit
for dash in ("‐", "‑", "‒", "–", "—", "−"):
    assert n(f"Foltyn{dash}Dumitru") == n("Foltyn-Dumitru"), f"U+{ord(dash):04X} not folded"
# the accent coverage that was already there must not regress
assert n("Çolakoğlu") == n("Colakoglu")
assert n("Müller") == n("Muller")
PY
ck "Unicode dashes fold to ASCII (no false MISMATCH)" 0 "$?"

# 2) a genuinely different surname must still MISMATCH — the fix must not blunt the check
python3 -B - "$V" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec); sys.modules["vr"] = m; spec.loader.exec_module(m)
n = m._normalize_surname
assert n("Foltyn-Dumitru") != n("Foltyn-Dumitrescu")
assert n("Eckel-Passow") != n("Eckel-Passov")
assert n("Smith") != n("Smyth")
PY
ck "a genuinely different surname still mismatches" 0 "$?"

# 3) BBT brace-protected surnames are PEOPLE — the author check must run, not be skipped
python3 -B - "$V" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec); sys.modules["vr"] = m; spec.loader.exec_module(m)
c = m.is_corporate_author_field
assert c("{Eckel-Passow}, Jeanette E. and {Ramos-Fresnedo}, Andres") is False
assert c("{von Deimling}, Andreas") is False
assert c("{Ramos-Fresnedo}, Andres") is False
PY
ck "BBT brace-protected surname is not 'corporate'" 0 "$?"

# 4) real collective authors are STILL skipped — the fix must not open a hole
python3 -B - "$V" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec); sys.modules["vr"] = m; spec.loader.exec_module(m)
c = m.is_corporate_author_field
assert c("{{KDIGO Working Group}}") is True
assert c("{{Alzheimer's Disease Neuroimaging Initiative}}") is True
assert c("{{The CRASH-3 Collaborators}}") is True
assert c("{{ADNI}}") is True          # lone braced blob, no name structure -> conservative skip
assert c("Smith, John and Doe, Jane") is False
PY
ck "real collective authors are still skipped" 0 "$?"

echo "----"
echo "test_author_normalization: $pass passed, $fail failed"
[ "$fail" -eq 0 ]

#!/usr/bin/env bash
# Regression test: the BibTeX `and others` sentinel is a declared truncation, not a mismatch.
#
# `and others` is BibTeX's own et-al.: Zotero, Mendeley, JabRef and a hand-written .bib all
# emit it for a list the author chose not to type out, and BibTeX and CSL both render it as
# "et al." verify_refs parsed it (it already dropped the literal family "others" from the
# cited list) but did not treat it as a declaration, so `cited=6 vs source=53` became
# AUTHOR MISMATCH — the render-aborting verdict — on the reference class clinical
# manuscripts cite most: multisociety guidelines and consortium papers.
#
# The escape hatch that did exist, `_audit_truncated = N`, is a field this toolkit invented.
# No reference manager writes it. So the universal way to declare a truncation failed and the
# private way succeeded, which is the wrong way round.
#
# The repo's own fixture proved the cost: skills/manage-refs/tests/fixtures/pre_submission_gate
# uses `and others`, and its Test 2 invariant broke when the author cross-check landed
# (v1.3.0, #41, 2026-05-31) and stayed broken because that fixture is not wired into CI.
#
# Network-free by construction: every assertion below is against the parser and the
# cross-check function, never against PubMed or CrossRef.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
V="$REPO_ROOT/skills/verify-refs/scripts/verify_refs.py"

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-58s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-58s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

run() {
  python3 - "$V" "$@" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec)
# @dataclass resolves cls.__module__ through sys.modules; register before exec.
sys.modules["vr"] = m
spec.loader.exec_module(m)
mode = sys.argv[2]

if mode == "sentinel":
    print("true" if m.has_bibtex_et_al(sys.argv[3]) else "false")

elif mode == "parsed_truncated":
    bib = (
        "@article{k,\n  author = {%s},\n  title = {T},\n  year = {2023},\n"
        "  doi = {10.1/x}\n}\n" % sys.argv[3]
    )
    rec = m.parse_bib(bib)[0]
    print("true" if rec.audit_truncated else "false")

elif mode == "xcheck":
    # cited families | source families | cited_n | actual_n | truncated
    cited = [s for s in sys.argv[3].split(",") if s]
    actual = [s for s in sys.argv[4].split(",") if s]
    mism, _notes = m.author_cross_check(
        cited, actual, int(sys.argv[5]), int(sys.argv[6]),
        corporate=False, soft=False,
        audit_truncated=(sys.argv[7] == "1"), first_author_guess="",
    )
    print("MISMATCH" if mism else "OK")
PY
}

echo "==== the sentinel is recognised only where BibTeX defines it ===="
ck "trailing 'and others'"          true  "$(run sentinel 'Rinella, M and Lazarus, J and others')"
ck "case-insensitive"               true  "$(run sentinel 'Rinella, M and Others')"
ck "no sentinel"                    false "$(run sentinel 'Rinella, M and Lazarus, J')"
ck "surname Others mid-list"        false "$(run sentinel 'Others, A B and Rinella, M')"
ck "empty field"                    false "$(run sentinel '')"

echo "==== the parser records it as a declared truncation ===="
ck "and others sets audit_truncated"   true  "$(run parsed_truncated 'Rinella, M and Lazarus, J and others')"
ck "plain list does not"               false "$(run parsed_truncated 'Rinella, M and Lazarus, J')"

echo "==== NEGATIVE CONTROLS — the check must still bite ===="
# 2 cited of 53 real authors, truncation declared: a count gap alone is not a finding.
ck "declared truncation, families agree" OK \
   "$(run xcheck 'Rinella,Lazarus' 'Rinella,Lazarus,Ratziu' 2 53 1)"
# Same declaration, but a cited family does not match the source: still a mismatch.
ck "declared truncation, wrong family"  MISMATCH \
   "$(run xcheck 'Rinella,Fabricated' 'Rinella,Lazarus,Ratziu' 2 53 1)"
# Same declaration, but MORE cited authors than the source has: invention, never intentional.
ck "declared truncation, extra author"  MISMATCH \
   "$(run xcheck 'Rinella,Lazarus,Ratziu,Ghost' 'Rinella,Lazarus,Ratziu' 4 3 1)"
# No declaration at all: a silent short list is exactly what the count check exists for.
ck "undeclared truncation still fires"  MISMATCH \
   "$(run xcheck 'Rinella,Lazarus' 'Rinella,Lazarus,Ratziu' 2 53 0)"

echo
echo "  passed=$pass failed=$fail"
[ "$fail" -eq 0 ] || exit 1
echo "OK: \`and others\` is a declaration, and declaring it does not buy silence on invention."

#!/usr/bin/env bash
# Regression test: a BibTeX field that is the entry's LAST one must still be read.
#
# BibTeX makes the comma after an entry's final field optional. Both the `doi` and `title` regexes
# required one, so any entry ending in `doi = {...}` produced `record.doi == ""` — and an empty DOI
# skips the CrossRef check outright and drops the record onto the soft-flagged OpenAlex title path,
# where the full-author cross-check is DISABLED. The anti-fabrication check this skill exists for was
# silently off for exactly those references. All three DOIs in the repo's own shipped fixture
# `tests/fixtures/corporate_author.bib` were being dropped this way.
#
# `title` had the identical defect with a different remedy: it cannot take an optional comma, because
# a non-greedy match would then stop at the inner `}` of a brace-protected group and return
# "A multisociety {Delphi". Both fields are now read by counting braces.
#
# Network-free: every assertion is against the parser.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
V="$REPO_ROOT/skills/verify-refs/scripts/verify_refs.py"

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-46s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-46s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

field() {  # field <which> <bib-body>
  python3 - "$V" "$1" "$2" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec)
sys.modules["vr"] = m
spec.loader.exec_module(m)
recs = m.parse_bib(sys.argv[3])
r = recs[0]
print(r.doi if sys.argv[2] == "doi" else r.title_guess)
PY
}

DOI_LAST='@article{k,
  author = {Smith, John},
  title  = {A title},
  doi    = {10.1000/xyz789}
}'
TITLE_LAST='@article{k,
  author = {Smith, John},
  doi    = {10.1000/xyz789},
  title  = {A plain title}
}'
BRACED='@article{k,
  author = {Smith, John},
  title  = {A multisociety {Delphi} consensus statement},
  doi    = {10.1000/xyz789}
}'
BRACED_LAST='@article{k,
  author = {Smith, John},
  doi    = {10.1000/xyz789},
  title  = {A multisociety {Delphi} consensus statement}
}'
QUOTED='@article{k,
  author = {Smith, John},
  title  = "A quoted title",
  doi    = {10.1000/xyz789}
}'
NO_DOI='@article{k,
  author = {Smith, John},
  title  = {A title}
}'
# doi genuinely mid-entry, with a trailing comma: the form that already worked and must keep working.
DOI_MID='@article{k,
  author = {Smith, John},
  doi    = {10.1000/xyz789},
  year   = {2024}
}'

echo "==== the last field is still a field ===="
ck "doi last"                     "10.1000/xyz789"  "$(field doi "$DOI_LAST")"
ck "doi mid-entry, trailing comma" "10.1000/xyz789" "$(field doi "$DOI_MID")"
ck "title last"                   "A plain title"   "$(field title "$TITLE_LAST")"

echo "==== brace-protected groups are not truncated (either position) ===="
ck "braced title, comma after"    "A multisociety {Delphi} consensus statement" "$(field title "$BRACED")"
ck "braced title, last field"     "A multisociety {Delphi} consensus statement" "$(field title "$BRACED_LAST")"
ck "quote-delimited title"        "A quoted title"  "$(field title "$QUOTED")"

echo "==== NEGATIVE CONTROLS — absent stays absent ===="
ck "no doi field: empty, not invented"  ""  "$(field doi "$NO_DOI")"
ck "title still read when doi absent"   "A title"  "$(field title "$NO_DOI")"

echo "==== the shipped fixture that was silently DOI-less ===="
n_empty="$(python3 - "$V" "$REPO_ROOT/skills/verify-refs/tests/fixtures/corporate_author.bib" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec)
sys.modules["vr"] = m
spec.loader.exec_module(m)
recs = m.parse_bib(open(sys.argv[2], encoding="utf-8").read())
print(sum(1 for r in recs if not r.doi))
PY
)"
ck "entries with no parsed DOI"   0  "$n_empty"

echo "==== duplicate detection needs the DOI it was not getting ===="
dup="$(python3 - "$V" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
m = importlib.util.module_from_spec(spec)
sys.modules["vr"] = m
spec.loader.exec_module(m)
bib = """@article{a,
  author = {Smith, John},
  title  = {First},
  doi    = {10.1000/same}
}

@article{b,
  author = {Doe, Jane},
  title  = {Second},
  doi    = {10.1000/same}
}
"""
print(len(m.detect_duplicates(m.parse_bib(bib))))
PY
)"
ck "two DOI-last entries sharing a DOI"  1  "$dup"

echo
echo "  passed=$pass failed=$fail"
[ "$fail" -eq 0 ] || exit 1
echo "OK: the last field parses, brace groups survive, and an absent DOI stays absent."

"""Synthetic source-evidence workflow controls; no private papers or network."""
from __future__ import annotations

import copy
import hashlib
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest

SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
sys.path.insert(0, str(SCRIPTS))
import check_claim_fidelity as fidelity
from _claim_evidence import binding_hash, render_table


def write_pdf(path: Path, text: str):
    """A real one-page PDF, using only stdlib and synthetic ASCII text."""
    escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
    stream = f"BT /F1 10 Tf 40 740 Td ({escaped}) Tj ET".encode()
    objects = [b"<< /Type /Catalog /Pages 2 0 R >>",
               b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
               b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
               b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
               b"<< /Length " + str(len(stream)).encode() + b" >>\nstream\n" + stream + b"\nendstream"]
    data = b"%PDF-1.4\n"; offsets = [0]
    for i, obj in enumerate(objects, 1):
        offsets.append(len(data)); data += f"{i} 0 obj\n".encode() + obj + b"\nendobj\n"
    start = len(data)
    data += f"xref\n0 {len(objects)+1}\n0000000000 65535 f \n".encode()
    data += b"".join(f"{n:010d} 00000 n \n".encode() for n in offsets[1:])
    data += f"trailer\n<< /Size {len(objects)+1} /Root 1 0 R >>\nstartxref\n{start}\n%%EOF\n".encode()
    path.write_bytes(data)


class ClaimEvidenceTests(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory()
        self.addCleanup(self.tmp.cleanup)
        self.root = Path(self.tmp.name)
        self.ft = self.root / "fulltext"; self.ft.mkdir()
        self.manuscript = self.root / "manuscript.md"
        self.manuscript.write_text("# Synthetic methods\n\nThe tool reduced latency by 18% in the complete sample [@demo].\n", encoding="utf-8")
        self.bib = self.root / "refs.bib"
        self.bib.write_text("@article{demo,\n title={Synthetic latency comparison},\n doi={10.1000/synthetic.latency}\n}\n", encoding="utf-8")
        self.text = self.ft / "10.1000_synthetic.latency.md"
        self.excerpt = "The tool reduced latency by 18% in the complete sample."
        self.text.write_text(self.excerpt + "\n" + "Synthetic background for a software experiment. " * 85, encoding="utf-8")
        self.pdf = self.ft / "10.1000_synthetic.latency.pdf"
        write_pdf(self.pdf, self.excerpt)
        self.retrieval = self.ft / "retrieval_report.json"
        self.retrieval.write_text(json.dumps({"schema_version":2,"items":[{
            "doi":"10.1000/synthetic.latency", "file":self.pdf.name,
            "file_sha256":hashlib.sha256(self.pdf.read_bytes()).hexdigest(),
            "source_identity":{"status":"consistent","reason":"synthetic first-page evidence"}}]}))
        self.audit = self.root / "reference_audit.json"
        self.audit.write_text(json.dumps({"schema_version":4,"records":[{
            "ref_id":"demo","doi":"10.1000/synthetic.latency","status":"OK"}]}))

    def report(self, reviewed=None, **kwargs):
        args = dict(retrieval_report=self.retrieval, reference_audit=self.audit, reviewed_report=reviewed)
        args.update(kwargs)
        return fidelity.build_report(self.manuscript, self.ft, self.bib, {}, **args)

    def review(self, report=None, verdict="supported"):
        report = copy.deepcopy(report or self.report())
        assessment = report["evidence_rows"][0]["assessment"]
        assessment.update(verdict=verdict, assessor="Synthetic test assessor", assessed_at="2026-01-01",
                          method="synthetic_fixture_annotation", source_scope="full_text", identity_checked=True,
                          source_pages=[1], source_excerpt=self.excerpt,
                          rationale="Synthetic source states the same latency change and population.")
        for side in ("claim", "source"):
            assessment[side] = dict(metric="latency change", unit="percent", denominator="all experimental runs",
                                    population="complete sample", direction="decrease")
        return report

    def save(self, value, name="reviewed.json"):
        path = self.root / name; path.write_text(json.dumps(value), encoding="utf-8"); return path

    def replace_source(self, old, new):
        self.text.write_text(self.text.read_text().replace(old, new))
        self.excerpt = self.excerpt.replace(old, new)
        write_pdf(self.pdf, self.excerpt)
        report = json.loads(self.retrieval.read_text())
        report["items"][0]["file_sha256"] = hashlib.sha256(self.pdf.read_bytes()).hexdigest()
        self.retrieval.write_text(json.dumps(report))

    def cli(self, *args):
        return subprocess.run([sys.executable, str(SCRIPTS / "check_claim_fidelity.py"),
                               "--manuscript", str(self.manuscript), "--fulltext-dir", str(self.ft),
                               "--bib", str(self.bib), *map(str, args)], capture_output=True, text=True)

    def test_no_warning_and_real_reference_do_not_establish_support(self):
        report = self.report(); row = report["evidence_rows"][0]
        self.assertEqual(report["findings"], [])
        self.assertEqual(row["reference_audit"]["recorded_status"], "OK")
        self.assertEqual(row["source"]["pdf"]["identity"]["status"], "consistent")
        self.assertEqual(row["verdict"], "not_assessed")

    def test_same_number_in_different_subgroup_stays_unassessed(self):
        self.replace_source("complete sample", "fastest subgroup")
        row = self.report()["evidence_rows"][0]
        self.assertEqual(row["verdict"], "not_assessed")
        reviewed = self.review(verdict="contradicted")
        a = reviewed["evidence_rows"][0]["assessment"]
        a["source"]["population"] = "fastest subgroup"
        a["source_excerpt"] = "The tool reduced latency by 18% in the fastest subgroup."
        a["rationale"] = "Same number describes a different population."
        row = self.report(self.save(reviewed))["evidence_rows"][0]
        self.assertEqual(row["verdict"], "contradicted")
        self.assertNotEqual(row["assessment"]["claim"]["population"], row["assessment"]["source"]["population"])

    def test_opposite_direction_is_not_inferred_from_matching_words(self):
        self.replace_source("reduced", "increased")
        self.assertEqual(self.report()["evidence_rows"][0]["verdict"], "not_assessed")

    def test_wrong_pdf_identity_cannot_carry_support(self):
        payload = json.loads(self.retrieval.read_text())
        payload["items"][0]["source_identity"]["status"] = "conflict"
        self.retrieval.write_text(json.dumps(payload))
        row = self.report(self.save(self.review()))["evidence_rows"][0]
        self.assertEqual(row["reference_audit"]["recorded_status"], "OK")
        self.assertEqual((row["verdict"], row["review_state"]), ("unresolved", "source_identity_conflict"))

    def test_abstract_only_review_remains_unresolved(self):
        self.text.write_text("Synthetic abstract with the same reported latency change.")
        reviewed = self.review(); reviewed["evidence_rows"][0]["assessment"]["source_scope"] = "abstract_only"
        row = self.report(self.save(reviewed))["evidence_rows"][0]
        self.assertEqual(row["verdict"], "unresolved")

    def test_unresolved_citation_is_inventoried_without_attribution_verb(self):
        self.manuscript.write_text("A latency difference of 18% [@missing].")
        row = self.report()["evidence_rows"][0]
        self.assertEqual(row["citation"], "missing"); self.assertIsNone(row["source"]["text_file"])
        self.assertEqual(row["verdict"], "not_assessed")

    def test_recorded_assessment_requires_pages_excerpt_and_comparison(self):
        for key, value in [("assessor", ""), ("source_pages", []), ("source_pages", [True]),
                           ("source_excerpt", ""), ("claim", {}), ("source", {})]:
            with self.subTest(key=key, value=value):
                reviewed = self.review(); reviewed["evidence_rows"][0]["assessment"][key] = value
                self.assertEqual(self.report(self.save(reviewed))["evidence_rows"][0]["verdict"], "unresolved")

    def test_complete_attributed_review_is_retained_without_changing_findings(self):
        initial = self.report(); reviewed = self.review(initial)
        report = self.report(self.save(reviewed)); row = report["evidence_rows"][0]
        self.assertEqual((row["verdict"], row["review_state"]), ("supported", "recorded"))
        self.assertEqual(report["findings"], initial["findings"])
        self.assertIn("not automatically verified facts", report["evidence_scope"])

    def test_changed_pdf_stays_stale_across_repeated_reruns(self):
        reviewed = self.save(self.review())
        write_pdf(self.pdf, "Different synthetic source version.")
        for _ in range(2):
            report = self.report(reviewed); row = report["evidence_rows"][0]
            self.assertEqual(row["verdict"], "unresolved"); self.assertEqual(row["review_state"], "stale_inputs")
            self.assertEqual(row["source"]["pdf"]["hash_status"], "changed_since_retrieval")
            reviewed = self.save(report)

    def test_changed_text_bibliography_and_manuscript_context_invalidate_review(self):
        for path in (self.text, self.bib, self.manuscript):
            with self.subTest(path=path.name):
                old = path.read_bytes(); reviewed = self.save(self.review())
                path.write_bytes(old + b"\nContext updated.\n")
                row = self.report(reviewed)["evidence_rows"][0]
                self.assertEqual(row["review_state"], "stale_inputs")
                path.write_bytes(old)

    def test_changed_sentence_preserves_orphaned_review(self):
        reviewed = self.save(self.review()); self.manuscript.write_text("A changed claim [@demo].")
        report = self.report(reviewed)
        self.assertEqual(report["evidence_rows"][0]["verdict"], "not_assessed")
        self.assertEqual(len(report["orphaned_evidence_reviews"]), 1)

    def test_explicit_reinspection_can_refresh_a_stale_assessment(self):
        reviewed = self.save(self.review())
        self.text.write_text(self.text.read_text() + "\nAdditional synthetic context.\n")
        stale = self.report(reviewed); row = stale["evidence_rows"][0]
        self.assertEqual(row["review_state"], "stale_inputs")
        row["assessment"]["binding_sha256"] = row["binding_sha256"]
        row["assessment"]["rationale"] = "Synthetic assessor rechecked the changed context."
        fresh = self.report(self.save(stale))["evidence_rows"][0]
        self.assertEqual((fresh["verdict"], fresh["review_state"]), ("supported", "recorded"))

    def test_numbered_citation_has_source_and_original_offsets(self):
        raw = "A synthetic latency result [1].\n\n## References\n1. Synthetic source. doi:10.1000/synthetic.latency\n"
        self.manuscript.write_text(raw)
        row = self.report()["evidence_rows"][0]
        self.assertEqual(row["citation"], "1")
        self.assertEqual(row["doi"], "10.1000/synthetic.latency")
        self.assertEqual(row["manuscript"]["char_start"], 0)

    def test_docx_coordinates_are_extracted_text_not_rendered_pages(self):
        from docx import Document
        doc = Document(); doc.add_paragraph("Synthetic latency evidence [@demo].")
        path = self.root / "manuscript.docx"; doc.save(path)
        report = fidelity.build_report(path, self.ft, self.bib, {})
        row = report["evidence_rows"][0]
        self.assertEqual(row["manuscript"]["line_start"], 1)
        self.assertIn("not rendered pages", row["manuscript"]["coordinate_system"])

    def test_duplicate_review_ids_rejected(self):
        reviewed = self.review(); reviewed["evidence_rows"].append(reviewed["evidence_rows"][0])
        with self.assertRaisesRegex(ValueError, "duplicate"): self.report(self.save(reviewed))

    def test_missing_or_duplicate_pdf_report_never_uses_first_match(self):
        payload = json.loads(self.retrieval.read_text()); payload["items"] *= 2
        self.retrieval.write_text(json.dumps(payload))
        self.assertEqual(self.report()["evidence_rows"][0]["source"]["pdf"]["hash_status"], "ambiguous_report")

    def test_report_paths_cannot_escape_the_pdf_directory(self):
        payload = json.loads(self.retrieval.read_text()); payload["items"][0]["file"] = "../outside.pdf"
        self.retrieval.write_text(json.dumps(payload))
        row = self.report()["evidence_rows"][0]
        self.assertIsNone(row["source"]["pdf"]["sha256"])

    def test_prose_positions_survive_exclusions_and_duplicate_sentences(self):
        line = "A synthetic claim [see @demo, p. 1; @missing]."
        raw = "---\ntitle: ignored [@meta]\n---\n# Heading\n\n```\nignored [@code]\n```\n\n| ignored [@table] |\n\n> ignored [@quote]\n\n" + line + "\n\n" + line + "\n\n## References\nignored [@bib]\n"
        self.manuscript.write_text(raw)
        rows = self.report()["evidence_rows"]
        self.assertEqual([r["citation"] for r in rows], ["demo", "missing", "demo", "missing"])
        self.assertEqual(len({r["id"] for r in rows}), 4)
        for row in rows:
            location = row["manuscript"]
            self.assertEqual(raw[location["char_start"]:location["char_end"]], line)
            self.assertEqual(raw.splitlines()[location["line_start"] - 1], line)

    def test_no_report_supplied_preserves_legacy_call_signature(self):
        report = fidelity.build_report(self.manuscript, self.ft, self.bib, {})
        self.assertEqual(report["evidence_rows"][0]["source"]["pdf"]["hash_status"], "not_available")
        self.assertEqual(report["evidence_rows"][0]["verdict"], "not_assessed")

    def test_table_escapes_source_markup(self):
        report = self.report(); report["evidence_rows"][0]["assessment"]["source_excerpt"] = "<img src=x>|[link](https://example.com)\nnext"
        table = render_table(report)
        self.assertNotIn("<img", table); self.assertNotIn("[link](", table); self.assertIn("&#124;", table)

    def test_cli_writes_derived_table_and_preserves_all_sources(self):
        paths = (self.manuscript, self.bib, self.text, self.pdf, self.retrieval, self.audit)
        before = {p:p.read_bytes() for p in paths}
        output = self.root / "claim_fidelity.json"; table = self.root / "claim_fidelity.md"
        result = self.cli("--retrieval-report", self.retrieval, "--reference-audit", self.audit,
                          "--out", output, "--evidence-table", table)
        self.assertEqual(result.returncode, 0, result.stderr)
        self.assertNotIn("every checkable claim is supported", result.stdout)
        self.assertIn("not been established", result.stdout)
        self.assertIn(r"not\_assessed", table.read_text())
        self.assertEqual(json.loads(output.read_text())["schema_version"], 2)
        self.assertEqual(before, {p:p.read_bytes() for p in paths})

    def test_cli_rejects_outputs_over_source_files(self):
        for path in (self.manuscript, self.text, self.pdf, self.retrieval):
            with self.subTest(path=path.name):
                before = path.read_bytes()
                result = self.cli("--retrieval-report", self.retrieval, "--out", path)
                self.assertEqual(result.returncode, 2, result.stderr)
                self.assertEqual(path.read_bytes(), before)

    def test_invalid_json_does_not_overwrite_existing_output(self):
        output = self.root / "claim_fidelity.json"; output.write_text("preserve existing output")
        self.retrieval.write_text("broken json")
        result = self.cli("--retrieval-report", self.retrieval, "--out", output)
        self.assertEqual(result.returncode, 2); self.assertEqual(output.read_text(), "preserve existing output")

    def test_table_requires_json_and_cannot_replace_it(self):
        output = self.root / "claim_fidelity.json"
        self.assertEqual(self.cli("--evidence-table", output).returncode, 2)
        self.assertEqual(self.cli("--out", output, "--evidence-table", output).returncode, 2)
        self.assertFalse(output.exists())


if __name__ == "__main__":
    unittest.main()

#!/usr/bin/env bash
# Regression test for verify-refs corporate/collective-author handling (B1).
# Offline (no network): a guideline body double-braced in BibTeX
# (`author = {{EASL} and {EASD}}`, `{{KDIGO CKD Work Group}}`) must be detected as
# a corporate author and EXEMPTED from the personal-name family cross-check — it
# must never be a MISMATCH (which would abort render_pandoc.sh on every
# guideline-citing cohort manuscript). A normal personal-author entry is not
# corporate. Stdlib-only (python3).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$HERE/../scripts/verify_refs.py"
BIB="$HERE/fixtures/corporate_author.bib"
ROOT="$(mktemp -d -t vrc_XXXX)"
trap 'rm -rf "$ROOT"' EXIT

fail=0
check() { local label="$1"; shift
    if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$label"
    else printf '  FAIL  %s\n' "$label"; fail=$((fail+1)); fi
}

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing" >&2; exit 2; }
[[ -f "$BIB" ]]    || { echo "ENV-ERR: fixture missing" >&2; exit 2; }

python3 "$SCRIPT" "$BIB" --project-root "$ROOT" --offline >/dev/null 2>&1
AUDIT="$ROOT/qc/reference_audit.json"
check "audit JSON written" test -s "$AUDIT"

assert_py() { python3 -c "
import json
d = json.load(open('$AUDIT'))
recs = {r['ref_id']: r for r in d['records']}
$1
"; }

# Double-braced guideline bodies detected as corporate, annotated, never MISMATCH.
for ref in easl2024masld kdigo2024ckd; do
    check "$ref corporate_author True" \
        assert_py "assert recs['$ref']['corporate_author'] is True, recs['$ref']"
    check "$ref annotated corporate (note)" \
        assert_py "assert 'corporate' in recs['$ref'].get('note','').lower(), recs['$ref']"
    check "$ref NOT a MISMATCH / no author-mismatch" \
        assert_py "assert recs['$ref']['status'] != 'MISMATCH' and 'AUTHOR MISMATCH' not in recs['$ref'].get('evidence',''), recs['$ref']"
done

# A normal personal-author entry must NOT be flagged corporate.
check "personal-author entry corporate_author False" \
    assert_py "assert recs['smith2024cohort']['corporate_author'] is False, recs['smith2024cohort']"

echo "fail=$fail"; [[ "$fail" -eq 0 ]] && echo "ALL PASS" || echo "FAILURES: $fail"
exit "$fail"

#!/usr/bin/env bash
# Regression test for verify-refs fabricated-author detection (network-free).
#
# author_cross_check() is the sole decision surface behind an AUTHOR MISMATCH status
# (family-by-family + author-count). It is the repo's most trust-critical citation
# check: a correct first author and DOI can coexist with fabricated co-author
# names. This test locks the logic so a refactor cannot silently drop the MISMATCH
# path. Stdlib-only, no network (the pure function is tested in isolation, so no
# PubMed/CrossRef/OpenAlex call is made).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPTS="$HERE/../scripts"
[[ -f "$SCRIPTS/verify_refs.py" ]] || { echo "ENV-ERR: verify_refs.py missing" >&2; exit 2; }

python3 - "$SCRIPTS" <<'PY'
import sys
sys.path.insert(0, sys.argv[1])
from verify_refs import author_cross_check

fails = 0
def check(label, cond):
    global fails
    print(("  PASS  " if cond else "  FAIL  ") + label)
    if not cond:
        fails += 1

# 1. Clean list — cited == actual — no mismatch.
m, n = author_cross_check(["Liu", "Chen", "Wang"], ["Liu", "Chen", "Wang"], 3, 3)
check("clean list -> no mismatch", m == [])

# 2. Fabricated co-authors — correct first author + count, wrong #2..#N families.
m, n = author_cross_check(["Liu", "Ingram", "Xue"], ["Liu", "Chen", "Wang"], 3, 3)
check("fabricated co-authors -> 2 family mismatches",
      len(m) == 2 and "#2 family" in m[0] and "#3 family" in m[1])

# 3. Extra cited author beyond the source list.
m, n = author_cross_check(["Liu", "Chen", "Ghost"], ["Liu", "Chen"], 3, 2)
check("extra cited author -> flagged", any("extra cited='Ghost'" in x for x in m))

# 4. Author-count mismatch (not truncated) -> AUTHOR COUNT flag.
m, n = author_cross_check(["Liu"], ["Liu", "Chen", "Wang"], 1, 3)
check("count mismatch -> AUTHOR COUNT flag", any("AUTHOR COUNT" in x for x in m))

# 5. Intentional CSL et-al truncation (cited < actual, marker set) -> note, NOT mismatch.
m, n = author_cross_check(["Liu"], ["Liu", "Chen", "Wang"], 1, 3, audit_truncated=True)
check("audit_truncated -> note not mismatch",
      m == [] and any("intentional truncate" in x for x in n))

# 6. Corporate/collective author -> exempt (must never fire MISMATCH).
m, n = author_cross_check(["Ghost"], ["KDIGO Working Group"], 1, 1, corporate=True)
check("corporate author -> exempt", m == [])

# 7. OpenAlex-soft list -> exempt from strict positional check.
m, n = author_cross_check(["Liu", "Ingram"], ["Liu", "Chen"], 2, 2, soft=True)
check("soft (OpenAlex) list -> exempt", m == [])

# 8. First-author degrade (no cited list; first_author_guess mismatches source).
m, n = author_cross_check([], ["Chen", "Wang"], 0, 2, first_author_guess="Liu")
check("first-author degrade -> #1 mismatch", any("#1 family" in x for x in m))

# 9. First-author degrade clean (first_author_guess matches source).
m, n = author_cross_check([], ["Liu", "Wang"], 0, 2, first_author_guess="Liu")
check("first-author degrade clean -> no mismatch", m == [])

print(f"\nfails={fails}")
sys.exit(1 if fails else 0)
PY
rc=$?
echo
[[ $rc -eq 0 ]] && echo "ALL PASS" || echo "FAILURES"
exit $rc

#!/usr/bin/env bash
# Regression test for the OpenAlex tertiary index (conference proceedings / non-DOI
# / non-biomedical recovery). Network-free: monkeypatches http_json so no live API
# is called. Motivation: NeurIPS/ICLR/ACL citations common in medical-AI papers fall
# through PubMed (not biomedical) and CrossRef (spotty proceedings) — OpenAlex is the
# free analogue of a journal portal's second index (e.g. Scopus). Stdlib-only.
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$HERE/../scripts/verify_refs.py"

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing" >&2; exit 2; }

python3 - "$SCRIPT" <<'PY'
import importlib.util, sys

spec = importlib.util.spec_from_file_location("vr", sys.argv[1])
vr = importlib.util.module_from_spec(spec)
sys.modules["vr"] = vr  # dataclass resolution needs the module registered (py3.14)
spec.loader.exec_module(vr)

fail = 0
def check(label, cond):
    global fail
    if cond:
        print(f"  PASS  {label}")
    else:
        print(f"  FAIL  {label}")
        fail += 1

# --- _title_similarity ----------------------------------------------------------
t = "Reflexion: Language agents with verbal reinforcement learning"
check("title sim exact == 1.0", abs(vr._title_similarity(t, t) - 1.0) < 1e-9)
check("title sim unrelated < 0.3",
      vr._title_similarity(t, "A meta-analysis of CT screening for lung cancer") < 0.3)
check("title sim empty == 0.0", vr._title_similarity("", t) == 0.0)

# --- _openalex_families ---------------------------------------------------------
work = {"authorships": [
    {"author": {"display_name": "Aman Madaan"}},
    {"author": {"display_name": "Niket Tandon"}},
    {"author": {"display_name": "Madaan A"}},   # trailing-initials form
]}
fams = vr._openalex_families(work)
check("families last-token parse", fams[:2] == ["Madaan", "Tandon"])
check("families strip trailing initials", fams[2] == "Madaan")

# --- verify_openalex via monkeypatched http_json --------------------------------
DOI_WORK = {"id": "https://openalex.org/W1",
            "title": "Self-Refine: Iterative refinement with self-feedback",
            "publication_year": 2023,
            "authorships": [{"author": {"display_name": "Aman Madaan"}}]}
TITLE_HIT = {"results": [
    {"title": "Reflexion: Language agents with verbal reinforcement learning",
     "publication_year": 2023,
     "authorships": [{"author": {"display_name": "Noah Shinn"}},
                     {"author": {"display_name": "Federico Cassano"}}]},
    {"title": "Some unrelated paper about kidneys", "authorships": []},
]}
TITLE_MISS = {"results": [
    {"title": "Completely different work on radiology", "authorships": []}]}

def make_http(mapping):
    def _http(url, timeout):
        for needle, payload in mapping.items():
            if needle in url:
                return payload
        return None
    return _http

# (a) DOI resolve
vr.http_json = make_http({"api.openalex.org/works/https://doi.org/": DOI_WORK})
st, ev, fams = vr.verify_openalex("10.5555/self-refine", "", 5)
check("openalex DOI resolve OK", st == "OK" and fams == ["Madaan"] and "via doi" in ev)

# (b) title.search with strong similarity → OK
vr.http_json = make_http({"api.openalex.org/works?": TITLE_HIT})
st, ev, fams = vr.verify_openalex(
    "", "Reflexion: Language agents with verbal reinforcement learning", 5)
check("openalex title hit OK", st == "OK" and fams[0] == "Shinn" and "via title" in ev)

# (c) title with no close match → UNVERIFIED (fabrication guard)
vr.http_json = make_http({"api.openalex.org/works?": TITLE_MISS})
st, ev, fams = vr.verify_openalex(
    "", "Reflexion: Language agents with verbal reinforcement learning", 5)
check("openalex weak-title rejected (UNVERIFIED)", st == "UNVERIFIED" and fams == [])

# (d) no DOI, no title, no match → UNVERIFIED, never FABRICATED
vr.http_json = make_http({})
st, ev, fams = vr.verify_openalex("", "", 5)
check("openalex empty never FABRICATED", st == "UNVERIFIED")

# --- integration through verify_record ------------------------------------------
# A conference paper: no PMID, no DOI, title only. OpenAlex resolves it and the
# cited first author matches → status OK, source includes openalex.
vr.http_json = make_http({"api.openalex.org/works?": TITLE_HIT})
rec = vr.RefRecord(
    ref_id="reflexion2023",
    raw="Shinn N, Cassano F, et al. Reflexion: Language agents with verbal reinforcement learning. NeurIPS 2023.",
    title_guess="Reflexion: Language agents with verbal reinforcement learning",
    cited_authors=["Shinn", "Cassano"],
    first_author_guess="Shinn",
)
out = vr.verify_record(rec, offline=False, timeout=5, use_openalex=True)
check("verify_record conference OK via openalex",
      out.status == "OK" and "openalex" in out.evidence)

# Same record with a fabricated first author → MISMATCH from OpenAlex authors.
vr.http_json = make_http({"api.openalex.org/works?": TITLE_HIT})
rec_bad = vr.RefRecord(
    ref_id="reflexion_bad",
    raw="Ebrahimi A, et al. Reflexion: Language agents with verbal reinforcement learning. NeurIPS 2023.",
    title_guess="Reflexion: Language agents with verbal reinforcement learning",
    cited_authors=["Ebrahimi", "Cassano"],
    first_author_guess="Ebrahimi",
)
out_bad = vr.verify_record(rec_bad, offline=False, timeout=5, use_openalex=True)
check("verify_record catches first-author hallucination via openalex",
      out_bad.status == "MISMATCH" and "AUTHOR MISMATCH" in out_bad.evidence)

# --no-openalex equivalent: use_openalex=False leaves a no-identifier record UNVERIFIED.
vr.http_json = make_http({"api.openalex.org/works?": TITLE_HIT})
rec_off = vr.RefRecord(
    ref_id="reflexion_off",
    raw="Reflexion NeurIPS 2023.",
    title_guess="Reflexion: Language agents with verbal reinforcement learning",
    cited_authors=["Shinn"],
    first_author_guess="Shinn",
)
out_off = vr.verify_record(rec_off, offline=False, timeout=5, use_openalex=False)
check("use_openalex=False skips OpenAlex (no openalex source)",
      "openalex" not in out_off.evidence)

print(f"fail={fail}")
print("ALL PASS" if fail == 0 else f"FAILURES: {fail}")
sys.exit(fail)
PY

#!/usr/bin/env bash
# Regression test for verify-refs Gate 6 (pagination-placeholder detection).
# Offline (no network): a bib entry whose pages are "e000--e000" with an "in press"
# note must get note="pagination_placeholder"; a normal entry must not. verify-refs
# stays manuscript-agnostic — it only flags; the P0/centrality call is /self-review's.
# Stdlib-only (python3).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$HERE/../scripts/verify_refs.py"
BIB="$HERE/fixtures/pagination_placeholder.bib"
ROOT="$(mktemp -d -t vrp_XXXX)"
trap 'rm -rf "$ROOT"' EXIT

fail=0
check() { local label="$1"; shift
    if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$label"
    else printf '  FAIL  %s\n' "$label"; fail=$((fail+1)); fi
}

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing" >&2; exit 2; }

python3 "$SCRIPT" "$BIB" --project-root "$ROOT" --offline >/dev/null 2>&1
AUDIT="$ROOT/qc/reference_audit.json"
check "audit JSON written" test -s "$AUDIT"

assert_py() { python3 -c "
import json
d = json.load(open('$AUDIT'))
recs = {r['ref_id']: r for r in d['records']}
$1
"; }

check "placeholder entry flagged note=pagination_placeholder" \
    assert_py "assert 'pagination_placeholder' in recs['methodref_inpress'].get('note',''), recs['methodref_inpress']"
check "placeholder entry status UNVERIFIED" \
    assert_py "assert recs['methodref_inpress']['status']=='UNVERIFIED', recs['methodref_inpress']['status']"
check "normal entry NOT flagged" \
    assert_py "assert 'pagination_placeholder' not in recs['normalref_2025'].get('note',''), recs['normalref_2025']"

echo "fail=$fail"; [[ "$fail" -eq 0 ]] && echo "ALL PASS" || echo "FAILURES: $fail"
exit "$fail"